PRODUCTION USE CASE

LLM API for RAG chatbots

An internal policy assistant answers questions across HR handbooks and regional addenda. Retrieval happens before the model call. Each passage keeps a document ID, section, effective date, and access label.

Production recipe

API, primary model, and failover configuration

Production choice	Recommendation	Why
API	POST /v1/chat/completions	OpenAI-compatible server-side request
Primary model	gemini-2.5-pro	Gemini 2.5 Pro is the primary route for long evidence sets and cross-document synthesis.
Fallback model	gemini-2.5-flash	Use Gemini 2.5 Flash for short, well-retrieved answers. Escalate ambiguous policy synthesis to Terra only after retrieval quality is verified.
Escalation model	gpt-5.6-terra	Use only when the primary route fails the defined quality or complexity boundary
Output contract	Task-specific text or patch	An answer with passage IDs after every material claim, or an explicit evidence-gap response.

Scenario

RAG chatbots in a production application

Gemini 2.5 Pro receives only the selected passages and must cite the passage IDs used. If regional policies disagree, the answer shows the conflict instead of choosing silently.

Architecture

How the RAG chatbots workflow operates

Authorize the user before retrieval.
Retrieve and rerank passages with document metadata intact.
Send the smallest sufficient evidence set to the model.
Require sentence-level passage IDs.
Verify citation support and abstain on missing evidence.

API request

Call gemini-2.5-pro through LLMFly AI

Send the request from your server. Replace the example content and placeholder tool schema with data and tools from your application.

request.exampleCopy-ready

curl https://app.llmfly.ai/v1/chat/completions \
  -H "Authorization: Bearer $LLMFLY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.5-pro",
    "messages": [
      {"role": "system", "content": "Answer only from the supplied passages. Cite passage IDs after each material sentence. If passages are missing or conflicting, state that explicitly."},
      {"role": "user", "content": "Can a Shanghai employee carry unused leave into next year? Use the supplied policy passages."}
    ]
  }'

Model choice

Why gemini-2.5-pro is the primary model

Gemini 2.5 Pro is the primary route for long evidence sets and cross-document synthesis.

Use Gemini 2.5 Flash for short, well-retrieved answers. Escalate ambiguous policy synthesis to Terra only after retrieval quality is verified.

Acceptance

Acceptance checks for RAG chatbots

Metric	Pass condition
Retrieval recall	Retrieval returns the known supporting passage for answerable test questions
Citation correctness	Each citation directly supports the sentence or field it is attached to
Unsupported-claim rate	Material claims without source support are absent from the final answer
Correct abstention rate	The assistant declines or qualifies when sources are missing, stale, or conflicting

Failure handling

Failures to handle before deployment

Blaming the model for retrieval misses
Stuffing every document into context
Showing irrelevant citations
Forcing answers without evidence

Output

Returned output and run records

An answer with passage IDs after every material claim, or an explicit evidence-gap response.

Record the model ID, request ID, token usage, retries, validation result, and final disposition for every production run.

Frequently asked questions

Which matters more: retrieval or the model?

Both matter, but evaluate them separately so a retrieval miss is not mistaken for generation failure.

How many passages should go into the prompt?

Use the smallest set that adequately supports the answer within budget.

What if sources disagree?

Surface the disagreement, source, and date instead of silently choosing one claim.

Test this setup with your own inputs

Compare the primary and fallback models with the same requests, tools, and validation rules.

Compare models