Retrieval-augmented generation, or RAG, finds relevant source passages at request time and supplies them to a model for an answer. It does not retrain the model or guarantee truth. Quality depends on document preparation, retrieval coverage, ranking, instructions, citation mapping, and a policy for answering when evidence is missing.
Who this is for: Developers, analysts, and knowledge owners building AI answers from private, changing, or source-sensitive documents.
- Evaluate retrieval and answer generation as separate stages with separate failure signals.
- Preserve source identity and location from ingestion through the final citation.
- Require the system to expose evidence gaps instead of filling them from model memory.
Index material for later search
A RAG system begins with approved documents. It parses content, divides it into retrievable units, attaches metadata, and stores a searchable representation. The representation may support keyword search, embedding similarity, or both. Keep document identifier, title, version, access scope, and original location attached to every unit.
Ingestion quality sets an upper bound on retrieval. Missing tables, broken headings, duplicate editions, and stripped page labels can make correct evidence unavailable or misleading. Validate a sample after parsing, define how updates replace older material, and record indexing failures. Retrieval cannot recover a paragraph that never entered the index correctly.
Retrieve evidence for the question
At request time, the system converts the user's need into one or more searches. It retrieves candidates, applies access controls and metadata filters, and may rerank results for relevance. The goal is not merely to find similar language. It is to include the evidence needed to answer while excluding unrelated passages that distract generation.
Questions often contain several claims. A single search for a broad sentence may retrieve one part and miss another. Query decomposition can search each subquestion, but it also adds complexity. Start with observed failures. Measure whether known supporting passages appear within the retrieved set before tuning the final answer prompt.
Generate within an evidence boundary
The model receives the question and selected passages with clear source labels. Instructions should say which material is authoritative, how to handle conflict, and what to do when evidence is insufficient. For a policy assistant, the correct behavior may be to state that the indexed policy does not answer the question and route it to an owner.
Citations should be built from stable source metadata, not invented by the model. Map each cited claim to the exact retrieved unit and provide a link or location a reader can inspect. Citation presence alone is weak: test whether the cited text actually supports the nearby claim and whether a newer source supersedes it.
Diagnose the pipeline by stage
When an answer is wrong, first ask whether the necessary evidence exists in the corpus. Then check parsing, chunking, retrieval, ranking, context assembly, and generation. Changing the prompt cannot fix an absent document. Increasing the number of passages may hide a ranking defect while adding irrelevant content and cost.
Maintain test questions with expected source units and acceptable answer properties. Include unanswerable questions, conflicting versions, access-restricted material, and wording unlike the source. Current retrieval APIs and model context behavior change, so consult official documentation quarterly while keeping your corpus-specific tests and provenance requirements stable.
Answer an employee leave-policy question
An employee asks whether unused leave carries over, and the company has policies for several countries and years.
- Index each policy with country, effective date, document version, section heading, and employee access scope.
- Use the employee's authorized country as a filter, retrieve passages about carryover, and prefer the currently effective policy.
- Provide the selected passages to the model with instructions to answer only from them and identify unresolved conflicts.
- Attach a citation to the exact policy section and verify that every stated limit or deadline appears there.
- Test a country with no published carryover rule and require an evidence-gap response rather than a generic answer.
RAG diagnosis table
Trace a bad answer through these checkpoints in order.
- Corpus: Is the correct, current, authorized source present and parseable?
- Retrieval: Did the expected source unit appear for the real user wording?
- Selection: Did filtering and ranking retain the best evidence and version?
- Generation: Does each claim stay within the supplied passages and uncertainty rule?
- Citation: Can a reviewer open the exact source location supporting the claim?
Common mistakes
- Calling a system grounded because it retrieved documents, without testing whether answers follow those documents.
- Removing version and permission metadata when creating embeddings, then trying to restore it after retrieval.
- Tuning only final-answer wording when the needed passage is missing from the retrieved candidates.
Try one
A benefits assistant cites a handbook but gives an outdated eligibility period. Explain how you would locate the failure.
A good diagnosis checks whether the current handbook was indexed, whether the old edition remained active, which passages retrieval returned, and whether ranking or generation selected the obsolete period. It verifies the citation target rather than trusting its label. The correction should include version filters or ingestion cleanup when needed, plus a test that asks the same question in different wording and expects the current source.
Sources
- OpenAI retrieval guideOfficial guidance for indexing, searching, and retrieving relevant source material.
- OpenAI embeddings guideOfficial explanation of embeddings and common semantic search uses.