RAG evaluation: the 12-check pre-production review

Most RAG failures are not model failures. They are retrieval failures, grounding failures, or evaluation failures that were never measured.

This checklist is deliberately narrow. It is the review I would run before letting a RAG system touch a customer, a compliance workflow, or a revenue decision.

1. Is the retrieval set representative?

Sample real queries, not just the clean examples in the demo. Include misspellings, multi-hop questions, ambiguous entities, long-tail documents, and questions with no answer in the corpus. If the evaluation set is synthetic, label it as such.

2. Does retrieval find the right evidence?

Measure recall@k and mean reciprocal rank on a labeled set. A correct answer built on the wrong document is a latent failure; it will break when the document changes or an auditor asks for the source.

3. Is the answer grounded?

Every material claim should map to retrieved evidence. Measure citation coverage and unsupported-claim rate. “The model sounded confident” is not a grounding metric.

4. Does it refuse when the corpus does not contain the answer?

Build a set of unanswerable questions. Measure false-answer rate and correct-refusal rate separately. A system that invents a policy is worse than one that says it does not know.

5. Is freshness handled?

Test documents that were updated, superseded, or deleted. Check index refresh latency and whether stale chunks can still be retrieved. For regulated workflows, record the document version used in the answer.

6. Are permissions enforced before generation?

Retrieval must respect document-level access control. Test a user who should not see a document: the content must not appear in retrieved context, the prompt, logs, or the final answer.

7. Is the system repeatable?

Run the same query at least three times. Measure answer consistency, citation consistency, and route changes. A RAG system that is correct one run and wrong the next is not production-ready.

8. What does it cost per successful answer?

Track input tokens, retrieved chunks, reranker calls, output tokens, and retries. Cost per successful outcome is the number that matters; a cheap average with expensive failures is not cheap.

9. What is the latency budget?

Measure p50 and p95 end-to-end latency, including retrieval, reranking, generation, and tool calls. Separate the model time from the retrieval time so you know what to fix.

10. Are prompt and model changes gated?

Every prompt, model, embedding, or index change should run the same regression suite. A green dashboard after an unlogged prompt change is not evidence.

11. Can you reconstruct an answer?

Keep the query, retrieved chunk IDs, document versions, prompt version, model version, output, and trace ID. If a customer disputes an answer six months later, you need the evidence trail.

12. Is there a human escalation path?

Define which answers require review, who reviews them, and how the review is recorded. Test the escalation path, not just the happy path.

The evidence pack

A defensible RAG review produces: the scenario set, retrieval metrics, groundedness and refusal results, k ≥ 3 repeatability results, cost/latency percentiles, access-control tests, regression-gate output, and a severity-ranked remediation plan.

If that sounds like more than a prompt review, it is. My own KYC Copilot case study, built commercially and then open-sourced, shows what the evidence looks like when the workflow is regulated. The scorecard will tell you which of these checks are already covered.