The Production RAG Checklist: from Demo to Governed Retrieval

Every engineering team we meet has a RAG demo that works. Almost none have a RAG system they would defend in front of a compliance officer. This article is a production RAG checklist: the concrete gates between a prototype that impresses in a meeting and a governed retrieval system that survives audit, adversarial users, and a Tuesday morning when the source data changed overnight. It is written for engineering leads in regulated industries, where "the model made it up" is not a bug report. It is an incident.

The gap is well documented. In a 2024 experience report across three real RAG deployments, researchers at Deakin University catalogued seven recurring failure points, from missing content in the index to answers that ignore retrieved context entirely. Their blunt conclusion: validating a RAG system is only feasible during operation, not before it. That finding should set your posture. You do not finish a RAG system and then govern it. You build the governance into the pipeline, or you ship a demo.

Why the demo lies to you

A demo runs on clean data, friendly questions, and one user with access to everything. Production runs on stale SharePoint folders, questions nobody anticipated, and users whose permissions differ by document, by matter, by client.

The failure modes are not exotic. Air Canada's support chatbot told a customer he could apply for a bereavement refund after travel, a policy that did not exist. The airline argued the chatbot was responsible for its own statements. The BC Civil Resolution Tribunal disagreed and ordered Air Canada to pay, holding that the company owes accurate information "whether it comes from a static page or a chatbot" (Moffatt v. Air Canada, 2024 BCCRT 149). That was a retrieval-and-generation failure with a legal invoice attached. In finance or healthcare, the invoice is larger.

On the security side, OWASP's 2025 LLM Top 10 added Vector and Embedding Weaknesses (LLM08) specifically for RAG systems: cross-tenant retrieval, embedding inversion, and missing access control on the retrieval API are now named, numbered risks. NIST's Generative AI Profile (AI 600-1) puts confabulation and data privacy among its twelve risk categories and maps suggested actions to each. The frameworks have caught up to what practitioners already knew. The demo hides the hard parts.

The production RAG checklist

This is the artifact. Six gate categories, each with a pass condition you can verify. Copy it into your launch review. If a box cannot be checked, the system is a demo wearing a production badge.

1. Evaluation gates

• [ ] A golden dataset of at least 100 real user questions with reference answers, owned and refreshed by domain experts, not by the team that built the pipeline.
• [ ] Retrieval recall measured separately from answer quality: did the right document appear in the top-k results, independent of what the model did with it.
• [ ] A CI eval gate that blocks deployment when faithfulness or answer-correctness scores drop below an agreed threshold. Evals without a gate are decoration (see CI evaluation gates).
• [ ] Regression runs triggered by every change: index rebuild, chunking change, prompt edit, model swap.

2. Citation enforcement

• [ ] Every factual sentence in an answer carries a citation to a specific retrieved chunk, verified programmatically, not requested politely in the prompt.
• [ ] Citations resolve to document, version, and date. A citation that cannot be clicked through to the source is a claim, not evidence.
• [ ] An abstention path: when retrieval returns nothing above the relevance threshold, the system says "I don't have a source for that" instead of improvising. This is the control that would have saved Air Canada CA$812 and a precedent.

3. Hybrid search, not vibes

Vector-only retrieval fails on the exact queries regulated industries care about: part numbers, statute citations, client codes, drug names. Pure embedding search treats "IRC Section 469" and "IRC Section 496" as near neighbors.

• [ ] Keyword (BM25) and vector search fused, for example with reciprocal rank fusion, with a reranker on top.
• [ ] Per-query-type measurement: identifier lookups, definitional questions, and multi-hop questions scored separately, because each fails differently.

4. Permission filtering at the retrieval layer

• [ ] Access control enforced at query time inside the retrieval layer, not appended to the prompt as instructions. The model should never see a chunk the user is not entitled to see.
• [ ] Permission metadata indexed alongside every chunk, sourced from the system of record (matter-level, client-level, chart-level).
• [ ] A negative test suite: queries from synthetic users who must not see specific documents, run in CI. This is your OWASP LLM08 regression test. Relationship-based access control is the pattern that holds up here (see ReBAC document permissions for RAG).

5. Freshness guarantees

• [ ] Every chunk carries a source timestamp, and every answer surfaces the "as of" date of its sources.
• [ ] Re-indexing triggered by source-system change events, not only by nightly cron. A policy document updated at 9:00 should not be contradicted by your assistant at 9:05.
• [ ] A staleness SLA per source: defined maximum lag, monitored, alerted.

6. Operational proof

• [ ] Query, retrieved chunks, answer, and citations logged as a single audit record. This is what you hand an auditor (see AI audit trail requirements).
• [ ] Rollback for index, prompt, and model versions, independently.
• [ ] Per-query cost and latency budgets, measured against real traffic, not the demo's.

A compact summary for the launch review:

The case against: just put it all in context

The strongest counterargument is that this checklist is engineering theater for a problem that context windows are deleting. Models with million-token windows are shipping. Prompt caching makes re-sending large documents cheap. The skeptic's plan is simpler: skip the chunking, embedding, and retrieval stack, dump the corpus into context, and let the model sort it out. Fewer moving parts, fewer failure points, and no stale index because there is no index.

This deserves a serious answer, because it is half right. For small, stable corpora, it genuinely works. A 50-page contract playbook does not need a vector database.

But the evidence does not support the general case. NVIDIA's ICLR 2024 study comparing retrieval against extended-context models found that retrieval improves performance regardless of context window size, and that a 4K-context model with simple retrieval matched a fine-tuned 16K model at a fraction of the compute. Long context and retrieval combined beat either alone. And the regulated-industry objections are structural, not benchmark-shaped. You cannot ACL-filter a prompt you already filled. You cannot prove "as of" freshness for a context window assembled by hand. You cannot produce an audit trail showing which document supported which claim when the answer came from 400 pages pasted into one prompt. Long context changes the economics of retrieval. It does not change the governance requirements.

Our read

Treat retrieval as a control plane, not a search feature. The checklist above is not a maturity aspiration; it is the minimum for any system whose answers carry regulatory weight. The teams that struggle are not the ones with weak models. They are the ones who treated evals, permissions, and freshness as phase two. In a regulated environment there is no phase two. There is the day an auditor, a client, or an opposing counsel asks where an answer came from, and you either replay the record or you do not. Controls as architecture: build the gates into the pipeline, and the demo-to-production gap stops being a gap.

What this means for you

• Score your current system against the six gates this week. Most teams we assess pass one or two. Permission filtering and citation enforcement are the two most commonly missing, and they are the two with legal exposure attached.
• Build the negative test suite first. Proving a user cannot retrieve what they must not see is harder than proving they can retrieve what they should, and it is what your CISO will ask about.
• Separate retrieval recall from answer quality in your metrics. When answers degrade, you need to know whether retrieval or generation broke. Averaged metrics hide the diagnosis.
• Put the staleness SLA in writing. "The index updates nightly" is not a freshness guarantee; it is a 24-hour window where your assistant contradicts your policy.
• If your corpus is small and static, long context may be the honest answer. Take the skeptic seriously for that case and skip the retrieval stack. The moment permissions, freshness, or auditability enter the requirements, retrieval is back on the table.

Book a free assessment: systemics.ai.

References

1. Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M. "Seven Failure Points When Engineering a Retrieval Augmented Generation System." arXiv:2401.05856, January 2024. https://arxiv.org/abs/2401.05856
2. OWASP GenAI Security Project. "OWASP Top 10 for Large Language Model Applications 2025." OWASP, November 2024. https://genai.owasp.org/llm-top-10/
3. National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)." NIST, July 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
4. Xu, P., Ping, W., Wu, X., et al. (NVIDIA). "Retrieval meets Long Context Large Language Models." ICLR 2024 / arXiv:2310.03025, October 2023. https://arxiv.org/abs/2310.03025
5. Moffatt v. Air Canada, 2024 BCCRT 149. British Columbia Civil Resolution Tribunal, February 14, 2024. https://canlii.ca/t/k2spq
6. LiquidMetal AI. "Is Retrieval Augmented Generation (RAG) Dead in the Era of Large Language Models?" April 2025. https://liquidmetal.ai/casesAndBlogs/rag-dead/