> ## Content Index > Fetch the complete content index at: https://insights.cloudacio.com/llms.txt > Use this file to discover other available public pages before exploring further. # Why your RAG demo works and your RAG product does not - URL: https://insights.cloudacio.com/moving-a-rag-prototype-to-production/ - Published: 2026-10-27T01:45:02.000Z - Updated: 2026-11-05T20:18:34.000Z - Description: A RAG demo is tested on questions and documents its builder chose. What changes in production, and why a fixed set of fifty real questions is the first thing to build. - Author: Juan Pablo Olivera - Tags: #author-juan-pablo-olivera The demo goes beautifully. You drop a folder of company documents into the system, ask it a few questions, and it answers in two seconds with the sources linked underneath. Everyone in the room agrees it is ready. Weeks later it is still "nearly ready", and nobody can tell you what is actually left to do. I want to explain why that gap is so common, and what closes it. The short version is that this is ordinary engineering, not research, and it gets far easier the moment you can measure what you are trying to improve. First, the term. **RAG**, or retrieval-augmented generation, means the model does not answer from memory. Before it writes anything, the system searches your documents, grabs the passages that look relevant, and pastes them into the prompt. It is an open-book exam instead of a closed-book one. The model is still the student, but now it has your handbook on the desk. ## The demo was rigged, and that was fine Not dishonestly. Rigged the way any first test is. The builder picked the documents, wrote the questions, and judged the answers with the source still fresh in their head, so a half-right answer read as right. Production gets none of that help. It gets questions nobody anticipated, a document pile nobody curated, and no expert reading over the model's shoulder. All fixable. You just have to see them first. ## The seven ways these systems break The most useful map I know is a paper called [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://arxiv.org/abs/2401.05856?ref=insights.cloudacio.com), by Barnett and colleagues. They wrote it from three real systems they built, in research, education and biomedicine. Their list walks the whole path, from your document pile to the sentence the user reads. The answer simply is not in your documents. Or it is there, but the search ranks it too low to find. Or the search finds it and the system drops it before it reaches the model. Or the model gets it and still fails to pull the answer out. Or the answer is right but arrives in the wrong format, or too vague, or weirdly over-specific, or missing half of what was asked. One of their conclusions deserves a sticky note on your monitor. A RAG system can only really be validated while it is running. It gets reliable through operation, not through clever design up front. The first failure on that list is the one a demo never catches, and it is the nastiest. When your documents have no answer, the model usually does not say so. It writes a confident, plausible paragraph in exactly the tone it uses when it is correct. Normal software breaks loudly, with a stack trace and a red alert. This breaks without an error, and it looks like it worked. ## Real users do not ask demo questions Demo questions are tidy. Real ones are not. Real users type three words and a typo. They ask two unrelated things in one message. They call the product by the name you retired two years ago. They send a follow-up that only makes sense if you remember what came before it. The two-questions-in-one case is worth watching for. The search finds documents about one of the topics, and the model answers that half with complete confidence, as if the other half was never there. None of this appears until you go and collect real questions. Support tickets, sales call notes and internal chat channels are the best source, since they are written in your users' words rather than your engineers'. ## Your real documents are a mess, and that is normal A curated demo folder hides a lot. Several versions of the same policy sit side by side, and nothing in the files says which one is current. A human knows; the index does not. Some PDFs are scans, which means the "text" is really a picture of text, so a plain extractor pulls out nothing and the file vanishes from the index without a warning. An OCR service such as [Amazon Textract](https://aws.amazon.com/textract/?ref=insights.cloudacio.com) reads those pictures back into text, but only if somebody checks which files came back empty. And two current documents from two departments flatly contradict each other. The model will pick one, and will not mention it had a choice. On a small clean corpus, almost any search setup looks fine. On a big messy one full of near-duplicates, the differences show up fast. Anthropic measured this when it published [contextual retrieval](https://www.anthropic.com/news/contextual-retrieval?ref=insights.cloudacio.com), a technique that tags each chunk of text with a one-line note about where it sits in its parent document before indexing it. It is the difference between filing a loose page and filing it with a header that says which report it came from. ![Column chart: share of relevant chunks missing from the top 20 results. Embeddings only 5.7%, contextual embeddings 3.7%, adding contextual BM25 2.9%, adding reranking 1.9%. Source: Anthropic, September 2024.](https://storage.ghost.io/c/62/8f/628feacd-48cf-4de5-85ac-9760eb6467c8/content/images/2026/11/retrieval.svg) Each step is a change to retrieval, not to the model. Combining embeddings with keyword search ([hybrid search](https://docs.opensearch.org/latest/vector-search/ai-search/hybrid-search/index/?ref=insights.cloudacio.com)) and then reranking cut missed chunks by 67% across Anthropic's test sets. The chart measures one thing, which is how often the passage holding the answer never reached the results at all. What matters is not the technique but where the gains came from. Every step there changes how the system searches. Not one changes the model. If a chunk is never retrieved, the model never saw the answer, and no model upgrade fixes that. So when answers come back wrong, read what the system actually retrieved before you touch the prompt. ## Who is allowed to see what Then there is access control, which turns a quality problem into a legal one. Salary figures, performance reviews, one customer's contract terms... these cannot be retrievable by whoever happens to ask. In a demo the flaw is invisible, since the only person asking is you. In production the search step has to filter by who is asking, which means tagging documents with metadata such as owner, department or tenant. Amazon Bedrock Knowledge Bases lets you attach [metadata to each document](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-metadata.html?ref=insights.cloudacio.com) and apply [metadata filters at query time](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html?ref=insights.cloudacio.com). Do this early. If permissions were not part of how documents get loaded in the first place, bolting them on later usually means re-indexing everything. ## Citations are not proof It is tempting to assume a well-built RAG system stops making things up, since look, it cites its sources. The best public evidence says otherwise. A Stanford team [tested commercial legal research tools](https://arxiv.org/abs/2405.20362?ref=insights.cloudacio.com) from LexisNexis and Thomson Reuters on 202 real legal questions. These are retrieval-based products, built by large vendors on top of carefully curated legal databases. Not weekend prototypes. Depending on the tool, between 17% and 33% of answers were hallucinated. That is somewhere between one in six and one in three. ![Donut chart: Lexis+ AI answers were 65% accurate, 18% incomplete and 17% hallucinated. Source: Magesh et al., Stanford, 2024.](https://storage.ghost.io/c/62/8f/628feacd-48cf-4de5-85ac-9760eb6467c8/content/images/2026/10/legal-1.svg) The best performer in the study. The hallucinated slice includes answers that cite a real source for a claim the source does not support, which a reader checking only that a citation exists would miss. Read that caption twice. The dangerous failure is a real, working source attached to a claim the source does not make; a made-up citation, by contrast, anyone would catch. A reader who checks that the link resolves will nod and move on. Where you put things in the prompt matters too. The paper [Lost in the Middle](https://arxiv.org/abs/2307.03172?ref=insights.cloudacio.com) found that models use what sits at the beginning and end of a long prompt far better than what sits in the middle. Rather like skimming a long email. So cramming in more retrieved passages is not a free upgrade, and past a point it buries the good one. You can put a guard at the exit. [Contextual grounding checks](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html?ref=insights.cloudacio.com) in Amazon Bedrock Guardrails score every response on two questions: is this supported by the source we retrieved, and does it answer what was asked? Anything below a threshold you set gets blocked. That catches some inventions before a user sees them, but it will not tell you whether the system is getting better or worse month over month. ## What it costs when everyone starts using it A demo answers one question at a time, for one patient person who is rooting for it. Production is different. Every question pays for the retrieved passages plus the answer, every time, and the bill grows with adoption. A wait that felt snappy in a meeting feels sluggish on the fortieth question of a shift. Before agreeing to a launch date, I would want two things measured rather than guessed: what one typical answer costs, including search and any reranking or guardrail calls, and how long answers take at the 95th percentile under the traffic you expect. That percentile is the slow end of the range, and the middle hides the experience people complain about. Both move with how many passages you retrieve, which is one more reason to measure retrieval first. ## Build the evaluation set before you build anything else If I could do exactly one thing on a stalled RAG project, this would be it. An **evaluation set** is a fixed list of real questions, each paired with the answer you expect and the document it should come from. It is a driving test for your system. The point is that it never changes, so results are comparable week to week. Three things make it work: - **Use real questions.** Pull them from support tickets and sales notes, in the words users actually typed. - **Include questions your documents cannot answer.** For these, "I don't have that information" is the correct answer, and you record it as such. This is the only way you will ever measure confident nonsense. - **Have the document owners confirm the expected answers.** Whoever owns the policy decides what it says. Otherwise your test is just an engineer's guesses, written down neatly. Fifty questions is plenty to start. You are looking for patterns, not statistical significance. It comes first for a simple reason. Every other fix depends on it. Change the chunk size, add hybrid search, switch to a cheaper model, tighten the access filters. Without a fixed set of questions you cannot tell whether the change helped. Worse, you cannot tell what it broke. With one, "nearly ready" stops being a feeling and becomes a scoreboard. Something like 43 right, 4 wrong, 3 correctly refused. Now the team can have a real conversation about whether that is good enough for internal staff, and what it would take before customers see it. You do not need to build a framework. [Amazon Bedrock evaluations](https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-kb.html?ref=insights.cloudacio.com) can score search-only or full question-and-answer runs against your own dataset, including systems hosted outside Bedrock. [LangSmith](https://docs.smith.langchain.com/evaluation?ref=insights.cloudacio.com) and [OpenAI Evals](https://platform.openai.com/docs/guides/evals?ref=insights.cloudacio.com) run your dataset against your pipeline and track results across versions. [Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available%5Fmetrics/faithfulness/?ref=insights.cloudacio.com) offers metrics such as faithfulness, which checks whether an answer's claims are supported by the passages retrieved. One caution on automated grading. Letting a model grade the answers is useful, since it lets you run the whole set on every change. But have a human grade a sample now and then. The judge is a model too, with the same blind spots as the thing it is judging. Then run the set on every release, and again whenever the source documents change. A routine policy update next month can break a dozen answers, and the evaluation run is the only thing that will tell you. ## What to do this week Build that set of fifty questions, run it against what you have today, and record three counts: right, wrong, and correctly refused. Then, for every wrong answer, look at what the system retrieved before you change a single setting. That list will tell you where next week belongs, whether that is your documents, your retrieval, or your model. A much better place to be than "nearly ready".