Your RAG problem is a retrieval problem

A retrieval system that returns the right document 95% of the time sounds excellent until you work out what the other 5% does to your answers.

Confident wrong answers are the failure mode

When retrieval misses, a language model does not say "I could not find this." It answers from whatever it did retrieve, in the same confident register as when it is right. The output is indistinguishable from a good answer unless the reader already knows the subject.

That is worse than an error. An error gets reported; a plausible wrong answer gets believed.

Measure retrieval on its own

The most common mistake we see is evaluating only the end-to-end answer. That conflates two systems with different failure modes and different fixes.

Evaluate retrieval separately:

• For a set of real questions, does the correct source appear in the top k results?
• When it does not, why — vocabulary mismatch, chunking, or genuinely missing content?

You cannot fix a retrieval problem by changing the model, and teams burn a lot of money discovering that.

Give the model somewhere to fail safely

Once retrieval is measured, the answer layer gets a cheap improvement: pass through the confidence, and let the model say it does not know. An assistant that occasionally declines is far more useful than one that never does.