Separating Retrieval Failures From Generation Failures in RAG Evaluation

Short answer: Score the two stages independently — retrieval with context precision and recall, generation with faithfulness and answer relevancy — and read the pair together. The combination localizes the fault: if the right context was retrieved but the answer is still wrong, it's a generation problem; if the context was missing or noisy, it's a retrieval problem. You cannot make this call without storing the exact chunks that were retrieved for each eval case.

Why you can't debug a single end-to-end score

A single "answer quality" number tells you the system failed, not why. RAG has two stages that fail for completely different reasons and get fixed in completely different places. Retrieval metrics diagnose the index and the retriever; generator metrics (faithfulness, answer relevancy) diagnose the LLM and the prompt. Collapse them into one score and every failure looks the same, so you end up guessing.

The distinction is only recoverable if you log the retrieved context alongside each answer. This is the single most important RAG debugging move (as Maxim and qaskills both stress): without stored context you literally cannot separate a bad-retrieval failure from a bad-generation one, because you have no way to check whether the evidence was even present when the model answered.

The four-quadrant diagnosis

Cross good/bad retrieval with good/bad generation and every failure lands in exactly one cell:

• Good retrieval / good generation — passing case, nothing to fix.
• Good retrieval / bad generation — the context was there and the model still got it wrong. Generation problem.
• Bad retrieval / good generation — the model faithfully used what it got, but it got the wrong chunks. Retrieval problem.
• Bad retrieval / bad generation — both broken; fix retrieval first, because generation can't recover from evidence it never saw.

Metrics per stage and what each combination implies

Using the RAGAS framing:

• Retrieval — context precision & context recall. Recall asks: were the chunks needed to answer actually retrieved? Precision asks: of what was retrieved, how much was relevant (vs. noise diluting the prompt)? Low recall means the answer's evidence never made it into the window. Low precision means the model had to find signal in noise.
• Generation — faithfulness & answer relevancy. Faithfulness asks: is every claim in the answer grounded in the retrieved context (or is the model hallucinating / ignoring it)? Answer relevancy asks: does the response actually address the question?

Read them as a pair:

• Recall high, faithfulness low → context was present, model ignored or contradicted it → prompt / model fix.
• Recall low, faithfulness high → model behaved correctly on bad inputs → retriever fix.
• Precision low, answer relevancy low → noisy context is derailing the model → retrieval fix (rerank / tighten chunks) before touching the prompt.

Storing retrieved chunks is non-negotiable

For every eval case, persist: the query, the ordered list of retrieved chunk IDs + text, the scores, the final answer, and (for context recall) the ground-truth reference. This is what lets a metric attribute blame to a stage.

{
  "query": "What is the refund window for damaged items?",
  "retrieved_chunks": [
    {"id": "doc_412#3", "score": 0.71, "text": "Returns must be initiated within 30 days..."},
    {"id": "doc_089#1", "score": 0.66, "text": "Shipping is calculated at checkout..."}
  ],
  "answer": "You have 14 days to request a refund.",
  "reference": "Damaged items: 30-day refund window."
}

Here the correct chunk was retrieved (recall is fine) but the answer says 14 days — faithfulness is low. Without the stored chunks you'd blame the retriever and waste a week reranking a retriever that worked.

Fix routing

Once the quadrant names the stage, the fix follows:

• Retrieval failures → embedding model, chunk size/overlap, adding a reranker, hybrid (dense + BM25) search, query rewriting, metadata filters. Do not touch the prompt.
• Generation failures → prompt instructions ("answer only from context"), stronger/larger model, output-format constraints, forcing citations, lowering temperature. Do not touch the index.

Fixing the wrong stage is the most common wasted-effort pattern in RAG — and the quadrant exists precisely to stop it.

Store the retrieved context, score both stages separately, and let the pair of metrics — not a hunch — route the fix.

Related: more writing.