Building a Golden Dataset for RAG Evaluation From Your Production Traces

Short answer: Mine your real failing traces (thumbs-down, bad answers, support tickets), and turn each into a case that stores three things: the input, the exact retrieved chunks, and the expected behavior. Keep the suite small and high-signal — 50-100 versioned cases — and store it in your repo so every change runs against real production failures instead of synthetic guesses.

Start from failures, not imagination

The fastest path to a useful eval set is to stop inventing questions and start harvesting the ones your system already got wrong. Begin with 50-100 focused cases built from real production failures. That range is deliberate: a small set of high-signal cases you actually curate beats thousands of synthetic ones, because synthetic questions cluster around the queries you already expected and miss the messy inputs that break you in production.

Store retrieved context per case

This is the part most teams skip, and it quietly ruins their eval loop. Each case should store the input, the retrieved chunks, and the expected behavior — so you can score retrieval separately from generation.

Without the chunks, a failing case only tells you the final answer was wrong. You can't tell whether the retriever pulled the wrong documents or the generator fumbled good context. With the chunks recorded, the split becomes mechanical:

• Chunks don't contain the answer → retrieval failure (fix chunking, embeddings, filters, or top-k).
• Chunks contain the answer but the output is wrong → generation failure (fix the prompt, model, or grounding).

A minimal case looks like this:

{
  "id": "prod-2026-0417-refund-window",
  "input": { "query": "can I get a refund after 40 days?", "user_tier": "pro" },
  "retrieved_chunks": [
    { "doc_id": "policy#refunds", "score": 0.71, "text": "Refunds within 30 days..." }
  ],
  "expected": {
    "must_include": ["30-day window", "no refund after 30 days"],
    "must_not_hallucinate": ["40-day", "exceptions for pro users"]
  },
  "source": "thumbs_down",
  "version": "v3"
}

Where the cases come from

Pull from every channel where reality disagrees with your system:

• Production logs — traces with low confidence, empty retrieval, or fallback answers.
• Thumbs-down — explicit user signal that an answer missed; the highest-value source you have.
• Support tickets — real questions phrased the way users actually phrase them, not the way you'd write a test.
• Deliberate edge cases — ambiguous queries, multi-hop questions, out-of-scope asks.
• Adversarial inputs — prompt injection, contradictory context, questions with no answer in the corpus (the answer should be "I don't know").

A workable minimum viable suite: 10-20 golden examples, plus 5-10 edge cases, plus 3-5 adversarial inputs. Grow toward the 50-100 range as production surfaces new failure modes.

Labeling: rubrics, humans, and judges

For factual lookups, store an exact expected answer. But most RAG outputs are open-ended, so use rubrics where exact answers don't fit — score against criteria like "cites the refund policy," "does not invent a timeframe," "declines when context is missing."

Grade with a combination of human review and an LLM judge. The LLM judge scales; humans keep it honest. Run periodic calibration: sample a batch, have a human grade it, and compare against the judge. If judge-human agreement drifts, fix the judge prompt before you trust another automated run. Track that agreement number over time — it's the health metric for your whole eval pipeline.

Version it, and guard against leakage

Store the dataset in-repo as versioned files (JSONL or similar) so every prompt change, model swap, and retriever tweak runs against the same fixed set, and diffs are reviewable in PRs. Bump a version when you add or change cases so results stay comparable across runs.

Then guard the boundary hard: golden cases must never leak into training data, few-shot prompt examples, or retrieved context. The moment your system has seen the answers, the eval measures memorization instead of capability. Keep evaluation data in a namespace your indexing and fine-tuning pipelines explicitly exclude.

Keep it fresh by continuously adding new failure cases as production surfaces them and re-running calibration on a schedule — a golden dataset is a living artifact, not a one-time deliverable.

Store the retrieved chunks with every case — it's the one detail that lets you tell a retrieval bug from a generation bug.

Related: more writing.