Reference-Free vs Reference-Based LLM Evaluation: When You Actually Need Ground Truth
Reference-based evals compare output to a known correct answer; reference-free evals score intrinsic properties like grounding without ground truth.
Contents
Short answer: Reference-based evaluation compares a model's output against a known correct answer (a gold label) and scores how close it is. Reference-free evaluation scores intrinsic properties of the output — like whether it's grounded in the retrieved context or relevant to the question — without any ground truth. You need reference-based when there's a single correct answer to check; reference-free when correctness is open-ended or you have no labels.
The core difference
Reference-based asks: does this match the answer we already know is right? You supply a gold answer per example, and the scorer measures similarity, exactness, or entailment against it. Classic examples: exact match, F1, or an LLM judge told "here is the correct answer, grade the response against it."
Reference-free asks: is this output well-formed on its own terms? No gold answer exists. Instead the scorer checks properties like faithfulness (are the claims supported by the retrieved context?), relevancy (does it actually address the question?), or fluency. Reference-free scorers check groundedness against the retrieved context, not a gold answer.
Comparison
| Reference-based | Reference-free | |
|---|---|---|
| Needs | Labeled gold answers per example | Just input + output (+ context for RAG) |
| Measures | Correctness vs known truth | Grounding, relevancy, coherence |
| Cost | High upfront (human labeling) | Low upfront; per-eval LLM-judge cost |
| Scales to prod | No — can't label live traffic | Yes — runs on every live request |
| Fails when | Multiple valid answers exist | The answer is fluent + grounded but factually wrong |
Where reference-free wins
- RAG faithfulness. You want to catch hallucination: claims the retrieved chunks don't support. That's a property of the output-plus-context, not a gold answer — so reference-free is the right tool.
- Open-ended generation. Summaries, rewrites, chat replies. There's no single correct string, so a gold answer is meaningless. You score relevancy and coherence instead.
- Live production monitoring. You can't hand-label millions of live queries. Reference-free scorers run online on real traffic.
Where it fails
Reference-free cannot verify factual correctness against a known answer. For a factual QA task with one right answer — "what year did X ship?", "which API returns Y?" — a response can be perfectly grounded in a retrieved chunk that happens to be wrong or outdated, and score high on faithfulness while being flat-out incorrect. Here you need a reference. If exact strings don't fit (dates phrased differently, paraphrased entities), golden-dataset guidance recommends rubric labeling instead of exact reference answers — you label what a correct answer must contain, and grade against the rubric.
The hybrid approach
You don't pick one. The pattern that actually works:
- Reference-free online — run faithfulness and relevancy on live traffic for continuous coverage and hallucination alerts.
- Reference-based on a golden set — maintain a curated, labeled evaluation set and run correctness checks against it in CI, before each release.
The golden set catches regressions in correctness the online scorers structurally can't see; the online scorers catch grounding failures at a scale the golden set never covers.
How this maps to RAGAS
RAGAS provides both families, which is why it fits the hybrid pattern cleanly:
- Reference-free:
faithfulness(claims entailed by retrieved context) andanswer_relevancy(does the answer address the question). No ground truth needed — run these online. - Reference-based:
context_recalland answer-correctness style metrics that compare against a reference. These need your golden set.
# Online: no ground truth needed
from ragas.metrics import faithfulness, answer_relevancy
online = [faithfulness, answer_relevancy]
# CI / golden set: needs reference answers
from ragas.metrics import context_recall, answer_correctness
golden = [context_recall, answer_correctness]Pick the metric by what you can supply. If you have a context but no gold answer, you're limited to reference-free — and that's fine for grounding. The moment you need to certify correctness, invest in the golden set.
Reference-free tells you the answer is well-built; reference-based tells you it's right — production needs both.
Related: more writing.
Frequently asked
- What is the difference between reference-free and reference-based LLM evaluation?
- Reference-based compares output against a known correct answer; reference-free scores intrinsic properties like grounding and relevancy without any ground truth.
- Is reference-free evaluation less reliable?
- Not inherently; it measures different things. It excels at grounding and relevancy but cannot verify factual correctness against a known answer.
- Do I need ground-truth answers for RAG evals?
- Not for faithfulness or answer relevancy, which are reference-free. You need references for context-recall-style and answer-correctness metrics.
- Can I combine both?
- Yes. Reference-free online for coverage and hallucination alerts, reference-based on a curated golden set for correctness checks in CI.
Keep reading
Why RAG fails in production — and where
When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.
ReadWhy I don't use LangChain in production RAG
Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.
ReadWhen NOT to use RAG (RAG vs fine-tuning vs prompting)
RAG became the default reflex for every LLM problem. Often it's the wrong tool. A straight decision framework for RAG vs fine-tuning vs just prompting.
ReadGet new essays by email
Occasional field notes on production RAG and LLM systems. No spam, unsubscribe anytime.