RAG evaluation answers two questions separately: did retrieval put the passage containing the answer into the context, and did the answer stay inside it? One end-to-end score says the system is wrong. It cannot say which half to fix, or whether the next chunking change improved anything.
Information retrieval settled this first. BEIR (Thakur and colleagues, NeurIPS 2021) benchmarked 10 retrieval systems over 18 public datasets and found BM25 hard to beat, with dense and sparse models often losing to it outside their training domain. That is zero-shot IR rather than RAG, and the reading is ours: when the ranking of methods moves with the corpus, someone else's benchmark is a hypothesis, and only a RAG evaluation run on your questions is evidence.
What RAG Evaluation Actually Measures
RAG evaluation is the practice of scoring two subsystems separately: the retriever, on whether it placed the passage containing the answer into the context, and the generator, on whether its answer stays inside that context. A quality drop then points at a component, not at the system.
Lewis and colleagues introduced retrieval-augmented generation in 2020 as parametric memory plus a non-parametric index: two moving parts, two ways to fail. The attribution rule: if the golden passage was not in the context, it is a retrieval failure and the answer score tells you nothing; if it was there and the answer still went wrong, it is a generation failure.
This article is about measuring a RAG system you already have: which numbers say retrieval failed, which say the model left the evidence behind, and what blocks a release. It does not cover how the retrieval stack itself is put together, whether retrieval or fine-tuning fits a use case, or how to supervise a live model.
RAG Evaluation Metrics: Retrieval First, Generation Second
Retrieval metrics are the cheap half: no model call, and they cap everything downstream, since no generator grounds an answer in a passage it never got.
- Recall@k: is the relevant passage anywhere in the top k? Position-blind, and the number to read first.
- Precision@k: how much of what came back is relevant? Low precision costs tokens and crowds evidence, a tax rather than a failure.
- MRR rewards an early first hit, which matters when one passage holds the answer.
- nDCG, normalized discounted cumulative gain, weights graded relevance by position, for labels that are not binary.
Position matters too. Liu and colleagues, in Lost in the Middle (TACL 2024), showed that performance tracks where the relevant information sits: highest near the beginning or the end, lower when the model must pull it from the middle of a long context, including in models sold as long-context. So passage count and order are evaluation variables, and k is one too, our recommendation: set it from the context budget you can ship, then measure recall@k at that k.
The generation half of RAG evaluation is where vocabulary stops being reliable: faithfulness, groundedness and relevance are names tools give to their own procedures, so two tools return different numbers under one label.
In Ragas, faithfulness is the share of claims in an answer inferable from the retrieved context, scored 0 to 1 by extracting claims and checking each against it, while non-LLM context recall counts how many reference passages reached the retrieved set. Azure AI Foundry cuts it differently: process evaluation scores a component, where Document Retrieval metrics such as Fidelity and NDCG need relevance labels while a separate Retrieval metric uses an LLM judge without them, and system evaluation scores the answer with Groundedness on a 1 to 5 scale, Relevance and Response Completeness.
A faithfulness of 0.86 means nothing without the tool, the judge model and the definition behind it. The NIST generative AI profile (AI 600-1, July 2024, voluntary) names the target: confabulation, "the production of confidently stated but erroneous or false content", a natural result of how generative models are designed.
The Golden Set: Reference Data You Can Actually Build
Teams stall here by assuming reference answers. RAG evaluation does not need one for every question. It needs a pointer to the passage that contains the answer: cheaper, more stable and directly measurable with retrieval metrics.
A usable golden set holds real questions, from traffic or domain experts, each with a golden passage attached and metadata to slice results by question type. Anthropic's guidance on building evaluations supplies the design rules: mirror the real distribution of tasks, cover edge cases including missing data, over-long inputs and questions where humans disagree, and automate grading. One line to keep: "More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals."
Two case types are almost always missing and they catch real regressions: questions the corpus cannot answer, where the right behavior is a refusal, and questions needing two documents. That is our recommendation. KILT (Petroni and colleagues, NAACL 2021) grounds its tasks in one Wikipedia snapshot and scores provenance as well as output, so pointing at evidence is no amateur substitute.
Golden sets age: a labeled passage can be deleted or stop holding the current answer while the label still claims it does, which makes an owner and a review rhythm part of the design. On size, RAG evaluation rewards coverage over volume: big enough means every important question type has several examples and no single change flips the result through two or three cases.
Offline and Online Evaluation Answer Different Questions
The LangSmith documentation draws the line at the data. Offline evaluation runs over datasets pairing an input with a reference output, which makes version comparison, regression testing and backtesting possible. Online evaluation runs over production traces with no reference output, so it can only spot anomalies and harvest cases for the offline set.
As one rule for RAG evaluation: offline can ask whether the answer was right, online only whether it looked grounded, because production traffic ships without an answer key. Cases caught online come back as labeled examples offline, which keeps a golden set current.
Without an answer key, online scoring leans on an LLM judge. Zheng and colleagues found strong judges reach over 80% agreement with human preferences, the level humans reach with each other, in open-ended chat comparisons rather than RAG and with 2023 judge models, alongside position, verbosity and self-enhancement bias. In retrieval, UMBRELA showed machine-generated relevance assessments correlate strongly with system rankings on TREC Deep Learning tracks, agreement about which system wins, not label by label. Clarke and Dietz are the counterweight: they submitted a system built to score well under an automatic assessor, and it did, without retrieving better. The TREC 2024 RAG procedure treats automatic assessment as a draft, post-edited by human assessors.
None of this is monitoring, which asks whether a live system's behavior has changed over time and is covered in watching a deployed model over time. Online evaluation asks about one response: does this answer hold up against its sources?
RAG Evaluation Gates: What Breaks When You Change the Chunker
Almost every change in a RAG system changes a search engine, so it has to be re-measured, and some need re-labeling first. Change the chunker and the old golden passages no longer exist under their old IDs, so a before-and-after comparison counts two things. Ragas ships context recall in three variants, LLM-based, text-based and ID-based, which makes the consequence explicit: a text label survives re-chunking, an ID label does not.
Embedding swaps deserve the same suspicion. MTEB (Muennighoff and colleagues) evaluated 33 models over 8 task types, 58 datasets and 112 languages and found no text embedding method dominating across all tasks. A leaderboard position is a hypothesis; your golden set is the test.
Comparability has one prerequisite: prompt, retrieval configuration, chunking parameters and generator version travel with the result as one artifact, because providers change model behavior under an unchanged name. OpenAI's evals guide assumes the same, describing runs as versioned and repeatable against a defined dataset. A RAG evaluation run that cannot be reproduced against the configuration that produced it is an anecdote with decimal places. The last two columns below are our recommendation.
| Change you ship | What it can break | Metric that catches it | Needs relabeling? |
|---|---|---|---|
| Chunking strategy or size | Which passages exist at all | recall@k, context recall | Yes, if labels use chunk IDs |
| Embedding model | Which passages come back | recall@k, nDCG | No |
| Top-k and context budget | Whether the answer passage survives | recall@k at the production k | No |
| Reranker | Order of passages the model reads | nDCG, MRR | No |
| Prompt template | Whether the model stays in evidence | faithfulness, answer relevance | No |
| Generator model or version | Wording, refusals, citations | faithfulness, refusal rate | No |
| Corpus update | Whether the labeled passage still answers | context recall, staleness review | Yes, for affected items |
Gates follow from the same split. A retrieval threshold blocks promotion, because that failure is unambiguous; a generation threshold warns until the judge is calibrated against human review; a drop on the critical-case subset blocks whatever the averages say, with thresholds agreed with the product owner before the change.
What RAG Evaluation Costs, and Where to Spend It
Nobody publishes this bill, so here is the model. One RAG evaluation run costs questions, times judge-scored metrics, times model calls per metric, times how often it fires in CI. The third term breaks estimates: faithfulness in Ragas splits the answer into claims and verifies each, so it is never a single call per question.
Spend in this order, our recommendation. Deterministic metrics first, because recall@k, precision@k and nDCG against labels are nearly free and catch the most common failure. LLM judges second, on a subset and only for generation metrics. Human review third, on a small sample, calibrating the judge rather than producing labels. ARES (Saad-Falcon and colleagues, NAACL 2024) shows how far that goes: lightweight judges fine-tuned on synthetic data, corrected with prediction-powered inference, across eight tasks with a few hundred human annotations.
So do not buy an evaluation platform before the golden set exists: the set is the asset, tooling is replaceable.
In our delivery work, evaluation is the first thing we stand up on a RAG project and the last we hand over, because it outlives the model, the vendor and the team. See our generative AI practice for the wider program of work. If your RAG system is live and all you have is one dashboard with one number, talk to our AI engineers about your corpus.
Frequently Asked Questions
What is RAG evaluation?
RAG evaluation is the separate scoring of retrieval, whether the passage containing the answer reached the context, and of generation, whether the answer stayed inside it, plus an attribution rule saying which number to act on. Retrieval uses recall@k, precision@k, MRR and nDCG; generation uses faithfulness, groundedness and answer relevance, defined by whichever tool computes them.
How big should a golden set for RAG evaluation be?
Judge it by coverage rather than count: large enough means every important question type has several examples, questions the corpus cannot answer are represented, and one change cannot flip the result through three cases. Anthropic's guidance breaks the tie: more questions graded automatically beats fewer graded by hand.
Can an LLM judge replace human evaluation?
Not entirely, though it can carry most runs. Strong judges reached over 80% agreement with human preferences in open-ended chat comparisons (Zheng and colleagues, 2023), which is agreement rather than correctness, and the TREC 2024 RAG procedure still puts human assessors over machine-generated labels. Watch the named biases: position, verbosity and self-enhancement.
How do you tell whether retrieval or generation caused a bad answer?
Check whether the golden passage was in the context. If not, this is a retrieval failure and the answer score carries no information. If it was there and the answer is still wrong, this is a generation failure: check faithfulness and where the passage sat among the results.
