LLM evaluation: a practical guide to evals, metrics, and judges

Guide · Evals

LLM evaluation is the practice of measuring whether an application built on a large language model produces outputs that are correct, safe, and useful for the tasks it serves. Unlike a model benchmark, LLM evaluation tests the whole system: the prompt, the retrieval step, the tools, the output format, and the model working together. This guide explains how to build an evaluation practice that engineers trust, from datasets and deterministic checks to LLM-as-a-judge, human review, RAG and agent evals, and release gates in CI. It belongs to our LLM evaluation and observability hub.

What is LLM evaluation?#

LLM evaluation is a repeatable process for scoring a language model application's outputs against explicit criteria, using datasets, automated graders, and human review. It answers one narrow question for each change: did this prompt edit, model swap, or retrieval tweak make the system better or worse on the tasks that users actually bring to it?

Every eval has the same four parts:

  • Dataset: inputs and, where possible, reference outputs or written grading notes.
  • Task runner: code that calls the application exactly as production does, with the same system prompt, retrieval, tools, and parameters.
  • Graders: code checks, model judges, or people that turn each output into a score.
  • Report: aggregate scores, per-case differences against a baseline, and the transcripts behind each failure.

Public leaderboards measure models on shared tasks, and our guide to LLM benchmarks covers how to read them. A benchmark score says little about whether your prompt handles your own customers' refund questions, which is why application evals exist. Outputs also vary between runs at any nonzero temperature, so one run of one case is an anecdote rather than a measurement.

Offline vs online evaluation#

Offline evaluation runs a fixed, versioned dataset through the application before a change ships, so results stay comparable across versions. Online evaluation scores live production traffic after release and catches inputs the dataset never anticipated. Mature teams run both: offline evals gate releases, and online evals feed new failure cases back into the offline dataset.

Offline evaluation Online evaluation
When it runs Before release, on every change After release, continuously or on a sample
Inputs Curated, versioned dataset Real user traffic
Ground truth Reference answers or rubrics Usually none, so graders must be reference-free
Typical graders Code checks, LLM judges, human review Reference-free judges, user feedback, guardrail hits
Main job Catch regressions and compare candidates Detect drift and discover new failure modes
Blind spot Covers only what someone thought to include Noisy and slow as a signal for one specific change

Online evaluation depends on capturing traces of real requests, which is the subject of our guide to LLM observability. A/B tests are also a form of online evaluation, but they measure product outcomes such as task completion or retention rather than output quality directly.

Offline evals run the eval dataset and gate releases; online evals score sampled production traces and turn new failures into dataset cases, closing the loop. OfflineOnlineEval datasetOffline evalsRelease gateProduction trafficOnline evalson sampled tracesNew failure casespass
Figure 1: Offline evals gate releases, and online evals turn production failures into new dataset cases.

How to build an LLM evaluation dataset#

Build an LLM evaluation dataset from real inputs rather than invented ones: production logs, support tickets, bug reports, and cases where the current system already failed. Start with a few dozen well-specified cases, write down what a correct answer must contain, and version the dataset so that scores from different weeks measure the same thing.

Anthropic's engineering guide to evals for AI agents recommends starting with 20 to 50 tasks drawn from real failures. The same guide sets a useful bar for each task: it should be unambiguous enough that two domain experts would reach the same pass or fail verdict. Cover four kinds of cases deliberately:

  • Common cases: the highest-volume intents in your traffic.
  • Edge cases: empty input, input close to the context window limit, mixed languages, and ambiguous requests.
  • Adversarial cases: prompt injection attempts and requests the application must refuse.
  • Regression cases: every bug that reached users, added on the day it was fixed.

Synthetic inputs generated by an LLM fill coverage gaps quickly, but they drift toward the generator's phrasing. Review them, label them, and keep them in a separate slice so you can see whether scores differ between real and synthetic cases.

Dataset size determines what an eval can detect. With 100 cases and an 80% pass rate, the standard error of the pass rate is 4 percentage points, so a 95% confidence interval spans roughly 72% to 88%. A move from 80% to 83% on that dataset falls within that noise. Grow the dataset, run several trials per case, or compare paired results case by case to see exactly which cases a change flipped. Keep a holdout slice that nobody tunes prompts against, or the dataset slowly becomes a training set for the prompt.

LLM evaluation metrics: start with deterministic checks#

The most useful LLM evaluation metrics are often the cheapest: deterministic checks that verify format, facts, and behavior with code. Schema validation, exact or normalized match, regular expressions, executed unit tests, and tool-call assertions are fast, reproducible, and free of grader bias. Reserve model-based grading for qualities that code cannot verify, such as tone or faithfulness.

Metric family Examples Good for Weakness
Deterministic JSON schema validation, exact match, regex, unit tests on generated code Extraction, classification, code, structured output Rejects valid paraphrases
Reference similarity BLEU, ROUGE, embedding similarity to a reference Translation and tightly constrained summaries Low correlation with human judgment on open-ended tasks
Model-graded Rubric scores, pairwise preference, natural-language assertions Helpfulness, tone, faithfulness, reasoning quality Cost, variance, and judge bias
Human Expert review, side-by-side ratings Ground truth, calibration, high-stakes domains Slow and expensive
Operational Latency, tokens, cost per task, refusal rate Trade-offs between candidate configurations Says nothing about correctness alone

The G-Eval paper reports that reference-based metrics such as BLEU and ROUGE correlate poorly with human judgments on tasks that require creativity and diversity, which is why they rarely suffice for chat or summarization. When an application returns structured outputs, most of its quality can be checked with code:

import json
 
def grade_extraction(output: str, expected: dict) -> dict:
    """Deterministic checks for a structured-output task."""
    try:
        data = json.loads(output)
    except json.JSONDecodeError:
        return {"valid_json": False, "fields_correct": 0.0}
    correct = sum(
        str(data.get(k, "")).strip().lower() == str(v).strip().lower()
        for k, v in expected.items()
    )
    return {"valid_json": True, "fields_correct": correct / len(expected)}

The same approach applies to tool calling: assert that the expected tool was called with arguments that parse and fall within allowed values. Checks like these return the same result on every run, so they make reliable release gates.

LLM as a judge: how it works and where it is biased#

LLM-as-a-judge uses a strong language model to grade another model's output against a rubric, either by scoring one response or by choosing the better of two. Research shows strong judges can approach the agreement rate between human raters, but judges also carry systematic biases toward position, length, and their own outputs, which you must measure and offset.

In Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al. (2023) found that GPT-4 as a judge reached over 80% agreement with human preferences, the same level of agreement measured between humans. The same paper documented position, verbosity, and self-enhancement biases. Later work has quantified each one:

  • Position bias: Wang et al. (2023) showed in Large Language Models are not Fair Evaluators that changing the order of two responses could skew results; with ChatGPT as the evaluator, Vicuna-13B could beat ChatGPT on 66 of 80 tested queries. Run both orders and keep only consistent verdicts.
  • Verbosity bias: judges tend to score longer answers higher. Add rubric criteria that penalize padding and compare responses of similar length where possible.
  • Self-preference bias: Panickssery et al. (2024) found that LLM evaluators rate their own outputs above other outputs that human annotators judge to be of equal quality, and linked the strength of this bias to the evaluator's ability to recognize its own text. Use a judge from a different model family than the system under test.
  • Vague rubrics: a prompt such as "rate helpfulness from 1 to 10" produces noisy scores. Use one criterion per judge call and binary or three-point scales.

Position bias is the easiest to neutralize in code:

def pairwise_verdict(judge, question, a, b):
    """Ask in both orders and keep only consistent verdicts."""
    first = judge(question, a, b)    # returns "A", "B", or "tie"
    second = judge(question, b, a)
    unswapped = {"A": "B", "B": "A", "tie": "tie"}[second]
    return first if first == unswapped else "inconsistent"

Three more habits keep judges trustworthy. Give the judge the reference answer or source documents whenever they exist. Require a short rationale before the verdict and store it for audits. Pin the judge model version and prompt, because upgrading the judge changes every historical score.

Human review and judge calibration#

Human review is the ground truth that every automated grader borrows its credibility from. Use domain experts to label a sample of outputs, measure how often the LLM judge agrees with them, and revise the rubric until agreement is acceptable. Once the judge is calibrated, people spot-check a rotating sample instead of grading every output.

A calibration pass takes a few hours and follows a fixed sequence:

  1. Sample 50 to 100 outputs that span good, borderline, and bad responses.
  2. Have two reviewers label them independently with the same rubric.
  3. Measure agreement between the reviewers first. If humans disagree, the rubric is ambiguous, and no judge prompt will fix it.
  4. Run the judge on the same items and compare it with the human consensus using Cohen's kappa, which corrects for agreement expected by chance, rather than raw percent agreement.
  5. Read every disagreement, tighten the rubric or add worked examples, and repeat.

Recalibrate whenever the judge model, the rubric, or the product's definition of a good answer changes. Reading raw transcripts remains the fastest way to discover failure modes that no metric yet tracks.

How to evaluate RAG systems and agents#

RAG systems and agents need evals at the component level as well as end to end, because a wrong final answer can come from retrieval, generation, a tool call, or planning. Score retrieval and generation separately for RAG. For agents, grade the final outcome, check the transcript for unsafe steps, and run several trials per task.

RAG evaluation metrics#

A retrieval-augmented generation pipeline fails in two distinct places, and the metrics should tell you which one. The Ragas paper introduced reference-free metrics for this purpose, and the Ragas library now lists context precision, context recall, faithfulness, and response relevancy among its RAG metrics.

  • Retrieval quality: context precision asks whether retrieved chunks are relevant and ranked well; context recall asks whether retrieval found everything the answer needs. Classic recall@k and mean reciprocal rank work when you have labeled relevant documents.
  • Generation quality: faithfulness asks whether every claim in the answer is supported by the retrieved context, which is the main defense against hallucination. Response relevancy asks whether the answer addresses the question.
  • End to end: answer correctness against a reference answer, when one exists.

Read the metrics together. Low context recall points to chunking, embeddings, or search configuration. High context recall with low faithfulness points to the prompt or the model. Our explainer on retrieval-augmented generation covers the pipeline stages these metrics map to.

Agent evaluation#

An AI agent run is a sequence of model calls and tool calls that changes external state, so the grader should inspect the end state, not only the final message. Anthropic's guide recommends grading what the agent produced rather than the exact path it took, because agent runs often reach the goal through valid sequences that eval designers did not anticipate. A booking agent passes when the reservation exists in the database, not when its reply claims success.

Agent runs are also inconsistent across trials. The τ-bench benchmark introduced pass^k, the probability that all k trials of a task succeed, and reported pass^8 below 25% in its retail domain for agents built on gpt-4o. The arithmetic is unforgiving: a 75% per-trial success rate over three trials gives about 42% for passing all three. Track pass^k for customer-facing agents, and add transcript checks for forbidden tool calls, step counts, and cost per task. Our explainer on AI agents covers the loop these checks inspect.

Eval-driven development and CI gating#

Eval-driven development means writing the eval before changing the prompt, model, or pipeline, the way test-driven development writes the test before the code. Every change then produces a score difference against a baseline, and CI blocks merges that push a gated metric below its threshold. Prompt edits become reviewable engineering changes instead of guesswork.

The loop has five steps:

  1. Reproduce the failure as an eval case.
  2. Confirm the case fails on the current version.
  3. Change the prompt, model, retrieval settings, or code.
  4. Run the full suite, not only the new case, and compare per-case results with the baseline.
  5. Merge only if every gated metric holds.

Anthropic's guide separates capability evals, which start with a low pass rate and measure progress, from regression evals, which should sit near 100%. Gate merges on the regression suite and track the capability suite over time. Set gates by metric type:

  • Deterministic metrics: gate strictly; schema validity should stay at 100%.
  • Model-graded metrics: gate with a tolerance wider than measured noise. Run the baseline twice to measure run-to-run variance before choosing the tolerance.
  • Cost and latency: gate on budgets too, especially when testing a cheaper model through LLM routing.
import json, sys
 
THRESHOLDS = {"schema_valid": 1.00, "faithfulness": 0.90, "task_success": 0.85}
TOLERANCE = 0.02  # measured run-to-run noise on model-graded metrics
 
results = json.load(open("eval_results.json"))
baseline = json.load(open("baseline_results.json"))
 
failures = [
    f"{m}: {results[m]:.3f} (min {lo}, baseline {baseline[m]:.3f})"
    for m, lo in THRESHOLDS.items()
    if results[m] < lo or results[m] < baseline[m] - TOLERANCE
]
if failures:
    sys.exit("Eval gate failed:\n" + "\n".join(failures))
print("Eval gate passed")

Eval runs spend tokens on every merge. Cache outputs keyed by prompt version, model version, and input, and force uncached runs on release branches.

Choosing an LLM evaluation framework#

An LLM evaluation framework supplies the plumbing: dataset management, a runner that calls your application, built-in graders, result storage, and CI integration. Open source libraries cover most needs for code-first teams, while hosted platforms add annotation queues, dashboards, and links between production traces and datasets. Choose based on where your graders and traces already live.

Widely used open source options include promptfoo, which uses declarative configs with command-line and CI integration; DeepEval; Ragas, which focuses on RAG and agent metrics; OpenAI Evals; and Inspect, developed by the UK AI Security Institute and Meridian Labs. Many tracing platforms also run judges on production traces and manage datasets, and our directory of LLM observability platforms lists them by category. Check any candidate against these criteria:

  • It accepts custom graders written as plain code.
  • It stores per-case results so you can diff two runs.
  • It exits nonzero on failure so CI can gate on it.
  • It handles multi-turn conversations and agent transcripts.
  • It versions datasets alongside prompts.
  • It can run self-hosted if your data cannot leave your infrastructure.

Where to go next: evaluation and tracing share the same data, so read our guide to LLM observability to see how production traces become eval cases. If you are comparing candidate models, start with LLM benchmarks, then confirm the shortlist with your own evals.

Frequently asked questions

What are LLM evals?

LLM evals are repeatable tests that score the outputs of a language model application against defined criteria. Each eval pairs inputs from a dataset with one or more graders, such as exact-match checks, schema validation, an LLM judge, or human review. Teams run evals before release to catch regressions and on production traffic to monitor quality over time.

What is LLM as a judge used for?

LLM-as-a-judge is used to grade outputs that have no single correct answer, such as summaries, chat replies, and explanations. A judge model receives the input, the output, and a rubric, then returns a score or a preference between two outputs. Teams use it to grade more outputs than human reviewers can cover, after checking that its verdicts agree with human labels.

What is a golden dataset in LLM evaluation?

A golden dataset is a curated set of inputs with reference answers or grading criteria that a team treats as ground truth for evaluation. Strong golden datasets are drawn from real traffic and known failures, cover edge cases on purpose, and are versioned so scores from different runs stay comparable. Most start with a few dozen cases and grow as new failures appear.

What is the difference between LLM evaluation and agent evaluation?

LLM evaluation usually scores a single model response to an input. Agent evaluation scores a multi-step run in which the agent calls tools, reads results, and changes external state. Agent evals therefore grade the final outcome, such as whether a record exists in a database, inspect the transcript for unsafe actions, and run several trials per task because results vary between runs.

How do you evaluate LLMs before production?

Build a dataset from realistic inputs, write deterministic checks for anything with a verifiable answer, add a calibrated LLM judge for open-ended quality, and review a sample by hand. Run the suite on every prompt, model, or retrieval change, compare per-case results against the current baseline, and block the release when a gated metric falls below its threshold.

What is the difference between LLM evaluation and benchmarking?

Benchmarking compares models on shared public test sets, such as coding or reasoning suites, so results are comparable across vendors. LLM evaluation tests one specific application, including its prompts, retrieval, tools, and output format, on data that reflects its users. Benchmarks help shortlist candidate models, while application evals determine whether a given configuration is ready to ship.

Terms in this guide

Cite this guide

The AI Engineering Editors. "LLM evaluation: a practical guide to evals, metrics, and judges." The AI Engineering, updated Aug 31, 2026. https://theaiengineering.com/articles/llm-evaluation

Title
LLM evaluation: a practical guide to evals, metrics, and judges
Author
The AI Engineering Editors
Publisher
The AI Engineering
Published
Updated
View as Markdown

The AI Engineering Editors ·