---
title: "LLM evaluation: a practical guide to evals, metrics, and judges"
description: "LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI."
summary: "LLM evaluation is the practice of measuring whether an application built on a language model produces correct, safe, and useful outputs for its real tasks. This guide covers offline and online evals, datasets, deterministic checks, judge bias, human review, RAG and agent evals, and CI gates."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/llm-evaluation"
markdown_url: "https://theaiengineering.com/articles/llm-evaluation.md"
topic: "LLM evaluation and observability"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-08-31"
updated: "2026-08-31"
keywords: ["llm evaluation","llm evals","llm evaluation metrics","llm evaluation framework","llm as a judge"]
citation: "The AI Engineering Editors. \"LLM evaluation: a practical guide to evals, metrics, and judges.\" The AI Engineering, updated Aug 31, 2026. https://theaiengineering.com/articles/llm-evaluation"
---

# LLM evaluation: a practical guide to evals, metrics, and judges

## TL;DR

- LLM evaluation tests your whole application, including prompts, retrieval, tools, and output format, which public model benchmarks do not measure.
- Deterministic checks such as schema validation, exact match, and executed unit tests should run before any model-graded metric because they are cheap, fast, and reproducible.
- LLM-as-a-judge scales grading of open-ended output, but judges show position, verbosity, and self-preference biases, so their verdicts need calibration against human labels.
- Agent evals should grade the final outcome and run several trials per task, because agent runs reach correct outcomes through varied paths and differ from one trial to the next.
- Eval results belong in CI as release gates, with tolerances set wider than the measured noise of the dataset and the judge.

LLM evaluation is the practice of measuring whether an application built on a large language model produces outputs that are correct, safe, and useful for the tasks it serves. Unlike a model benchmark, LLM evaluation tests the whole system: the prompt, the retrieval step, the tools, the output format, and the model working together. This guide explains how to build an evaluation practice that engineers trust, from datasets and deterministic checks to LLM-as-a-judge, human review, RAG and agent evals, and release gates in CI. It belongs to our [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) hub.

## What is LLM evaluation?

[LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation) is a repeatable process for scoring a language model application's outputs against explicit criteria, using datasets, automated graders, and human review. It answers one narrow question for each change: did this prompt edit, model swap, or retrieval tweak make the system better or worse on the tasks that users actually bring to it?

Every eval has the same four parts:

- **Dataset:** inputs and, where possible, reference outputs or written grading notes.
- **Task runner:** code that calls the application exactly as production does, with the same system prompt, retrieval, tools, and parameters.
- **Graders:** code checks, model judges, or people that turn each output into a score.
- **Report:** aggregate scores, per-case differences against a baseline, and the transcripts behind each failure.

Public leaderboards measure models on shared tasks, and our guide to [LLM benchmarks](https://theaiengineering.com/articles/llm-benchmarks) covers how to read them. A benchmark score says little about whether your prompt handles your own customers' refund questions, which is why application evals exist. Outputs also vary between runs at any nonzero [temperature](https://theaiengineering.com/glossary/temperature), so one run of one case is an anecdote rather than a measurement.

## Offline vs online evaluation

Offline evaluation runs a fixed, versioned dataset through the application before a change ships, so results stay comparable across versions. [Online evaluation](https://theaiengineering.com/glossary/online-evaluation) scores live production traffic after release and catches inputs the dataset never anticipated. Mature teams run both: offline evals gate releases, and online evals feed new failure cases back into the offline dataset.

| | Offline evaluation | Online evaluation |
|---|---|---|
| When it runs | Before release, on every change | After release, continuously or on a sample |
| Inputs | Curated, versioned dataset | Real user traffic |
| Ground truth | Reference answers or rubrics | Usually none, so graders must be reference-free |
| Typical graders | Code checks, LLM judges, human review | Reference-free judges, user feedback, guardrail hits |
| Main job | Catch regressions and compare candidates | Detect drift and discover new failure modes |
| Blind spot | Covers only what someone thought to include | Noisy and slow as a signal for one specific change |

Online evaluation depends on capturing traces of real requests, which is the subject of our guide to [LLM observability](https://theaiengineering.com/articles/llm-observability). A/B tests are also a form of online evaluation, but they measure product outcomes such as task completion or retention rather than output quality directly.

**Figure 1: Offline evals gate releases, and online evals turn production failures into new dataset cases.**

Offline evals run the eval dataset and gate releases; online evals score sampled production traces and turn new failures into dataset cases, closing the loop.

Components:

- **Eval dataset**
- **Offline evals**
- **Release gate**
- **Production traffic**
- **Online evals**: on sampled traces
- **New failure cases**

Groups:

- Offline: Eval dataset, Offline evals, Release gate
- Online: New failure cases, Online evals, Production traffic

Connections:

- Eval dataset → Offline evals
- Offline evals → Release gate
- Release gate → Production traffic: pass
- Production traffic → Online evals
- Online evals → New failure cases
- New failure cases → Eval dataset

## How to build an LLM evaluation dataset

Build an [LLM evaluation dataset](https://theaiengineering.com/glossary/golden-dataset) from real inputs rather than invented ones: production logs, support tickets, bug reports, and cases where the current system already failed. Start with a few dozen well-specified cases, write down what a correct answer must contain, and version the dataset so that scores from different weeks measure the same thing.

Anthropic's engineering guide to [evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) recommends starting with 20 to 50 tasks drawn from real failures. The same guide sets a useful bar for each task: it should be unambiguous enough that two domain experts would reach the same pass or fail verdict. Cover four kinds of cases deliberately:

- **Common cases:** the highest-volume intents in your traffic.
- **Edge cases:** empty input, input close to the context window limit, mixed languages, and ambiguous requests.
- **Adversarial cases:** prompt injection attempts and requests the application must refuse.
- **Regression cases:** every bug that reached users, added on the day it was fixed.

Synthetic inputs generated by an LLM fill coverage gaps quickly, but they drift toward the generator's phrasing. Review them, label them, and keep them in a separate slice so you can see whether scores differ between real and synthetic cases.

Dataset size determines what an eval can detect. With 100 cases and an 80% pass rate, the standard error of the pass rate is 4 percentage points, so a 95% confidence interval spans roughly 72% to 88%. A move from 80% to 83% on that dataset falls within that noise. Grow the dataset, run several trials per case, or compare paired results case by case to see exactly which cases a change flipped. Keep a holdout slice that nobody tunes prompts against, or the dataset slowly becomes a training set for the prompt.

## LLM evaluation metrics: start with deterministic checks

The most useful LLM evaluation metrics are often the cheapest: deterministic checks that verify format, facts, and behavior with code. Schema validation, exact or normalized match, regular expressions, executed unit tests, and tool-call assertions are fast, reproducible, and free of grader bias. Reserve model-based grading for qualities that code cannot verify, such as tone or faithfulness.

| Metric family | Examples | Good for | Weakness |
|---|---|---|---|
| Deterministic | JSON schema validation, exact match, regex, unit tests on generated code | Extraction, classification, code, structured output | Rejects valid paraphrases |
| Reference similarity | BLEU, ROUGE, embedding similarity to a reference | Translation and tightly constrained summaries | Low correlation with human judgment on open-ended tasks |
| Model-graded | Rubric scores, pairwise preference, natural-language assertions | Helpfulness, tone, faithfulness, reasoning quality | Cost, variance, and judge bias |
| Human | Expert review, side-by-side ratings | Ground truth, calibration, high-stakes domains | Slow and expensive |
| Operational | Latency, tokens, cost per task, refusal rate | Trade-offs between candidate configurations | Says nothing about correctness alone |

The G-Eval paper reports that reference-based metrics such as BLEU and ROUGE correlate poorly with human judgments on tasks that require creativity and diversity, which is why they rarely suffice for chat or summarization. When an application returns structured outputs, most of its quality can be checked with code:

```python
import json

def grade_extraction(output: str, expected: dict) -> dict:
    """Deterministic checks for a structured-output task."""
    try:
        data = json.loads(output)
    except json.JSONDecodeError:
        return {"valid_json": False, "fields_correct": 0.0}
    correct = sum(
        str(data.get(k, "")).strip().lower() == str(v).strip().lower()
        for k, v in expected.items()
    )
    return {"valid_json": True, "fields_correct": correct / len(expected)}
```

The same approach applies to tool calling: assert that the expected tool was called with arguments that parse and fall within allowed values. Checks like these return the same result on every run, so they make reliable release gates.

## LLM as a judge: how it works and where it is biased

[LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) uses a strong language model to grade another model's output against a rubric, either by scoring one response or by choosing the better of two. Research shows strong judges can approach the agreement rate between human raters, but judges also carry systematic biases toward position, length, and their own outputs, which you must measure and offset.

In [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685), Zheng et al. (2023) found that GPT-4 as a judge reached over 80% agreement with human preferences, the same level of agreement measured between humans. The same paper documented position, verbosity, and self-enhancement biases. Later work has quantified each one:

- **Position bias:** Wang et al. (2023) showed in [Large Language Models are not Fair Evaluators](https://arxiv.org/abs/2305.17926) that changing the order of two responses could skew results; with ChatGPT as the evaluator, Vicuna-13B could beat ChatGPT on 66 of 80 tested queries. Run both orders and keep only consistent verdicts.
- **Verbosity bias:** judges tend to score longer answers higher. Add rubric criteria that penalize padding and compare responses of similar length where possible.
- **Self-preference bias:** Panickssery et al. (2024) found that LLM evaluators rate their own outputs above other outputs that human annotators judge to be of equal quality, and linked the strength of this bias to the evaluator's ability to recognize its own text. Use a judge from a different model family than the system under test.
- **Vague rubrics:** a prompt such as "rate helpfulness from 1 to 10" produces noisy scores. Use one criterion per judge call and binary or three-point scales.

Position bias is the easiest to neutralize in code:

```python
def pairwise_verdict(judge, question, a, b):
    """Ask in both orders and keep only consistent verdicts."""
    first = judge(question, a, b)    # returns "A", "B", or "tie"
    second = judge(question, b, a)
    unswapped = {"A": "B", "B": "A", "tie": "tie"}[second]
    return first if first == unswapped else "inconsistent"
```

Three more habits keep judges trustworthy. Give the judge the reference answer or source documents whenever they exist. Require a short rationale before the verdict and store it for audits. Pin the judge model version and prompt, because upgrading the judge changes every historical score.

## Human review and judge calibration

Human review is the ground truth that every automated grader borrows its credibility from. Use domain experts to label a sample of outputs, measure how often the LLM judge agrees with them, and revise the rubric until agreement is acceptable. Once the judge is calibrated, people spot-check a rotating sample instead of grading every output.

A calibration pass takes a few hours and follows a fixed sequence:

1. Sample 50 to 100 outputs that span good, borderline, and bad responses.
2. Have two reviewers label them independently with the same rubric.
3. Measure agreement between the reviewers first. If humans disagree, the rubric is ambiguous, and no judge prompt will fix it.
4. Run the judge on the same items and compare it with the human consensus using Cohen's kappa, which corrects for agreement expected by chance, rather than raw percent agreement.
5. Read every disagreement, tighten the rubric or add worked examples, and repeat.

Recalibrate whenever the judge model, the rubric, or the product's definition of a good answer changes. Reading raw transcripts remains the fastest way to discover failure modes that no metric yet tracks.

## How to evaluate RAG systems and agents

RAG systems and agents need evals at the component level as well as end to end, because a wrong final answer can come from retrieval, generation, a tool call, or planning. Score retrieval and generation separately for RAG. For agents, grade the final outcome, check the transcript for unsafe steps, and run several trials per task.

### RAG evaluation metrics

A [retrieval-augmented generation](https://theaiengineering.com/glossary/retrieval-augmented-generation) pipeline fails in two distinct places, and the metrics should tell you which one. The [Ragas paper](https://arxiv.org/abs/2309.15217) introduced reference-free metrics for this purpose, and the Ragas library now lists context precision, context recall, faithfulness, and response relevancy among its RAG metrics.

- **Retrieval quality:** context precision asks whether retrieved chunks are relevant and ranked well; context recall asks whether retrieval found everything the answer needs. Classic [recall@k](https://theaiengineering.com/glossary/recall-at-k) and mean reciprocal rank work when you have labeled relevant documents.
- **Generation quality:** [faithfulness](https://theaiengineering.com/glossary/faithfulness) asks whether every claim in the answer is supported by the retrieved context, which is the main defense against [hallucination](https://theaiengineering.com/glossary/hallucination). Response relevancy asks whether the answer addresses the question.
- **End to end:** answer correctness against a reference answer, when one exists.

Read the metrics together. Low context recall points to chunking, embeddings, or search configuration. High context recall with low faithfulness points to the prompt or the model. Our explainer on [retrieval-augmented generation](https://theaiengineering.com/articles/what-is-rag) covers the pipeline stages these metrics map to.

### Agent evaluation

An [AI agent](https://theaiengineering.com/glossary/ai-agent) run is a sequence of model calls and tool calls that changes external state, so the grader should inspect the end state, not only the final message. Anthropic's guide recommends grading what the agent produced rather than the exact path it took, because agent runs often reach the goal through valid sequences that eval designers did not anticipate. A booking agent passes when the reservation exists in the database, not when its reply claims success.

Agent runs are also inconsistent across trials. The τ-bench benchmark introduced [pass^k](https://theaiengineering.com/glossary/pass-at-k), the probability that all k trials of a task succeed, and reported pass^8 below 25% in its retail domain for agents built on gpt-4o. The arithmetic is unforgiving: a 75% per-trial success rate over three trials gives about 42% for passing all three. Track pass^k for customer-facing agents, and add transcript checks for forbidden tool calls, step counts, and cost per task. Our explainer on [AI agents](https://theaiengineering.com/articles/what-are-ai-agents) covers the loop these checks inspect.

## Eval-driven development and CI gating

Eval-driven development means writing the eval before changing the prompt, model, or pipeline, the way test-driven development writes the test before the code. Every change then produces a score difference against a baseline, and CI blocks merges that push a gated metric below its threshold. Prompt edits become reviewable engineering changes instead of guesswork.

The loop has five steps:

1. Reproduce the failure as an eval case.
2. Confirm the case fails on the current version.
3. Change the prompt, model, retrieval settings, or code.
4. Run the full suite, not only the new case, and compare per-case results with the baseline.
5. Merge only if every gated metric holds.

Anthropic's guide separates capability evals, which start with a low pass rate and measure progress, from regression evals, which should sit near 100%. Gate merges on the regression suite and track the capability suite over time. Set gates by metric type:

- **Deterministic metrics:** gate strictly; schema validity should stay at 100%.
- **Model-graded metrics:** gate with a tolerance wider than measured noise. Run the baseline twice to measure run-to-run variance before choosing the tolerance.
- **Cost and latency:** gate on budgets too, especially when testing a cheaper model through [LLM routing](https://theaiengineering.com/articles/llm-routing).

```python
import json, sys

THRESHOLDS = {"schema_valid": 1.00, "faithfulness": 0.90, "task_success": 0.85}
TOLERANCE = 0.02  # measured run-to-run noise on model-graded metrics

results = json.load(open("eval_results.json"))
baseline = json.load(open("baseline_results.json"))

failures = [
    f"{m}: {results[m]:.3f} (min {lo}, baseline {baseline[m]:.3f})"
    for m, lo in THRESHOLDS.items()
    if results[m] < lo or results[m] < baseline[m] - TOLERANCE
]
if failures:
    sys.exit("Eval gate failed:\n" + "\n".join(failures))
print("Eval gate passed")
```

Eval runs spend tokens on every merge. Cache outputs keyed by prompt version, model version, and input, and force uncached runs on release branches.

## Choosing an LLM evaluation framework

An LLM evaluation framework supplies the plumbing: dataset management, a runner that calls your application, built-in graders, result storage, and CI integration. Open source libraries cover most needs for code-first teams, while hosted platforms add annotation queues, dashboards, and links between production traces and datasets. Choose based on where your graders and traces already live.

Widely used open source options include promptfoo, which uses declarative configs with command-line and CI integration; DeepEval; Ragas, which focuses on RAG and agent metrics; OpenAI Evals; and Inspect, developed by the UK AI Security Institute and Meridian Labs. Many tracing platforms also run judges on production traces and manage datasets, and our directory of [LLM observability platforms](https://theaiengineering.com/platforms/llm-observability) lists them by category. Check any candidate against these criteria:

- It accepts custom graders written as plain code.
- It stores per-case results so you can diff two runs.
- It exits nonzero on failure so CI can gate on it.
- It handles multi-turn conversations and agent transcripts.
- It versions datasets alongside prompts.
- It can run self-hosted if your data cannot leave your infrastructure.

Where to go next: evaluation and tracing share the same data, so read our guide to [LLM observability](https://theaiengineering.com/articles/llm-observability) to see how production traces become eval cases. If you are comparing candidate models, start with [LLM benchmarks](https://theaiengineering.com/articles/llm-benchmarks), then confirm the shortlist with your own evals.

## Frequently asked questions

### What are LLM evals?

LLM evals are repeatable tests that score the outputs of a language model application against defined criteria. Each eval pairs inputs from a dataset with one or more graders, such as exact-match checks, schema validation, an LLM judge, or human review. Teams run evals before release to catch regressions and on production traffic to monitor quality over time.

### What is LLM as a judge used for?

LLM-as-a-judge is used to grade outputs that have no single correct answer, such as summaries, chat replies, and explanations. A judge model receives the input, the output, and a rubric, then returns a score or a preference between two outputs. Teams use it to grade more outputs than human reviewers can cover, after checking that its verdicts agree with human labels.

### What is a golden dataset in LLM evaluation?

A golden dataset is a curated set of inputs with reference answers or grading criteria that a team treats as ground truth for evaluation. Strong golden datasets are drawn from real traffic and known failures, cover edge cases on purpose, and are versioned so scores from different runs stay comparable. Most start with a few dozen cases and grow as new failures appear.

### What is the difference between LLM evaluation and agent evaluation?

LLM evaluation usually scores a single model response to an input. Agent evaluation scores a multi-step run in which the agent calls tools, reads results, and changes external state. Agent evals therefore grade the final outcome, such as whether a record exists in a database, inspect the transcript for unsafe actions, and run several trials per task because results vary between runs.

### How do you evaluate LLMs before production?

Build a dataset from realistic inputs, write deterministic checks for anything with a verifiable answer, add a calibrated LLM judge for open-ended quality, and review a sample by hand. Run the suite on every prompt, model, or retrieval change, compare per-case results against the current baseline, and block the release when a gated metric falls below its threshold.

### What is the difference between LLM evaluation and benchmarking?

Benchmarking compares models on shared public test sets, such as coding or reasoning suites, so results are comparable across vendors. LLM evaluation tests one specific application, including its prompts, retrieval, tools, and output format, on data that reflects its users. Benchmarks help shortlist candidate models, while application evals determine whether a given configuration is ready to ship.

## Terms in this guide

- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Hallucination](https://theaiengineering.com/glossary/hallucination): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name.
- [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data.
- [Structured outputs](https://theaiengineering.com/glossary/structured-outputs): Structured outputs is an LLM API feature that constrains generation so the model's response always conforms to a developer-supplied JSON Schema, producing output that code can parse without retries.
- [AI agent](https://theaiengineering.com/glossary/ai-agent): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.
- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Faithfulness](https://theaiengineering.com/glossary/faithfulness): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1.
- [JSON Schema](https://theaiengineering.com/glossary/json-schema): JSON Schema is a declarative vocabulary, itself written in JSON, for describing the structure of JSON data; LLM APIs and MCP use it to define tool arguments and constrain model output.
- [LMArena](https://theaiengineering.com/glossary/lmarena): LMArena, originally Chatbot Arena and branded Arena since January 2026, is a public leaderboard that ranks language models from crowdsourced human votes between pairs of anonymous model responses.
- [Recall@k](https://theaiengineering.com/glossary/recall-at-k): Recall@k is a retrieval metric that measures the fraction of a query's relevant documents that appear in the top k results, averaged over a set of test queries.
- [pass@k](https://theaiengineering.com/glossary/pass-at-k): pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [Prompt injection](https://theaiengineering.com/glossary/prompt-injection): Prompt injection is an attack in which text supplied to a large language model, typed by a user or hidden in content the model reads, alters the application's intended behavior or output.

## Related guides

- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [What is retrieval-augmented generation (RAG)? How RAG works](https://theaiengineering.com/articles/what-is-rag): Retrieval-augmented generation (RAG) grounds LLM answers in your documents. Learn the RAG pipeline, hybrid search, agentic RAG, evaluation, and failure modes.

Part of the [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) topic hub.

## Cite this guide

The AI Engineering Editors. "LLM evaluation: a practical guide to evals, metrics, and judges." The AI Engineering, updated Aug 31, 2026. https://theaiengineering.com/articles/llm-evaluation

- Title: LLM evaluation: a practical guide to evals, metrics, and judges
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-08-31
- Updated: 2026-08-31
- URL: https://theaiengineering.com/articles/llm-evaluation
- Markdown: https://theaiengineering.com/articles/llm-evaluation.md

To cite one section, link to its anchor:

- [What is LLM evaluation?](https://theaiengineering.com/articles/llm-evaluation#what-is-llm-evaluation)
- [Offline vs online evaluation](https://theaiengineering.com/articles/llm-evaluation#offline-vs-online-evaluation)
- [How to build an LLM evaluation dataset](https://theaiengineering.com/articles/llm-evaluation#how-to-build-an-llm-evaluation-dataset)
- [LLM evaluation metrics: start with deterministic checks](https://theaiengineering.com/articles/llm-evaluation#llm-evaluation-metrics-start-with-deterministic-checks)
- [LLM as a judge: how it works and where it is biased](https://theaiengineering.com/articles/llm-evaluation#llm-as-a-judge-how-it-works-and-where-it-is-biased)
- [Human review and judge calibration](https://theaiengineering.com/articles/llm-evaluation#human-review-and-judge-calibration)
- [How to evaluate RAG systems and agents](https://theaiengineering.com/articles/llm-evaluation#how-to-evaluate-rag-systems-and-agents)
- [Eval-driven development and CI gating](https://theaiengineering.com/articles/llm-evaluation#eval-driven-development-and-ci-gating)
- [Choosing an LLM evaluation framework](https://theaiengineering.com/articles/llm-evaluation#choosing-an-llm-evaluation-framework)
- [Frequently asked questions](https://theaiengineering.com/articles/llm-evaluation#faq)
