---
title: "What is LLM evaluation? Methods, metrics, and graders"
description: "LLM evaluation measures whether a model or LLM application meets quality, safety, and cost goals, using test sets, code checks, LLM judges, and human review."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/llm-evaluation"
markdown_url: "https://theaiengineering.com/glossary/llm-evaluation.md"
topic: "LLM evaluation and observability"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["LLM evaluation","LLM evals","evals","model evaluation"]
citation: "The AI Engineering Editors. \"What is LLM evaluation? Methods, metrics, and graders.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-evaluation"
---

# What is LLM evaluation? Methods, metrics, and graders

Also called LLM evals, evals, model evaluation.

> **Definition of LLM evaluation:** LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.

LLM evaluation, usually shortened to evals, is how teams decide whether a prompt change, model upgrade, or new agent design made an application better or worse. LLM outputs vary from run to run and rarely have a single correct string, so evaluation relies on test sets of representative inputs, explicit success criteria, and graders that score each output. OpenAI's [evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) recommend evaluating early and often and growing the test set from production logs.

## How LLM evaluation works

An eval runs a set of tasks through the system, scores each result, and aggregates the scores. Anthropic's engineering guide to [evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) groups graders into three types:

| Grader | How it scores | Strengths | Weaknesses |
|---|---|---|---|
| Code-based | Exact match, regex, unit tests, schema checks | Fast, cheap, reproducible | Limited on subjective quality |
| Model-based | An [LLM judge](https://theaiengineering.com/glossary/llm-as-a-judge) applies a rubric or compares outputs | Flexible; scales to open-ended output | Non-deterministic; needs calibration |
| Human | Expert review, spot checks, A/B tests | Highest quality | Slow and expensive |

Most suites combine all three: code checks for hard requirements, model judges for qualities such as faithfulness or tone, and periodic human review to calibrate the judges. For [AI agents](https://theaiengineering.com/glossary/ai-agent), the same guide separates the transcript, the full record of steps and tool calls, from the outcome, the final state of the environment, such as whether a reservation actually exists in the database.

Anthropic also distinguishes capability evals from regression evals. Capability evals start with a low pass rate and measure progress. Regression evals should stay near a 100% pass rate and catch backsliding when prompts or models change.

## LLM evaluation vs benchmarks

Public [LLM benchmarks](https://theaiengineering.com/articles/llm-benchmarks) compare models on fixed, shared tasks. Application evals test your system, including its prompts, retrieval, and tools, on your own data and success criteria. A model that leads a leaderboard can still fail your eval, so benchmarks help shortlist candidates and application evals decide what ships.

## Why LLM evaluation matters

Without evals, every change is judged by spot-checking a few outputs, which misses regressions in cases nobody looked at. Offline evals run before deployment; online evaluation scores sampled production traffic, which is where evaluation and [LLM observability](https://theaiengineering.com/glossary/llm-observability) meet. Evals are also how teams measure [hallucination](https://theaiengineering.com/glossary/hallucination) rates and [guardrail](https://theaiengineering.com/glossary/guardrails) false positives instead of guessing at them.

The [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation) goes deeper on designing eval suites.

## Related terms

- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [Hallucination](https://theaiengineering.com/glossary/hallucination): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [AI agent](https://theaiengineering.com/glossary/ai-agent): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
- [Benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination): Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures.

## Guides that cover llm evaluation

- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [How to build an AI agent: a step-by-step guide in Python](https://theaiengineering.com/articles/how-to-build-an-ai-agent): Learn how to build an AI agent from scratch in Python: when you need one, tool schemas, the agent loop, stop conditions, guardrails, evals, and tracing.

Part of the [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) topic hub.

## Cite this definition

The AI Engineering Editors. "What is LLM evaluation? Methods, metrics, and graders." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-evaluation

- Title: What is LLM evaluation? Methods, metrics, and graders
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/llm-evaluation
- Markdown: https://theaiengineering.com/glossary/llm-evaluation.md
