AI agent evaluation: metrics, graders, and how to test agents
AI agent evaluation is the practice of measuring whether an AI agent completes its tasks correctly, safely, and efficiently, by grading the state it leaves behind, the tool calls it made along the way, and how consistently it succeeds across repeated runs. It extends LLM evaluation from one model response to a multi-step run that reads data, writes data, and spends money. This guide covers metrics, reliability across runs, graders, eval sets, CI gates, and public agent benchmarks, and belongs to our LLM evaluation and observability hub.
What is AI agent evaluation?#
AI agent evaluation is the repeatable testing of an agent system on tasks with defined success criteria, scored by graders that inspect the final environment state and the transcript. Because agent runs vary, each task runs several times, and the scores describe the model, prompts, tools, and orchestration code together, never the model alone.
Anthropic's engineering guide, Demystifying evals for AI agents, defines the core vocabulary:
- Task: one test with defined inputs and success criteria.
- Trial: one attempt at a task; several run per task.
- Transcript (also trace or trajectory): the full record of a trial, including messages, tool calls, and results.
- Outcome: the final state of the environment, such as whether a reservation actually exists.
- Grader: logic that scores one aspect of a trial.
- Scaffold: the code that turns a model into an agent by orchestrating its tool calls.
Because the scaffold is part of the system under test, a prompt edit, a new tool, and a model swap each need a fresh eval run. The parts are those of any LLM evaluation, plus an environment. Our AI agents explainer covers the loop the scaffold drives.
How agent evaluation differs from evaluating an LLM call#
Evaluating a single LLM call scores one output against one input. Evaluating an agent scores a run of many model calls and tool calls that changes external state, can fail at any step, takes a different path on each trial, and varies in cost. The grader needs access to the environment, not only to the agent's final reply.
| Single LLM call | Agent run | |
|---|---|---|
| Unit graded | One response | A trajectory of model turns and tool calls |
| What correct means | Output matches a reference or rubric | Environment ends in the goal state, by any valid path |
| Side effects | None | Writes to databases, files, tickets, or payments |
| Where it fails | The response | Any step: plan, tool choice, arguments, error handling, stopping |
| Run-to-run variation | Sampling noise in one output | Noise compounds across steps and changes the path |
Errors compound: if each of 10 steps were independent and 95% reliable, the run would succeed about 60% of the time. The final message is not evidence: an agent can report success after a failed tool call, a form of hallucination, so the grader checks the database. And many valid paths exist: the WebArena paper criticizes comparing an agent's actions with a reference action sequence, because that ignores functional correctness and alternative solutions.
What to measure: AI agent evaluation metrics#
AI agent evaluation metrics fall into five groups: outcome (did the task succeed), trajectory (right tools, right arguments, no wasted steps), safety and policy (did the agent stay within its rules), efficiency (cost, latency, and turns), and reliability (how often it succeeds across repeated trials). Track all five, and gate releases on outcome and safety first.
| Group | Metric | How to measure it | Typical grader |
|---|---|---|---|
| Outcome | Task success rate | End state matches the goal state, or tests pass | Code |
| Trajectory | Tool selection | Required tools called, no unexpected tools | Code |
| Trajectory | Argument correctness | Arguments fit the schema and match expected values | Code |
| Trajectory | Redundant steps | Repeated calls, loops, steps beyond a reference path | Code or LLM judge |
| Safety | Policy violations | Forbidden tools, writes without confirmation, data leaks | Code plus LLM judge |
| Safety | Injection resistance | Tasks whose tool results contain planted instructions | Code on end state |
| Efficiency | Cost, latency, turns | Token cost, wall-clock time, and model turns per task | Trace data |
| Reliability | pass@k and pass^k | Success across k trials of the same task | Computed from trials |
Outcome and task success#
Measure task success on the environment: query the database, run the tests, or inspect the files written. The τ-bench paper (tau-bench) compares the final database with an annotated goal state and checks that the agent's replies contained the information the user needed. For multi-part tasks, Anthropic's guide recommends partial credit.
Trajectory and tool-use quality#
Trajectory metrics explain why an outcome failed and catch runs that succeeded wastefully. LangChain's MIT-licensed agentevals package names four ways to compare a trajectory with a reference: strict (same tool calls, same order), unordered (same calls, any order), subset (no tools beyond the expected ones), and superset (expected calls appear, extras allowed). Strict matching is brittle because agents find valid paths nobody anticipated, so check writes and critical arguments exactly and let reads vary.
Safety and policy compliance#
An outcome check can pass while the agent broke a rule. The τ-bench authors note that a successful reward is necessary but not sufficient, because an agent might issue a return without the explicit user confirmation the policy requires. Check that forbidden tools were never called and that writes followed confirmation. Include tasks where the agent should refuse or ask, and tool results that carry prompt injection attempts. Runtime guardrails block some violations; evals measure how often the agent attempts them.
Cost, latency, and turn count#
Measure efficiency per completed task, next to success rate: succeeding slightly more often at three times the cost is a product decision. Runs that hit the step limit count as failures. The data comes from traces, covered in our LLM observability guide.
pass@k vs pass^k: measuring reliability across runs#
pass@k is the probability that at least one of k independent trials of a task succeeds; pass^k is the probability that all k trials succeed. pass@k rises with k and fits tasks where a person or verifier can pick the one good attempt. pass^k falls with k and fits agents that must handle the same request correctly every time.
In code generation, Kulal et al. (2019) counted a problem as solved if any of k samples passed its unit tests, and Chen et al. (2021), in the paper that introduced HumanEval, gave the unbiased estimator still in use: generate n ≥ k samples, count the c that pass, and average 1 - C(n-c, k) / C(n, k) over problems. The τ-bench paper proposed pass^k (read "pass hat k") for agent tasks that need consistency, defining it as the chance that all k i.i.d. trials succeed, averaged across tasks, with the estimator C(c, k) / C(n, k). At k = 1 both equal the average success rate.
from math import comb
def pass_at_k(n, c, k):
"""P(at least one of k trials passes), given c of n trials passed."""
return 1.0 - comb(n - c, k) / comb(n, k)
def pass_hat_k(n, c, k):
"""P(all k trials pass): the pass^k estimator from tau-bench."""
return comb(c, k) / comb(n, k)
# One task, 10 trials, 8 passed.
for k in (1, 3, 5):
at, hat = pass_at_k(10, 8, k), pass_hat_k(10, 8, k)
print(f"k={k} pass@k={at:.3f} pass^k={hat:.3f}")
# k=1 pass@k=0.800 pass^k=0.800
# k=3 pass@k=1.000 pass^k=0.467
# k=5 pass@k=1.000 pass^k=0.222A task that passes 8 of 10 trials looks solved under pass@3 and fails more often than it succeeds under pass^3. Run at least as many trials as the largest k you report, reset the environment so trials stay independent, and keep per-task values so one flaky task does not vanish into an average.
Graders for agents and how they fail#
Agent evals use three kinds of graders: code that checks the end state and the transcript, LLM judges that apply a rubric to the transcript, and human reviewers. Code graders are the default because they are fast, cheap, and reproducible. Judges cover qualities code cannot check, and humans calibrate both.
| Grader | Best at | Failure mode | Mitigation |
|---|---|---|---|
| Code: end-state checks | Whether the task happened | Misses how it happened, such as a skipped confirmation | Add trajectory and policy checks |
| Code: trajectory match | Tool choice, arguments, forbidden calls | Rejects valid alternative paths when order is strict | Match writes and critical arguments only |
| LLM judge on transcript | Plan quality, error recovery, policy in conversation | Trusts the agent's claims; misses details in long transcripts | Give it tool results and the end state; ask one question per call |
| Human review | Ground truth and new failure modes | Slow, costly, reviewers disagree | Use for calibration and spot checks |
An LLM-as-a-judge grading a trajectory carries the position, verbosity, and self-preference biases covered in our LLM evaluation guide, plus an agent-specific weakness: the agent wrote much of the transcript, so a judge that reads only its messages will believe "the refund is done." Pass the judge the raw tool results and ask narrow binary questions, such as whether the agent confirmed before every write.
Graders also fail when the environment leaks. Anthropic's guide reports that in some internal evals, Claude examined git history left by previous trials to gain an unfair advantage. Reset state between trials, keep grading files outside the agent's writable workspace, and read sampled transcripts with their grades to tell agent mistakes from graders that rejected valid solutions.
A code-based grader for an agent trajectory#
A code-based grader for an agent takes the recorded transcript and the environment's end state and returns named pass or fail checks. The example grades a refund task: the refund must exist in the database, and the agent must write once with the right arguments, ask for confirmation first, stay within a tool budget, and tell the user the amount.
WRITE_TOOLS = {"issue_refund", "cancel_order", "update_address"}
EXPECTED = [{"name": "issue_refund",
"args": {"order_id": "W1001", "amount": 54.04}}]
def grade_refund_trial(trajectory, end_state, max_tool_calls=6):
"""Code-based grader for one trial of a refund task."""
steps = list(enumerate(trajectory))
tools = [(i, s) for i, s in steps if s["type"] == "tool"]
writes = [(i, s) for i, s in tools if s["name"] in WRITE_TOOLS]
def confirmed(i):
# Policy: the agent asked, then the user replied, before the write.
turns = [s["type"] for s in trajectory[:i] if s["type"] != "tool"]
return turns[-2:] == ["assistant", "user"]
final = [s["text"] for s in trajectory if s["type"] == "assistant"][-1]
refunds = [{"order_id": "W1001", "amount": 54.04}]
checks = {
"end_state": end_state["refunds"] == refunds
and end_state["orders"]["W1001"]["status"] == "refunded",
"expected_write": [{"name": s["name"], "args": s["args"]}
for _, s in writes] == EXPECTED,
"confirmed_before_write": all(confirmed(i) for i, _ in writes),
"step_budget": len(tools) <= max_tool_calls,
"told_user_amount": "54.04" in final,
}
return {"passed": all(checks.values()), **checks}
good = [
{"type": "user", "text": "Refund order W1001, please."},
{"type": "tool", "name": "get_order", "args": {"order_id": "W1001"}},
{"type": "assistant", "text": "W1001 qualifies for $54.04. Proceed?"},
{"type": "user", "text": "Yes."},
{"type": "tool", "name": "issue_refund",
"args": {"order_id": "W1001", "amount": 54.04}},
{"type": "assistant", "text": "Refunded $54.04 to your card."},
]
eager = [good[0], good[1], good[4], good[5]] # no confirmation step
state = {"orders": {"W1001": {"status": "refunded"}},
"refunds": [{"order_id": "W1001", "amount": 54.04}]}
print(grade_refund_trial(good, state)["passed"]) # True
print(grade_refund_trial(eager, state))
# {'passed': False, 'end_state': True, 'expected_write': True,
# 'confirmed_before_write': False, 'step_budget': True,
# 'told_user_amount': True}The eager trajectory leaves the same database as the good one, so an end-state check alone would pass it; only the trajectory check catches the missing confirmation. The check verifies the order of turns, not that the user agreed, so a production grader would read a structured confirmation flag from the tool layer or ask an LLM judge that one question. In a real run, the trajectory comes from the trace export and the end state from querying the sandbox. Per-check failure rates tell an engineer what to fix.
How to build an agent eval set from real traces#
Build agent eval tasks from production traces and known failures rather than invented scenarios. Each task needs an instruction, a reproducible initial state, the tools or mocks the agent may use, success criteria two domain experts would agree on, and ideally a reference solution that proves the task is solvable in that environment.
- Sample traces through your LLM observability pipeline, weighted toward failures, escalations, retries, and negative feedback.
- Read and label each failure: wrong tool, wrong arguments, missed policy, gave up early, or looped.
- Convert it to a task. Rebuild the initial state with personal data removed, write the instruction, and define the goal state. For conversational agents, give a simulated user a goal and persona, as τ-bench does with a language model.
- Add a reference solution that proves the task can pass; Terminal-Bench ships a human-written one with every task, according to its paper.
- Balance the set with tasks where the agent should act and tasks where it should refuse, ask, or hand off.
- Version it and start small: Anthropic recommends 20 to 50 simple tasks drawn from real failures.
Isolation matters as much as coverage: Anthropic's guide warns that leftover files, cached data, or exhausted resources from one trial can cause correlated failures that reflect infrastructure, not the agent.
Offline evals in CI vs online evals in production#
Offline agent evals run versioned tasks in sandboxed environments before a change ships, so they can check end states and gate merges. Online evals score sampled production runs after release, where no goal state exists, using reference-free judges, policy checks on traces, and user signals. Offline evals catch regressions; online evals find the failures that become next week's tasks.
| Offline evals in CI | Online evals in production | |
|---|---|---|
| Environment | Sandboxes, seeded databases, mocked or test-mode APIs | Real tools and real users |
| Ground truth | Goal state and success criteria per task | Usually none |
| Graders | State checks, trajectory checks, calibrated judges | Reference-free judges, policy checks on traces, escalation rates |
| Trials | Several per task, to estimate pass^k | One per user request |
| Main job | Block regressions and compare configurations | Detect drift and discover new failure modes |
Anthropic's guide separates capability evals, which start with a low pass rate, from regression evals, which should sit near 100%. Gate merges on the regression suite: pass^k on critical tasks, zero policy violations, and cost per task within budget. Move capability tasks into it once they pass reliably. Since 50 tasks at 5 trials each is 250 full agent runs, run a sample on pull requests and the full suite on release branches. The gate script in our LLM evaluation guide works unchanged for agent metrics.
Online, trajectory checks still apply, because a forbidden tool call or an unconfirmed write is visible in logged spans. Many LLM observability and evaluation platforms run judges on sampled traces and turn flagged runs into dataset items.
Public agent benchmarks and what they measure#
Public agent benchmarks score a model and its scaffold in a fixed environment: real repositories, simulated customer service, web apps, desktops, or terminals. They help shortlist models and supply proven grading designs, but they do not measure your agent on your tasks, and a reported score depends on the scaffold, tools, and turn limits used.
| Benchmark | Environment | Task | Grading |
|---|---|---|---|
| SWE-bench Verified | Real Python repositories | Resolve a GitHub issue with a patch | Fail-to-pass and pass-to-pass tests; 500 human-validated tasks |
| τ-bench | Retail and airline support with API tools, a policy, and an LLM-simulated user | Complete the request within policy | Final database against a goal state, plus required facts in replies; reports pass^k |
| τ²-bench | Telecom support where agent and user both act on shared state | Resolve the issue while guiding the user | Verifiable tasks from a compositional generator |
| GAIA | The open web and the agent's tools | Real-world questions needing browsing, multimodality, and tool use | Quasi exact match on a short answer; 466 questions |
| WebArena | Four self-hosted web apps: shopping, forums, code collaboration, content management | 812 long-horizon tasks | Programmatic validators of functional correctness |
| OSWorld | Real Ubuntu, Windows, and macOS computers | 369 tasks across apps, files, and multi-app workflows | A per-task evaluation script from a defined initial state |
| Terminal-Bench | Containerized terminals | 89 hard tasks in version 2.0 | Tests on the final container state; each task has a human-written solution |
OpenAI stopped reporting SWE-bench Verified in February 2026, citing flawed tests and contamination; our LLM benchmarks guide covers that decision and SWE-bench Pro. Terminal-Bench's site now leads with Terminal-Bench 4.0, so check which version a result used. Coding-agent results depend heavily on the scaffold, as our AI coding agents comparison shows.
Where to go next: our guide on how to build an AI agent writes the loop these evals test, the AI agent frameworks comparison shows which frameworks emit traces, and the tool calling guide covers tool design behind many trajectory failures.
Frequently asked questions
How do you evaluate AI agents?
- Give the agent realistic tasks in a sandboxed environment with a known initial state, run each task several times, and grade every trial on two artifacts: the end state, such as database records or passing tests, and the transcript of tool calls. Add checks for policy, cost, and step counts, calibrate any LLM judge against human labels, and gate releases on the results.
What are the most important AI agent evaluation metrics?
- Task success rate, measured against the final environment state, is the primary metric. Supporting metrics cover the trajectory (correct tool selection, correct arguments, redundant steps), safety (policy violations, forbidden tool calls, resistance to prompt injection), efficiency (cost, latency, and turns per task), and reliability across repeated trials, usually reported as pass^k for agents that must succeed every time.
What is the difference between pass@k and pass^k?
- pass@k is the probability that at least one of k independent trials of a task succeeds, so it rises as k grows. pass^k, introduced with the τ-bench benchmark in 2024, is the probability that all k trials succeed, so it falls as k grows. Both equal the average success rate when k is 1. pass^k measures the consistency that users actually experience.
How do you test AI agents before production?
- Build 20 to 50 tasks from real traces and known failures, each with a seeded environment, success criteria, and a reference solution. Run every task several times in a fresh sandbox, with mocked or test-mode tools for anything that has real side effects and a simulated user for conversational agents. Gate merges on a regression suite and track a harder capability suite separately.
What tools are used for AI agent evaluation?
- Agent evaluation tools fall into three groups. Open source frameworks such as Inspect, from the UK AI Security Institute and Meridian Labs, run agent tasks with tool calling and Docker sandboxes. Harbor, from the creators of Terminal-Bench, runs agents in container environments. Observability platforms add trace-based datasets, judges on sampled production traffic, and annotation queues. Most teams also write custom graders in plain code.
Terms in this guide
Cite this guide
The AI Engineering Editors. "AI agent evaluation: metrics, graders, and how to test agents." The AI Engineering, updated Oct 31, 2026. https://theaiengineering.com/articles/ai-agent-evaluation
- Title
- AI agent evaluation: metrics, graders, and how to test agents
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Published
- Updated