---
title: "What is online evaluation? Scoring live LLM traffic"
description: "Online evaluation scores live LLM traffic with reference-free judges, code checks, and user feedback. Learn how it works and how it differs from offline evals."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/online-evaluation"
markdown_url: "https://theaiengineering.com/glossary/online-evaluation.md"
topic: "LLM evaluation and observability"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Online evaluation","online evals","production evaluation"]
citation: "The AI Engineering Editors. \"What is online evaluation? Scoring live LLM traffic.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/online-evaluation"
---

# What is online evaluation? Scoring live LLM traffic

Also called online evals, production evaluation.

> **Definition of Online evaluation:** Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.

Online evaluation is the scoring of an LLM application's real production traffic after release, as opposed to running a fixed test set before release. Evaluators read captured traces, usually a filtered sample, and attach scores for qualities such as relevance, faithfulness, policy compliance, or task success. It catches failures that no test case anticipated and shows whether quality drifts as users, data, and upstream models change. It depends on [LLM observability](https://theaiengineering.com/glossary/llm-observability), because evaluators need the stored prompt, retrieved context, and output of each request.

## How online evaluation works

Online evaluation runs graders on production traces as they arrive. A rule selects which traces to score using filters and a sampling rate, graders score each selected trace without a reference answer, and the scores are stored on the trace, where dashboards aggregate them and low scores are routed to human review.

Tracing platforms such as LangSmith and Langfuse configure online evaluators this way. Langfuse's [online evaluation docs](https://langfuse.com/docs/evaluation/get-started/online) define a rule as a set of filters, a sampling rate, and one or more evaluators, and advise lowering the sampling rate when the estimated cost of an LLM judge is too high. Score a random sample of normal traffic plus every trace with an error or negative user feedback, so rare failures are not sampled away.

Live traffic has no reference answers, so online graders are reference-free:

- **Model-based:** an [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) checks faithfulness to the retrieved context, relevance, or tone.
- **Code-based:** schema validation, forbidden tool calls, step counts, and cost per task, computed from span data.
- **Signals:** user ratings, retries, escalations to a human, and [guardrail](https://theaiengineering.com/glossary/guardrails) triggers.

The [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry) GenAI conventions define a [`gen_ai.evaluation.result` event](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md) that carries the metric name, a score value or label, and an explanation, parented to the span being evaluated. Scores recorded this way travel with the trace in a vendor-neutral format. A/B tests are a related form of online evaluation that compares product outcomes, such as task completion, between two live versions.

## Online vs offline evaluation

Offline evaluation runs a fixed, versioned [golden dataset](https://theaiengineering.com/glossary/golden-dataset) before a change ships and can compare outputs with reference answers. Online evaluation scores real traffic after release, where references rarely exist. Offline evals gate releases. Online evals detect drift and surface new failure modes, which then become offline test cases.

| | Offline evaluation | Online evaluation |
|---|---|---|
| When it runs | Before release, on every change | After release, on live or sampled traffic |
| Inputs | Curated, versioned dataset | Real user requests |
| Ground truth | Reference answers or rubrics | Usually none |
| Typical graders | Code checks, LLM judges, human review | Reference-free judges, policy checks, user feedback |
| Main job | Block regressions and compare candidates | Detect drift and find new failure modes |
| Blind spot | Only covers cases someone wrote down | Noisy signal for judging one specific change |

Anthropic's guide to [evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) describes production monitoring as the post-launch layer that detects distribution drift and unanticipated real-world failures, with A/B tests for significant changes once traffic allows, and user feedback and transcript review filling the remaining gaps. For agents, online evaluation cannot check a goal state the way a sandboxed offline task can, but trajectory checks still apply, because a forbidden tool call or an unconfirmed write is visible in the logged spans.

Low-scoring production traces go to review, confirmed failures join the offline dataset, and the next release is gated on them. The [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation) shows that loop end to end.

## Related terms

- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.

## Guides that cover online evaluation

- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.

Part of the [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) topic hub.

## Cite this definition

The AI Engineering Editors. "What is online evaluation? Scoring live LLM traffic." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/online-evaluation

- Title: What is online evaluation? Scoring live LLM traffic
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/online-evaluation
- Markdown: https://theaiengineering.com/glossary/online-evaluation.md
