What is online evaluation? Scoring live LLM traffic
Also called online evals, production evaluation
Definition
Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
Online evaluation is the scoring of an LLM application's real production traffic after release, as opposed to running a fixed test set before release. Evaluators read captured traces, usually a filtered sample, and attach scores for qualities such as relevance, faithfulness, policy compliance, or task success. It catches failures that no test case anticipated and shows whether quality drifts as users, data, and upstream models change. It depends on LLM observability, because evaluators need the stored prompt, retrieved context, and output of each request.
How online evaluation works#
Online evaluation runs graders on production traces as they arrive. A rule selects which traces to score using filters and a sampling rate, graders score each selected trace without a reference answer, and the scores are stored on the trace, where dashboards aggregate them and low scores are routed to human review.
Tracing platforms such as LangSmith and Langfuse configure online evaluators this way. Langfuse's online evaluation docs define a rule as a set of filters, a sampling rate, and one or more evaluators, and advise lowering the sampling rate when the estimated cost of an LLM judge is too high. Score a random sample of normal traffic plus every trace with an error or negative user feedback, so rare failures are not sampled away.
Live traffic has no reference answers, so online graders are reference-free:
- Model-based: an LLM-as-a-judge checks faithfulness to the retrieved context, relevance, or tone.
- Code-based: schema validation, forbidden tool calls, step counts, and cost per task, computed from span data.
- Signals: user ratings, retries, escalations to a human, and guardrail triggers.
The OpenTelemetry GenAI conventions define a gen_ai.evaluation.result event that carries the metric name, a score value or label, and an explanation, parented to the span being evaluated. Scores recorded this way travel with the trace in a vendor-neutral format. A/B tests are a related form of online evaluation that compares product outcomes, such as task completion, between two live versions.
Online vs offline evaluation#
Offline evaluation runs a fixed, versioned golden dataset before a change ships and can compare outputs with reference answers. Online evaluation scores real traffic after release, where references rarely exist. Offline evals gate releases. Online evals detect drift and surface new failure modes, which then become offline test cases.
| Offline evaluation | Online evaluation | |
|---|---|---|
| When it runs | Before release, on every change | After release, on live or sampled traffic |
| Inputs | Curated, versioned dataset | Real user requests |
| Ground truth | Reference answers or rubrics | Usually none |
| Typical graders | Code checks, LLM judges, human review | Reference-free judges, policy checks, user feedback |
| Main job | Block regressions and compare candidates | Detect drift and find new failure modes |
| Blind spot | Only covers cases someone wrote down | Noisy signal for judging one specific change |
Anthropic's guide to evals for AI agents describes production monitoring as the post-launch layer that detects distribution drift and unanticipated real-world failures, with A/B tests for significant changes once traffic allows, and user feedback and transcript review filling the remaining gaps. For agents, online evaluation cannot check a goal state the way a sandboxed offline task can, but trajectory checks still apply, because a forbidden tool call or an unconfirmed write is visible in the logged spans.
Low-scoring production traces go to review, confirmed failures join the offline dataset, and the next release is gated on them. The LLM evaluation guide shows that loop end to end.
Related terms
Guides that cover online evaluation
Cite this definition
The AI Engineering Editors. "What is online evaluation? Scoring live LLM traffic." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/online-evaluation
- Title
- What is online evaluation? Scoring live LLM traffic
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated