---
title: "LLM evaluation and observability"
description: "LLM evaluation and observability for production teams: eval datasets, metrics and LLM-as-a-judge, plus tracing, monitoring and the tools that support them."
type: "topic-hub"
canonical_url: "https://theaiengineering.com/topics/llm-evaluation"
markdown_url: "https://theaiengineering.com/topics/llm-evaluation.md"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
keywords: ["llm evaluation","llm evals","llm evaluation metrics","llm observability","llm monitoring","llm as a judge"]
citation: "The AI Engineering Editors. \"LLM evaluation and observability.\" The AI Engineering. https://theaiengineering.com/topics/llm-evaluation"
---

# LLM evaluation and observability

Measuring whether LLM systems work, before and after they ship.

LLM evaluation measures whether a model-based system produces acceptable outputs for its task; LLM observability records what the system actually did in production so failures can be found and explained. Together they replace intuition with evidence.

This hub covers offline and online evals, judge models and their biases, tracing standards, the metrics worth tracking, and the categories of tooling available.

## LLM evaluation and observability guides

- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.

## Key terms

- [Faithfulness](https://theaiengineering.com/glossary/faithfulness): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Hallucination](https://theaiengineering.com/glossary/hallucination): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.

## Tools and platforms

- [LLM observability and evaluation](https://theaiengineering.com/platforms/llm-observability): 34 platforms. LLM observability is the practice of recording each step of an LLM application, including prompts, model responses, tool calls, retrieval results, latency, token usage, and cost, so engineers can debug failures and measure quality.

## Cite this page

The AI Engineering Editors. "LLM evaluation and observability." The AI Engineering. https://theaiengineering.com/topics/llm-evaluation

- Title: LLM evaluation and observability
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- URL: https://theaiengineering.com/topics/llm-evaluation
- Markdown: https://theaiengineering.com/topics/llm-evaluation.md
