---
title: "What is LLM observability? Traces, metrics, and tools"
description: "LLM observability captures traces, prompts, outputs, token usage, latency, and quality scores from LLM apps in production so teams can debug and improve them."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/llm-observability"
markdown_url: "https://theaiengineering.com/glossary/llm-observability.md"
topic: "LLM evaluation and observability"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["LLM observability","LLM monitoring","AI observability","LLM tracing"]
citation: "The AI Engineering Editors. \"What is LLM observability? Traces, metrics, and tools.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-observability"
---

# What is LLM observability? Traces, metrics, and tools

Also called LLM monitoring, AI observability, LLM tracing.

> **Definition of LLM observability:** LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.

LLM observability is the extension of application monitoring to the parts of an AI system that traditional APM tools do not capture: what the model was asked, what it returned, which tools it called, and whether the answer was any good. One user request to an agent can involve several model calls, retrievals, and tool executions, so observability tools record them as a trace of nested spans that shows the full path from request to response.

## How LLM observability works

Instrumentation emits a span for each operation. The source is usually an SDK, an [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry) exporter, or an [AI gateway](https://theaiengineering.com/glossary/ai-gateway) that already sees all model traffic.

**Figure 1: One agent request traced as nested spans with token and latency data. Model call durations are illustrative.**

A 3.9 second POST /chat trace containing four child spans in order: a 180 ms retrieval, a model call, a 310 ms tool call, and a second model call.

Spans:

- POST /chat: 0 to 3900 ms
  - retrieve (vector search): 10 to 190 ms (180 ms)
  - llm.chat: 200 to 1700 ms (3,100 in · 220 out)
  - tool.call get_order_status: 1710 to 2020 ms (310 ms)
  - llm.chat: 2030 to 3500 ms (3,600 in · 140 out)

Markers:

- 840 ms: TTFT 640 ms
- 2620 ms: TTFT 590 ms

The [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), still marked as in development, standardize attribute names such as `gen_ai.request.model`, `gen_ai.usage.input_tokens`, and `gen_ai.tool.name`, so traces from different frameworks and vendors share one schema. On top of traces, teams track three groups of signals:

- **Performance:** [time to first token](https://theaiengineering.com/glossary/time-to-first-token), end-to-end latency, and error and rate-limit rates.
- **Usage and cost:** input and output [tokens](https://theaiengineering.com/glossary/tokens) per request, user, feature, and model.
- **Quality:** scores from online evaluators such as [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) graders, user feedback, and guardrail triggers.

Because traces contain full prompts and outputs, they can include personal data, so redaction and retention rules belong in the design from the start.

## LLM observability vs monitoring

Monitoring answers known questions with dashboards and alerts: is p95 latency above target, did the error rate spike. Observability keeps enough detail, including full prompts, outputs, and intermediate steps, to answer questions nobody planned for, such as why an agent called the wrong tool for one customer. LLM applications need both, because quality failures rarely raise an error.

## LLM observability vs evaluation

[LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation) scores outputs against criteria; observability collects the production data those scores run on. Failing production traces become new eval cases, which closes the loop between the two. For more, see the [LLM observability guide](https://theaiengineering.com/articles/llm-observability) and the [LLM observability platforms](https://theaiengineering.com/platforms/llm-observability) directory.

## Related terms

- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.
- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.

## Guides that cover llm observability

- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [How to build an MCP server: a Python and TypeScript tutorial](https://theaiengineering.com/articles/how-to-build-an-mcp-server): Learn how to build an MCP server with the official Python and TypeScript SDKs: tools, resources, transports, testing, client config, and production security.
- [How to build an AI agent: a step-by-step guide in Python](https://theaiengineering.com/articles/how-to-build-an-ai-agent): Learn how to build an AI agent from scratch in Python: when you need one, tool schemas, the agent loop, stop conditions, guardrails, evals, and tracing.
- [AI agent frameworks compared: LangGraph, CrewAI, ADK, and more](https://theaiengineering.com/articles/ai-agent-frameworks): An AI agent frameworks comparison of LangGraph, LangChain, CrewAI, the OpenAI, Claude, and Google SDKs, Pydantic AI, Mastra, and more, by language and license.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [What is the Model Context Protocol (MCP)? How it works](https://theaiengineering.com/articles/what-is-model-context-protocol): The Model Context Protocol (MCP) is an open standard for connecting AI apps to tools and data. Learn its architecture, transports, tool calls, and security.

Part of the [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) topic hub.

## Cite this definition

The AI Engineering Editors. "What is LLM observability? Traces, metrics, and tools." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-observability

- Title: What is LLM observability? Traces, metrics, and tools
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/llm-observability
- Markdown: https://theaiengineering.com/glossary/llm-observability.md
