What is LLM observability? Traces, metrics, and tools

Also called LLM monitoring, AI observability, LLM tracing

Definition

LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.

LLM observability is the extension of application monitoring to the parts of an AI system that traditional APM tools do not capture: what the model was asked, what it returned, which tools it called, and whether the answer was any good. One user request to an agent can involve several model calls, retrievals, and tool executions, so observability tools record them as a trace of nested spans that shows the full path from request to response.

How LLM observability works#

Instrumentation emits a span for each operation. The source is usually an SDK, an OpenTelemetry exporter, or an AI gateway that already sees all model traffic.

A 3.9 second POST /chat trace containing four child spans in order: a 180 ms retrieval, a model call, a 310 ms tool call, and a second model call. 0 ms1000 ms2000 ms3000 ms4000 msPOST /chatretrieve (vector search)180 msllm.chat3,100 in · 220 outtool.call get_order_status310 msllm.chat3,600 in · 140 outTTFT 640 msTTFT 590 ms
Figure 1: One agent request traced as nested spans with token and latency data. Model call durations are illustrative.

The OpenTelemetry GenAI semantic conventions, still marked as in development, standardize attribute names such as gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.tool.name, so traces from different frameworks and vendors share one schema. On top of traces, teams track three groups of signals:

  • Performance: time to first token, end-to-end latency, and error and rate-limit rates.
  • Usage and cost: input and output tokens per request, user, feature, and model.
  • Quality: scores from online evaluators such as LLM-as-a-judge graders, user feedback, and guardrail triggers.

Because traces contain full prompts and outputs, they can include personal data, so redaction and retention rules belong in the design from the start.

LLM observability vs monitoring#

Monitoring answers known questions with dashboards and alerts: is p95 latency above target, did the error rate spike. Observability keeps enough detail, including full prompts, outputs, and intermediate steps, to answer questions nobody planned for, such as why an agent called the wrong tool for one customer. LLM applications need both, because quality failures rarely raise an error.

LLM observability vs evaluation#

LLM evaluation scores outputs against criteria; observability collects the production data those scores run on. Failing production traces become new eval cases, which closes the loop between the two. For more, see the LLM observability guide and the LLM observability platforms directory.

Cite this definition

The AI Engineering Editors. "What is LLM observability? Traces, metrics, and tools." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-observability

Title
What is LLM observability? Traces, metrics, and tools
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM evaluation and observability hub · Updated