---
title: "LLM observability: what to capture, which metrics, and tools"
description: "LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private."
summary: "LLM observability is the practice of recording traces, tokens, latency, cost, and errors from language model applications so any output can be explained later. This guide covers what to capture, the OpenTelemetry GenAI conventions, monitoring versus observability, tool categories, and privacy."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/llm-observability"
markdown_url: "https://theaiengineering.com/articles/llm-observability.md"
topic: "LLM evaluation and observability"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-11-01"
updated: "2026-11-11"
keywords: ["llm observability","llm observability tools","llm monitoring","llm observability open source","llm observability metrics"]
citation: "The AI Engineering Editors. \"LLM observability: what to capture, which metrics, and tools.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/llm-observability"
---

# LLM observability: what to capture, which metrics, and tools

## TL;DR

- LLM observability records every model call, retrieval, and tool call in a request as spans in one trace, with tokens, latency, cost, and errors attached to each span.
- The OpenTelemetry GenAI semantic conventions define standard span names and attributes such as gen_ai.operation.name and gen_ai.usage.input_tokens, but they still carry Development status.
- OpenTelemetry guidance says instrumentations should not capture prompts and outputs by default, and recommends external storage with separate access controls for content in production.
- Monitoring answers questions you defined in advance with metrics and alerts, while observability keeps enough context in traces to answer new questions after an incident.
- Production traces are the best source of new eval cases, so observability and evaluation work best as one loop.

LLM observability is the practice of recording what a language model application did on each request, including prompts, outputs, model calls, tool calls, token usage, latency, cost, and errors, so that any output can be explained after the fact. LLM observability matters because LLM failures are rarely crashes: a request returns HTTP 200 with a fluent, wrong answer, and only the trace shows which retrieval or tool step went astray. This guide covers what to capture, which metrics to track, where the [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry) GenAI conventions stand, how tools differ, and how to keep prompt data private. It is part of our [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) hub.

## What is LLM observability?

[LLM observability](https://theaiengineering.com/glossary/llm-observability) is the collection and analysis of traces, metrics, and logs from language model applications, detailed enough to explain why a specific response was slow, expensive, or wrong. It applies distributed tracing to LLM workloads and adds signals that ordinary application monitoring lacks, such as token counts, model parameters, and the content of prompts and outputs.

Traditional APM assumes deterministic code: the same input takes the same path. An LLM application breaks that assumption in three ways:

- **Outputs vary:** the same prompt can produce different answers, so you need the actual output, not only the status code.
- **Cost scales with content:** a request's price depends on its [tokens](https://theaiengineering.com/glossary/tokens), which depend on prompt length, retrieved context, and output length.
- **Requests fan out:** one user question can trigger retrieval, several model calls, and tool calls, and an [AI agent](https://theaiengineering.com/glossary/ai-agent) can loop through dozens of steps.

## What to capture in an LLM trace

Capture one trace per user request, with a span for every model call, retrieval, and tool execution inside it. Each model call span should record the provider, requested and returned model, parameters, token counts, latency, finish reason, and errors. Record prompt and output content only under an explicit policy, because content is both sensitive and large.

**Figure 1: One agent request as a trace, with a span for each model call, retrieval, and tool call. Timings are illustrative.**

A 4.8 second agent trace with child spans for retrieval, a first model call, a tool call with a nested HTTP request, and a second model call.

Spans:

- invoke_agent support-bot: 0 to 4800 ms (4.8 s)
  - retrieval: 40 to 340 ms (0.3 s · 8 chunks)
  - chat gpt-4.1: 380 to 2280 ms (1.9 s · 2.1k in / 180 out)
  - execute_tool lookup_order: 2320 to 2920 ms (0.6 s)
    - GET /orders/{id}: 2360 to 2860 ms (0.5 s)
  - chat gpt-4.1: 2960 to 4760 ms (1.8 s · 2.6k in / 240 out)

### Traces and spans

A trace is the tree of spans for one request; a span is one timed operation with attributes. Propagate trace context across services so a request that passes through an API, an [AI gateway](https://theaiengineering.com/glossary/ai-gateway), and a worker still appears as one trace. Attach a conversation or session ID so multi-turn chats can be read in order.

### Prompts, outputs, and parameters

Record the prompt template name and version separately from the rendered prompt. A version label lets you group traces by prompt release without storing full text. Record parameters such as temperature, maximum tokens, and tool definitions, since a changed parameter explains many sudden quality shifts.

### Tokens, cost, and latency

Record input, output, cached, and reasoning tokens separately. [Prompt caching](https://theaiengineering.com/glossary/prompt-caching) and [reasoning models](https://theaiengineering.com/glossary/reasoning-model) both change cost in ways a single total hides. For latency, record total duration and, for streaming calls, [time to first token](https://theaiengineering.com/glossary/time-to-first-token), which is what users perceive as responsiveness.

### Errors and finish reasons

Record the error class (rate limit, timeout, provider 5xx, content filter, malformed output) and the finish reason for successful calls. A response that stopped because it hit the token limit returns successfully but is usually truncated, and only the finish reason reveals it.

## LLM observability metrics

LLM observability metrics fall into four groups: latency, usage and cost, reliability, and quality. Derive them from span attributes rather than separate counters so every chart can be broken down by model, provider, prompt version, feature, or customer. The table lists the metrics that most often explain incidents and budget surprises in production.

| Metric | What it tells you | Useful breakdowns |
|---|---|---|
| End-to-end latency (p50, p95, p99) | User-perceived speed | Route, model, provider |
| Time to first token | Streaming responsiveness | Model, prompt size |
| [Time per output token](https://theaiengineering.com/glossary/inter-token-latency) | Generation speed | Model, provider, region |
| Input, output, cached, and reasoning tokens | Cost drivers and context growth | Feature, prompt version |
| Cost per request and per completed task | Unit economics | Feature, customer, model |
| Error rate by class | Reliability by dependency | Provider, error type |
| Finish reasons | Truncation and refusals | Prompt version, model |
| Tool calls and steps per agent run | Agent efficiency and loops | Agent, tool |
| Online eval scores and user feedback | Output quality | Prompt version, model |

Cost per completed task matters more than cost per request for agents, because one task can trigger many model calls. Track averages and tails together: a feature with a modest average token count can still carry a long tail of requests that fill the context window.

## OpenTelemetry GenAI semantic conventions

The OpenTelemetry GenAI semantic conventions define standard span names, attributes, metrics, and events for model calls, embeddings, retrieval, tool execution, agents, and MCP. They let any instrumentation library emit spans that any compatible backend can read. As of November 2026 the conventions still carry Development status, so attribute names can change between releases.

In 2026 the conventions moved out of the main semantic conventions repository into a dedicated [OpenTelemetry GenAI semantic conventions repository](https://github.com/open-telemetry/semantic-conventions-genai), which also holds provider-specific conventions for Anthropic, AWS Bedrock, Azure AI Inference, and OpenAI. Development is the earliest of OpenTelemetry's [maturity levels](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/maturity-levels.md), and its definition warns that configuration options might break often. Pin the convention version your instrumentation emits and record it, so dashboards do not break silently when attribute names change.

An inference span is named `{gen_ai.operation.name} {gen_ai.request.model}`, and its core attributes look like this:

```json
{
  "name": "chat gpt-4.1",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "openai",
    "gen_ai.request.model": "gpt-4.1",
    "gen_ai.request.temperature": 0.2,
    "gen_ai.response.model": "gpt-4.1-2025-04-14",
    "gen_ai.response.finish_reasons": ["stop"],
    "gen_ai.usage.input_tokens": 2104,
    "gen_ai.usage.cache_read.input_tokens": 1536,
    "gen_ai.usage.output_tokens": 180,
    "gen_ai.conversation.id": "conv_5j66UpCpwteGg4YSxUnt7lPY"
  }
}
```

The conventions mark `gen_ai.operation.name` and `gen_ai.provider.name` as required, and token usage, response model, and finish reasons as recommended. Agent work uses separate spans such as `invoke_agent`, `plan`, and `execute_tool {gen_ai.tool.name}`. Metrics include `gen_ai.client.operation.duration` on the client side and `gen_ai.server.time_to_first_token` for model servers. Content attributes such as `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.system_instructions` are opt-in, which the privacy section below covers.

Gateways can emit these spans without changes to application code. [Bifrost](https://docs.getbifrost.ai/features/observability/otel), for example, exports traces in the GenAI conventions format over OTLP HTTP or gRPC, with token counts, cost, and time to first token as attributes, to backends such as Grafana Cloud, Datadog, New Relic, Honeycomb, and Langfuse.

## LLM monitoring vs observability

LLM monitoring tracks predefined metrics and alerts when they cross thresholds, such as error rate, p95 latency, or daily spend. Observability keeps enough context in traces to answer questions nobody predicted, such as why refund answers degraded after Tuesday's prompt release. You need both: monitoring detects that something changed, and observability explains the cause.

| | LLM monitoring | LLM observability |
|---|---|---|
| Question type | Known in advance | Discovered during investigation |
| Data | Aggregated metrics and alerts | Traces with attributes and, where allowed, content |
| Typical output | Dashboard, page, budget alert | A specific trace or cohort that explains a failure |
| Example | "Error rate on provider A is above 2%" | "Answers fail when retrieval returns docs from the old index" |

Monitoring is cheap to run continuously. Observability costs more storage, which is why sampling and retention policies matter.

## How to monitor LLM applications in production

Monitor LLM applications in production by alerting on a small set of metrics tied to user impact: error rate by provider, p95 latency and time to first token, spend against budget, and online quality scores. Keep full traces for a sample of traffic and for every error, then review flagged traces on a regular schedule.

- **Sampling:** keep 100% of error traces and traces with negative user feedback, plus a random sample of successful ones. Sample by trace, not by span, so retained traces stay complete.
- **Alerts:** page on provider error spikes and latency regressions; send cost anomalies and quality drops to a daily review instead of a pager.
- **Release markers:** annotate dashboards with prompt, model, and retrieval index versions so a shift lines up with its cause.
- **Gateway logs:** if traffic passes through an AI gateway such as Bifrost or LiteLLM, its request logs give provider-level latency, errors, and spend without code changes. Our explainer on [AI gateways](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers that layer.

Per-provider error metrics from your own traffic are an outage signal that does not depend on a provider updating its status page.

## LLM observability tools: open source and managed

LLM observability tools fall into five categories: instrumentation libraries that emit spans, open source platforms you can self-host, tracing built into development frameworks or ML platforms, general APM products with LLM features, and AI gateways that log at the proxy. Many teams combine an OpenTelemetry-based instrumentation layer with one storage and analysis backend.

| Category | What it does | Examples |
|---|---|---|
| Instrumentation libraries | Emit OpenTelemetry-compatible spans from SDKs and frameworks | OpenLLMetry, OpenInference |
| Open source platforms | Store traces, run evals, manage prompts; self-hostable | Langfuse, Helicone, Opik, Arize Phoenix |
| Framework and ML platform tracing | Tracing bundled with a framework or ML platform | LangSmith, MLflow Tracing |
| General APM with LLM features | Add LLM spans to existing infrastructure monitoring | Datadog |
| AI gateways | Request-level logs, tokens, and cost at the proxy | Bifrost, LiteLLM, Helicone AI Gateway |

Licenses differ inside the open source category. Langfuse's core is MIT-licensed with enterprise directories under separate terms, Helicone and Opik use Apache 2.0, and Arize Phoenix uses the Elastic License 2.0, which is source-available rather than an OSI-approved open source license. OpenInference describes itself as a set of conventions and plugins complementary to OpenTelemetry, and MLflow describes its tracing as fully OpenTelemetry-compatible.

When comparing tools, check OTLP ingestion, which convention versions are supported, self-hosting options, data residency, and whether the tool can run evaluations on stored traces. Our directory of [LLM observability platforms](https://theaiengineering.com/platforms/llm-observability) lists options by category, and the [AI gateways directory](https://theaiengineering.com/platforms/ai-gateways) covers proxy-level logging.

## Privacy: handling prompts and outputs safely

Prompts and outputs routinely contain personal data, credentials pasted by users, and confidential documents retrieved by RAG. Treat them as user data, not operational telemetry. Capture content only where policy allows, redact before export, store content separately with tighter access controls and shorter retention, and keep metadata such as tokens and latency in the standard pipeline.

The OpenTelemetry [guidance on capturing instructions, inputs, and outputs](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) states that instrumentations should not capture content by default and should offer an opt-in. It describes three patterns:

1. **Do not record content.** This is the default.
2. **Record content on span attributes.** The spec describes this as suited to pre-production or to cases where telemetry storage complies with privacy regulations.
3. **Store content externally and record references on spans.** The spec recommends this for production, because external storage allows separate access controls. Reference attributes such as `gen_ai.input.messages_ref` hold the storage URI.

Additional controls worth adopting:

- **Redact at the source:** strip emails, phone numbers, keys, and account numbers before spans leave the process, not in the backend.
- **Pseudonymize users:** record a stable hashed user ID instead of names or emails.
- **Set retention by data class:** keep metrics for months and raw content for days, unless a review process needs longer.
- **Respect residency:** confirm the region where a hosted backend stores trace content.
- **Drop content at the gateway:** a gateway can keep tokens, cost, and latency while discarding prompt and response bodies. [Bifrost](https://docs.getbifrost.ai/features/observability/content-logging) sets this globally or per virtual key, and its documentation notes that each export connector has its own setting, so turn content off there as well.
- **Never log secrets:** provider API keys and [guardrails](https://theaiengineering.com/glossary/guardrails) configuration belong in a secret manager, not in span attributes.

## From observability to evaluation

Observability and evaluation share the same data. Production traces show which inputs fail, [online judges](https://theaiengineering.com/glossary/online-evaluation) score sampled traces, and the worst cases become new entries in the [offline eval dataset](https://theaiengineering.com/glossary/golden-dataset). Connecting the two turns every incident into a regression test, so the same failure cannot ship twice without the eval suite flagging it first.

The loop works in four steps:

1. Run reference-free graders, including an [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) for [faithfulness](https://theaiengineering.com/glossary/faithfulness) or relevance, on a sample of production traces.
2. Route low-scoring and user-flagged traces to an annotation queue for human review.
3. Copy confirmed failures, with their inputs and retrieved context, into the versioned eval dataset.
4. Re-run the offline suite on every change and compare scores with the baseline.

Where to go next: read our guide to [LLM evaluation](https://theaiengineering.com/articles/llm-evaluation) for datasets, judges, and CI gates, and our explainer on [AI agents](https://theaiengineering.com/articles/what-are-ai-agents) for the multi-step runs that make tracing necessary.

## Frequently asked questions

### What are LLM observability tools?

LLM observability tools collect, store, and visualize traces from language model applications. They record prompts, outputs, token counts, latency, cost, and errors for each model and tool call, then let teams search traces, build dashboards, and run evaluations on sampled traffic. The category includes instrumentation libraries, open source platforms, framework-specific tracing, general APM products with LLM features, and AI gateways such as Bifrost that log requests.

### What is LLM tracing?

LLM tracing records the full path of one request through an LLM application as a tree of spans. Each span represents one operation, such as a model call, a vector search, or a tool execution, and carries its own timing, inputs, outputs, and token usage. Tracing shows which step caused a slow or wrong answer, which aggregate metrics alone cannot show.

### What is the difference between observability and monitoring?

Monitoring tracks predefined metrics and fires alerts when they cross thresholds, such as error rate or p95 latency. Observability captures enough context, including full traces and their attributes, to investigate questions nobody predicted, such as why answers about one product degraded after a prompt change. Monitoring tells you that something is wrong; observability helps you find out why.

### Is there an open source LLM observability platform?

Yes. Langfuse, Helicone, and Opik are open source LLM observability platforms that can be self-hosted, and Arize Phoenix is source-available under the Elastic License 2.0. Instrumentation libraries such as OpenLLMetry and OpenInference emit OpenTelemetry-compatible spans that these platforms and many general-purpose tracing backends can ingest. Check each license, since some projects keep enterprise features under separate terms.

### What is the difference between LLM observability and agent observability?

LLM observability focuses on individual model calls: prompts, outputs, tokens, latency, and cost. Agent observability extends the same tracing to multi-step runs, adding spans for planning, tool execution, and agent invocations, plus metrics such as tool calls per run. The OpenTelemetry GenAI conventions cover both, with dedicated agent spans such as invoke_agent and execute_tool.

### How do you track LLM cost?

Record input, output, cached, and reasoning token counts on every model call span, then multiply by the provider's current per-token prices in your observability backend or gateway. Attribute cost to features, customers, and prompt versions through span attributes, and aggregate it per completed task rather than per request, since one agent task can trigger many model calls.

## Terms in this guide

- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Faithfulness](https://theaiengineering.com/glossary/faithfulness): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1.

## Related guides

- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [What are AI agents? How agents work and when to use one](https://theaiengineering.com/articles/what-are-ai-agents): AI agents are LLM systems that call tools in a loop until a task is done. Learn how the loop works, agents vs agentic AI, examples, failure modes, and limits.

Part of the [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation) topic hub.

## Cite this guide

The AI Engineering Editors. "LLM observability: what to capture, which metrics, and tools." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/llm-observability

- Title: LLM observability: what to capture, which metrics, and tools
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-11-01
- Updated: 2026-11-11
- URL: https://theaiengineering.com/articles/llm-observability
- Markdown: https://theaiengineering.com/articles/llm-observability.md

To cite one section, link to its anchor:

- [What is LLM observability?](https://theaiengineering.com/articles/llm-observability#what-is-llm-observability)
- [What to capture in an LLM trace](https://theaiengineering.com/articles/llm-observability#what-to-capture-in-an-llm-trace)
- [LLM observability metrics](https://theaiengineering.com/articles/llm-observability#llm-observability-metrics)
- [OpenTelemetry GenAI semantic conventions](https://theaiengineering.com/articles/llm-observability#opentelemetry-genai-semantic-conventions)
- [LLM monitoring vs observability](https://theaiengineering.com/articles/llm-observability#llm-monitoring-vs-observability)
- [How to monitor LLM applications in production](https://theaiengineering.com/articles/llm-observability#how-to-monitor-llm-applications-in-production)
- [LLM observability tools: open source and managed](https://theaiengineering.com/articles/llm-observability#llm-observability-tools-open-source-and-managed)
- [Privacy: handling prompts and outputs safely](https://theaiengineering.com/articles/llm-observability#privacy-handling-prompts-and-outputs-safely)
- [From observability to evaluation](https://theaiengineering.com/articles/llm-observability#from-observability-to-evaluation)
- [Frequently asked questions](https://theaiengineering.com/articles/llm-observability#faq)
