LLM observability and evaluation

LLM observability is the practice of recording each step of an LLM application, including prompts, model responses, tool calls, retrieval results, latency, token usage, and cost, so engineers can debug failures and measure quality. The tools in this category collect those traces, score them with evaluations, or both.

The category has four groups. Open-source platforms such as Langfuse, Opik, MLflow, Laminar, and Agenta can be self-hosted. Evaluation frameworks such as DeepEval, Ragas, promptfoo, and Inspect run tests in code and CI. Commercial platforms such as LangSmith, Braintrust, Maxim AI, Galileo, and HoneyHive combine tracing, datasets, experiments, and online scoring. Application monitoring vendors, including Datadog, New Relic, Dynatrace, Grafana, and Sentry, add LLM views to existing APM.

Most tools accept OpenTelemetry data, which matters because ownership is changing: ClickHouse acquired Langfuse, Cisco acquired Galileo, OpenAI agreed to buy promptfoo, Traceloop is joining ServiceNow, and Helicone moved to maintenance mode in 2026.

LLM observability and evaluation compared

PlatformOpen sourceDeploymentBest for
AgentaYes (MIT)Self-hosted, Managed cloudTeams where non-engineers edit and evaluate prompts
Arize AXNoManaged cloudEnterprise agent monitoring beyond self-hosted Phoenix
Arize PhoenixNoLocal, Self-hosted, Managed cloudLocal and self-hosted tracing with OpenTelemetry
BraintrustNoManaged cloudEval-driven development using production traces
Confident AI and DeepEvalYes (Apache-2.0)Library, Managed cloudPytest-style LLM tests with a hosted dashboard
Datadog LLM ObservabilityNoManaged cloudTeams already monitoring infrastructure with Datadog
Dynatrace AI ObservabilityNoManaged cloudDynatrace customers adding LLM monitoring
EvidentlyYes (Apache-2.0)Library, Self-hostedTeams monitoring both classic ML and LLM systems
Fiddler AINoManaged cloud, Self-hostedRegulated enterprises monitoring ML and agents together
GalileoNoManaged cloudTurning offline evals into production guardrails
Grafana Cloud AI ObservabilityNoManaged cloudTeams standardized on Grafana and OpenTelemetry
HeliconeYes (Apache-2.0)Managed cloud, Self-hostedExisting users needing proxy-based request logging
HoneyHiveNoManaged cloud, Self-hostedEnterprises with strict data residency for traces
InspectYes (MIT)Library, CLIResearch-grade model and agent evaluations
LaminarYes (Apache-2.0)Managed cloud, Self-hostedDebugging long-running and browser agents
LangfuseYes (MIT)Self-hosted, Managed cloudSelf-hosted tracing and prompt management
LangSmithNoManaged cloud, Self-hostedTeams building with LangChain or LangGraph
LangWatchYes (Apache-2.0)Managed cloud, Self-hostedPlatform teams governing agents and coding assistants
LunaryNoManaged cloud, Self-hostedSmall teams combining logs, prompts, and reviews
Maxim AINoManaged cloud, Self-hostedPre-release agent simulation plus production monitoring
MLflowYes (Apache-2.0)Self-hosted, Managed cloud, LibraryTeams already using MLflow for ML lifecycle work
New Relic AI observabilityNoManaged cloudTeams already using New Relic APM
OpenLITYes (Apache-2.0)Self-hosted, LibrarySelf-hosted OpenTelemetry stack with GPU monitoring
OpikYes (Apache-2.0)Self-hosted, Managed cloudOpen-source evals with built-in prompt optimization
promptfooYes (MIT)CLI, Library, Managed cloudPrompt and model regression tests in CI
PromptLayerNoManaged cloudPrompt iteration by domain experts
Pydantic LogfireNoManaged cloud, Self-hostedPython teams using Pydantic AI or FastAPI
RagasYes (Apache-2.0)LibraryRAG metrics and synthetic test data
RaindropNoManaged cloudFinding silent failures in production agent conversations
Respan (formerly Keywords AI)Yes (Apache-2.0)Managed cloud, Self-hostedGateway and tracing from one vendor
Sentry AI observabilityNoManaged cloudDevelopers already using Sentry for errors
Traceloop and OpenLLMetryYes (Apache-2.0)Library, Managed cloudOpenTelemetry-native instrumentation for LLM apps
TruLensYes (MIT)LibraryScoring RAG pipelines step by step
W&B WeaveYes (Apache-2.0)Managed cloud, LibraryTeams already using Weights & Biases

Profiles

  • Agenta logo

    Agenta

    agenta.ai

    Agenta is an open-source workspace, MIT-licensed outside its enterprise directories, for building agents through chat and improving them with versioned prompts and tools, test-set evaluations with LLM-as-a-judge or custom evaluators, and tracing of every run.

    • Open source · MIT
    • Self-hosted
    • Managed cloud
    • Python
    • TypeScript
  • Arize AX logo

    Arize AX

    Arize AI · arize.com

    Arize AX is Arize AI's managed platform for observing, evaluating, and improving AI agents, with tracing built on OpenInference, the open convention Arize created, plus span, trace, and session evaluations, experiments, and production monitoring.

    • Managed cloud
    • Python
    • TypeScript
  • Arize Phoenix logo

    Arize Phoenix

    Arize AI · arize.com

    Arize Phoenix is a self-hostable, source-available tool under the Elastic License 2.0 for tracing LLM applications and agents with OpenTelemetry and OpenInference, running evals, and comparing experiments on datasets.

    • Local
    • Self-hosted
    • Managed cloud
    • Python
    • TypeScript
  • Braintrust

    braintrust.dev

    Braintrust is a commercial platform for evaluating and observing AI agents that traces production traffic, turns traces into versioned datasets, scores outputs with LLM, code, or human scorers, and catches regressions before release.

    • Managed cloud
    • Python
    • TypeScript
    • Go
  • Confident AI and DeepEval logo

    Confident AI and DeepEval

    Confident AI · confident-ai.com

    DeepEval is an Apache-2.0 Python framework for testing LLM applications with plug-and-play metrics for agents, RAG, multi-turn chat, and voice, and Confident AI is the commercial platform from the same team for running those evals and observing production traces.

    • Open source · Apache-2.0
    • Library
    • Managed cloud
    • Python
  • Datadog LLM Observability

    Datadog · datadoghq.com

    Datadog LLM Observability, part of Datadog's Agent Observability offering, traces prompts, retrieval, and tool calls in agent workflows, runs offline experiments and built-in or custom evaluations, and flags hallucinations, prompt injection, and PII exposure.

    • Managed cloud
  • Dynatrace AI Observability

    Dynatrace · dynatrace.com

    Dynatrace AI Observability monitors AI applications, LLMs, and agents from the user experience through orchestration to models and infrastructure, tracking token usage, latency, and cost and detecting hallucinations, prompt injection, PII leakage, and toxicity.

    • Managed cloud
  • Evidently logo

    Evidently

    Evidently AI · evidentlyai.com

    Evidently is an Apache-2.0 Python framework for evaluating, testing, and monitoring ML and LLM systems, with a library of built-in metrics that combines rules, classifiers, and LLM judges for testing AI applications, RAG systems, and multi-agent workflows.

    • Open source · Apache-2.0
    • Library
    • Self-hosted
    • Python
  • Fiddler AI logo

    Fiddler AI

    fiddler.ai

    Fiddler AI is a commercial control plane for enterprise AI agents and models that combines evaluation, monitoring, guardrail policy enforcement, governance, and ML observability, with SaaS, VPC, and AWS GovCloud deployments.

    • Managed cloud
    • Self-hosted
  • Galileo logo

    Galileo

    Cisco · galileo.ai · Burlingame, California, United States

    Galileo is an evaluation and observability platform for AI agents with built-in and custom evaluators, an insights engine that finds failure modes, and Luna models distilled from evaluators to monitor production traffic and drive guardrails.

    Cisco completed its acquisition of Galileo on May 22, 2026, and is bringing its capabilities into Splunk Observability Cloud.

    • Managed cloud
  • Grafana Cloud AI Observability

    Grafana Labs · grafana.com

    Grafana Cloud AI Observability, including Agent Observability, traces production agents with OpenTelemetry, runs live evaluations for hallucinations and quality drift, and tracks latency, tokens, and cost across agents, runs, and models.

    • Managed cloud
  • Helicone logo

    Helicone

    helicone.ai

    Helicone is an Apache-2.0 LLM observability platform and AI gateway that logs requests through a proxy or SDK and shows sessions, users, and costs, with prompt management, datasets, a playground, rate limits, and alerts.

    Mintlify acquired Helicone in March 2026 and moved it to maintenance mode; Helicone's site now states that it is no longer part of Mintlify.

    • Open source · Apache-2.0
    • Managed cloud
    • Self-hosted
    • TypeScript
  • HoneyHive logo

    HoneyHive

    honeyhive.ai

    HoneyHive is a commercial platform for tracing, evaluating, monitoring, and improving production AI agents across teams, with OpenTelemetry ingestion and hybrid or fully self-hosted deployments that keep trace data inside the customer's network.

    • Managed cloud
    • Self-hosted
  • Inspect

    UK AI Security Institute · inspect.aisi.org.uk

    Inspect is an MIT-licensed Python framework from the UK AI Security Institute for building and running LLM evaluations, with components for datasets, solvers, scorers, tool use, agents, and sandboxed execution, plus a log viewer for results.

    • Open source · MIT
    • Library
    • CLI
    • Python
  • Laminar logo

    Laminar

    laminar.sh

    Laminar is an Apache-2.0 observability platform for AI agents that traces runs, extracts agent inputs automatically, evaluates results, and surfaces failures, with integrations for agent SDKs and browser agents and Helm charts for self-hosting.

    • Open source · Apache-2.0
    • Managed cloud
    • Self-hosted
    • TypeScript
    • Python
  • Langfuse logo

    Langfuse

    ClickHouse · langfuse.com

    Langfuse is an open-source platform for tracing, evaluating, and improving LLM applications and agents, with prompt management and datasets, MIT-licensed outside its enterprise directories and available self-hosted or as Langfuse Cloud.

    ClickHouse acquired Langfuse in January 2026, and the project remains MIT-licensed outside its enterprise directories.

    • Open source · MIT
    • Self-hosted
    • Managed cloud
    • Python
    • TypeScript
  • LangSmith logo

    LangSmith

    LangChain · langchain.com

    LangSmith is LangChain's commercial platform for tracing, monitoring, and evaluating LLM and agent applications; it works with or without LangChain, accepts OpenTelemetry data, and runs as managed cloud, bring-your-own-cloud, or self-hosted.

    • Managed cloud
    • Self-hosted
    • Python
    • TypeScript
    • Go
  • LangWatch

    langwatch.ai

    LangWatch is an Apache-2.0 platform for AI platform teams that traces, tests, routes, and governs LLM calls across a company, from internal agents to the coding assistants engineers use, with evaluations and observability in one place.

    • Open source · Apache-2.0
    • Managed cloud
    • Self-hosted
    • TypeScript
    • Python
  • Lunary logo

    Lunary

    lunary.ai

    Lunary is an observability and prompt management platform for LLM applications that logs traces and errors, versions prompt templates with A/B testing, supports human review of responses, and reports usage and cost, in the cloud or self-hosted.

    • Managed cloud
    • Self-hosted
    • Python
    • TypeScript
  • Maxim AI logo

    Maxim AI

    getmaxim.ai

    Maxim AI is a commercial platform for simulating, evaluating, and observing AI agents, with AI-generated scenario simulations, LLM-as-a-judge, programmatic, statistical, and human evaluators, and real-time trace logging with alerts.

    • Managed cloud
    • Self-hosted
    • Python
    • TypeScript
    • Java
  • MLflow

    mlflow.org

    MLflow is an open-source platform for agents, LLMs, and machine learning models that adds OpenTelemetry-compatible tracing, LLM evaluation, prompt management, and an AI gateway to its experiment tracking and model registry, self-hosted or as managed services.

    • Open source · Apache-2.0
    • Self-hosted
    • Managed cloud
    • Library
    • Python
    • TypeScript
  • New Relic AI observability

    New Relic · newrelic.com

    New Relic's AI observability extends its full-stack monitoring to AI applications, showing response time, quality, and token metrics next to APM signals, mapping interactions between agents and tools, and integrating with OpenAI, vector databases, and LangChain.

    • Managed cloud
  • OpenLIT logo

    OpenLIT

    openlit.io

    OpenLIT is an Apache-2.0 platform built on OpenTelemetry for agent observability, evaluations, guardrails, prompt management, and cost and GPU monitoring, covering both custom agents and coding agents, with auto-instrumentation SDKs and a self-hostable dashboard.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
    • TypeScript
  • Opik logo

    Opik

    Comet · comet.com

    Opik is an Apache-2.0 platform from Comet for tracing agents, scoring traces with LLM-as-a-judge metrics, running test suites, optimizing prompts, and applying guardrails, available self-hosted or on Comet's cloud.

    • Open source · Apache-2.0
    • Self-hosted
    • Managed cloud
    • Python
    • TypeScript
  • promptfoo logo

    promptfoo

    promptfoo.dev

    promptfoo is an MIT-licensed CLI and library for testing prompts, agents, and RAG pipelines against test cases and comparing models side by side, with red teaming and a commercial platform for teams.

    OpenAI agreed to acquire Promptfoo in March 2026 and said it would keep developing the MIT-licensed open-source project.

    • Open source · MIT
    • CLI
    • Library
    • Managed cloud
    • TypeScript
  • PromptLayer logo

    PromptLayer

    promptlayer.com

    PromptLayer is a prompt management platform with a prompt registry and visual editor, LLM evaluations and regression sets, and production tracing and logs, designed so domain experts can iterate on prompts without changing the codebase.

    • Managed cloud
  • Pydantic Logfire logo

    Pydantic Logfire

    Pydantic · pydantic.dev

    Pydantic Logfire is an observability platform built on OpenTelemetry that traces LLM calls and agents together with the rest of an application, with an MIT-licensed Python SDK and support for JavaScript, Rust, Go, Java, and other OpenTelemetry languages.

    • Managed cloud
    • Self-hosted
    • Python
    • TypeScript
    • Rust
  • Ragas logo

    Ragas

    ragas.io

    Ragas is an Apache-2.0 Python library for evaluating LLM applications with LLM-based and traditional metrics, generating synthetic test sets, and building feedback loops from production data; it began with metrics for retrieval-augmented generation.

    • Open source · Apache-2.0
    • Library
    • Python
  • Raindrop

    raindrop.ai

    Raindrop is a commercial monitoring platform for AI agents that traces every run, detects silent failures in production conversations, and lets teams simulate changes before shipping them.

    • Managed cloud
  • Respan (formerly Keywords AI) logo

    Respan (formerly Keywords AI)

    Respan · respan.ai

    Respan, formerly Keywords AI, pairs an open-source AI gateway written in Rust and licensed Apache-2.0 with a hosted platform for tracing, monitoring, and evaluating every agent run.

    Keywords AI rebranded as Respan in 2026 and released its Rust gateway as open source.

    • Open source · Apache-2.0
    • Managed cloud
    • Self-hosted
    • Rust
  • Sentry AI observability

    Sentry · sentry.io

    Sentry's AI observability traces agent runs, LLM calls, and tool calls in production applications and flags errors, latency, and token cost spikes, with SDK integrations for the OpenAI, Anthropic, Vercel AI, LangChain, and Pydantic AI libraries.

    • Managed cloud
  • Traceloop and OpenLLMetry logo

    Traceloop and OpenLLMetry

    Traceloop · traceloop.com

    Traceloop is an LLM reliability platform that runs evaluators and monitors on production traces, built on OpenLLMetry, its Apache-2.0 set of OpenTelemetry instrumentations that can also export to other observability backends.

    Traceloop announced in 2026 that it is joining ServiceNow; OpenLLMetry remains an Apache-2.0 open-source project.

    • Open source · Apache-2.0
    • Library
    • Managed cloud
    • Python
    • TypeScript
  • TruLens logo

    TruLens

    Snowflake · trulens.org

    TruLens is an MIT-licensed Python library, created by TruEra and maintained by Snowflake since the acquisition, that instruments AI agents with OpenTelemetry and scores each step with LLM judges such as the RAG triad of context relevance, groundedness, and answer relevance.

    • Open source · MIT
    • Library
    • Python
  • W&B Weave logo

    W&B Weave

    Weights & Biases (CoreWeave) · wandb.ai

    W&B Weave is an Apache-2.0 toolkit from Weights & Biases, now part of CoreWeave, for tracing agent sessions, running evaluations with built-in and custom scorers, applying guardrails, and comparing models and prompts in a playground.

    • Open source · Apache-2.0
    • Managed cloud
    • Library
    • Python
    • TypeScript

How to choose

01Tracing and standards
Support for OpenTelemetry and OpenInference conventions, nested spans for agents, and multi-turn sessions.
02Evaluation
Datasets, experiments, LLM-as-a-judge and code scorers, human annotation, and online evaluation of production traffic.
03Deployment
Self-hosted open source, managed SaaS, or bring-your-own-cloud, which matters when traces contain sensitive data.
04Integrations
SDKs and auto-instrumentation for the frameworks, gateways, and model providers you already use.
05Prompt management
Versioned prompts, playgrounds, and links between prompt versions and evaluation results.
06Cost and data controls
Token and cost tracking, sampling, retention settings, and PII redaction.

Frequently asked questions

What is LLM observability?

LLM observability is the ability to see what an LLM application did for each request: the prompts sent, the responses returned, the tools and retrievers called, and the latency, tokens, and cost of every step. Observability tools record this as traces and add evaluations, so teams can find failures, detect regressions, and compare changes with evidence.

Langfuse vs LangSmith: what is the difference?

Langfuse is an open-source platform, MIT-licensed outside its enterprise directories, that teams can self-host or use as a managed cloud, and ClickHouse acquired it in January 2026. LangSmith is LangChain's commercial platform, with managed, bring-your-own-cloud, and self-hosted options. Both trace any framework and support evaluations, and LangSmith integrates closely with LangChain and LangGraph.

Is Langfuse open source?

Yes. Langfuse's repository is MIT-licensed except for code in its enterprise directories, which carries a separate commercial license. Teams can self-host the core platform, including tracing, evaluations, prompt management, and datasets, or use Langfuse Cloud. ClickHouse acquired Langfuse in January 2026, and the license notice now names ClickHouse as the copyright holder while keeping the MIT terms.

DeepEval vs Ragas: what is the difference?

DeepEval and Ragas are both Apache-2.0 Python libraries for scoring LLM outputs. DeepEval is organized like a unit-testing framework, with pytest integration and metrics for agents, RAG, chatbots, and voice, and connects to the Confident AI platform. Ragas began with retrieval-augmented generation metrics and adds synthetic test set generation. Many teams use either one to score traces collected by a separate observability tool.

Are there open source LangSmith alternatives?

Yes. Langfuse, Opik, MLflow, Laminar, Agenta, and OpenLIT can be self-hosted under open-source licenses, and Arize Phoenix is self-hostable under the source-available Elastic License 2.0. Most accept OpenTelemetry traces, so instrumentation such as OpenLLMetry or OpenInference can send the same data to a new backend. LangSmith also offers a self-hosted edition, but it is commercial software.

Other categories

Product details change quickly; check each vendor’s site for current features, licensing and pricing.