---
title: "What is pass@k? pass@k vs pass^k in LLM evaluation"
description: "pass@k is the probability that at least one of k sampled LLM outputs passes a task's tests. How the unbiased estimator works and how pass@k differs from pass^k."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/pass-at-k"
markdown_url: "https://theaiengineering.com/glossary/pass-at-k.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["pass@k","pass at k","pass@1","pass^k"]
citation: "The AI Engineering Editors. \"What is pass@k? pass@k vs pass^k in LLM evaluation.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/pass-at-k"
---

# What is pass@k? pass@k vs pass^k in LLM evaluation

Also called pass at k, pass@1, pass^k.

> **Definition of pass@k:** pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks.

pass@k is a metric for tasks with an automatic pass or fail check, such as unit tests, that estimates the chance that at least one of k independent attempts succeeds. Kulal et al. (2019) used it for generating code from pseudocode, and [Chen et al. (2021)](https://arxiv.org/abs/2107.03374), the paper that introduced Codex and the HumanEval benchmark, defined an unbiased estimator for it. pass@1 is the expected success rate of a single attempt. Larger k describes what a system can reach when a person or a verifier picks the passing attempt.

## How pass@k is calculated

pass@k is calculated by generating n samples per problem, with n at least k, counting the c samples that pass, and computing the chance that a random subset of k samples contains at least one pass. The benchmark score averages that value over all problems.

The estimator from the Codex paper is `pass@k = 1 - C(n-c, k) / C(n, k)`, where `C` is the binomial coefficient, averaged over problems. Chen et al. used n = 200 and k ≤ 100, and noted that the simpler estimate `1 - (1 - p)^k`, built from the pass@1 rate p, is biased.

Sampling settings change the result. The paper found that higher [temperatures](https://theaiengineering.com/glossary/temperature) are optimal for larger k: for a 679M-parameter model, the best temperature was 0.2 for pass@1 and 0.8 for pass@100. With one sample per problem, Codex solved 28.8% of HumanEval; with 100 samples per problem, it solved 70.2%. Drawing more samples is a form of [test-time compute](https://theaiengineering.com/glossary/test-time-compute), and it pays off in production only when a test or verifier can tell which sample passed.

## pass@k vs pass^k

pass@k asks whether any of k attempts succeeds, so it rises as k grows. pass^k, proposed with the τ-bench agent benchmark in 2024, asks whether all k attempts succeed, so it falls as k grows. At k = 1 the two are equal. pass@k fits tasks with a verifier that can pick a winner; pass^k fits agents that must succeed every time.

| | pass@k | pass^k |
|---|---|---|
| Question | Does at least one of k trials pass? | Do all k trials pass? |
| As k grows | Rises | Falls |
| Estimator | `1 - C(n-c, k) / C(n, k)` | `C(c, k) / C(n, k)` |
| Fits | Code generation with tests, best-of-N with a verifier | Customer-facing [agents](https://theaiengineering.com/glossary/ai-agent), repeated workflows |
| Source | Kulal et al. 2019; estimator from Chen et al. 2021 | Yao et al. 2024 (τ-bench) |

The [τ-bench paper](https://arxiv.org/abs/2406.12045) defines pass^k, read "pass hat k," as the chance that all k i.i.d. trials of a task succeed, averaged across tasks. Its authors reported that function-calling agents built on gpt-4o succeeded on fewer than 50% of tasks and had a pass^8 below 25% in the retail domain.

The two metrics diverge fast. For one task with 10 trials and 5 passes, pass@1 and pass^1 are both 0.5. At k = 3, pass@3 is 0.917 and pass^3 is 0.083; at k = 5, pass@5 is 0.996 and pass^5 is 0.004.

Report k, n, and the sampling settings next to any pass@k number, because a pass@10 score is not comparable with a pass@1 score. [SWE-bench](https://theaiengineering.com/glossary/swe-bench) reports the percentage of tasks resolved, so check how many attempts per task a reported result allowed. The [AI agent evaluation guide](https://theaiengineering.com/articles/ai-agent-evaluation) includes Python implementations of both estimators and shows how to gate releases on pass^k.

## Related terms

- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [SWE-bench](https://theaiengineering.com/glossary/swe-bench): SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests.
- [Temperature](https://theaiengineering.com/glossary/temperature): Temperature is a sampling parameter that rescales a language model's next-token probabilities: low values concentrate probability on the likeliest tokens, and high values flatten the distribution for more varied output.
- [Test-time compute](https://theaiengineering.com/glossary/test-time-compute): Test-time compute is the computation a language model spends while answering a query, through longer reasoning, multiple samples, or search, as opposed to the train-time compute spent building the model.
- [AI agent](https://theaiengineering.com/glossary/ai-agent): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached.

## Guides that cover pass@k

- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [Best LLM for coding: models compared on agentic coding benchmarks](https://theaiengineering.com/articles/best-llm-for-coding): The best LLM for coding depends on the job: 11 models compared on Terminal-Bench 4.0, DeepSWE, and SWE-bench Pro, plus context, open weights, and cost.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is pass@k? pass@k vs pass^k in LLM evaluation." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/pass-at-k

- Title: What is pass@k? pass@k vs pass^k in LLM evaluation
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/pass-at-k
- Markdown: https://theaiengineering.com/glossary/pass-at-k.md
