What is pass@k? pass@k vs pass^k in LLM evaluation
Also called pass at k, pass@1, pass^k
Definition
pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks.
pass@k is a metric for tasks with an automatic pass or fail check, such as unit tests, that estimates the chance that at least one of k independent attempts succeeds. Kulal et al. (2019) used it for generating code from pseudocode, and Chen et al. (2021), the paper that introduced Codex and the HumanEval benchmark, defined an unbiased estimator for it. pass@1 is the expected success rate of a single attempt. Larger k describes what a system can reach when a person or a verifier picks the passing attempt.
How pass@k is calculated#
pass@k is calculated by generating n samples per problem, with n at least k, counting the c samples that pass, and computing the chance that a random subset of k samples contains at least one pass. The benchmark score averages that value over all problems.
The estimator from the Codex paper is pass@k = 1 - C(n-c, k) / C(n, k), where C is the binomial coefficient, averaged over problems. Chen et al. used n = 200 and k ≤ 100, and noted that the simpler estimate 1 - (1 - p)^k, built from the pass@1 rate p, is biased.
Sampling settings change the result. The paper found that higher temperatures are optimal for larger k: for a 679M-parameter model, the best temperature was 0.2 for pass@1 and 0.8 for pass@100. With one sample per problem, Codex solved 28.8% of HumanEval; with 100 samples per problem, it solved 70.2%. Drawing more samples is a form of test-time compute, and it pays off in production only when a test or verifier can tell which sample passed.
pass@k vs pass^k#
pass@k asks whether any of k attempts succeeds, so it rises as k grows. pass^k, proposed with the τ-bench agent benchmark in 2024, asks whether all k attempts succeed, so it falls as k grows. At k = 1 the two are equal. pass@k fits tasks with a verifier that can pick a winner; pass^k fits agents that must succeed every time.
| pass@k | pass^k | |
|---|---|---|
| Question | Does at least one of k trials pass? | Do all k trials pass? |
| As k grows | Rises | Falls |
| Estimator | 1 - C(n-c, k) / C(n, k) |
C(c, k) / C(n, k) |
| Fits | Code generation with tests, best-of-N with a verifier | Customer-facing agents, repeated workflows |
| Source | Kulal et al. 2019; estimator from Chen et al. 2021 | Yao et al. 2024 (τ-bench) |
The τ-bench paper defines pass^k, read "pass hat k," as the chance that all k i.i.d. trials of a task succeed, averaged across tasks. Its authors reported that function-calling agents built on gpt-4o succeeded on fewer than 50% of tasks and had a pass^8 below 25% in the retail domain.
The two metrics diverge fast. For one task with 10 trials and 5 passes, pass@1 and pass^1 are both 0.5. At k = 3, pass@3 is 0.917 and pass^3 is 0.083; at k = 5, pass@5 is 0.996 and pass^5 is 0.004.
Report k, n, and the sampling settings next to any pass@k number, because a pass@10 score is not comparable with a pass@1 score. SWE-bench reports the percentage of tasks resolved, so check how many attempts per task a reported result allowed. The AI agent evaluation guide includes Python implementations of both estimators and shows how to gate releases on pass^k.
Related terms
Guides that cover pass@k
Cite this definition
The AI Engineering Editors. "What is pass@k? pass@k vs pass^k in LLM evaluation." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/pass-at-k
- Title
- What is pass@k? pass@k vs pass^k in LLM evaluation
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated