LLM benchmarks explained: what they measure and how to read them

Explainer · Models

LLM benchmarks are standardized test sets, each with a fixed scoring rule, used to compare how large language models perform on tasks such as knowledge recall, scientific reasoning, coding, and human preference. Benchmark scores headline model announcements and leaderboards, but a score means little without three facts: what the test measures, how it was run, and whether the model could have seen the answers in training. This explainer covers all three, part of the models topic hub.

What are LLM benchmarks?#

An LLM benchmark is a fixed dataset of tasks plus a scoring method, such as exact-match accuracy, unit-test pass rate, or preference votes, applied the same way to every model. A reported score depends on the dataset, the evaluation setup that formats prompts and samples outputs, and the scorer, so changing any of the three changes the number.

A dataset of tasks and answers passes through an evaluation setup to the model, and a scorer turns the model's outputs into the reported score. Datasettasks and answersEval setupprompt template, few-shoteffort, samplingagent scaffold, retriesModelScorermatch, tests, votesScore
Figure 1: A benchmark score is produced by a dataset, an evaluation setup, and a scorer, not by the model alone.

The evaluation setup causes most disagreements between published numbers. Common sources of variation:

  • Prompt format: few-shot examples, chain-of-thought instructions, and answer extraction rules.
  • Sampling: temperature, and whether the score is the first attempt (pass@1) or the best of several (pass@k).
  • Reasoning effort: the same reasoning model can score differently at low and high effort settings.
  • Agent scaffold: for agentic benchmarks, the tools, turn limits, and time budget given to the model.

Knowledge and reasoning benchmarks#

Knowledge and reasoning benchmarks test whether a model can answer questions across academic subjects and expert domains, usually in multiple-choice format scored by exact match. MMLU defined the category, MMLU-Pro and GPQA raised the difficulty, and Humanity's Last Exam was built because earlier tests stopped separating frontier models.

MMLU (Hendrycks et al., 2020) covers 57 tasks, including elementary mathematics, US history, computer science, and law, with four answer options per question. The Humanity's Last Exam paper reports that frontier models now exceed 90% accuracy on MMLU.

MMLU-Pro (Wang et al., 2024) expands each question to ten answer options, adds reasoning-focused questions, and removes trivial and noisy items. The authors report accuracy 16% to 33% lower than on MMLU, lower sensitivity to prompt wording (about 2% variation across 24 prompt styles, versus 4% to 5% on MMLU), and a benefit from chain-of-thought prompting that MMLU did not show.

GPQA (Rein et al., 2023) contains 448 multiple-choice questions in biology, physics, and chemistry written by domain experts. Experts with or pursuing PhDs in the field reached 65% accuracy, while skilled non-experts with unrestricted web access and more than 30 minutes per question reached 34%. Leaderboards usually report GPQA Diamond, a 198-question subset that expert validators answered correctly and most non-experts did not.

Humanity's Last Exam (Phan et al., 2025), organized by the Center for AI Safety and Scale AI, contains 2,500 expert-written questions across dozens of subjects, in multiple-choice and short-answer formats suited to automated grading. The benchmark is multimodal, each question has a verifiable answer that cannot be found quickly online, and the organizers keep a private held-out set to check for overfitting.

LLM benchmarks for coding#

Coding benchmarks score models by executing generated code against tests, which makes grading objective. They range from single-function puzzles (HumanEval) to competitive programming problems collected over time (LiveCodeBench) to multi-file repository changes driven by real GitHub issues (SWE-bench), and the gap between those levels is wide.

HumanEval (Chen et al., 2021) contains 164 hand-written Python problems, each with a function signature, a docstring, and unit tests. The HumanEval paper reports results as pass@k, the probability that at least one of k sampled solutions passes all tests, and defines an unbiased estimator for it. The original Codex model solved 28.8% of problems on the first attempt; current models score far higher, so HumanEval no longer separates frontier models.

LiveCodeBench (Jain et al., 2024) continuously collects new problems from LeetCode, AtCoder, and Codeforces contests and records each problem's release date. Evaluators can therefore score a model only on problems published after its training cutoff. The benchmark also tests self-repair, code execution, and test output prediction.

SWE-bench (Jimenez et al., ICLR 2024) contains 2,294 tasks built from real GitHub issues and pull requests in 12 Python repositories. A model receives the repository and the issue, edits the code, and passes only if the repository's tests pass. OpenAI released SWE-bench Verified in August 2024, a 500-task subset that human annotators screened for clear issue descriptions and fair tests.

In February 2026, OpenAI published why it no longer evaluates SWE-bench Verified, citing tests that reject functionally correct solutions and evidence that frontier models had seen benchmark solutions during training. OpenAI recommended SWE-bench Pro, a Scale AI benchmark with 1,865 long-horizon tasks from 41 repositories, split into a public set, a held-out set, and a commercial set built from proprietary repositories.

A SWE-bench score measures a model together with its agent scaffold: the tools, retry policy, and time budget matter as much as the model. The AI coding agents comparison covers how those scaffolds differ in practice.

Major LLM benchmarks at a glance#

The table below summarizes what each major benchmark measures, how it is scored, and the most common pitfall when reading its results. It deliberately omits current scores, which change with every model release; the model release tracker records new models as they ship.

Benchmark Measures Format Scoring Watch for
MMLU Broad academic knowledge 4-option multiple choice, 57 tasks Accuracy Saturated above 90% at the frontier
MMLU-Pro Knowledge plus reasoning 10-option multiple choice Accuracy Chain-of-thought settings change results
GPQA Diamond Graduate-level science 198 multiple-choice questions Accuracy Small set; a few points can be noise
Humanity's Last Exam Expert knowledge at the frontier 2,500 multimodal questions Accuracy, calibration error Low calibration: confident wrong answers are common
HumanEval Single-function Python 164 problems with unit tests pass@k Near ceiling at the frontier; public since 2021
LiveCodeBench Competitive programming Problems dated by contest release pass@1 Compare only on problems after a model's cutoff
SWE-bench Verified Repository-level bug fixing 500 real GitHub issues Percent resolved Flawed tests and contamination per OpenAI
SWE-bench Pro Long-horizon software tasks 1,865 tasks, public and private sets Percent resolved Results depend on the agent scaffold
ARC-AGI-2 Novel abstract reasoning Grid puzzles from a few examples Percent of tasks solved Read scores together with cost per task
ARC-AGI-3 Interactive, agentic reasoning Turn-based game environments Environments completed Results depend on the agent scaffold
LMArena Human preference Pairwise anonymous battles Bradley-Terry rating Preference is not correctness; check style control

On GPQA Diamond, one question is worth about half a percentage point, and the standard error of an accuracy near 80% on 198 questions is roughly 2.8 points, so small gaps between models are within noise.

What is ARC-AGI?#

ARC-AGI is a benchmark series created by François Chollet that measures skill-acquisition efficiency: how quickly a system learns to solve a task it has never seen. Tasks rely only on core knowledge priors that humans acquire early, such as objects and counting, and are designed to be easy for people and hard for AI systems.

Chollet introduced the original benchmark in his 2019 paper "On the Measure of Intelligence." ARC-AGI-1 and ARC-AGI-2 present a few demonstration pairs, typically 2 to 5, each showing an input grid transformed into an output grid by an unstated rule; grids are at most 30 by 30 cells with up to 10 colors. The solver must infer the rule and apply it to a new input.

ARC-AGI-3, launched by ARC Prize on March 25, 2026, is the first interactive version. It contains hundreds of turn-based game environments with no instructions or stated goals, so an agent must explore, infer the rules, identify what winning means, and carry what it learns into harder levels. ARC Prize reported that humans solve the environments while frontier AI systems scored a fraction of a percent at launch.

What is LMArena?#

LMArena, formerly Chatbot Arena and rebranded as Arena in January 2026, ranks language models by crowdsourced human preference. A user submits a prompt, two anonymous models respond, the user votes for the better answer, and the model identities are revealed. The votes feed a statistical model that produces a rating and confidence interval for each model.

The project began at UC Berkeley in April 2023, became an independent company in April 2025, and moved to arena.ai with its rebrand. In December 2023, the team replaced online Elo with a Bradley-Terry model, which assumes fixed model strength and gives more stable ratings and tighter confidence intervals. Style control, introduced in August 2024, adjusts ratings for answer length and markdown formatting, which voters otherwise tend to reward.

Arena-style rankings have known weaknesses. The Leaderboard Illusion (Singh et al., 2025) reported that undisclosed private testing let some providers evaluate many model variants and publish only the best score, citing 27 private variants Meta tested before the Llama 4 release, and that proprietary models received a larger share of battles than open-weight models. The Llama 4 episode itself, in which the version tested on the leaderboard differed from the public release, led LMArena to update its policies. Treat an Arena rating as evidence of user preference on general chat prompts, not as a measure of accuracy on your task.

Contamination and saturation#

Contamination means benchmark questions or solutions leaked into a model's training data, so the score reflects recall instead of capability. Saturation means top models cluster near the maximum score, so remaining differences increasingly reflect noise rather than ability. Both problems shorten the useful life of every public benchmark, and both are why new benchmarks keep appearing.

Contamination is hard to rule out because benchmarks are published on the web and pretraining corpora are scraped from the web. One clear demonstration is GSM1k (Zhang et al., 2024), a new set of grade-school math problems written to mirror the GSM8k benchmark. Some models scored up to 8% lower on GSM1k than on GSM8k, several model families showed signs of systematic overfitting, and the size of the gap correlated with how likely each model was to generate GSM8k examples. Many frontier models showed minimal overfitting.

Benchmark designers counter contamination in three ways:

  • Private held-out sets: SWE-bench Pro's held-out and commercial sets, Humanity's Last Exam's private set, and ARC-AGI's semi-private and private evaluation sets.
  • Time-based splits: LiveCodeBench lets evaluators score only problems released after a model's training cutoff.
  • Fresh replacements: new sets written to match an old benchmark's style, such as GSM1k for GSM8k.

Saturation has a familiar pattern: MMLU passed 90% for frontier models, MMLU-Pro and Humanity's Last Exam followed, and HumanEval gave way to LiveCodeBench and SWE-bench. When a benchmark saturates, its remaining errors can include ambiguous or mislabeled questions, and further gains stop meaning much.

How to read an LLM leaderboard#

Read an LLM leaderboard as a list of experiments, not a ranking of models. Before comparing two rows, check that both scores came from the same evaluation setup, sampling settings, reasoning effort, and agent scaffold, that the test set is unlikely to sit in either model's training data, and that the gap exceeds the benchmark's noise.

AI model comparison charts often compress these details into one number, so look for the following before trusting a gap:

  1. Who ran it. Self-reported scores use the developer's own setup; independent evaluations use a shared one, so the two are often not comparable.
  2. Attempts and selection. pass@1, pass@k, majority voting, and best-of-N selection can produce very different scores for the same model.
  3. Effort and budget. Reasoning model results depend on the stated effort level or token budget, and higher settings cost more per task. The reasoning models explainer covers that trade-off.
  4. Sample size. Small benchmarks need confidence intervals; a two-point gap on 198 questions is not a meaningful difference.
  5. Cost and latency. A slightly lower score at a fraction of the cost may be the better production choice.
  6. Relevance. A coding leaderboard says little about extraction, summarization, or support chat quality.

How to build your own eval set#

A task-specific eval set is a collection of real or realistic inputs from your application, each paired with a reference answer or grading rubric, scored automatically on every change. It measures what public benchmarks cannot: performance on your prompts, documents, output formats, and failure modes, at the effort and cost settings you plan to ship.

The process, covered in depth in the LLM evaluation guide, follows six steps:

  1. Sample real inputs. Draw from production logs, stratified by task type and difficulty, including known failures. Remove personal data first.
  2. Write references or rubrics. Use exact expected outputs where possible; for open-ended answers, write rubrics and use LLM-as-a-judge scoring calibrated against human labels.
  3. Prefer deterministic graders. Exact match, JSON schema validation, and unit tests are cheaper and more stable than judge models.
  4. Hold out a test split. Tune prompts on a development split and report results only on the frozen test split.
  5. Record cost and latency. Log tokens, latency, and price per case alongside quality.
  6. Re-run on every change. Model upgrades, prompt edits, and quantization all warrant a full run, ideally in CI.

A single eval case can be as simple as this:

{
  "id": "refund-042",
  "input": "I was charged twice for order A1234, please fix this.",
  "expected": {"intent": "refund_request", "order_id": "A1234"},
  "grader": "exact_match_fields",
  "tags": ["refunds", "hard"]
}

Track hallucination and format failures as separate tags, because they usually need different fixes. Once a model is in production, feed sampled traces from your LLM observability pipeline back into the eval set so it keeps pace with real traffic.

Public LLM benchmarks are a filter for building a shortlist, and your own eval set makes the final call. Next, read the LLM evaluation guide to set up scoring, the reasoning models explainer for effort settings, and the release tracker for new models worth testing.

Frequently asked questions

What are LLM benchmarks?

LLM benchmarks are standardized datasets of tasks, each paired with a scoring method, used to compare language models under the same conditions. Examples include multiple-choice knowledge tests such as MMLU-Pro, expert science questions such as GPQA, coding tests such as SWE-bench and LiveCodeBench, abstract reasoning puzzles such as ARC-AGI, and human preference rankings such as LMArena. Each benchmark measures a narrow capability, so no single score describes overall model quality.

What is the MMLU benchmark?

MMLU is a multiple-choice benchmark introduced by Hendrycks and colleagues in 2020 that covers 57 tasks, including elementary mathematics, US history, computer science, and law, with four answer options per question. MMLU became the standard broad-knowledge test for language models. Frontier models now score above 90%, according to the Humanity's Last Exam paper, so MMLU-Pro, with ten options and harder reasoning questions, has largely replaced it for comparing top models.

What is SWE-bench Verified?

SWE-bench Verified is a 500-task subset of the SWE-bench coding benchmark, released by OpenAI in August 2024 after human annotators screened tasks for clear issue descriptions and fair tests. Each task gives a model a real GitHub issue and repository and checks whether its patch passes the tests. In February 2026, OpenAI stopped reporting SWE-bench Verified, citing flawed tests and contamination, and recommended SWE-bench Pro.

What is GPQA Diamond?

GPQA Diamond is the highest-quality subset of GPQA, a benchmark of graduate-level multiple-choice questions in biology, physics, and chemistry written by domain experts. The full GPQA set has 448 questions; Diamond keeps 198 questions that expert validators answered correctly and most non-expert validators did not. Questions are designed to be Google-proof: skilled non-experts with unrestricted web access reached only 34% accuracy on the full set.

What is benchmark contamination?

Benchmark contamination occurs when test questions, answers, or solutions appear in a model's training data, so a high score can reflect memorization rather than capability. Contamination is common because benchmarks are published online and pretraining corpora are scraped from the web. Evidence includes accuracy drops on freshly written test sets that mirror older benchmarks. Mitigations include private held-out sets and benchmarks that add new problems after model training cutoffs.

What is LMArena used for?

LMArena, now branded Arena, is used to compare language models by human preference. Users submit a prompt, read anonymous responses from two models, and vote for the better one; votes feed a Bradley-Terry statistical model that produces ratings with confidence intervals. The leaderboard captures which answers people prefer, including style and formatting effects, so it complements correctness benchmarks rather than replacing them.

Terms in this guide

Cite this guide

The AI Engineering Editors. "LLM benchmarks explained: what they measure and how to read them." The AI Engineering, updated Sep 15, 2026. https://theaiengineering.com/articles/llm-benchmarks

Title
LLM benchmarks explained: what they measure and how to read them
Author
The AI Engineering Editors
Publisher
The AI Engineering
Published
Updated
View as Markdown

The AI Engineering Editors ·