---
title: "What is benchmark contamination in LLM evaluation?"
description: "Benchmark contamination happens when test questions or answers leak into an LLM's training data and inflate scores. How it is detected and how teams limit it."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/benchmark-contamination"
markdown_url: "https://theaiengineering.com/glossary/benchmark-contamination.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Benchmark contamination","data contamination","test set contamination","benchmark leakage","training data contamination"]
citation: "The AI Engineering Editors. \"What is benchmark contamination in LLM evaluation?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/benchmark-contamination"
---

# What is benchmark contamination in LLM evaluation?

Also called data contamination, test set contamination, benchmark leakage, training data contamination.

> **Definition of Benchmark contamination:** Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures.

Benchmark contamination occurs when a benchmark's test items, or their solutions, appear in the data a model was trained on. Public benchmarks are posted on the web and [pretraining](https://theaiengineering.com/glossary/pretraining) corpora are scraped from the web, so leakage happens unless someone filters for it. A contaminated model can score well by recall, which makes the benchmark overstate how the model will perform on inputs it has never seen. For an engineer choosing a model, contamination is one reason a leaderboard gap can vanish on production traffic.

## How benchmark contamination is detected

Benchmark contamination is detected in three ways: searching the training corpus for text that overlaps the test set, probing the model for signs that it memorized test items, and comparing scores on the original benchmark with scores on fresh questions of matching style and difficulty. Only the model developer can search the corpus; outside researchers rely on probes and fresh sets.

- **Overlap search.** A developer with the training data looks for long n-gram matches between the corpus and each test item, then reports scores on a clean subset with the matches removed.
- **Memorization probes.** [Oren et al. (2023)](https://arxiv.org/abs/2310.17623) test whether a model assigns higher likelihood to a benchmark's original example order than to shuffled orders, which needs neither the training data nor the weights. The authors report that the test detects contamination in models as small as 1.4 billion parameters and test sets as small as 1,000 examples, and their audit of five popular models found little evidence of pervasive contamination.
- **Fresh replicas.** [GSM1k](https://arxiv.org/abs/2405.00332) (Zhang et al., 2024) is a new set of grade-school math problems written to mirror GSM8k. Some models scored up to 8% lower on GSM1k, several model families showed systematic overfitting across almost all model sizes, and a model's probability of generating GSM8k examples correlated with its score gap (Spearman's r² = 0.36). Many frontier models showed minimal signs of overfitting.
- **Verbatim recall.** OpenAI found that every frontier model it tested could reproduce the original human-written fix, or verbatim details of the problem statement, for some [SWE-bench](https://theaiengineering.com/glossary/swe-bench) Verified tasks. That finding was one of two reasons it gave for [no longer reporting the benchmark](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) in February 2026.

## How teams limit benchmark contamination

Benchmark designers limit contamination by keeping part of the test set private, by adding new questions after model training cutoffs, and by writing fresh replacements for leaked benchmarks. Teams choosing a model add a fourth defense: a private [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation) set built from their own traffic, which no public training corpus contains.

| Defense | How it works | Example |
|---|---|---|
| Private held-out set | Part of the test set is never published | SWE-bench Pro's held-out and commercial sets |
| Time-based questions | Questions are added regularly from recently published sources | LiveBench |
| Fresh replacement | A new set matches an old benchmark's style | GSM1k for GSM8k |
| Your own eval set | Cases come from your logs and stay private | A [golden dataset](https://theaiengineering.com/glossary/golden-dataset) |

[LiveBench](https://arxiv.org/abs/2406.19314) (White et al., 2024) adds and updates questions monthly, drawing on recent math competitions, arXiv papers, news articles, and datasets, and scores answers automatically against objective ground truth instead of using an LLM judge. A time-based score is clean only for questions that postdate the model's training cutoff, so check the cutoff before comparing models.

Contamination differs from saturation. Contamination inflates a score through exposure to the answers; saturation means models cluster near the maximum score, as on [MMLU](https://theaiengineering.com/glossary/mmlu), so the remaining differences are small or reflect label errors. Both shorten a public benchmark's useful life. The [LLM benchmarks explainer](https://theaiengineering.com/articles/llm-benchmarks) covers contamination, saturation, and how to read a leaderboard.

## Related terms

- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [SWE-bench](https://theaiengineering.com/glossary/swe-bench): SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests.
- [MMLU](https://theaiengineering.com/glossary/mmlu): MMLU (Massive Multitask Language Understanding) is a benchmark of 15,908 four-option multiple-choice questions across 57 subjects, from elementary mathematics to law, that measures a language model's broad academic knowledge.
- [Pretraining](https://theaiengineering.com/glossary/pretraining): Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.

## Guides that cover benchmark contamination

- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [AI coding agents: how they work and how the major tools compare](https://theaiengineering.com/articles/ai-coding-agents): AI coding agents compared: how they differ from autocomplete, the four main categories, a comparison table of major tools, and how to evaluate and secure them.
- [Best LLM for coding: models compared on agentic coding benchmarks](https://theaiengineering.com/articles/best-llm-for-coding): The best LLM for coding depends on the job: 11 models compared on Terminal-Bench 4.0, DeepSWE, and SWE-bench Pro, plus context, open weights, and cost.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is benchmark contamination in LLM evaluation?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/benchmark-contamination

- Title: What is benchmark contamination in LLM evaluation?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/benchmark-contamination
- Markdown: https://theaiengineering.com/glossary/benchmark-contamination.md
