---
title: "What is MMLU? The LLM knowledge benchmark explained"
description: "MMLU is a 57-subject multiple-choice benchmark of LLM knowledge. How it is scored, why it saturated, what errors it contains, and how MMLU-Pro differs."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/mmlu"
markdown_url: "https://theaiengineering.com/glossary/mmlu.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["MMLU","Massive Multitask Language Understanding","MMLU benchmark"]
citation: "The AI Engineering Editors. \"What is MMLU? The LLM knowledge benchmark explained.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/mmlu"
---

# What is MMLU? The LLM knowledge benchmark explained

Also called Massive Multitask Language Understanding, MMLU benchmark.

> **Definition of MMLU:** MMLU (Massive Multitask Language Understanding) is a benchmark of 15,908 four-option multiple-choice questions across 57 subjects, from elementary mathematics to law, that measures a language model's broad academic knowledge.

MMLU, short for Massive Multitask Language Understanding, is a multiple-choice benchmark that tests how much academic and professional knowledge a [large language model](https://theaiengineering.com/glossary/large-language-model) can apply. [Hendrycks et al.](https://arxiv.org/abs/2009.03300) released it in 2020, and it appeared at ICLR 2021, with 57 tasks in STEM, the humanities, the social sciences, and other fields, from elementary mathematics and US history to computer science and law. It became the standard broad-knowledge score in model announcements. In 2025, the [Humanity's Last Exam](https://arxiv.org/abs/2501.14249) authors reported that LLMs exceed 90% accuracy on MMLU, so it no longer separates frontier models.

## How MMLU works

MMLU works as a four-option multiple-choice exam. Each of its 15,908 questions has one correct letter, A through D, and a model's score is its accuracy on the 14,079-question test set. The original evaluation prompts the model zero-shot or with up to five solved examples per subject, then takes the answer letter the model rates most probable.

Graduate and undergraduate students collected the questions by hand from freely available sources, including practice questions for exams such as the GRE and the USMLE. The paper estimated expert-level accuracy at about 89.8%. Unspecialized Amazon Mechanical Turk workers scored 34.5%, and random guessing scores 25%. When MMLU was released, the largest GPT-3 model beat random chance by almost 20 percentage points on average and stayed near random on subjects including morality and law.

The prompt setup changes the number. The [few-shot](https://theaiengineering.com/glossary/few-shot-prompting) example count and the prompt template both move MMLU scores, so two published results are comparable only when the setup matches.

## Known problems with MMLU

MMLU has three known problems: saturation, label errors, and contamination. Frontier models score above 90%, near the paper's 89.8% estimate for experts, so the remaining gaps are small. A 2024 re-annotation estimated that 6.49% of the questions contain errors. The questions have also been public since 2020, so they can appear in training data.

The re-annotation study, [Are We Done with MMLU?](https://arxiv.org/abs/2406.04127) (Gema et al., 2024), built MMLU-Redux from 5,700 manually re-checked questions across all 57 subjects. It found errors in 57% of the analyzed Virology questions and reported significant differences between model results on the corrected questions and the originally reported scores. Public availability raises the risk of [benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination), which inflates scores through memorized answers.

## MMLU vs MMLU-Pro

MMLU-Pro is a harder successor that keeps the multiple-choice format but raises the answer options from four to ten, removes trivial and noisy questions, and adds reasoning-focused problems. Its authors report accuracy 16% to 33% lower than on MMLU, less sensitivity to prompt wording, and a benefit from chain-of-thought prompting that MMLU did not show.

| | MMLU | MMLU-Pro |
|---|---|---|
| Released | 2020 | 2024 |
| Questions | 15,908 | 12,032 |
| Coverage | 57 tasks | 14 disciplines |
| Options per question | 4 | 10 |
| Random-guess accuracy | 25% | 10% |
| Score variation across 24 prompt styles | 4% to 5% | About 2% |
| [Chain-of-thought](https://theaiengineering.com/glossary/chain-of-thought) benefit | Not shown | Shown |

[MMLU-Pro](https://arxiv.org/abs/2406.01574) (Wang et al., 2024, NeurIPS 2024 Datasets and Benchmarks track) draws its new questions from STEM websites, TheoremQA, and SciBench alongside filtered MMLU items. Treat any MMLU or MMLU-Pro score as one input to [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation), and confirm it on your own tasks. The [LLM benchmarks explainer](https://theaiengineering.com/articles/llm-benchmarks) compares MMLU with GPQA, Humanity's Last Exam, and coding benchmarks.

## Related terms

- [Benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination): Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [Chain-of-thought prompting](https://theaiengineering.com/glossary/chain-of-thought): Chain-of-thought prompting is a technique that has a language model write intermediate reasoning steps before its final answer, which improves large models' accuracy on arithmetic, commonsense, and symbolic reasoning tasks.
- [Few-shot prompting](https://theaiengineering.com/glossary/few-shot-prompting): Few-shot prompting is a technique that puts a few example inputs paired with desired outputs in a prompt, so a language model infers the task, format, and labels without changing its weights.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.

## Guides that cover mmlu

- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is MMLU? The LLM knowledge benchmark explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/mmlu

- Title: What is MMLU? The LLM knowledge benchmark explained
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/mmlu
- Markdown: https://theaiengineering.com/glossary/mmlu.md
