What is MMLU? The LLM knowledge benchmark explained
Also called Massive Multitask Language Understanding, MMLU benchmark
Definition
MMLU (Massive Multitask Language Understanding) is a benchmark of 15,908 four-option multiple-choice questions across 57 subjects, from elementary mathematics to law, that measures a language model's broad academic knowledge.
MMLU, short for Massive Multitask Language Understanding, is a multiple-choice benchmark that tests how much academic and professional knowledge a large language model can apply. Hendrycks et al. released it in 2020, and it appeared at ICLR 2021, with 57 tasks in STEM, the humanities, the social sciences, and other fields, from elementary mathematics and US history to computer science and law. It became the standard broad-knowledge score in model announcements. In 2025, the Humanity's Last Exam authors reported that LLMs exceed 90% accuracy on MMLU, so it no longer separates frontier models.
How MMLU works#
MMLU works as a four-option multiple-choice exam. Each of its 15,908 questions has one correct letter, A through D, and a model's score is its accuracy on the 14,079-question test set. The original evaluation prompts the model zero-shot or with up to five solved examples per subject, then takes the answer letter the model rates most probable.
Graduate and undergraduate students collected the questions by hand from freely available sources, including practice questions for exams such as the GRE and the USMLE. The paper estimated expert-level accuracy at about 89.8%. Unspecialized Amazon Mechanical Turk workers scored 34.5%, and random guessing scores 25%. When MMLU was released, the largest GPT-3 model beat random chance by almost 20 percentage points on average and stayed near random on subjects including morality and law.
The prompt setup changes the number. The few-shot example count and the prompt template both move MMLU scores, so two published results are comparable only when the setup matches.
Known problems with MMLU#
MMLU has three known problems: saturation, label errors, and contamination. Frontier models score above 90%, near the paper's 89.8% estimate for experts, so the remaining gaps are small. A 2024 re-annotation estimated that 6.49% of the questions contain errors. The questions have also been public since 2020, so they can appear in training data.
The re-annotation study, Are We Done with MMLU? (Gema et al., 2024), built MMLU-Redux from 5,700 manually re-checked questions across all 57 subjects. It found errors in 57% of the analyzed Virology questions and reported significant differences between model results on the corrected questions and the originally reported scores. Public availability raises the risk of benchmark contamination, which inflates scores through memorized answers.
MMLU vs MMLU-Pro#
MMLU-Pro is a harder successor that keeps the multiple-choice format but raises the answer options from four to ten, removes trivial and noisy questions, and adds reasoning-focused problems. Its authors report accuracy 16% to 33% lower than on MMLU, less sensitivity to prompt wording, and a benefit from chain-of-thought prompting that MMLU did not show.
| MMLU | MMLU-Pro | |
|---|---|---|
| Released | 2020 | 2024 |
| Questions | 15,908 | 12,032 |
| Coverage | 57 tasks | 14 disciplines |
| Options per question | 4 | 10 |
| Random-guess accuracy | 25% | 10% |
| Score variation across 24 prompt styles | 4% to 5% | About 2% |
| Chain-of-thought benefit | Not shown | Shown |
MMLU-Pro (Wang et al., 2024, NeurIPS 2024 Datasets and Benchmarks track) draws its new questions from STEM websites, TheoremQA, and SciBench alongside filtered MMLU items. Treat any MMLU or MMLU-Pro score as one input to LLM evaluation, and confirm it on your own tasks. The LLM benchmarks explainer compares MMLU with GPQA, Humanity's Last Exam, and coding benchmarks.
Related terms
Guides that cover mmlu
Cite this definition
The AI Engineering Editors. "What is MMLU? The LLM knowledge benchmark explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/mmlu
- Title
- What is MMLU? The LLM knowledge benchmark explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated