What is LLM-as-a-judge? How it works, pros and cons
Also called LLM judge, LLM as a judge, model-based grader
Definition
LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
LLM-as-a-judge is the practice of using a capable model to grade qualities that code cannot check, such as whether an answer is helpful, faithful to its sources, or consistent with a style guide. The judge receives the input, the output to grade, the criteria, and sometimes a reference answer, and returns a score or verdict, often with a written rationale. The name was popularized by Zheng et al. in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023), and judges are now a standard grader type in LLM evaluation.
How LLM-as-a-judge works#
Three formats cover most uses:
- Single-output grading: score one response against a rubric, for example pass or fail on "cites only the provided sources," a common check for hallucination.
- Pairwise comparison: show two responses and ask which is better, which is useful for comparing prompts or models.
- Reference-guided grading: compare the response with a known good answer.
Zheng et al. found that GPT-4 as a judge agreed with human preferences more than 80% of the time, the same level of agreement humans reach with each other. The paper also documented biases that still shape judge design:
| Bias | What happens | Common mitigation |
|---|---|---|
| Position | Scores a response higher because of where it appears | Swap the order and grade twice |
| Verbosity | Scores longer answers higher | Rubrics that reward concision |
| Self-enhancement | Scores outputs resembling its own higher | Use a judge from a different model family |
| Limited reasoning | Misgrades math or logic | Provide reference answers |
OpenAI's evaluation guidance recommends pairwise or pass/fail formats over open-ended scales, and validating every judge against human labels.
LLM-as-a-judge vs human evaluation#
Human review remains the reference standard, but it is slow and expensive. An LLM judge costs less and can run on every commit or on sampled production traffic through LLM observability pipelines. The practical pattern is to label a set of outputs by hand, measure how often the judge agrees with those labels, refine the judge prompt until agreement is acceptable, and recheck agreement whenever the judge model changes. A judge that has never been compared with human labels is an unmeasured metric.
Judges also appear in training. RLAIF, a variant of RLHF, uses model-generated preference labels in place of human ones. For how judges combine with code-based and human graders, see the LLM evaluation guide.
Related terms
Guides that cover llm-as-a-judge
- LLM evaluation: a practical guide to evals, metrics, and judges
- AI agent evaluation: metrics, graders, and how to test agents
- LLM benchmarks explained: what they measure and how to read them
- Agentic workflows: five patterns and when to use each
- Semantic caching for LLMs: how it works and when it pays off
- How to build an AI agent: a step-by-step guide in Python
- AI coding agents: how they work and how the major tools compare
- AI agent frameworks compared: LangGraph, CrewAI, ADK, and more
- LLM observability: what to capture, which metrics, and tools
- AI engineer roadmap: skills, projects, and resources by stage
- What is retrieval-augmented generation (RAG)? How RAG works
- What is AI engineering? The discipline, the stack, and the work
Cite this definition
The AI Engineering Editors. "What is LLM-as-a-judge? How it works, pros and cons." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-as-a-judge
- Title
- What is LLM-as-a-judge? How it works, pros and cons
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated