What is LLM-as-a-judge? How it works, pros and cons

Also called LLM judge, LLM as a judge, model-based grader

Definition

LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.

LLM-as-a-judge is the practice of using a capable model to grade qualities that code cannot check, such as whether an answer is helpful, faithful to its sources, or consistent with a style guide. The judge receives the input, the output to grade, the criteria, and sometimes a reference answer, and returns a score or verdict, often with a written rationale. The name was popularized by Zheng et al. in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023), and judges are now a standard grader type in LLM evaluation.

How LLM-as-a-judge works#

Three formats cover most uses:

  • Single-output grading: score one response against a rubric, for example pass or fail on "cites only the provided sources," a common check for hallucination.
  • Pairwise comparison: show two responses and ask which is better, which is useful for comparing prompts or models.
  • Reference-guided grading: compare the response with a known good answer.

Zheng et al. found that GPT-4 as a judge agreed with human preferences more than 80% of the time, the same level of agreement humans reach with each other. The paper also documented biases that still shape judge design:

Bias What happens Common mitigation
Position Scores a response higher because of where it appears Swap the order and grade twice
Verbosity Scores longer answers higher Rubrics that reward concision
Self-enhancement Scores outputs resembling its own higher Use a judge from a different model family
Limited reasoning Misgrades math or logic Provide reference answers

OpenAI's evaluation guidance recommends pairwise or pass/fail formats over open-ended scales, and validating every judge against human labels.

LLM-as-a-judge vs human evaluation#

Human review remains the reference standard, but it is slow and expensive. An LLM judge costs less and can run on every commit or on sampled production traffic through LLM observability pipelines. The practical pattern is to label a set of outputs by hand, measure how often the judge agrees with those labels, refine the judge prompt until agreement is acceptable, and recheck agreement whenever the judge model changes. A judge that has never been compared with human labels is an unmeasured metric.

Judges also appear in training. RLAIF, a variant of RLHF, uses model-generated preference labels in place of human ones. For how judges combine with code-based and human graders, see the LLM evaluation guide.

Cite this definition

The AI Engineering Editors. "What is LLM-as-a-judge? How it works, pros and cons." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-as-a-judge

Title
What is LLM-as-a-judge? How it works, pros and cons
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM evaluation and observability hub · Updated