What is LMArena (Chatbot Arena)? How it ranks LLMs
Also called Chatbot Arena, LMSYS Chatbot Arena
Definition
LMArena, originally Chatbot Arena and branded Arena since January 2026, is a public leaderboard that ranks language models from crowdsourced human votes between pairs of anonymous model responses.
LMArena is a crowdsourced leaderboard that ranks large language models by human preference. A user enters a prompt, two anonymous models answer, the user votes for the better response, and the model identities are revealed only after the vote. A statistical model turns the accumulated votes into a rating and a confidence interval for each model. The prompts come from real users rather than a fixed test set, so the leaderboard measures which answers people prefer on open-ended chat, including the effect of style and length, not correctness on a defined task.
The project began at UC Berkeley in 2023 as LMSYS Chatbot Arena. It moved to its own domain, lmarena.ai, in September 2024 and became known as LMArena, and in April 2025 the team started a company to support the platform. On January 28, 2026, LMArena rebranded as Arena at arena.ai, saying it had outgrown the "LM" in its name as it grew from a research project into a broader platform.
How LMArena ranks models#
LMArena fits a Bradley-Terry model to the pairwise votes. Each model gets a strength coefficient, and the gap between two coefficients predicts how often one model beats the other. A model's rank is one plus the number of models whose confidence interval sits entirely above its own, so close models can share a rank.
Chiang et al. describe the method in Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024), written when the platform had collected over 240,000 votes. The authors report that crowdsourced votes agree well with expert raters, and they choose which model pairs to show with an active sampling rule that favors pairs whose comparison most shrinks the confidence intervals. The leaderboard first used online Elo ratings and switched to Bradley-Terry in December 2023. Bradley-Terry assumes each model's strength is fixed, so the order in which votes arrive no longer moves the ratings.
Style control, added in August 2024, puts answer length and counts of markdown headers, bold text, and list items into the same regression as separate covariates. A model's style-controlled rating then reflects preference after those effects are accounted for; the authors found length to be the strongest style factor.
LMArena vs static benchmarks#
Static benchmarks such as MMLU or SWE-bench score answers against fixed references, so they measure correctness on a defined task. LMArena measures preference on prompts users choose, which captures helpfulness, tone, and formatting but not whether an answer is right. The Chatbot Arena paper notes that static test sets can suffer benchmark contamination, while crowdsourcing gathers fresh prompts.
Arena rankings have documented weaknesses. The Leaderboard Illusion (Singh et al., 2025) reported that undisclosed private testing let a few providers try many variants and publish only the best score, citing 27 private variants Meta tested before the Llama 4 release. The authors estimated that Google and OpenAI received 19.2% and 20.4% of arena data, while 83 open-weight models together received about 29.7%, and argued that these dynamics reward overfitting to the arena rather than general model quality.
Arena votes are also training and validation data. Zheng et al. used Chatbot Arena preferences to measure how often an LLM-as-a-judge agrees with humans, and RouteLLM trained its LLM routers on 80,000 Chatbot Arena battles.
Treat an Arena rating as evidence of user preference on general chat, and confirm a shortlist with your own LLM evaluation suite. The LLM benchmarks explainer compares LMArena with correctness benchmarks.
Related terms
Guides that cover lmarena
Cite this definition
The AI Engineering Editors. "What is LMArena (Chatbot Arena)? How it ranks LLMs." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/lmarena
- Title
- What is LMArena (Chatbot Arena)? How it ranks LLMs
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated