---
title: "What is SWE-bench? The coding agent benchmark explained"
description: "SWE-bench is a benchmark that tests whether a model can resolve real GitHub issues in Python repositories. How tasks are scored and what each variant changes."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/swe-bench"
markdown_url: "https://theaiengineering.com/glossary/swe-bench.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["SWE-bench","SWE-bench Verified","SWE-bench Lite","SWE bench"]
citation: "The AI Engineering Editors. \"What is SWE-bench? The coding agent benchmark explained.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/swe-bench"
---

# What is SWE-bench? The coding agent benchmark explained

Also called SWE-bench Verified, SWE-bench Lite, SWE bench.

> **Definition of SWE-bench:** SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests.

SWE-bench is a benchmark that measures whether a language model, usually running inside an [AI agent](https://theaiengineering.com/glossary/ai-agent) scaffold, can resolve real software issues. [Jimenez et al.](https://arxiv.org/abs/2310.06770) built it from 2,294 GitHub issues, and the pull requests that fixed them, across 12 popular Python repositories, and presented it at ICLR 2024. Each task gives the model a codebase and an issue description and asks for a patch. The paper reported that the best model it tested, Claude 2, resolved 1.96% of the issues.

## How a SWE-bench task is scored

A SWE-bench task counts as resolved when the model's patch applies to the repository and two sets of tests pass: fail-to-pass tests, which failed before the original fix and pass after it, and pass-to-pass tests, which passed before the fix and must still pass. The benchmark score is the percentage of tasks resolved.

**Figure 1: A SWE-bench task is resolved only when the patch applies and both test sets pass.**

A model reads an issue and a codebase and writes a patch; the evaluation code applies it in a container and runs fail-to-pass and pass-to-pass tests, and the task is resolved only if both pass.

Components:

- **Issue and codebase**: Repository at the issue's commit
- **Model or agent**: Edits files
- **Patch**: Applied in a container
- **Fail-to-pass tests**: Prove the issue is fixed
- **Pass-to-pass tests**: Prove nothing broke
- **Resolved**: Both sets pass

Connections:

- Issue and codebase → Model or agent
- Model or agent → Patch
- Patch → Fail-to-pass tests
- Patch → Pass-to-pass tests
- Fail-to-pass tests → Resolved
- Pass-to-pass tests → Resolved

The tests come from the original pull request, and the model never sees them. To build the dataset, the authors collected merged pull requests that resolved an issue and changed test files, then kept only instances where at least one test went from failing to passing and the environment installed and ran cleanly. That filter narrowed about 90,000 pull requests to 2,294 tasks. The reference fixes edit 32.8 lines across 1.7 files on average, and the paper reports a median of 51 pass-to-pass tests per task. The current evaluation code runs every task in Docker containers, according to [swebench.com](https://www.swebench.com/SWE-bench/).

## SWE-bench variants

SWE-bench has several variants that trade size for reliability or widen coverage. SWE-bench Lite is a 300-task subset of simpler, self-contained issues; SWE-bench Verified is a 500-task subset screened by human annotators; SWE-bench Multimodal uses issues that contain images; and SWE-bench Multilingual covers nine programming languages beyond Python.

| Variant | Tasks | What changes |
|---|---|---|
| SWE-bench | 2,294 | The original set from 12 Python repositories |
| Lite | 300 test, 23 development | Filtered to single-file fixes and self-contained issue text |
| Verified | 500 | Human-screened for well-specified issues and fair tests |
| Multimodal | 517 at release; version 2 keeps 480 | Issues include screenshots, mockups, or diagrams |
| Multilingual | 300 from 42 repositories | C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust |

OpenAI released [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) on August 13, 2024, after 93 software developers reviewed 1,699 random samples from the test set for underspecified issues and tests that reject valid solutions. In February 2026, OpenAI [stopped reporting SWE-bench Verified](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/), citing [contamination](https://theaiengineering.com/glossary/benchmark-contamination) and an audit in which 59.4% of 138 problems had material issues in test design or problem description, and recommended SWE-bench Pro from Scale AI.

A SWE-bench score measures a model together with its scaffold, tools, retry policy, and time budget, so compare results only under the same setup. Run the agent in an isolated [sandbox](https://theaiengineering.com/glossary/agent-sandbox) when reproducing a result. The [AI coding agents comparison](https://theaiengineering.com/articles/ai-coding-agents) shows how scaffolds differ and how to run a trial on your own repositories.

## Related terms

- [Benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination): Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures.
- [pass@k](https://theaiengineering.com/glossary/pass-at-k): pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [AI agent](https://theaiengineering.com/glossary/ai-agent): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached.
- [Agent sandbox](https://theaiengineering.com/glossary/agent-sandbox): An agent sandbox is an isolated execution environment in which an AI agent runs code, shell commands, or a browser, with enforced limits on the files, network hosts, and credentials it can reach.

## Guides that cover swe-bench

- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.
- [AI coding agents: how they work and how the major tools compare](https://theaiengineering.com/articles/ai-coding-agents): AI coding agents compared: how they differ from autocomplete, the four main categories, a comparison table of major tools, and how to evaluate and secure them.
- [Best LLM for coding: models compared on agentic coding benchmarks](https://theaiengineering.com/articles/best-llm-for-coding): The best LLM for coding depends on the job: 11 models compared on Terminal-Bench 4.0, DeepSWE, and SWE-bench Pro, plus context, open weights, and cost.
- [Claude Code vs Cursor: how the two coding agents differ](https://theaiengineering.com/articles/claude-code-vs-cursor): Claude Code vs Cursor compared: terminal agent vs AI editor, models, subagents, hooks, rules files, code review, CI use, pricing, and enterprise controls.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is SWE-bench? The coding agent benchmark explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/swe-bench

- Title: What is SWE-bench? The coding agent benchmark explained
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/swe-bench
- Markdown: https://theaiengineering.com/glossary/swe-bench.md
