What is SWE-bench? The coding agent benchmark explained
Also called SWE-bench Verified, SWE-bench Lite, SWE bench
Definition
SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests.
SWE-bench is a benchmark that measures whether a language model, usually running inside an AI agent scaffold, can resolve real software issues. Jimenez et al. built it from 2,294 GitHub issues, and the pull requests that fixed them, across 12 popular Python repositories, and presented it at ICLR 2024. Each task gives the model a codebase and an issue description and asks for a patch. The paper reported that the best model it tested, Claude 2, resolved 1.96% of the issues.
How a SWE-bench task is scored#
A SWE-bench task counts as resolved when the model's patch applies to the repository and two sets of tests pass: fail-to-pass tests, which failed before the original fix and pass after it, and pass-to-pass tests, which passed before the fix and must still pass. The benchmark score is the percentage of tasks resolved.
The tests come from the original pull request, and the model never sees them. To build the dataset, the authors collected merged pull requests that resolved an issue and changed test files, then kept only instances where at least one test went from failing to passing and the environment installed and ran cleanly. That filter narrowed about 90,000 pull requests to 2,294 tasks. The reference fixes edit 32.8 lines across 1.7 files on average, and the paper reports a median of 51 pass-to-pass tests per task. The current evaluation code runs every task in Docker containers, according to swebench.com.
SWE-bench variants#
SWE-bench has several variants that trade size for reliability or widen coverage. SWE-bench Lite is a 300-task subset of simpler, self-contained issues; SWE-bench Verified is a 500-task subset screened by human annotators; SWE-bench Multimodal uses issues that contain images; and SWE-bench Multilingual covers nine programming languages beyond Python.
| Variant | Tasks | What changes |
|---|---|---|
| SWE-bench | 2,294 | The original set from 12 Python repositories |
| Lite | 300 test, 23 development | Filtered to single-file fixes and self-contained issue text |
| Verified | 500 | Human-screened for well-specified issues and fair tests |
| Multimodal | 517 at release; version 2 keeps 480 | Issues include screenshots, mockups, or diagrams |
| Multilingual | 300 from 42 repositories | C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust |
OpenAI released SWE-bench Verified on August 13, 2024, after 93 software developers reviewed 1,699 random samples from the test set for underspecified issues and tests that reject valid solutions. In February 2026, OpenAI stopped reporting SWE-bench Verified, citing contamination and an audit in which 59.4% of 138 problems had material issues in test design or problem description, and recommended SWE-bench Pro from Scale AI.
A SWE-bench score measures a model together with its scaffold, tools, retry policy, and time budget, so compare results only under the same setup. Run the agent in an isolated sandbox when reproducing a result. The AI coding agents comparison shows how scaffolds differ and how to run a trial on your own repositories.
Related terms
Guides that cover swe-bench
- LLM benchmarks explained: what they measure and how to read them
- AI agent evaluation: metrics, graders, and how to test agents
- AI coding agents: how they work and how the major tools compare
- Best LLM for coding: models compared on agentic coding benchmarks
- Claude Code vs Cursor: how the two coding agents differ
Cite this definition
The AI Engineering Editors. "What is SWE-bench? The coding agent benchmark explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/swe-bench
- Title
- What is SWE-bench? The coding agent benchmark explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated