What is SWE-bench? The coding agent benchmark explained

Also called SWE-bench Verified, SWE-bench Lite, SWE bench

Definition

SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests.

SWE-bench is a benchmark that measures whether a language model, usually running inside an AI agent scaffold, can resolve real software issues. Jimenez et al. built it from 2,294 GitHub issues, and the pull requests that fixed them, across 12 popular Python repositories, and presented it at ICLR 2024. Each task gives the model a codebase and an issue description and asks for a patch. The paper reported that the best model it tested, Claude 2, resolved 1.96% of the issues.

How a SWE-bench task is scored#

A SWE-bench task counts as resolved when the model's patch applies to the repository and two sets of tests pass: fail-to-pass tests, which failed before the original fix and pass after it, and pass-to-pass tests, which passed before the fix and must still pass. The benchmark score is the percentage of tasks resolved.

A model reads an issue and a codebase and writes a patch; the evaluation code applies it in a container and runs fail-to-pass and pass-to-pass tests, and the task is resolved only if both pass. Issue and codebaseRepository at the issue's commitModel or agentEdits filesPatchApplied in a containerFail-to-pass testsProve the issue is fixedPass-to-pass testsProve nothing brokeResolved
Figure 1: A SWE-bench task is resolved only when the patch applies and both test sets pass.

The tests come from the original pull request, and the model never sees them. To build the dataset, the authors collected merged pull requests that resolved an issue and changed test files, then kept only instances where at least one test went from failing to passing and the environment installed and ran cleanly. That filter narrowed about 90,000 pull requests to 2,294 tasks. The reference fixes edit 32.8 lines across 1.7 files on average, and the paper reports a median of 51 pass-to-pass tests per task. The current evaluation code runs every task in Docker containers, according to swebench.com.

SWE-bench variants#

SWE-bench has several variants that trade size for reliability or widen coverage. SWE-bench Lite is a 300-task subset of simpler, self-contained issues; SWE-bench Verified is a 500-task subset screened by human annotators; SWE-bench Multimodal uses issues that contain images; and SWE-bench Multilingual covers nine programming languages beyond Python.

Variant Tasks What changes
SWE-bench 2,294 The original set from 12 Python repositories
Lite 300 test, 23 development Filtered to single-file fixes and self-contained issue text
Verified 500 Human-screened for well-specified issues and fair tests
Multimodal 517 at release; version 2 keeps 480 Issues include screenshots, mockups, or diagrams
Multilingual 300 from 42 repositories C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust

OpenAI released SWE-bench Verified on August 13, 2024, after 93 software developers reviewed 1,699 random samples from the test set for underspecified issues and tests that reject valid solutions. In February 2026, OpenAI stopped reporting SWE-bench Verified, citing contamination and an audit in which 59.4% of 138 problems had material issues in test design or problem description, and recommended SWE-bench Pro from Scale AI.

A SWE-bench score measures a model together with its scaffold, tools, retry policy, and time budget, so compare results only under the same setup. Run the agent in an isolated sandbox when reproducing a result. The AI coding agents comparison shows how scaffolds differ and how to run a trial on your own repositories.

Cite this definition

The AI Engineering Editors. "What is SWE-bench? The coding agent benchmark explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/swe-bench

Title
What is SWE-bench? The coding agent benchmark explained
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM models and benchmarks hub · Updated