Each sample in the SWE-bench test set is created from a resolved GitHub issue in one of 12 open-source Python repositories on GitHub. Each sample has an associated pull request (PR), which includes both the solution code and unit tests to verify code correctness. These unit tests fail before the solution code in the PR is added, but pass afterwards, and are therefore called FAIL_TO_PASS tests. Each sample also has associated PASS_TO_PASS tests, which pass both before and after the PR is merged, and are used to check that existing unrelated functionality in the codebase has not been broken by the PR. 

For each sample in SWE-bench, agents are provided with the original text from the GitHub issue, known as the problem statement, and are given access to the codebase. Given these, agents must edit the files in the codebase to resolve the issue. The tests are not shown to the agent.

A proposed edit is evaluated by running both the FAIL_TO_PASS and PASS_TO_PASS tests. If the FAIL_TO_PASS tests pass, this means the edit solves the issue. If the PASS_TO_PASS tests pass, then the edit has not inadvertently broken unrelated sections of the codebase. Both sets of tests are required to pass for the edit to fully resolve the original GitHub issue.

To address these issues, we launched a human annotation campaign with professional software developers to screen each sample of the SWE-bench test set for appropriately scoped unit tests and well-specified issue descriptions.

Together with the authors of SWE-bench, we are releasing SWE-bench Verified: a subset of the original test set from SWE-bench, consisting of 500 samples verified to be non-problematic by our human annotators. This version supersedes the original SWE-bench and SWE-bench Lite test sets. Additionally, we are releasing our human annotations for all SWE-bench test samples. These annotations enable slicing the dataset by difficulty. The 'easy' subset is composed of 196 <15-minute fix tasks, while the 'hard' subset is composed of 45 >1-hour tasks.

We also collaborated with the SWE-bench authors to develop a new evaluation harness for SWE-bench⁠(opens in a new window) which uses containerized Docker environments to make evaluating on SWE-bench easier and more reliable.

On SWE-bench Verified, GPT‑4o resolves 33.2% of samples³, with the best performing open-source scaffold, Agentless, doubling its previous score of 16% on SWE-bench.

We worked with 93 software developers experienced in Python to manually screen SWE-bench samples for quality. We annotated 1,699 random samples from the SWE-bench test set to produce SWE-bench Verified. The following analysis is based on the 1,699 samples.

We annotate samples to capture:

• Whether we consider the issue description to be underspecified and hence unfair to be testing on.
• Whether the FAIL_TO_PASS unit tests filter out valid solutions.

Each annotation criterion has a label ranging [0, 1, 2, 3] in increasing severity. Labels 0 and 1 are minor; labels 2 and 3 are severe and indicate that the sample is inadequate in some way and should be discarded. We choose to annotate across four ordinal categories rather than a single binary label of severe/not severe to capture more granular detail.

Additionally, we rate the difficulty of each sample by having annotators estimate how long it would take for a developer to decide upon and implement the solution, assuming the sample is non-problematic. Finally, we provide a freeform input option to flag any other major issues with the sample (for example, if the FAIL_TO_PASS unit tests are easily gamed, this could lead to an invalid solution being marked as correct).

Our team of engineers first hand-labeled 50 samples to a high degree of confidence for use in annotator onboarding tests. To take part in the annotation campaign, each prospective annotator had to pass our onboarding tests. We provided detailed feedback to each annotator throughout onboarding to better train them for the task.  Annotators were not necessarily prior experts in the codebases relevant to SWE-bench, but were given time to familiarize themselves with each codebase they worked with.

To ensure a high-quality dataset, each sample is labeled 3 times by separate annotators. It is easy to accidentally miss potential issues, and issues themselves can be ambiguous, so we conservatively ensemble annotations by taking the highest-severity label amongst the 3 annotators.

The full text of our annotation rubric can be found here⁠(opens in a new window).

To construct SWE-bench Verified, we filter out any sample from the original test set where either the problem statement or the FAIL_TO_PASS unit tests have an ensemble label of 2 or above in severity. We also filter out all samples that have other major issues flagged. Given our ensembling method, this is equivalent to filtering out samples where any single annotator of three has flagged an issue with the sample. This approach leads to a higher false-positive rate in removing samples, but helps increase our confidence in sample quality for the final dataset. 

We include as many samples with difficulty 1-4 hours and >4 hours as possible, and then we randomly sample the remainder to arrive at the 500 samples that constitute SWE-bench Verified.

The chart below compares the difficulty distributions of the original SWE-bench datasets and our new SWE-bench Verified dataset. We estimate the difficulty distribution of SWE-bench based on our random subset of 1699 samples. Note that while these results provide estimates of the effort necessary to implement a solution (refer to our annotation instructions for the precise wording), they assume a software engineer who is able to figure out the solution. In practice, we expect the baseline solve rate of a typical human software engineer to be lower than 100%.  

We observe that most (77.8%) of the samples in the original SWE-bench dataset were estimated to take less than an hour for an experienced software engineer to complete. Both SWE-bench Lite and our new SWE-bench Verified dataset skews this further, leaving fewer than 10% of issues estimated to take longer than an hour. However, the mechanism underlying this shift is importantly different: SWE-bench Lite subsampled the original dataset to make the benchmark easier, whereas SWE-bench Verified attempts to remove infeasible samples from the dataset. We further explore this effect in the next section.

With our new SWE-bench Verified dataset, we tested GPT‑4o’s performance using several open-source scaffolds that performed well on the original SWE-bench leaderboards⁴.

We found that GPT‑4o’s performance on the best-performing scaffold reaches 33.2% on SWE-bench Verified, more than doubling its score of 16% on the original SWE-bench. In general, this validates our initial suspicion that the original SWE-bench dataset underestimates agent abilities. Note that the jump from SWE-bench Lite to SWE-bench Verified is not as significant, because SWE-bench Lite was already filtered in a way that makes it easier⁠(opens in a new window) than the full dataset, though that process would not fully capture the same issues as our filtering procedure.

The increase in performance when evaluating on SWE-bench Verified may partly be explained by shifting the distribution toward easier samples (as shown in earlier analyses). However, our goal is not to inflate benchmark scores, but to make sure that the benchmark faithfully represents model capability at any given difficulty level.

We investigate this by plotting performance stratified by difficulty. If our new dataset merely shifted the difficulty distribution to contain more easy samples, the stratified performance within each category would not change, as appears to be the case going from the original SWE-bench to SWE-bench Lite. We instead observe that performance increases within individual difficulty categories when moving to SWE-bench Verified, which is consistent with the intended effect of removing impossible samples from all categories instead of removing difficult samples. The effect is clearest in the easiest two buckets of difficulty, where we have the most samples.

We use SWE-bench as one of several evaluations tracking the Medium risk level of the Model Autonomy risk category in our Preparedness Framework. Tracking catastrophic risk levels via evaluations relies on ensuring that we can trust evaluation results and are calibrated about what the scores entail.

Our experiences suggest that we should:

Invest in deeply understanding our benchmarks. Although SWE-bench was designed thoughtfully, it underestimates model capabilities due to the issues mentioned in this blogpost. As our systems get closer to AGI, we need to evaluate them on increasingly more challenging tasks. This also elevates the level of expertise and care needed to curate and verify benchmarks to ensure that they are sufficiently challenging and robust (a case where work like CriticGPT⁠, which explores ways in which AI can assist with annotation pipelines, may be helpful).

Account for progress in the ecosystem. Community-led progress in agent scaffolding highlights the need to consider potential external enhancements to a model when assessing risk. Looking at the difference between the worst- and best-performing scaffolds for a given model on the SWE-bench leaderboards⁠(opens in a new window), we can see that, for example, GPT‑4’s performance on SWE-bench Lite varies between 2.7% using an early RAG-based scaffold and 28.3% using CodeR. Thus the Preparedness Framework calls for evaluations to be run continually and as often as needed to identify any non-trivial capability change; which includes before, during, and even after training, where models can be enhanced via integration with external systems. Furthermore, curating evaluations is an ecosystem-wide effort, and we hope to continue collaborating with researchers on building trustworthy, high-quality evaluations.

Be cognizant of limitations. Evaluations based on static datasets are inherently limited, and SWE-bench is no exception. Given that the benchmark is composed of scrapes of public GitHub repos, large foundation models that are pre-trained on internet text are likely to be contaminated on the tasks. Furthermore, SWE-bench only covers a narrow distribution of the Medium risk level for model autonomy and so must be supplemented with other evaluations. 

We believe in an empirical and scientific approach to tracking and protecting against catastrophic risk. Building and continually improving evaluations is a key element of this work. There remains much to be done, and we’re eager to see more work from the community in contributing valuable benchmarks like SWE-bench.