Coding and terminal agents
SWE-bench Verified
Resolve 500 human-validated GitHub issues.
About SWE-bench Verified
What it measures
Percentage resolved by the official test harness.
Scope & limitations
Hold repository commit, dependency images, attempts and agent scaffold fixed; historical public tasks can be contaminated.
Evaluation environment
Repository-editing agent and isolated executable tests.