Software engineering / Benchmark leaderboard
RepoBench-500
Expert-verified GitHub issues paired with a service, repository, pull-request, incident, ADR, and standup world.
No published company result yet.
The existing Blobfish benchmark page retains its task definitions, baseline evidence and release status. Company comparisons will appear here after a reviewed evaluation.
View existing benchmark evidence and release status →
Arrange an evaluation for your company →What this benchmark measures
Metrics
Expert-verified, solvable real-world GitHub issues scored by repository tests.
Execution requirements
The upstream task environments use repository-specific Docker images and test contracts.
Scope and limitations
This Blobfish suite has its own task and environment definitions. Upstream scores and reference controls are not company performance scores.
Evaluation availability
Catalog entry. Request a managed evaluation to qualify your agent interface and the benchmark’s native grading requirements.
Sources and company fit
- Original benchmark ↗Blobfish benchmark suite
- Dataset / catalog ↗