Domain source
SWE-bench Verified contributes the upstream evaluation contract and 500 public tasks.
RepoBench-500Evidence 2026-08-26Dataset page live
Can an agent resolve the issue in the repository—not just propose a patch?
Expert-verified GitHub issues paired with a service, repository, pull-request, incident, ADR, and standup world.
500upstream Harbor tasks
4released world tasks
100%reference ceiling
0%seeded random floor
Baseline leaderboard
Solvable, discriminating, and honest about what ran.
All companion tasks are solvable by their reference traces and reject the seeded random floor. No paid or frontier model run is claimed. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.
EntryRoleTasksPass rateMean rewardTool calls
Reference solverDeterministic trace replay; not a model rank.
Solvability ceiling4/4100%1.009/9 successfulSeeded random floorSeeded random tool policy; not a model rank.
Discrimination floor0/40%0.0020/24 successfulFrontier model submissionsAgent, model revision, trials, costs, and full traces required.
Model leaderboard———Awaiting first runWhy these controls matter
A 100% reference ceiling shows the tasks are executable, not all failing. A 0% seeded-random floor shows the verifier does not simply reward any activity. Neither row measures model capability.
Task browser
Read every released task.
These are the four independent Blobfish diagnostic tasks in this world—not excerpts relabeled from the larger Harbor source suite.
train4 reference stepshard · generator label
eng_resolve_ingest_incident
Incident 9401 on the ingest service was mitigated at minute 175. Close it out: set incident 9401's resolved_minute to 175 and status to 'resolved', work out how long it lasted (resolved_minute minus detected_minute), then record the follow-up decision by creating an ADR on service 9001 titled 'Rate-limit the ingest queue' with status 'accepted', decided_on '2026-07-21' and a decision that states the incident duration as 'N minutes'; finally post a standup note on service 9001 by author 'Rowan Novak' for day_label 'tuesday' whose body references incident 9401.
Required tool surface
- 01
get_incident - 02
update_incident - 03
create_adr_record - 04
create_standup_note
- Reference
- PASS
- Random floor
- FAIL
- Verifier
- vcode
- Gradable
- Yes
Generator label only; no empirical model difficulty claim is published.
Episode trace
Follow the tools, outputs, and verdict.
Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.
deterministic solvability ceiling
Reference solver
A checked-in reference trace replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.
- Task
- eng_resolve_ingest_incident
- Steps
- 4
- Reward
- 1.0
- Status
- passed
- 00Task loadeduser → agent
Fresh seeded database copied into an isolated episode.
- 01
get_incidenttool successArguments{ "incident_id": 9401 }Result{ "incident_id": 9401, "service_id": 9001, "summary": "Ingest queue saturated, consumers stalled", "severity": "sev1", "status": "open", "detected_minute": 100, "resolved_minute": null } - 02
update_incidenttool successArguments{ "incident_id": 9401, "resolved_minute": 175, "status": "resolved" }Result{ "incident_id": 9401, "service_id": 9001, "summary": "Ingest queue saturated, consumers stalled", "severity": "sev1", "status": "resolved", "detected_minute": 100, "resolved_minute": 175 } - 03
create_adr_recordtool successArguments{ "service_id": 9001, "title": "Rate-limit the ingest queue", "decision": "Incident 9401 lasted 75 minutes; rate-limit producers to keep consumers from stalling.", "status": "accepted", "decided_on": "2026-07-21" }Result{ "adr_id": 9302, "service_id": 9001, "title": "Rate-limit the ingest queue", "decision": "Incident 9401 lasted 75 minutes; rate-limit producers to keep consumers from stalling.", "status": "accepted", "decided_on": "2026-07-21" } - 04
create_standup_notetool successArguments{ "service_id": 9001, "author": "Rowan Novak", "day_label": "tuesday", "body": "Incident 9401 resolved; writing the rate-limit ADR follow-up." }Result{ "note_id": 9503, "service_id": 9001, "author": "Rowan Novak", "day_label": "tuesday", "body": "Incident 9401 resolved; writing the rate-limit ADR follow-up." } - 05Deterministic verifierpassed
- Basis
- vcode
- Reward
- 1.0
- Gradable
- yes
Released assets
Inspect the world down to each digest.
The executable database, schema, seed, tools, tasks, MCP manifest, and world manifest are checked into the same repository and linked individually.
Code + worldGitHub repositorybenchmark/worlds/repobench-500Runnable companionHarbor datasetblobfishai/domainbench-24Companion datasetHugging FaceBlobfish DomainBench-24Domain sourceUpstream Harbor suiteswe-bench/swe-bench-verified
Environment
A stateful MCP system, not a prompt file.
6 related entity tables expose 23 read-only and 21 mutating tools. Every episode gets a fresh copy of the 115-row seed state.
- 01Seed snapshotImmutable SQLite state
- 02MCP gateway44 typed tools
- 03Agent episodeIsolated scratch copy
- 04Trace + stateCalls and full diff
- 05VerifierDeterministic verdict
01Services
services · 19 rows02Repositories
repositories · 20 rows03Pull Requests
pull_requests · 20 rows04Adr Records
adr_records · 18 rows05Incidents
incidents · 19 rows06Standup Notes
standup_notes · 19 rowsPASStask admission
4/4reference replay
0/4random floor
Fresh copyper episode
MCP + CLI
Reproduce every layer locally.
Harbor preserves the upstream suite. Blobfish serves and evaluates the independent companion world. The reference policy is keyless; a model run requires an explicit provider and cost cap.
01Run this domain on Harbor
harbor run -d blobfishai/domainbench-24 \
-i '*repobench-500*' -a <agent> -m <provider/model>02Run the upstream suite
harbor run -d swe-bench/swe-bench-verified
-a <agent> -m <provider/model>03Serve the world over MCP
blobfish info benchmark/worlds/repobench-500
blobfish serve benchmark/worlds/repobench-50004Replay the reference ceiling
blobfish eval benchmark/worlds/repobench-500 \
--policy oracle --split all --run release-oracleMethodology
Archipelago-shaped evaluation, with deterministic grading.
Mercor’s open-source Archipelago clarified the public contract: environment, agent runner, trajectory, and grading should be inspectable together. Blobfish already has the same layers, plus fail-closed state policies and Harbor release gates.
LayerArchipelagoBlobfish implementation
EnvironmentDocker environment + MCP gateway + snapshotsSQLite world + MCP runtime + sandbox copy
Agent runnerLoop/ReAct registry with sandbox lifecycleMulti-provider harness + long-horizon trajectory runner
TrajectoryMessages, tool calls, outputs, and artifact editsFull-fidelity episode JSONL, calls, outputs, errors, and state evidence
GradingStatic, domain, and LLM verifiers over outputs and snapshotsState, trace, VCode, rubric, composite, and full-world diff verifiers
Release gateDataset/task package contractOracle admission, random-floor probe, immutable Harbor export, and replay
Independent world
The companion system is domain-aligned, generated with a fixed seed, and clearly labeled as separate from the source task set.
Two-sided probe
All 4 tasks must pass the reference replay and reject the random floor before release evidence is generated.
Model publication
No model row appears without a pinned model, agent, trials, runtime, costs, trajectories, and verifier outputs.
Across the portfolio