Domain source
CyberDefenseBench contributes the upstream evaluation contract and 100 public tasks.
DefenseBench-100Evidence 2026-08-26Dataset page live
Can an agent harden a service without reaching beyond the sandbox?
Local defensive-hardening tasks paired with a permission-aware search, document, employee, and access-grant world.
100upstream Harbor tasks
4released world tasks
100%reference ceiling
0%seeded random floor
Baseline leaderboard
Solvable, discriminating, and honest about what ran.
All companion tasks are solvable by their reference traces and reject the seeded random floor. No paid or frontier model run is claimed. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.
EntryRoleTasksPass rateMean rewardTool calls
Reference solverDeterministic trace replay; not a model rank.
Solvability ceiling4/4100%1.0011/11 successfulSeeded random floorSeeded random tool policy; not a model rank.
Discrimination floor0/40%0.0023/24 successfulFrontier model submissionsAgent, model revision, trials, costs, and full traces required.
Model leaderboard———Awaiting first runWhy these controls matter
A 100% reference ceiling shows the tasks are executable, not all failing. A 0% seeded-random floor shows the verifier does not simply reward any activity. Neither row measures model capability.
Task browser
Read every released task.
These are the four independent Blobfish diagnostic tasks in this world—not excerpts relabeled from the larger Harbor source suite.
train5 reference stepshard · generator label
es_lock_down_confidential_document
Document 9201 ('FY27 Budget Model') is confidential and may only be readable by the Finance team. Review its access grants: revoke every still-active grant held by an employee who is not on the Finance team by setting that grant's status to 'revoked', leave Finance grants and already-revoked grants alone, do not touch grants on other documents, and finally set document 9201's status to 'restricted'.
Required tool surface
- 01
search_access_grants - 02
search_employees - 03
update_access_grant - 04
update_indexed_document
- Reference
- PASS
- Random floor
- FAIL
- Verifier
- vcode
- Gradable
- Yes
Generator label only; no empirical model difficulty claim is published.
Episode trace
Follow the tools, outputs, and verdict.
Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.
deterministic solvability ceiling
Reference solver
A checked-in reference trace replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.
- Task
- es_lock_down_confidential_document
- Steps
- 5
- Reward
- 1.0
- Status
- passed
- 00Task loadeduser → agent
Fresh seeded database copied into an isolated episode.
- 01
search_access_grantstool successArguments{ "document_id": 9201, "limit": 50 }Result[ { "grant_id": 9401, "document_id": 9201, "employee_id": 9101, "level": "read", "status": "active" }, { "grant_id": 9402, "document_id": 9201, "employee_id": 9102, "level": "read", "status": "active" }, { "grant_id": 9403, "document_id": 9201, "employee_id": 9103, "level": "read", "status": "active" }, { "grant_id": 9404, "document_id": 9201, "employee_id": 9104, "level": "read", "status": "revoked" } ] - 02
search_employeestool successArguments{ "limit": 50 }Result[ { "employee_id": 1, "name": "Standard Internal Employee", "email": "elizabethlee@gmail.com", "team": "Standard Team E80" }, { "employee_id": 2, "name": "Recurring Internal Employee", "email": "john_jackson713@work.org", "team": "Consolidated Team C88" }, { "employee_id": 3, "name": "Consolidated Partner Employee", "email": "jessicajackson@gmail.com", "team": "Consolidated Team A59" }, { "employee_id": 4, "name": "Standard Domestic Employee", "email": "elizabeth_gonzalez83@work.org", "team": "Preferred Team A46" }, { "employee_id": 5, "name": "Quarterly Regional Employee", "email": "barbara.martinez606@work.org", "team": "Backup Team A22" }, { "employee_id": 6, "name": "Standard Partner Employee", "email": "lisawilson@company.com", "team": "General Team E78" }, { "employee_id": 7, "name": "Recurring Internal Employee", "email": "sandra_thompson1@outlook.com", "team": "General Team D87" }, { "employee_id": 8, "name": "Quarterly Regional Employee", "email": "kennethwalker478@work.org", "team": "Backup Team A56" }, { "employee_id": 9, "name": "Expedited Partner Employee", "email": "karen_gonzalez@outlook.com", "team": "Standard Team C17" }, { "employee_id": 10, "name": "Consolidated Retail Employee", "email": "sarah_wilson@outlook.com", "team": "Backup Team B44" }, { "employee_id": 11, "name": "Expedited Partner Employee", "email": "betty.smith@outlook.com", "team": "Preferred Team A60" }, { "employee_id": 12, "name": "Consolidated Partner Employee", "email": "sandra.martin63@work.org", "team": "Standard Team D35" }, { "employee_id": 13, "name": "Expedited Export Employee", "email": "karenperez@outlook.com", "team": "Preferred Team B21" }, { "employee_id": 14, "name": "Expedited Export Employee", "email": "joshua.smith@company.com", "team": "Preferred Team C80" }, { "employee_id": 15, "name": "Annual Export Employee", "email": "jessicarobinson@outlook.com", "team": "Backup Team F96" }, { "employee_id": 16, "name": "Quarterly Partner Employee", "email": "lisa_wilson@work.org", "team": "Secondary Team F88" }, { "employee_id": 17, "name": "Recurring Domestic Employee", "email": "david_brown782@yahoo.com", "team": "Standard Team C36" }, { "employee_id": 9101, "name": "Ines Bakker", "email": "ines@acme.example.com", "team": "Finance" }, { "employee_id": 9102, "name": "Rowan Ferreira", "email": "rowan.f@acme.example.com", "team": "Sales" }, { "employee_id": 9103, "name": "Rowan Ferreira", "email": "r.ferreira@acme.example.com", "team": "Engineering" }, { "employee_id": 9104, "name": "Tobias Lund", "email": "tobias@acme.example.com", "team": "Finance" } ] - 03
update_access_granttool successArguments{ "grant_id": 9402, "status": "revoked" }Result{ "grant_id": 9402, "document_id": 9201, "employee_id": 9102, "level": "read", "status": "revoked" } - 04
update_access_granttool successArguments{ "grant_id": 9403, "status": "revoked" }Result{ "grant_id": 9403, "document_id": 9201, "employee_id": 9103, "level": "read", "status": "revoked" } - 05
update_indexed_documenttool successArguments{ "document_id": 9201, "status": "restricted" }Result{ "document_id": 9201, "source_id": 9001, "title": "FY27 Budget Model", "owner_email": "ines@acme.example.com", "updated_on": "2026-07-10", "sensitivity": "confidential", "status": "restricted" } - 06Deterministic verifierpassed
- Basis
- vcode
- Reward
- 1.0
- Gradable
- yes
Released assets
Inspect the world down to each digest.
The executable database, schema, seed, tools, tasks, MCP manifest, and world manifest are checked into the same repository and linked individually.
Code + worldGitHub repositorybenchmark/worlds/defensebench-100Runnable companionHarbor datasetblobfishai/domainbench-24Companion datasetHugging FaceBlobfish DomainBench-24Domain sourceUpstream Harbor suitepolyvorlabs/cyberdefense-bench
Environment
A stateful MCP system, not a prompt file.
6 related entity tables expose 23 read-only and 19 mutating tools. Every episode gets a fresh copy of the 121-row seed state.
- 01Seed snapshotImmutable SQLite state
- 02MCP gateway42 typed tools
- 03Agent episodeIsolated scratch copy
- 04Trace + stateCalls and full diff
- 05VerifierDeterministic verdict
01Sources
sources · 20 rows02Employees
employees · 21 rows03Indexed Documents
indexed_documents · 20 rows04Document Chunks
document_chunks · 20 rows05Access Grants
access_grants · 22 rows06Search Queries
search_queries · 18 rowsPASStask admission
4/4reference replay
0/4random floor
Fresh copyper episode
MCP + CLI
Reproduce every layer locally.
Harbor preserves the upstream suite. Blobfish serves and evaluates the independent companion world. The reference policy is keyless; a model run requires an explicit provider and cost cap.
01Run this domain on Harbor
harbor run -d blobfishai/domainbench-24 \
-i '*defensebench-100*' -a <agent> -m <provider/model>02Run the upstream suite
harbor run -d polyvorlabs/cyberdefense-bench
-a <agent> -m <provider/model>03Serve the world over MCP
blobfish info benchmark/worlds/defensebench-100
blobfish serve benchmark/worlds/defensebench-10004Replay the reference ceiling
blobfish eval benchmark/worlds/defensebench-100 \
--policy oracle --split all --run release-oracleMethodology
Archipelago-shaped evaluation, with deterministic grading.
Mercor’s open-source Archipelago clarified the public contract: environment, agent runner, trajectory, and grading should be inspectable together. Blobfish already has the same layers, plus fail-closed state policies and Harbor release gates.
LayerArchipelagoBlobfish implementation
EnvironmentDocker environment + MCP gateway + snapshotsSQLite world + MCP runtime + sandbox copy
Agent runnerLoop/ReAct registry with sandbox lifecycleMulti-provider harness + long-horizon trajectory runner
TrajectoryMessages, tool calls, outputs, and artifact editsFull-fidelity episode JSONL, calls, outputs, errors, and state evidence
GradingStatic, domain, and LLM verifiers over outputs and snapshotsState, trace, VCode, rubric, composite, and full-world diff verifiers
Release gateDataset/task package contractOracle admission, random-floor probe, immutable Harbor export, and replay
Independent world
The companion system is domain-aligned, generated with a fixed seed, and clearly labeled as separate from the source task set.
Two-sided probe
All 4 tasks must pass the reference replay and reject the random floor before release evidence is generated.
Model publication
No model row appears without a pinned model, agent, trials, runtime, costs, trajectories, and verifier outputs.
Across the portfolio