All benchmarksScientific discovery

DiscoveryBench-102Evidence 2026-08-26Dataset page live

Can an agent turn scientific data into a reproducible result?

Scientific-programming tasks paired with an assay, compound, target, QC-result, and literature world.

Explore all tasks ↓World repository ↗Run on Harbor ↗

102upstream Harbor tasks

4released world tasks

100%reference ceiling

0%seeded random floor

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces and reject the seeded random floor. No paid or frontier model run is claimed. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.009/9 successful

Seeded random floorSeeded random tool policy; not a model rank.

Discrimination floor0/40%0.0020/24 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% seeded-random floor shows the verifier does not simply reward any activity. Neither row measures model capability.

Task browser

Read every released task.

These are the four independent Blobfish diagnostic tasks in this world—not excerpts relabeled from the larger Harbor source suite.

train4 reference stepshard · generator label

bio_promote_hit_compound

Call the potency of assay run 9301. Compute the mean ic50_nm across its replicates that PASSED QC (qc_flag 'pass') — replicates flagged 'fail' are excluded. A mean below 100 nM makes the compound a hit, so: set compound 9101's status to 'hit' and record the mean in its screening_note, set assay run 9301's status to 'complete', and queue confirmation by creating an assay run on assay 9201 for compound 9101 with run_on '2026-08-01', plate_label 'CONF-001' and status 'pending'.

Required tool surface

01search_assay_results
02update_compound
03update_assay_run
04create_assay_run

Reference: PASS
Random floor: FAIL
Verifier: vcode
Gradable: Yes

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in reference trace replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: bio_promote_hit_compound
Steps: 4
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

search_assay_resultstool success

Arguments

{
  "run_id": 9301,
  "limit": 50
}

Result

[
  {
    "result_id": 9401,
    "run_id": 9301,
    "replicate": 1,
    "ic50_nm": 20,
    "percent_inhibition": 91,
    "qc_flag": "pass"
  },
  {
    "result_id": 9402,
    "run_id": 9301,
    "replicate": 2,
    "ic50_nm": 24,
    "percent_inhibition": 89,
    "qc_flag": "pass"
  },
  {
    "result_id": 9403,
    "run_id": 9301,
    "replicate": 3,
    "ic50_nm": 22,
    "percent_inhibition": 90,
    "qc_flag": "pass"
  },
  {
    "result_id": 9404,
    "run_id": 9301,
    "replicate": 4,
    "ic50_nm": 400,
    "percent_inhibition": 11,
    "qc_flag": "fail"
  }
]

update_compoundtool success

Arguments

{
  "compound_id": 9101,
  "status": "hit",
  "screening_note": "Mean QC-passing IC50 22 nM on run 9301."
}

Result

{
  "compound_id": 9101,
  "code": "BF-1042",
  "name": "Velatinib",
  "molecular_weight": 421.5,
  "status": "hit",
  "screening_note": "Mean QC-passing IC50 22 nM on run 9301."
}

update_assay_runtool success

Arguments

{
  "run_id": 9301,
  "status": "complete"
}

Result

{
  "run_id": 9301,
  "assay_id": 9201,
  "compound_id": 9101,
  "run_on": "2026-07-14",
  "plate_label": "PLT-0091",
  "status": "complete"
}

create_assay_runtool success

Arguments

{
  "assay_id": 9201,
  "compound_id": 9101,
  "run_on": "2026-08-01",
  "plate_label": "CONF-001",
  "status": "pending"
}

Result

{
  "run_id": 9303,
  "assay_id": 9201,
  "compound_id": 9101,
  "run_on": "2026-08-01",
  "plate_label": "CONF-001",
  "status": "pending"
}

05
Deterministic verifierpassed
Basis
vcode
Reward
1.0
Gradable
yes

Released assets

Inspect the world down to each digest.

The executable database, schema, seed, tools, tasks, MCP manifest, and world manifest are checked into the same repository and linked individually.

Code + worldGitHub repositorybenchmark/worlds/discoverybench-102 Runnable companionHarbor datasetblobfishai/domainbench-24 Companion datasetHugging FaceBlobfish DomainBench-24 Domain sourceUpstream Harbor suitescienceagentbench/scienceagentbench

AssetTypeSizeSHA-256Open

Seeded SQLite snapshotPristine state copied for every episode.

SQLite32 KB44b907f65127934c…View ↗

Relational schemaHuman-readable table and constraint definitions.

SQL1.5 KB4dfc04a1e3d89b38…View ↗

Seed dataHuman-readable source for the released world state.

SQL12 KBe3e3c20b66558ba0…View ↗

MCP tool catalogPublic tool names, schemas, and implementations.

JSON95 KB511ab9dc9a947171…View ↗

Task manifestInstructions, splits, reference traces, and isolated verifier programs.

JSON101 KB73b32ea4a6834a88…View ↗

MCP manifestStdio server declaration for the world.

JSON1.8 KB6dfe7eff3b010b68…View ↗

World manifestContent identity, counts, split, and admission evidence.

JSON2.4 KB9788c5cbc28caa5c…View ↗

Environment

A stateful MCP system, not a prompt file.

6 related entity tables expose 23 read-only and 20 mutating tools. Every episode gets a fresh copy of the 119-row seed state.

01Seed snapshotImmutable SQLite state
02MCP gateway43 typed tools
03Agent episodeIsolated scratch copy
04Trace + stateCalls and full diff
05VerifierDeterministic verdict

01Targetstargets · 19 rows

02Compoundscompounds · 20 rows

03Assaysassays · 19 rows

04Assay Runsassay_runs · 19 rows

05Assay Resultsassay_results · 24 rows

06Literature Refsliterature_refs · 18 rows

PASStask admission

4/4reference replay

0/4random floor

Fresh copyper episode

MCP + CLI

Reproduce every layer locally.

Harbor preserves the upstream suite. Blobfish serves and evaluates the independent companion world. The reference policy is keyless; a model run requires an explicit provider and cost cap.

01Run this domain on Harbor

harbor run -d blobfishai/domainbench-24 \
  -i '*discoverybench-102*' -a <agent> -m <provider/model>

02Run the upstream suite

harbor run -d scienceagentbench/scienceagentbench
  -a <agent> -m <provider/model>

03Serve the world over MCP

blobfish info benchmark/worlds/discoverybench-102
blobfish serve benchmark/worlds/discoverybench-102

04Replay the reference ceiling

blobfish eval benchmark/worlds/discoverybench-102 \
  --policy oracle --split all --run release-oracle

Methodology

Archipelago-shaped evaluation, with deterministic grading.

Mercor’s open-source Archipelago clarified the public contract: environment, agent runner, trajectory, and grading should be inspectable together. Blobfish already has the same layers, plus fail-closed state policies and Harbor release gates.

LayerArchipelagoBlobfish implementation

EnvironmentDocker environment + MCP gateway + snapshotsSQLite world + MCP runtime + sandbox copy

Agent runnerLoop/ReAct registry with sandbox lifecycleMulti-provider harness + long-horizon trajectory runner

TrajectoryMessages, tool calls, outputs, and artifact editsFull-fidelity episode JSONL, calls, outputs, errors, and state evidence

GradingStatic, domain, and LLM verifiers over outputs and snapshotsState, trace, VCode, rubric, composite, and full-world diff verifiers

Release gateDataset/task package contractOracle admission, random-floor probe, immutable Harbor export, and replay

Read Archipelago ↗Inspect this Blobfish world ↗Inspect the upstream suite ↗

Domain source

ScienceAgentBench contributes the upstream evaluation contract and 102 public tasks.

Independent world

The companion system is domain-aligned, generated with a fixed seed, and clearly labeled as separate from the source task set.

Two-sided probe

All 4 tasks must pass the reference replay and reject the random floor before release evidence is generated.

Model publication

No model row appears without a pinned model, agent, trials, runtime, costs, trajectories, and verifier outputs.

Across the portfolio

Keep testing the boundary.

View all seven →

Legal operationsCounselBench-100

100 source tasks

Financial researchFilingBench-50

50 source tasks

Customer operationsServiceBench-375

375 source tasks