All benchmarksFinancial research

FilingBench-50Evidence 2026-08-26Dataset page live

Can an agent find the number, show its provenance, and update the model?

Public financial-research questions paired with a deterministic DCF, assumptions, period-data, and review-note world.

Explore all tasks ↓World repository ↗Run on Harbor ↗

50upstream Harbor tasks

4released world tasks

100%reference ceiling

0%seeded random floor

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces and reject the seeded random floor. No paid or frontier model run is claimed. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.009/9 successful

Seeded random floorSeeded random tool policy; not a model rank.

Discrimination floor0/40%0.0019/24 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% seeded-random floor shows the verifier does not simply reward any activity. Neither row measures model capability.

Task browser

Read every released task.

These are the four independent Blobfish diagnostic tasks in this world—not excerpts relabeled from the larger Harbor source suite.

train4 reference stepshard · generator label

fa_finalize_solstice_dcf

Finalise the Solstice base-case DCF (model_id 9201, company 9001). Total the free_cash_flow across every financial period belonging to company 9001 — other companies' periods do not count. Set model 9201's enterprise_value to that total and its status to 'final', mark its WACC assumption (assumption_id 9301) as status 'verified', and file the review: create a review note on model 9201 from reviewer 'Head of Research' with status 'open' and a body containing the summed cash flow.

Required tool surface

01search_financial_periods
02update_dcf_model
03update_dcf_assumption
04create_review_note

Reference: PASS
Random floor: FAIL
Verifier: vcode
Gradable: Yes

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in reference trace replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: fa_finalize_solstice_dcf
Steps: 4
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

search_financial_periodstool success

Arguments

{
  "company_id": 9001,
  "limit": 50
}

Result

[
  {
    "period_id": 9101,
    "company_id": 9001,
    "period_label": "2024",
    "revenue": 800,
    "ebitda": 200,
    "capex": 60,
    "free_cash_flow": 100
  },
  {
    "period_id": 9102,
    "company_id": 9001,
    "period_label": "2025",
    "revenue": 900,
    "ebitda": 230,
    "capex": 65,
    "free_cash_flow": 120
  },
  {
    "period_id": 9103,
    "company_id": 9001,
    "period_label": "2026",
    "revenue": 1000,
    "ebitda": 260,
    "capex": 70,
    "free_cash_flow": 140
  }
]

update_dcf_modeltool success

Arguments

{
  "model_id": 9201,
  "enterprise_value": 360,
  "status": "final"
}

Result

{
  "model_id": 9201,
  "company_id": 9001,
  "name": "Solstice base case",
  "discount_rate": 0.09,
  "terminal_growth": 0.02,
  "status": "final",
  "enterprise_value": 360
}

update_dcf_assumptiontool success

Arguments

{
  "assumption_id": 9301,
  "status": "verified"
}

Result

{
  "assumption_id": 9301,
  "model_id": 9201,
  "name": "WACC",
  "value": 0.09,
  "source": "Capital IQ",
  "status": "verified"
}

create_review_notetool success

Arguments

{
  "model_id": 9201,
  "reviewer_name": "Head of Research",
  "body": "Unlevered free cash flow across the period set totals 360.",
  "status": "open"
}

Result

{
  "note_id": 9502,
  "model_id": 9201,
  "reviewer_name": "Head of Research",
  "body": "Unlevered free cash flow across the period set totals 360.",
  "status": "open"
}

05
Deterministic verifierpassed
Basis
vcode
Reward
1.0
Gradable
yes

Released assets

Inspect the world down to each digest.

The executable database, schema, seed, tools, tasks, MCP manifest, and world manifest are checked into the same repository and linked individually.

Code + worldGitHub repositorybenchmark/worlds/filingbench-50 Runnable companionHarbor datasetblobfishai/domainbench-24 Companion datasetHugging FaceBlobfish DomainBench-24 Domain sourceUpstream Harbor suitevals/financeagent

AssetTypeSizeSHA-256Open

Seeded SQLite snapshotPristine state copied for every episode.

SQLite32 KBf70a5b84565d5aa9…View ↗

Relational schemaHuman-readable table and constraint definitions.

SQL1.5 KB510bf7e2943495cb…View ↗

Seed dataHuman-readable source for the released world state.

SQL14 KBc888acecc74c6705…View ↗

MCP tool catalogPublic tool names, schemas, and implementations.

JSON102 KB6653f9d2a6e13304…View ↗

Task manifestInstructions, splits, reference traces, and isolated verifier programs.

JSON104 KB96388b0f39e51ed1…View ↗

MCP manifestStdio server declaration for the world.

JSON2.0 KBe92914c7b33fd3e5…View ↗

World manifestContent identity, counts, split, and admission evidence.

JSON2.5 KB8475c86667f578b5…View ↗

Environment

A stateful MCP system, not a prompt file.

6 related entity tables expose 23 read-only and 21 mutating tools. Every episode gets a fresh copy of the 116-row seed state.

01Seed snapshotImmutable SQLite state
02MCP gateway44 typed tools
03Agent episodeIsolated scratch copy
04Trace + stateCalls and full diff
05VerifierDeterministic verdict

01Fa Companiesfa_companies · 20 rows

02Financial Periodsfinancial_periods · 21 rows

03Dcf Modelsdcf_models · 19 rows

04Dcf Assumptionsdcf_assumptions · 20 rows

05Lbo Modelslbo_models · 18 rows

06Review Notesreview_notes · 18 rows

PASStask admission

4/4reference replay

0/4random floor

Fresh copyper episode

MCP + CLI

Reproduce every layer locally.

Harbor preserves the upstream suite. Blobfish serves and evaluates the independent companion world. The reference policy is keyless; a model run requires an explicit provider and cost cap.

01Run this domain on Harbor

harbor run -d blobfishai/domainbench-24 \
  -i '*filingbench-50*' -a <agent> -m <provider/model>

02Run the upstream suite

harbor run -d vals/financeagent
  -a <agent> -m <provider/model>

03Serve the world over MCP

blobfish info benchmark/worlds/filingbench-50
blobfish serve benchmark/worlds/filingbench-50

04Replay the reference ceiling

blobfish eval benchmark/worlds/filingbench-50 \
  --policy oracle --split all --run release-oracle

Methodology

Archipelago-shaped evaluation, with deterministic grading.

Mercor’s open-source Archipelago clarified the public contract: environment, agent runner, trajectory, and grading should be inspectable together. Blobfish already has the same layers, plus fail-closed state policies and Harbor release gates.

LayerArchipelagoBlobfish implementation

EnvironmentDocker environment + MCP gateway + snapshotsSQLite world + MCP runtime + sandbox copy

Agent runnerLoop/ReAct registry with sandbox lifecycleMulti-provider harness + long-horizon trajectory runner

TrajectoryMessages, tool calls, outputs, and artifact editsFull-fidelity episode JSONL, calls, outputs, errors, and state evidence

GradingStatic, domain, and LLM verifiers over outputs and snapshotsState, trace, VCode, rubric, composite, and full-world diff verifiers

Release gateDataset/task package contractOracle admission, random-floor probe, immutable Harbor export, and replay

Read Archipelago ↗Inspect this Blobfish world ↗Inspect the upstream suite ↗

Domain source

Finance Agent Benchmark contributes the upstream evaluation contract and 50 public tasks.

Independent world

The companion system is domain-aligned, generated with a fixed seed, and clearly labeled as separate from the source task set.

Two-sided probe

All 4 tasks must pass the reference replay and reject the random floor before release evidence is generated.

Model publication

No model row appears without a pinned model, agent, trials, runtime, costs, trajectories, and verifier outputs.

Across the portfolio

Keep testing the boundary.

View all seven →

Legal operationsCounselBench-100

100 source tasks

Customer operationsServiceBench-375

375 source tasks

Scientific discoveryDiscoveryBench-102

102 source tasks