All benchmarksEnterprise operations

OpsBench-14Evidence 2026-08-26Dataset page live

Can an agent join the systems behind a real operational decision?

Cross-functional enterprise questions paired with vendors, processes, changes, controls, and review workflows.

Explore all tasks ↓World repository ↗Run on Harbor ↗

14upstream Harbor tasks

4released world tasks

100%reference ceiling

0%seeded random floor

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces and reject the seeded random floor. No paid or frontier model run is claimed. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.009/9 successful

Seeded random floorSeeded random tool policy; not a model rank.

Discrimination floor0/40%0.0022/24 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% seeded-random floor shows the verifier does not simply reward any activity. Neither row measures model capability.

Task browser

Read every released task.

These are the four independent Blobfish diagnostic tasks in this world—not excerpts relabeled from the larger Harbor source suite.

heldout4 reference stepshard · generator label

ops_reject_uncontrolled_change

Change request 9301 is a high-risk change to the Vendor Onboarding process (doc_id 9101). Policy: a high-risk change cannot be approved while any step of that process has no control_ref mapped. Count the DISTINCT steps of doc 9101 with an empty control_ref — note step 9202 is a duplicate of 9201 and counts once. Because at least one step is uncontrolled: set change request 9301's status to 'rejected', set process doc 9101's status to 'needs_controls', and open the remediation by creating a compliance control named 'Step control backfill' with framework 'SOC2' and status 'untested'.

Required tool surface

01search_process_steps
02update_change_request
03update_process_doc
04create_compliance_control

Reference: PASS
Random floor: FAIL
Verifier: vcode
Gradable: Yes

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in reference trace replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: ops_reject_uncontrolled_change
Steps: 4
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

search_process_stepstool success

Arguments

{
  "doc_id": 9101,
  "limit": 50
}

Result

[
  {
    "step_id": 9201,
    "doc_id": 9101,
    "step_number": 1,
    "description": "Collect vendor tax documentation",
    "control_ref": "CTRL-014"
  },
  {
    "step_id": 9202,
    "doc_id": 9101,
    "step_number": 1,
    "description": "Collect vendor tax documentation",
    "control_ref": "CTRL-014"
  },
  {
    "step_id": 9203,
    "doc_id": 9101,
    "step_number": 2,
    "description": "Run sanctions screening",
    "control_ref": "CTRL-021"
  },
  {
    "step_id": 9204,
    "doc_id": 9101,
    "step_number": 3,
    "description": "Grant system access",
    "control_ref": ""
  }
]

update_change_requesttool success

Arguments

{
  "request_id": 9301,
  "status": "rejected"
}

Result

{
  "request_id": 9301,
  "doc_id": 9101,
  "title": "Skip sanctions screening for tier3",
  "risk_level": "high",
  "status": "rejected",
  "requested_by": "Riley Garcia"
}

update_process_doctool success

Arguments

{
  "doc_id": 9101,
  "status": "needs_controls"
}

Result

{
  "doc_id": 9101,
  "name": "Vendor Onboarding",
  "owner_name": "Jordan Patel",
  "status": "needs_controls",
  "version": 3
}

create_compliance_controltool success

Arguments

{
  "name": "Step control backfill",
  "framework": "SOC2",
  "status": "untested",
  "last_tested_on": ""
}

Result

{
  "control_id": 9402,
  "name": "Step control backfill",
  "framework": "SOC2",
  "status": "untested",
  "last_tested_on": ""
}

05
Deterministic verifierpassed
Basis
vcode
Reward
1.0
Gradable
yes

Released assets

Inspect the world down to each digest.

The executable database, schema, seed, tools, tasks, MCP manifest, and world manifest are checked into the same repository and linked individually.

Code + worldGitHub repositorybenchmark/worlds/opsbench-14 Runnable companionHarbor datasetblobfishai/domainbench-24 Companion datasetHugging FaceBlobfish DomainBench-24 Domain sourceUpstream Harbor suiteEnterprise-Bench/l1-l2-bench

AssetTypeSizeSHA-256Open

Seeded SQLite snapshotPristine state copied for every episode.

SQLite32 KBd6ca2df3416d0c44…View ↗

Relational schemaHuman-readable table and constraint definitions.

SQL1.4 KB0e2ed2c463b00b18…View ↗

Seed dataHuman-readable source for the released world state.

SQL13 KB9457c0e86a30ba26…View ↗

MCP tool catalogPublic tool names, schemas, and implementations.

JSON97 KBc8fbd5a4df7d08a3…View ↗

Task manifestInstructions, splits, reference traces, and isolated verifier programs.

JSON104 KB0994054741ca6f75…View ↗

MCP manifestStdio server declaration for the world.

JSON1.9 KBd431118d99a969b5…View ↗

World manifestContent identity, counts, split, and admission evidence.

JSON2.5 KB42447057cbe5a476…View ↗

Environment

A stateful MCP system, not a prompt file.

6 related entity tables expose 21 read-only and 21 mutating tools. Every episode gets a fresh copy of the 116-row seed state.

01Seed snapshotImmutable SQLite state
02MCP gateway42 typed tools
03Agent episodeIsolated scratch copy
04Trace + stateCalls and full diff
05VerifierDeterministic verdict

01Vendorsvendors · 20 rows

02Process Docsprocess_docs · 19 rows

03Process Stepsprocess_steps · 22 rows

04Change Requestschange_requests · 19 rows

05Compliance Controlscompliance_controls · 18 rows

06Vendor Reviewsvendor_reviews · 18 rows

PASStask admission

4/4reference replay

0/4random floor

Fresh copyper episode

MCP + CLI

Reproduce every layer locally.

Harbor preserves the upstream suite. Blobfish serves and evaluates the independent companion world. The reference policy is keyless; a model run requires an explicit provider and cost cap.

01Run this domain on Harbor

harbor run -d blobfishai/domainbench-24 \
  -i '*opsbench-14*' -a <agent> -m <provider/model>

02Run the upstream suite

harbor run -d Enterprise-Bench/l1-l2-bench
  -a <agent> -m <provider/model>

03Serve the world over MCP

blobfish info benchmark/worlds/opsbench-14
blobfish serve benchmark/worlds/opsbench-14

04Replay the reference ceiling

blobfish eval benchmark/worlds/opsbench-14 \
  --policy oracle --split all --run release-oracle

Methodology

Archipelago-shaped evaluation, with deterministic grading.

Mercor’s open-source Archipelago clarified the public contract: environment, agent runner, trajectory, and grading should be inspectable together. Blobfish already has the same layers, plus fail-closed state policies and Harbor release gates.

LayerArchipelagoBlobfish implementation

EnvironmentDocker environment + MCP gateway + snapshotsSQLite world + MCP runtime + sandbox copy

Agent runnerLoop/ReAct registry with sandbox lifecycleMulti-provider harness + long-horizon trajectory runner

TrajectoryMessages, tool calls, outputs, and artifact editsFull-fidelity episode JSONL, calls, outputs, errors, and state evidence

GradingStatic, domain, and LLM verifiers over outputs and snapshotsState, trace, VCode, rubric, composite, and full-world diff verifiers

Release gateDataset/task package contractOracle admission, random-floor probe, immutable Harbor export, and replay

Read Archipelago ↗Inspect this Blobfish world ↗Inspect the upstream suite ↗

Domain source

Enterprise-Bench L1–L2 contributes the upstream evaluation contract and 14 public tasks.

Independent world

The companion system is domain-aligned, generated with a fixed seed, and clearly labeled as separate from the source task set.

Two-sided probe

All 4 tasks must pass the reference replay and reject the random floor before release evidence is generated.

Model publication

No model row appears without a pinned model, agent, trials, runtime, costs, trajectories, and verifier outputs.

Across the portfolio

Keep testing the boundary.

View all seven →

Legal operationsCounselBench-100

100 source tasks

Financial researchFilingBench-50

50 source tasks

Customer operationsServiceBench-375

375 source tasks