All benchmarksCustomer operations

ServiceBench-375Evidence 2026-08-26Dataset page live

Can an agent follow policy while the customer and the system keep changing?

Multi-turn service-agent evaluation paired with an SLA, ticket, escalation, message, and knowledge-base world.

Explore all tasks ↓World repository ↗Run on Harbor ↗

375upstream Harbor tasks

4released world tasks

100%reference ceiling

0%seeded random floor

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces and reject the seeded random floor. No paid or frontier model run is claimed. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.009/9 successful

Seeded random floorSeeded random tool policy; not a model rank.

Discrimination floor0/40%0.0018/24 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% seeded-random floor shows the verifier does not simply reward any activity. Neither row measures model capability.

Task browser

Read every released task.

These are the four independent Blobfish diagnostic tasks in this world—not excerpts relabeled from the larger Harbor source suite.

train4 reference stepshard · generator label

cs_escalate_sla_breach

It is now minute 180. Ticket 9201 belongs to an enterprise-plan customer and still has no first response. Look up the first_response_minutes of the SLA policy for that customer's plan and work out how many minutes past the SLA deadline the response now is (deadline = opened_minute + first_response_minutes). Then set ticket 9201's priority to 'urgent' and status to 'escalated', create an escalation on ticket 9201 with reason 'sla_first_response_breach', tier 'tier2' and status 'open', and post a holding reply: a ticket message on 9201 from author 'agent' at sent_minute 180 whose body states the overdue amount as 'N minutes'.

Required tool surface

01search_sla_policies
02update_ticket
03create_escalation
04create_ticket_message

Reference: PASS
Random floor: FAIL
Verifier: vcode
Gradable: Yes

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in reference trace replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: cs_escalate_sla_breach
Steps: 4
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

search_sla_policiestool success

Arguments

{
  "plan": "enterprise",
  "limit": 10
}

Result

[
  {
    "policy_id": 9101,
    "name": "Enterprise SLA",
    "plan": "enterprise",
    "first_response_minutes": 30,
    "resolve_minutes": 240
  }
]

update_tickettool success

Arguments

{
  "ticket_id": 9201,
  "priority": "urgent",
  "status": "escalated"
}

Result

{
  "ticket_id": 9201,
  "customer_id": 9001,
  "subject": "Fleet controller offline",
  "status": "escalated",
  "priority": "urgent",
  "opened_minute": 100,
  "first_response_minute": null,
  "resolved_minute": null
}

create_escalationtool success

Arguments

{
  "ticket_id": 9201,
  "reason": "sla_first_response_breach",
  "tier": "tier2",
  "status": "open"
}

Result

{
  "escalation_id": 9402,
  "ticket_id": 9201,
  "reason": "sla_first_response_breach",
  "tier": "tier2",
  "status": "open"
}

create_ticket_messagetool success

Arguments

{
  "ticket_id": 9201,
  "author": "agent",
  "body": "Apologies — our first response is 50 minutes past the enterprise SLA. An engineer is now on this.",
  "sent_minute": 180,
  "channel": "email"
}

Result

{
  "message_id": 9303,
  "ticket_id": 9201,
  "author": "agent",
  "body": "Apologies — our first response is 50 minutes past the enterprise SLA. An engineer is now on this.",
  "sent_minute": 180,
  "channel": "email"
}

05
Deterministic verifierpassed
Basis
vcode
Reward
1.0
Gradable
yes

Released assets

Inspect the world down to each digest.

The executable database, schema, seed, tools, tasks, MCP manifest, and world manifest are checked into the same repository and linked individually.

Code + worldGitHub repositorybenchmark/worlds/servicebench-375 Runnable companionHarbor datasetblobfishai/domainbench-24 Companion datasetHugging FaceBlobfish DomainBench-24 Domain sourceUpstream Harbor suitesierra-research/tau3-bench

AssetTypeSizeSHA-256Open

Seeded SQLite snapshotPristine state copied for every episode.

SQLite32 KBf1c39d517436dd11…View ↗

Relational schemaHuman-readable table and constraint definitions.

SQL1.5 KBaae113321a7e8003…View ↗

Seed dataHuman-readable source for the released world state.

SQL15 KB9b5b097b96bcd595…View ↗

MCP tool catalogPublic tool names, schemas, and implementations.

JSON99 KBbbfffdac46d3bd61…View ↗

Task manifestInstructions, splits, reference traces, and isolated verifier programs.

JSON103 KB1a7d4d40cff99e0a…View ↗

MCP manifestStdio server declaration for the world.

JSON1.9 KBd7fc28450ed8676d…View ↗

World manifestContent identity, counts, split, and admission evidence.

JSON2.4 KB79dadcb06d047317…View ↗

Environment

A stateful MCP system, not a prompt file.

6 related entity tables expose 22 read-only and 21 mutating tools. Every episode gets a fresh copy of the 115-row seed state.

01Seed snapshotImmutable SQLite state
02MCP gateway43 typed tools
03Agent episodeIsolated scratch copy
04Trace + stateCalls and full diff
05VerifierDeterministic verdict

01Support Customerssupport_customers · 20 rows

02Sla Policiessla_policies · 19 rows

03Ticketstickets · 20 rows

04Ticket Messagesticket_messages · 19 rows

05Escalationsescalations · 18 rows

06Kb Articleskb_articles · 19 rows

PASStask admission

4/4reference replay

0/4random floor

Fresh copyper episode

MCP + CLI

Reproduce every layer locally.

Harbor preserves the upstream suite. Blobfish serves and evaluates the independent companion world. The reference policy is keyless; a model run requires an explicit provider and cost cap.

01Run this domain on Harbor

harbor run -d blobfishai/domainbench-24 \
  -i '*servicebench-375*' -a <agent> -m <provider/model>

02Run the upstream suite

harbor run -d sierra-research/tau3-bench
  -a <agent> -m <provider/model>

03Serve the world over MCP

blobfish info benchmark/worlds/servicebench-375
blobfish serve benchmark/worlds/servicebench-375

04Replay the reference ceiling

blobfish eval benchmark/worlds/servicebench-375 \
  --policy oracle --split all --run release-oracle

Methodology

Archipelago-shaped evaluation, with deterministic grading.

Mercor’s open-source Archipelago clarified the public contract: environment, agent runner, trajectory, and grading should be inspectable together. Blobfish already has the same layers, plus fail-closed state policies and Harbor release gates.

LayerArchipelagoBlobfish implementation

EnvironmentDocker environment + MCP gateway + snapshotsSQLite world + MCP runtime + sandbox copy

Agent runnerLoop/ReAct registry with sandbox lifecycleMulti-provider harness + long-horizon trajectory runner

TrajectoryMessages, tool calls, outputs, and artifact editsFull-fidelity episode JSONL, calls, outputs, errors, and state evidence

GradingStatic, domain, and LLM verifiers over outputs and snapshotsState, trace, VCode, rubric, composite, and full-world diff verifiers

Release gateDataset/task package contractOracle admission, random-floor probe, immutable Harbor export, and replay

Read Archipelago ↗Inspect this Blobfish world ↗Inspect the upstream suite ↗

Domain source

τ³-bench contributes the upstream evaluation contract and 375 public tasks.

Independent world

The companion system is domain-aligned, generated with a fixed seed, and clearly labeled as separate from the source task set.

Two-sided probe

All 4 tasks must pass the reference replay and reject the random floor before release evidence is generated.

Model publication

No model row appears without a pinned model, agent, trials, runtime, costs, trajectories, and verifier outputs.

Across the portfolio

Keep testing the boundary.

View all seven →

Legal operationsCounselBench-100

100 source tasks

Financial researchFilingBench-50

50 source tasks

Scientific discoveryDiscoveryBench-102

102 source tasks