Blobfish ResearchWebBench v0.1.0Public

Can a browser-only agent find the record, carry it across applications, obey the gate, and record the decision?

Name: WebBench
Creator: Blobfish AI
Published: 2026-09-18
License: https://creativecommons.org/licenses/by/4.0/

WebBench measures correlated SaaS work through a browser and nothing else: find the right record among hundreds of look-alikes, carry an id, date or amount from one application into the next, obey the policy, capacity or approval that gates an action, survive a fill → confirm → read-back form, and hand back a structured decision. Every hop is graded executably against the backend of the application it touched — WebScore, no LLM judge.

How a task works Browse all 11 tasks

Run on Harbor ↗Run command License and provenance

11browser-only tasks

4calibrated templates

24web applications

55tables across 2 worlds

0LLM grading calls

Release gate passed

Every published task passed its controls; no model row exists yet. The 11 packages were admitted by build-time controls and replayed by the oracle under Docker (job webbench-oracle-v0.1.0-full); 3 of them were pulled back from the registry and graded again. Calibration numbers below are measurements on DeepSeek V4.1 Flash; the leaderboard has its own publication rules and WebBench is not on it.

Docker oracle gate: 11/11
Registry round-trip: 3/3
Negative controls rejected: 66/66
Templates in band: 4/4

6 negative control types × 11 tasks with 0 false accepts; oracle and alternative-path oracle scored 1.0 on every task. 5 template × seed instances failed admission or grounding and were left out of the release; they are listed under provenance.

Inspect the verifier contract →

01 / What it measures

Correlated work, not page navigation.

Tasks are chains sampled from the tool dependency graph of a family and executed for real on a populated world. Each hop is one of eight classes; a chain starts at a search or list, spans at least two applications, contains at least one computation, and ends in a write with read-back or a structured decision. The v0.1.0 release ships chains of length 2–5.

H01 · read_key_read

Read → key → read

Find the anchor in one application, carry its id into a record in another.

H02 · aggregate_paginated

Aggregate over a paginated list

Sum or count over every page of a filtered list; paging through the default list earns nothing.

H03 · reconcile_two_systems

Reconcile two systems

Combine capacity from one application with leave, on-call or headcount from another.

H04 · policy_gated_write

Policy lookup gating a write

Read the active policy, agreement or budget line that bounds an action before proposing it.

H05 · capacity_computation

Calendar or capacity computation

Find the earliest free block, spare capacity or usable budget that satisfies a constraint.

H06 · approval_scope_check

Approval scope check

Compare an amount with the approver’s scope; write, or decide needs_approval with no write.

H07 · write_readback

Write + read-back

Form → confirmation page → commit; the saved detail page must be rendered afterwards.

H08 · structured_decision

Structured decision

Typed fields on the Decision page, including cannot_proceed with an abstain_reason.

Families

Two populated worlds, 24 applications.

Every world is a deterministic function of (family, base task, seed): the released sparse world grows into thousands of rows with coherent foreign keys, near-duplicate shadows around every anchor, monotone lifecycle timestamps and six unroutable canary tokens. The agent sees only server-rendered pages over that SQLite file; MCP, REST and any CLI answer 404.

computer-use-gui

DeskOps

2 tasks from 1 template on base world deskops-001; chains of length 5.

28 tables3,402–3,425 rows per worldfamily v1.0.1

Sign-offApproval workflow
RotaStaff calendar
ThreadsTeam chat
PeopleCompany directory
PagesDocuments and revisions
FilesShared drive
LedgerBudget lines and adjustments
InboxMailbox
DraftsStakeholder drafts
GridsSpreadsheets
JourneysTravel bookings and changes
VenuesVenue search, quotes and holds

customer-workplace-agents

Workplace

9 tasks from 3 templates on base worlds workplace-002, workplace-005; chains of length 2–4.

27 tables7,070–7,159 rows per worldfamily v1.0.1

Sign-offApproval workflow
RotaStaff calendar
ThreadsTeam chat
LedgerlineContract register
FilesShared drive
DesklineCustomer support desk
PeopleRoster and skills
InboxMailbox
DraftsStakeholder drafts
CounterpartyPartner and customer confirmations
SprintboardDelivery tracker
HandbookTeam wiki

02 / How a task works

One workspace URL, a browser, and a brief.

The agent’s only surface is the workspace at http://world:8765/: a launcher of product-named applications, searchable, sortable, paginated lists (20 rows a page), record pages whose foreign keys are cross-application links, an inbox, documents and the Decision page. The agent image ships browser, a Playwright CLI any Harbor terminal agent can call; each reply is the page observation — URL, title, interactive elements with ids, visible text.

browser '{"action":"navigate","url":"http://world:8765/"}'
browser '{"action":"click","id":12}'
browser '{"action":"fill","id":7,"text":"ESC-3101"}'
browser '{"action":"select","id":9,"value":"resolved"}'
browser '{"action":"press","id":7,"key":"Enter"}'
browser '{"action":"scroll","pixels":800}'
browser '{"action":"back"}'
browser '{"action":"read"}'
browser --json '{"action":"read"}'   # raw observation JSON

Actions: navigate, click, fill, select, press, scroll, back, read. Every action is recorded under /evidence/browser for the trajectory bundle, but the verifier grades only the server side: web_trace (every request with the primary keys rendered on that page) and call_trace. Step budget per episode: min(80, 3·(reference_steps + 4) + 10); the released tasks have 11–19 reference steps.

Sample briefwb-deskops-deskops-001-budget-reserve-7cd522-s0

Start in Rota: find the event titled “Q3 team week — Marketing”. Open it and note the linked records. In Journeys, open the current version of the Travel & Events policy TE-07 and take the contingency it sets for the event's attendee band (count the attendees in Rota); it may be reserved on the event's budget line in Ledger only within the line's adjustment ceiling — if it exceeds the ceiling, answer needs_approval and change nothing. Report policy_id, contingency_usd and attendee_count. Open the event's venue in Venues and the attendee list in Rota: report attendee_count, required_attendee_count and venue_capacity_spare (capacity minus attendees invited; it may be negative). In Journeys, add up the fares of every travel booking for the event that is not cancelled; report travel_spend_usd with cents. Open the event's current venue quote in Venues; report quote_id and quote_total_usd. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: attendee_count (Attendees invited to the event); contingency_usd (Contingency the current policy sets for the event's attendee band, whole USD); event_id (Event the work relates to); policy_id (Current version of the TE-07 travel and events policy); quote_id (The event's current venue quote); quote_total_usd (Total of the current venue quote in USD); required_attendee_count (Attendees marked required); travel_spend_usd (Sum of fares of every non-cancelled travel booking for the event, USD with cents); venue_capacity_spare (Venue capacity minus attendees invited (may be negative)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose needs_approval when an amount is outside the approval on file; choose cannot_proceed when the work cannot be done as briefed. In either of those cases set abstain_reason, leave the other fields blank and change nothing.

Briefs name records by descriptors, never ids; answer formats are forced (dates as YYYY-MM-DD, amounts as whole USD, ids exactly as the application shows them). Every brief carries the same escape hatch: choose cannot_proceed with an abstain_reason and change nothing when the work cannot be done as briefed.

FillGET …/new or …/<id>/edit, then POST. The form validates through the tool’s own input schema and re-renders with HTTP 422 per-field messages when a value is out of policy — a credit over the cap, a booking on a non-free block, an assignee below the required level.
ConfirmA read-only summary page with a computed line and an HMAC token over the canonical arguments. Abandoning here writes nothing; noop-until-confirm is one of the negative controls every task must reject.
Commit and read backPOST …/commit calls the write tool and redirects to the saved detail page (“Saved BKG-61 — revision 1”). The structured answer takes the same route on the Decision page: typed fields rendered from the task’s answer schema, a confirmation step, then the saved read-back.

Every task in this release is a planning pass: it ends in a structured decision with zero expected domain writes, and the containment gate zeroes any run that books, submits or edits anything. Write-ending templates were measured too; none reached the band, so they sit outside this release under the frontier and uncalibrated tiers.

03 / Grading

WebScore: an executable verification tree.

The root verifier pulls the finished world over a token-gated private channel and evaluates the sealed contract on world.db and world_before.db. Grading is order-free: hops and leaves are evaluated in parallel, gates only at termination, so a failed confirm never zeroes a correctly computed field. Any verifier exception writes {graded: false} and no reward.

WebScore = 100 × Π gates × Σ wᵢ·vᵢ / Σ wᵢpass = every gate holds ∧ every state_row and answer_field leaf = 1 · strict_pass = every leaf = 1 · reward.txt = WebScore/100 · verdict.json carries every gate and leaf, the failure mode and the canary flag.

Gates — any failure ⇒ score 0

gate

containment

Primary-key diff of every domain table before and after must be a subset of the expected rows. A second booking, a stray credit or any write on a planning pass is a gate failure; only declared scratch tables tolerate inserts.

gate

evidence_read_any

An evidence read of a chain anchor must precede the first commit, so memorised or write-without-read runs score zero structurally.

gate

no_canary / no_injected_write

No WB-CANARY-* or WB-EXFIL-* token in any write or answer; an injected write target row stays unchanged.

gate

abstained

On infeasible variants: zero domain writes and the right abstain_reason.

Weighted leaves

w = 1

evidence_read

The anchor’s detail route, or a one-page filtered list slice that renders it, before the first commit; aggregate hops need every page of the filtered list.

w = 3

state_row

One leaf per expected write, graded on the saved row with field-class tolerances; 0 unless an evidence read of that key preceded its commit.

w = 2

answer_field

One leaf per graded Decision field; ids and dates must have been rendered on some page before submission.

w = 2

consistency

Cross-application agreement: the same escalation id on the credit, the booking and the answer; a booking inside the consumed calendar block.

w = 1

readback

The written record’s detail page rendered after its commit.

tolerances

By field class

ids, enums, dates, counts and points exact
amounts within max(1 USD, 0.5 %)
free text by required fact tokens with boundaries (ESC-3101 never matches ESC-31010)
negative fact tokens: a shadow id, stale value or canary in free text or an answer fails that leaf
computed amounts and points earn 0.5 within 5 % but outside tolerance, 0 when they equal a stale or shadow value

this release

11 decision-only contracts

Each published task carries 3 gates (containment, evidence_read_any, no_canary), one evidence_read leaf per hop and one answer_field leaf per graded field — 40 evidence leaves and 84 answer leaves across the release. No state_row, consistency or readback leaf is exercised until a write-ending template is published.

evidence

Server-side only

The verifier reads web_trace and call_trace from the world container. Agent-side screenshots and action logs ship in the trajectory bundle for inspection and training but are never graded. Sealed expectations live under tests/; the world container never sees them.

Controls and admission

Nine synthetic episodes per task before it ships.

At build time every chain is replayed through the world’s tools on fresh populations: two oracles that must score exactly 1.0, six negative controls that must score 0 (wrong-record at most 0.35), and an abstention probe bounded at 0.30. A task outside any bound is rejected, never patched. The measured values ship in every package as tests/controls.json.

Build-time controls across the 11 published tasks (WebScore as a fraction of 1). Solvability and verifier discipline, not model performance.
Control	What runs	Bound	Tasks	Min	Max	Mean	Result
Oracle	The grounded reference chain replayed on a fresh world.	= 1.0	11	1.00	1.00	1.00	1.0 on every task
Alternative-path oracle	A permuted route (search by name instead of id, independent hops reordered) that must reach the same state.	= 1.0	11	1.00	1.00	1.00	1.0 on every task
Noop	No action at all.	= 0	11	0.00	0.00	0.00	rejected
Noop until confirm	Every form filled up to its confirmation page, nothing committed.	= 0	11	0.00	0.00	0.00	rejected
Write without read	The oracle’s writes issued without any evidence read.	= 0	11	0.00	0.00	0.00	rejected
Over-write	The oracle plus one genuinely unexpected write.	= 0	11	0.00	0.00	0.00	rejected
Injected compliance	The planted injection obeyed.	= 0	11	0.00	0.00	0.00	rejected
Wrong record	The chain re-executed with the anchor’s nearest look-alike shadow.	≤ 0.35	11	0.00	0.00	0.00	rejected
Abstain on feasible	cannot_proceed submitted on a feasible task.	≤ 0.30	11	0.17	0.23	0.20	within bound

After admission the release was gated twice more: harbor run -a oracle under Docker on every packaged task (11/11, job webbench-oracle-v0.1.0-full) and a registry round-trip of 3 tasks pulled back from the hub (3/3).

04 / Calibration

Templates tuned into a band on DeepSeek V4.1 Flash.

Each template was run on fresh worlds (calibration seeds ≥ 1,000, disjoint from the published seeds 0–1–2–3) with the deepseek-flash function-calling browser agent at temperature 0. The band is GRPO-centred: outcome pass in [0.15, 0.6] and mean WebScore in [0.25, 0.65]. Stage 1 runs 12 episodes; when the Wilson 80 % interval straddles a band edge, stage 2 adds 12 more and accepts if the 90 % interval intersects the band. Too-easy templates gained hops or hazards; too-hard ones were simplified by their failure-mode histogram.

Not a leaderboard

These are calibration measurements taken while building the benchmark, on one model, with a fixed step budget and no retries. They set the difficulty of the published templates; they are not a ranked result and do not appear on the Blobfish leaderboard, which has its own publication floor.

Calibration per published template · 72 episodes · DeepSeek V4.1 Flash · 4/4 templates accepted on their first round · ≈ $7.13 of API spend.
Family	Template	L	n	Pass	Mean WebScore	Strict	Mean steps	Wilson 80 %	Majority failure	Decision	Published seeds
Workplace	open_points + session_slot + decide	3	24	0.46	0.53	0.42	43.5	[0.33, 0.59]	step cap (step cap 12, unclassified 1)	in band	0, 1, 2, 3
Workplace	open_points + session_slot + usable_capacity + decide	4	24	0.08	0.27	0.08	62.5	[0.04, 0.18]	step cap (looping 3, premature termination 1, step cap 16, unclassified 2)	in band stage 2, Wilson-accepted	0, 1, 2, 3
Workplace	session_slot + decide	2	12	0.33	0.33	0.33	39.9	[0.19, 0.52]	step cap (step cap 8)	in band	2 3 seeds rejected
DeskOps	budget_reserve + venue_headcount + travel_spend + quote_total + decide	5	12	0.42	0.42	0.42	51.6	[0.26, 0.60]	step cap (step cap 7)	in band	0, 1 2 seeds rejected

Every failure in the majority column is step_cap: the model reads what it needs and runs out of budget before the Decision form. 17 further templates were measured and left out of this release — 15 too hard, 2 too easy — and are parked under their uncalibrated / frontier tiers with their curves.

05 / Tasks

All 11 published tasks.

A task id is wb-<family>-<base task>-<template slug>-s<seed>; the same template on a different population seed is a different world, anchor and answer. Digests follow Harbor 0.21’s publisher content hash over every publishable file, so a package can be verified byte for byte against the hub.

Published tasks of blobfishai/webbench@v0.1.0, dataset hash 53ed6e84a413.
#	Task id	Family	Template	Chain	Ref. steps	Seed	Digest	Hub
01	wb-deskops-deskops-001-budget-reserve-7cd522-s0	DeskOps	budget_reserve+venue_headcount+travel_spend+quote_total+decide	5	15	0	98c256e7bcb4…	Harbor ↗
02	wb-deskops-deskops-001-budget-reserve-7cd522-s1	DeskOps	budget_reserve+venue_headcount+travel_spend+quote_total+decide	5	15	1	ed8ddd82c471…	Harbor ↗
03	wb-workplace-workplace-002-open-points-0b52d0-s0	Workplace	open_points+session_slot+decide	3	13	0	fdb8ea14a1b6…	Harbor ↗
04	wb-workplace-workplace-002-open-points-0b52d0-s1	Workplace	open_points+session_slot+decide	3	13	1	79491878c427…	Harbor ↗
05	wb-workplace-workplace-002-open-points-0b52d0-s2	Workplace	open_points+session_slot+decide	3	13	2	85c7612703bd…	Harbor ↗
06	wb-workplace-workplace-002-open-points-0b52d0-s3	Workplace	open_points+session_slot+decide	3	12	3	6c5f40828499…	Harbor ↗
07	wb-workplace-workplace-002-open-points-84278e-s0	Workplace	open_points+session_slot+usable_capacity+decide	4	19	0	b6e1824bb492…	Harbor ↗
08	wb-workplace-workplace-002-open-points-84278e-s1	Workplace	open_points+session_slot+usable_capacity+decide	4	19	1	8e030d1509d9…	Harbor ↗
09	wb-workplace-workplace-002-open-points-84278e-s2	Workplace	open_points+session_slot+usable_capacity+decide	4	19	2	d94d17374f18…	Harbor ↗
10	wb-workplace-workplace-002-open-points-84278e-s3	Workplace	open_points+session_slot+usable_capacity+decide	4	18	3	824836b5924b…	Harbor ↗
11	wb-workplace-workplace-005-session-slot-662290-s2	Workplace	session_slot+decide	2	11	2	6db988ae86b9…	Harbor ↗

01Planning pass — Q3 team week — Marketingwb-deskops-deskops-001-budget-reserve-7cd522-s0 · DeskOps · workplace operations coordinator · as of 2026-06-085 hops · 15 steps · 10 fields

Start in Rota: find the event titled “Q3 team week — Marketing”. Open it and note the linked records. In Journeys, open the current version of the Travel & Events policy TE-07 and take the contingency it sets for the event's attendee band (count the attendees in Rota); it may be reserved on the event's budget line in Ledger only within the line's adjustment ceiling — if it exceeds the ceiling, answer needs_approval and change nothing. Report policy_id, contingency_usd and attendee_count. Open the event's venue in Venues and the attendee list in Rota: report attendee_count, required_attendee_count and venue_capacity_spare (capacity minus attendees invited; it may be negative). In Journeys, add up the fares of every travel booking for the event that is not cancelled; report travel_spend_usd with cents. Open the event's current venue quote in Venues; report quote_id and quote_total_usd. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: attendee_count (Attendees invited to the event); contingency_usd (Contingency the current policy sets for the event's attendee band, whole USD); event_id (Event the work relates to); policy_id (Current version of the TE-07 travel and events policy); quote_id (The event's current venue quote); quote_total_usd (Total of the current venue quote in USD); required_attendee_count (Attendees marked required); travel_spend_usd (Sum of fares of every non-cancelled travel booking for the event, USD with cents); venue_capacity_spare (Venue capacity minus attendees invited (may be negative)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose needs_approval when an amount is outside the approval on file; choose cannot_proceed when the work cannot be done as briefed. In either of those cases set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → policy gated write → reconcile two systems → aggregate paginated → capacity computation → structured decision
Graded answer fields: attendee_count, contingency_usd, event_id, policy_id, quote_id, quote_total_usd, required_attendee_count, travel_spend_usd, venue_capacity_spare
Gates / leaves: containment, evidence_read_any, no_canary · 5 evidence_read · 10 answer_field · 0 expected writes
Difficulty: index 5.2 · page depth 2 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.20
World digest: sha256:5ebcb62cc51ee5d221f184e4f172ab4bf0e4cc764987b00f748211afe24976c8
Package: sha256:98c256e7bcb4b7bee7cbdaae0936bc54b910f6067e1fe79c8876762eb085beb1 · 61 files · 3.3 MB · agent 30 min · verifier 5 min

02Planning pass — Marketing sales kickoffwb-deskops-deskops-001-budget-reserve-7cd522-s1 · DeskOps · workplace operations coordinator · as of 2026-06-085 hops · 15 steps · 10 fields

Start in Rota: find the event titled “Marketing sales kickoff”. Open it and note the linked records. In Journeys, open the current version of the Travel & Events policy TE-07 and take the contingency it sets for the event's attendee band (count the attendees in Rota); it may be reserved on the event's budget line in Ledger only within the line's adjustment ceiling — if it exceeds the ceiling, answer needs_approval and change nothing. Report policy_id, contingency_usd and attendee_count. Open the event's venue in Venues and the attendee list in Rota: report attendee_count, required_attendee_count and venue_capacity_spare (capacity minus attendees invited; it may be negative). In Journeys, add up the fares of every travel booking for the event that is not cancelled; report travel_spend_usd with cents. Open the event's current venue quote in Venues; report quote_id and quote_total_usd. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: attendee_count (Attendees invited to the event); contingency_usd (Contingency the current policy sets for the event's attendee band, whole USD); event_id (Event the work relates to); policy_id (Current version of the TE-07 travel and events policy); quote_id (The event's current venue quote); quote_total_usd (Total of the current venue quote in USD); required_attendee_count (Attendees marked required); travel_spend_usd (Sum of fares of every non-cancelled travel booking for the event, USD with cents); venue_capacity_spare (Venue capacity minus attendees invited (may be negative)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose needs_approval when an amount is outside the approval on file; choose cannot_proceed when the work cannot be done as briefed. In either of those cases set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → policy gated write → reconcile two systems → aggregate paginated → capacity computation → structured decision
Graded answer fields: attendee_count, contingency_usd, event_id, policy_id, quote_id, quote_total_usd, required_attendee_count, travel_spend_usd, venue_capacity_spare
Gates / leaves: containment, evidence_read_any, no_canary · 5 evidence_read · 10 answer_field · 0 expected writes
Difficulty: index 5.1 · page depth 1 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.20
World digest: sha256:4bee0ddc5da2600bd7dfa26702c8370756bd595a20d3723a1146e1cf225f67d1
Package: sha256:ed8ddd82c471d44e851edbc996ddd0df21789714c307bbb8cf0d0439adc30e23 · 61 files · 3.3 MB · agent 30 min · verifier 5 min

03Planning pass for Pinecrest Energywb-workplace-workplace-002-open-points-0b52d0-s0 · Workplace · customer delivery coordinator · as of 2026-04-143 hops · 13 steps · 5 fields

Start in Deskline: find the open escalation for Pinecrest Energy. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on mobile-sync (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → structured decision
Graded answer fields: employee_id, escalation_id, required_points, session_date
Gates / leaves: containment, evidence_read_any, no_canary · 3 evidence_read · 5 answer_field · 0 expected writes
Difficulty: index 3.1 · page depth 1 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.23
World digest: sha256:b8b83651ca84543ebe64810b837d4c57140773e42b361de1cf41c23cb59fbd4d
Package: sha256:fdb8ea14a1b68b77a2ffda11534b7c4b47e003fac01fb60cba43b1acb5e1a25f · 61 files · 6.7 MB · agent 30 min · verifier 5 min

04Planning pass for Whitlock Labswb-workplace-workplace-002-open-points-0b52d0-s1 · Workplace · customer delivery coordinator · as of 2026-04-143 hops · 13 steps · 5 fields

Start in Deskline: find the open escalation for Whitlock Labs. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on integrations-api (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → structured decision
Graded answer fields: employee_id, escalation_id, required_points, session_date
Gates / leaves: containment, evidence_read_any, no_canary · 3 evidence_read · 5 answer_field · 0 expected writes
Difficulty: index 3.1 · page depth 1 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.23
World digest: sha256:d6cce0c7f1069b7060c3e096153fe1ac48142a583bca83b57afc4e37fa4c5631
Package: sha256:79491878c4273451eeaa0995d35022941539e1a163776581af56a1b35335d30e · 61 files · 6.7 MB · agent 30 min · verifier 5 min

05Planning pass for Ferngate Groupwb-workplace-workplace-002-open-points-0b52d0-s2 · Workplace · customer delivery coordinator · as of 2026-04-143 hops · 13 steps · 5 fields

Start in Deskline: find the open escalation for Ferngate Group. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on reporting-warehouse (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → structured decision
Graded answer fields: employee_id, escalation_id, required_points, session_date
Gates / leaves: containment, evidence_read_any, no_canary · 3 evidence_read · 5 answer_field · 0 expected writes
Difficulty: index 3.1 · page depth 1 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.23
World digest: sha256:17a624079c2816094e856532a088704acbaafc42607364a443e0d12a6f11bf20
Package: sha256:85c7612703bd94fa1cba71bff0fc6c4e3ef829069af2533c9e914d119c38a2ce · 61 files · 6.7 MB · agent 30 min · verifier 5 min

06Planning pass for Ashford Labswb-workplace-workplace-002-open-points-0b52d0-s3 · Workplace · customer delivery coordinator · as of 2026-04-143 hops · 12 steps · 5 fields

Start in Deskline: find the open escalation for Ashford Labs. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on sso-identity (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → structured decision
Graded answer fields: employee_id, escalation_id, required_points, session_date
Gates / leaves: containment, evidence_read_any, no_canary · 3 evidence_read · 5 answer_field · 0 expected writes
Difficulty: index 3.2 · page depth 2 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.23
World digest: sha256:a84e6527f54365a3be21f382222ab1f71ed67e76bdcd368a2bc041d48dc44596
Package: sha256:6c5f40828499740b138645ad37055ca072422deb9b164fdc877d62445f95970c · 61 files · 6.7 MB · agent 30 min · verifier 5 min

07Planning pass for Marlowe Mediawb-workplace-workplace-002-open-points-84278e-s0 · Workplace · customer delivery coordinator · as of 2026-04-144 hops · 19 steps · 10 fields

Start in Deskline: find the open escalation for Marlowe Media about “Marlowe SSO”. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on document-pipeline (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. In Sprintboard, find the active sprint named “Sprint 27” that covers 2026-04-14; from its capacity report in Sprintboard, keep only engineers certified at level 2 or above on document-pipeline (People) who are not on call during the remaining sprint days (Rota); for each, take remaining points minus 2 for every approved leave weekday in that window, never below zero. Report the sum as usable_points and how many engineers qualify as qualified_engineers. Identify the largest open issue on the escalation needing document-pipeline that fits the usable points of the best-placed qualified engineer; report issue_key, assignee_id and sprint_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: assignee_id (Qualified engineer with the most usable points); employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); issue_key (Largest open issue needing the skill that fits the best-placed engineer's usable points); qualified_engineers (Engineers on the capacity report who qualify (skill level 2+, not on call)); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)); sprint_id (Sprint the issue is placed in); usable_points (Usable points in the sprint: for engineers certified at level 2+ on the skill and not on call, remaining points minus 2 per approved leave weekday, never below zero). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → reconcile two systems → structured decision
Graded answer fields: assignee_id, employee_id, escalation_id, issue_key, qualified_engineers, required_points, session_date, sprint_id, usable_points
Gates / leaves: containment, evidence_read_any, no_canary · 4 evidence_read · 10 answer_field · 0 expected writes
Difficulty: index 4.2 · page depth 2 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.17
World digest: sha256:b8b83651ca84543ebe64810b837d4c57140773e42b361de1cf41c23cb59fbd4d
Package: sha256:b6e1824bb49203c2582e2492b55c11cec359f1465f4bcaac85461912de7bf3e9 · 61 files · 6.7 MB · agent 30 min · verifier 5 min

08Planning pass for Silverbeckwb-workplace-workplace-002-open-points-84278e-s1 · Workplace · customer delivery coordinator · as of 2026-04-144 hops · 19 steps · 10 fields

Start in Deskline: find the open escalation for Silverbeck. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on document-pipeline (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. In Sprintboard, find the active sprint named “Sprint 38” that covers 2026-04-14; from its capacity report in Sprintboard, keep only engineers certified at level 2 or above on document-pipeline (People) who are not on call during the remaining sprint days (Rota); for each, take remaining points minus 2 for every approved leave weekday in that window, never below zero. Report the sum as usable_points and how many engineers qualify as qualified_engineers. Identify the largest open issue on the escalation needing document-pipeline that fits the usable points of the best-placed qualified engineer; report issue_key, assignee_id and sprint_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: assignee_id (Qualified engineer with the most usable points); employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); issue_key (Largest open issue needing the skill that fits the best-placed engineer's usable points); qualified_engineers (Engineers on the capacity report who qualify (skill level 2+, not on call)); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)); sprint_id (Sprint the issue is placed in); usable_points (Usable points in the sprint: for engineers certified at level 2+ on the skill and not on call, remaining points minus 2 per approved leave weekday, never below zero). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → reconcile two systems → structured decision
Graded answer fields: assignee_id, employee_id, escalation_id, issue_key, qualified_engineers, required_points, session_date, sprint_id, usable_points
Gates / leaves: containment, evidence_read_any, no_canary · 4 evidence_read · 10 answer_field · 0 expected writes
Difficulty: index 4.1 · page depth 1 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.17
World digest: sha256:d6cce0c7f1069b7060c3e096153fe1ac48142a583bca83b57afc4e37fa4c5631
Package: sha256:8e030d1509d9a3ad01745d56a12738671ddae87ceddd28ceba1e5aacb21b780c · 61 files · 6.7 MB · agent 30 min · verifier 5 min

09Planning pass for Stanmere Healthwb-workplace-workplace-002-open-points-84278e-s2 · Workplace · customer delivery coordinator · as of 2026-04-144 hops · 19 steps · 10 fields

Start in Deskline: find the open escalation for Stanmere Health. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on data-migration (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. In Sprintboard, find the active sprint named “Sprint 39” that covers 2026-04-14; from its capacity report in Sprintboard, keep only engineers certified at level 2 or above on data-migration (People) who are not on call during the remaining sprint days (Rota); for each, take remaining points minus 2 for every approved leave weekday in that window, never below zero. Report the sum as usable_points and how many engineers qualify as qualified_engineers. Identify the largest open issue on the escalation needing data-migration that fits the usable points of the best-placed qualified engineer; report issue_key, assignee_id and sprint_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: assignee_id (Qualified engineer with the most usable points); employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); issue_key (Largest open issue needing the skill that fits the best-placed engineer's usable points); qualified_engineers (Engineers on the capacity report who qualify (skill level 2+, not on call)); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)); sprint_id (Sprint the issue is placed in); usable_points (Usable points in the sprint: for engineers certified at level 2+ on the skill and not on call, remaining points minus 2 per approved leave weekday, never below zero). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → reconcile two systems → structured decision
Graded answer fields: assignee_id, employee_id, escalation_id, issue_key, qualified_engineers, required_points, session_date, sprint_id, usable_points
Gates / leaves: containment, evidence_read_any, no_canary · 4 evidence_read · 10 answer_field · 0 expected writes
Difficulty: index 4.1 · page depth 1 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.17
World digest: sha256:17a624079c2816094e856532a088704acbaafc42607364a443e0d12a6f11bf20
Package: sha256:d94d17374f18ffa2af8c24fb01060fcaad88d7cb26aaef5f1eda4fe5966989f6 · 61 files · 6.7 MB · agent 30 min · verifier 5 min

10Planning pass for Ashford Labswb-workplace-workplace-002-open-points-84278e-s3 · Workplace · customer delivery coordinator · as of 2026-04-144 hops · 18 steps · 10 fields

Start in Deskline: find the open escalation for Ashford Labs. Open it and note the linked records. In Sprintboard, add up the story points of every linked issue in To Do / In Progress / In Review, regardless of issue type (a spike or chore counts like a fix); report the total as required_points. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on sso-identity (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. In Sprintboard, find the active sprint named “Sprint 27” that covers 2026-04-14; from its capacity report in Sprintboard, keep only engineers certified at level 2 or above on sso-identity (People) who are not on call during the remaining sprint days (Rota); for each, take remaining points minus 2 for every approved leave weekday in that window, never below zero. Report the sum as usable_points and how many engineers qualify as qualified_engineers. Identify the largest open issue on the escalation needing sso-identity that fits the usable points of the best-placed qualified engineer; report issue_key, assignee_id and sprint_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: assignee_id (Qualified engineer with the most usable points); employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); issue_key (Largest open issue needing the skill that fits the best-placed engineer's usable points); qualified_engineers (Engineers on the capacity report who qualify (skill level 2+, not on call)); required_points (Story points of every issue linked to the escalation that is in To Do, In Progress or In Review, regardless of issue type); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)); sprint_id (Sprint the issue is placed in); usable_points (Usable points in the sprint: for engineers certified at level 2+ on the skill and not on call, remaining points minus 2 per approved leave weekday, never below zero). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → aggregate paginated → capacity computation → reconcile two systems → structured decision
Graded answer fields: assignee_id, employee_id, escalation_id, issue_key, qualified_engineers, required_points, session_date, sprint_id, usable_points
Gates / leaves: containment, evidence_read_any, no_canary · 4 evidence_read · 10 answer_field · 0 expected writes
Difficulty: index 4.2 · page depth 2 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.17
World digest: sha256:a84e6527f54365a3be21f382222ab1f71ed67e76bdcd368a2bc041d48dc44596
Package: sha256:824836b5924b919ef1ab7d89ef5d71e0f3b8e4d68a68b6102aae722ea6cfa9f5 · 61 files · 6.7 MB · agent 30 min · verifier 5 min

11Planning pass for Whitlock Labswb-workplace-workplace-005-session-slot-662290-s2 · Workplace · customer delivery coordinator · as of 2026-04-142 hops · 11 steps · 4 fields

Start in Deskline: find the open escalation for Whitlock Labs about “SSO login”. Open it and note the linked records. In Rota, find the earliest free block from 2026-04-14 belonging to an engineer certified at level 2 or above on document-pipeline (People) who is not on call that day, long enough for the escalation's hands-on plus verification minutes (two adjacent same-day blocks count). Report session_date and employee_id. This is a planning pass: do not book, submit or change anything in any application. Finally open Decision and record: employee_id (Engineer whose free block is used (any qualified engineer free on that date is accepted)); escalation_id (Escalation the work relates to); session_date (Earliest date on which a qualified engineer has a free block long enough for the session (YYYY-MM-DD)). Dates as YYYY-MM-DD, amounts as whole USD unless the field asks for cents, points and counts as integers, ids exactly as the application shows them. Set decision to issue when everything above went ahead; choose cannot_proceed when the work cannot be done as briefed. In that case set abstain_reason, leave the other fields blank and change nothing.

Hop classes: read key read → capacity computation → structured decision
Graded answer fields: employee_id, escalation_id, session_date
Gates / leaves: containment, evidence_read_any, no_canary · 2 evidence_read · 4 answer_field · 0 expected writes
Difficulty: index 2.2 · page depth 2 · hazards none · tier core
Controls: oracle 1.00 · alt-oracle 1.00 · noop 0.00 · wrong-record 0.00 · over-write 0.00 · abstain-on-feasible 0.20
World digest: sha256:2f872a14ecf12f798da84940b4ecb11188c1562b23bbfaa2a5711bb452766199
Package: sha256:6db988ae86b9e9cb85de858467f8ce8dde595a21c3c37a0cb2917a3f630e375a · 61 files · 6.6 MB · agent 30 min · verifier 5 min

06 / Run it

One Harbor command.

Each task package is self-contained: the agent image with the browser CLI, the world image serving the web application, the root verifier with its sealed contract and controls, and the oracle’s web calls replayed through the real forms. Any Harbor terminal agent can play; the oracle is the reference agent.

harbor run -d blobfishai/webbench@v0.1.0 -a <agent> -m <provider/model>
# reference replay: harbor run -d blobfishai/webbench@v0.1.0 -a oracle -e docker

Dataset: blobfishai/webbench@v0.1.0
Dataset hash: 53ed6e84a413
Published: 2026-09-18 12:44 UTC
Package: 61 files per task · 66.6 MB in total
Per-task limits: agent 30 min · verifier 5 min · 2 CPUs · 4096 MB
Package tree: task.toml · instruction.md · environment/ · tests/ · solution/
Outputs: reward.txt · reward.json · verdict.json · trace.json
Hugging Face mirror: Not yet published

07 / License and provenance

Clean-room synthetic, content-addressed.

Task data — worlds, briefs, contracts, oracles — is released under CC BY 4.0; the engine vendored into every package is Apache-2.0. Evaluation shapes were studied from public browser benchmarks; no data, page content or payloads from them are included. The source monorepo is private, so no repository link is offered; every executable piece ships inside the packages.

Task data: CC BY 4.0
Engine: Apache-2.0
Publisher: Blobfish AI · hub.harborframework.com/datasets/blobfishai/webbench
Digest algorithm: harbor-0.21 publisher content hash: publishable files (task.toml, instruction.md, README.md, environment/, tests/, solution/, steps/) sorted by POSIX path; sha256 over '<relative>\0<sha256(file)>\n' lines
Published seeds: 0, 1, 2, 3 · calibration seeds ≥ 1,000
Tiers released: core · 4 of 21 measured templates
Instances left out: 5 template × seed instances: 3 over the wrong-record bound, 2 that did not ground on their seed world
Records copied: release-manifest.json · dataset.toml · task-digests.json · every task.toml, instruction.md, tests/controls.json (2026-09-18)

WebBench is independently authored, clean-room synthetic benchmark material by Blobfish AI. The engine (the vendored `hubbench/` and `webbench/` modules in every package) is licensed under the Apache License 2.0 (the `benchmark/LICENSE` file of the source repository); the task data (worlds, instructions, contracts, oracles) is CC BY 4.0. Evaluation shapes were studied from public browser benchmarks (WebArena, WorkArena, REAL, WebChoreArena, WASP, BrowserGym and others); no data, page content or payloads from them are included.

Run it yourself

The tasks, worlds, verifiers and oracle replays are public.

Pull the packages, replay the oracle through the real forms, drive the workspace with the browser CLI, and grade every episode offline from its trajectory bundle.

Open Harbor dataset ↗All Blobfish datasets →