All benchmarksCustomer operations

ServiceBench-375Evidence 2026-08-29Independent companionDataset page live

Can an agent follow policy while the customer and the system keep changing?

Multi-turn service-agent evaluation paired with an SLA, ticket, escalation, message, and knowledge-base world.

Explore 4 companion tasks ↓World repository ↗Run on Harbor ↗

375upstream Harbor tasks

4independent companion tasks

28files per task room

57+criteria per task

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces, exact replays, and reject thirteen task-applicable controls including correct state with wrong reasoning, wrong target, and keyword stuffing. No paid or frontier model run is claimed. These 4 tasks are Blobfish-authored companions, not conversions of the 375-task upstream suite. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.00217/217 successful

Wrong-evidence causal controlExact-release deterministic negative control; not a model rank.

Adversarial discrimination floor0/40%0.00217/217 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% wrong-evidence floor shows the verifier rejects a plausible workflow that relies on the wrong source record. Neither row measures model capability.

Task browser

Read every released task.

These are 4 independent Blobfish v3.1 realism tasks—not excerpts relabeled from the upstream suite. Each is a high-level employee request backed by a multi-source evidence room, three operating options, a task-specific causal evaluation narrative, and deterministic state checks.

train60 reference stepshard · generator label

cs_escalate_sla_breach

Bluepeak Robotics says its fleet issue has gone unanswered, and the account team thinks we are already outside the contractual first-response window. Can you establish whether we have actually breached the customer's applicable commitment as of now? If we have, make the bounded escalation the service policy calls for and send the customer an honest update. Give the service lead the amount of time overdue and enough evidence to defend the action; if the clock has not expired, do not manufacture an escalation.

How the employee outcome is evaluated

Evidence has to support the decision, the write, and the final answer.

Provider evidence: 26 exact reads
Live records: 25 correlated reads
Supported writes: 4
Proof: 4 readbacks

Reasoning contract

It is now minute 180. Ticket 9201 belongs to an enterprise-plan customer and still has no first response. Look up the first_response_minutes of the SLA policy for that customer's plan and work out how many minutes past the SLA deadline the response now is (deadline = opened_minute + first_response_minutes). Then set ticket 9201's priority to 'urgent' and status to 'escalated', create an escalation on ticket 9201 with reason 'sla_first_response_breach', tier 'tier2' and status 'open', and post a holding reply: a ticket message on 9201 from author 'agent' at sent_minute 180 whose body states the overdue amount as 'N minutes'.

Deterministic success

All exact provider and live-domain evidence is read before the first write; the task-specific predicates support the chosen branch; only the authorized state delta and one completion row are created; the persisted result is reopened.

Exact total call order is not graded. Equivalent investigation is accepted when the same task-specific evidence is established before mutation, the supported state is produced, and get_escalation → get_ticket_message → get_ticket → get_task_result reopens every changed provider row and the persisted result.

Required evidence systems

01gmail.messages.list
02gmail.messages.get
03google_drive.files.list
04google_drive.files.get
05google_sheets.spreadsheets.values.get
06slack.search_messages
07slack.conversations_replies
08search_support_customers
09count_support_customers
10get_support_customer
11search_tickets
12count_tickets
13get_ticket
14search_sla_policies
15count_sla_policies
16get_sla_policy
17search_ticket_messages
18count_ticket_messages
19get_ticket_message
20search_escalations
21count_escalations

Reference: PASS
Wrong-evidence floor: FAIL
Verifier: exact final tables, argument-aware causal trace, persisted result, and readback
Gradable: Yes

Operating options

✓
Complete the bounded supported outcomeCurrent authority, usable live evidence, and the available internal review window all support this task-scoped outcome.
02
Hold for an independent evidence reviewThis remains within authority but misses the documented need date even though the current evidence has no unresolved control conflict.
03
Extend remediation to neighboring recordsThe wider change exceeds both the approved immutable-record scope and the current incremental-cost authority.

Evidence files: 28
Exact criteria: 65
Allowed writes: 4
Call order: Not graded

Task evidenceTypeSizeSHA-256Open

operating-policy.mdassets/cs_escalate_sla_breach/operating-policy.md

MD1.2 KBa17a292cbe31e2b2…View ↗

source-request.emlassets/cs_escalate_sla_breach/source-request.eml

EML878 B5951b74cb9c03405…View ↗

customer-service-policy.pdfassets/cs_escalate_sla_breach/customer-service-policy.pdf

PDF930 B3cec0ed90f0acc26…View ↗

sla-and-queue-review.xlsxassets/cs_escalate_sla_breach/sla-and-queue-review.xlsx

XLSX8.8 KBac7d98cf99b173e8…View ↗

approval-decision.jsonassets/cs_escalate_sla_breach/approval-decision.json

JSON713 B5b47b9e793b130ec…View ↗

team-slack-intake.jsonassets/cs_escalate_sla_breach/team-slack-intake.json

JSON823 Bc4704a1890a11482…View ↗

drive-source-index.jsonassets/cs_escalate_sla_breach/drive-source-index.json

JSON377 B00c7e29811cec3e8…View ↗

provider-contracts.jsonassets/cs_escalate_sla_breach/provider-contracts.json

JSON21.1 KB9a171bef6c4f0cd3…View ↗

former-owner-suggestion.emlassets/cs_escalate_sla_breach/former-owner-suggestion.eml

EML619 B578b55ccbc53c999…View ↗

approval-scope.emlassets/cs_escalate_sla_breach/approval-scope.eml

EML824 B70e44dce74bde646…View ↗

operations-slack-thread.jsonassets/cs_escalate_sla_breach/operations-slack-thread.json

JSON432 B4259256f94f3d57d…View ↗

drive-approval-record.jsonassets/cs_escalate_sla_breach/drive-approval-record.json

JSON532 Bc3955457ff3aaaf5…View ↗

authority-revision-2026-3.mdassets/cs_escalate_sla_breach/authority-revision-2026-3.md

MD279 B0699ea79c87f032a…View ↗

authority-revision-2025-4.mdassets/cs_escalate_sla_breach/authority-revision-2025-4.md

MD221 B7a150b5272ce3b62…View ↗

control-standard-2025-4.pdfassets/cs_escalate_sla_breach/control-standard-2025-4.pdf

PDF817 B496930c8c515e49e…View ↗

review-calendar.xlsxassets/cs_escalate_sla_breach/review-calendar.xlsx

XLSX2.1 KB79e48a78a21d1523…View ↗

record-lineage.csvassets/cs_escalate_sla_breach/record-lineage.csv

CSV510 B37b7fcdb256ad2e4…View ↗

system-audit.logassets/cs_escalate_sla_breach/system-audit.log

LOG871 B4908cbdc171af4ec…View ↗

live-snapshot.jsonassets/cs_escalate_sla_breach/live-snapshot.json

JSON14.2 KBc0c75fb363eedc7c…View ↗

evidence-status.yamlassets/cs_escalate_sla_breach/evidence-status.yaml

YAML222 B47227a9821419520…View ↗

custodian-notes.txtassets/cs_escalate_sla_breach/custodian-notes.txt

TXT208 B8ac5245b6a5ef890…View ↗

support_customers-system-export.csvassets/cs_escalate_sla_breach/support_customers-system-export.csv

CSV1.9 KB760ea9e4253908c2…View ↗

tickets-system-export.csvassets/cs_escalate_sla_breach/tickets-system-export.csv

CSV1.9 KB0d4d17185ac85841…View ↗

sla_policies-system-export.csvassets/cs_escalate_sla_breach/sla_policies-system-export.csv

CSV1.9 KB3dafd310ea9866b2…View ↗

ticket_messages-system-export.csvassets/cs_escalate_sla_breach/ticket_messages-system-export.csv

CSV4.4 KB5d497d94e6a63580…View ↗

escalations-system-export.csvassets/cs_escalate_sla_breach/escalations-system-export.csv

CSV1.6 KBfe5f931516aaca3d…View ↗

kb_articles-system-export.csvassets/cs_escalate_sla_breach/kb_articles-system-export.csv

CSV4.5 KB9b942a50f95bc230…View ↗

asset-manifest.jsonassets/cs_escalate_sla_breach/asset-manifest.json

JSON4.4 KBb61f4ab79059d670…View ↗

Inspect all 65 deterministic criteria

01
identity · identity_escalationsFor cs_escalate_sla_breach, resolve escalations by immutable escalation_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_escalations"}, {"args": {}, "tool": "count_escalations"}, {"args": {"limit": 50, "ticket_id": 9201}, "tool": "search_escalations"}, {"args": {"ticket_id": 9201}, "tool": "count_escalations"}].
02
investigation · investigation_escalationsEstablish the relevant and competing escalations records before deciding the cs_escalate_sla_breach outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
03
identity · identity_sla_policiesFor cs_escalate_sla_breach, resolve sla_policies by immutable policy_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_sla_policies"}, {"args": {}, "tool": "count_sla_policies"}, {"args": {"limit": 50, "name": "Enterprise SLA"}, "tool": "search_sla_policies"}, {"args": {"name": "Enterprise SLA"}, "tool": "count_sla_policies"}, {"args": {"policy_id": 9101}, "tool": "get_sla_policy"}, {"args": {"limit": 10, "plan": "enterprise"}, "tool": "search_sla_policies"}].
04
investigation · investigation_sla_policiesEstablish the relevant and competing sla_policies records before deciding the cs_escalate_sla_breach outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
05
identity · identity_support_customersFor cs_escalate_sla_breach, resolve support_customers by immutable customer_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_support_customers"}, {"args": {}, "tool": "count_support_customers"}, {"args": {"limit": 50, "name": "Bluepeak Robotics"}, "tool": "search_support_customers"}, {"args": {"name": "Bluepeak Robotics"}, "tool": "count_support_customers"}, {"args": {"customer_id": 9001}, "tool": "get_support_customer"}].
06
investigation · investigation_support_customersEstablish the relevant and competing support_customers records before deciding the cs_escalate_sla_breach outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
07
identity · identity_ticket_messagesFor cs_escalate_sla_breach, resolve ticket_messages by immutable message_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_ticket_messages"}, {"args": {}, "tool": "count_ticket_messages"}, {"args": {"limit": 50, "ticket_id": 9201}, "tool": "search_ticket_messages"}, {"args": {"ticket_id": 9201}, "tool": "count_ticket_messages"}, {"args": {"message_id": 9301}, "tool": "get_ticket_message"}].
08
investigation · investigation_ticket_messagesEstablish the relevant and competing ticket_messages records before deciding the cs_escalate_sla_breach outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
09
identity · identity_ticketsFor cs_escalate_sla_breach, resolve tickets by immutable ticket_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_tickets"}, {"args": {}, "tool": "count_tickets"}, {"args": {"customer_id": 9001, "limit": 50}, "tool": "search_tickets"}, {"args": {"customer_id": 9001}, "tool": "count_tickets"}, {"args": {"ticket_id": 9201}, "tool": "get_ticket"}].
10
investigation · investigation_ticketsEstablish the relevant and competing tickets records before deciding the cs_escalate_sla_breach outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
11
investigation · investigation_provider_gmailFor cs_escalate_sla_breach, use task-scoped gmail discovery and exact record reads to inspect source-request.eml (msg-e03774d4fb6a840b7773), former-owner-suggestion.eml (msg-6871a5c44fab7d8021d3), approval-scope.eml (msg-dc90a4c698690fb4b5f5). The remaining 0 listed gmail records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"maxResults": 100, "q": "\"DB24-SERVICEBENCH-375-CS_ESCALATE_SLA_BREACH\"", "userId": "me"}, "tool": "gmail.messages.list"}, {"args": {"format": "raw", "id": "msg-e03774d4fb6a840b7773", "userId": "me"}, "tool": "gmail.messages.get"}, {"args": {"format": "raw", "id": "msg-6871a5c44fab7d8021d3", "userId": "me"}, "tool": "gmail.messages.get"}, {"args": {"format": "raw", "id": "msg-dc90a4c698690fb4b5f5", "userId": "me"}, "tool": "gmail.messages.get"}].
12
investigation · investigation_provider_google_driveFor cs_escalate_sla_breach, use task-scoped google_drive discovery and exact record reads to inspect operating-policy.md (file-cc93d194ef55489378e9), customer-service-policy.pdf (file-f73e51a20b918a55986e), approval-decision.json (file-1eed20baeac829904ba6), drive-source-index.json (file-ca3ba70ad52bf7cd34ab), drive-approval-record.json (file-a0fa18cac1ad466d9ac2), authority-revision-2026-3.md (file-e382c605326d17ec3009), authority-revision-2025-4.md (file-e52576f8afeae220e8ec), record-lineage.csv (file-24660208b49359388709), live-snapshot.json (file-54512f4a7d32f3c0a4b3), support_customers-system-export.csv (file-3a7111786ef3f5819250), tickets-system-export.csv (file-a311ca3a4bb10b2140d5), sla_policies-system-export.csv (file-f4bcafe9a879e28209b2), ticket_messages-system-export.csv (file-c70b5d1aa0c53ff57280), escalations-system-export.csv (file-b9a585a1dea79eb6d724), kb_articles-system-export.csv (file-463a367ae757a7c5eafd), asset-manifest.json (file-538d71d2ca41522f1e08). The remaining 5 listed google_drive records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"fields": "files(id,name,mimeType,modifiedTime,version)", "pageSize": 1000, "q": "fullText contains 'DB24-SERVICEBENCH-375-CS_ESCALATE_SLA_BREACH'"}, "tool": "google_drive.files.list"}, {"args": {"alt": "media", "fileId": "file-cc93d194ef55489378e9"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-f73e51a20b918a55986e"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-1eed20baeac829904ba6"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-ca3ba70ad52bf7cd34ab"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-b9a585a1dea79eb6d724"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-463a367ae757a7c5eafd"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-f4bcafe9a879e28209b2"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-3a7111786ef3f5819250"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-c70b5d1aa0c53ff57280"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-a311ca3a4bb10b2140d5"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-a0fa18cac1ad466d9ac2"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-e382c605326d17ec3009"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-e52576f8afeae220e8ec"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-24660208b49359388709"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-54512f4a7d32f3c0a4b3"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-538d71d2ca41522f1e08"}, "tool": "google_drive.files.get"}].
13
investigation · investigation_provider_google_sheetsFor cs_escalate_sla_breach, use task-scoped google_sheets discovery and exact record reads to inspect sla-and-queue-review.xlsx (sheet-2d36b0b7ee8ac04aee9a), review-calendar.xlsx (sheet-b9879849ffa8f571cfe9). The remaining 0 listed google_sheets records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"range": "A:Z", "spreadsheetId": "sheet-2d36b0b7ee8ac04aee9a"}, "tool": "google_sheets.spreadsheets.values.get"}, {"args": {"range": "A:Z", "spreadsheetId": "sheet-b9879849ffa8f571cfe9"}, "tool": "google_sheets.spreadsheets.values.get"}].
14
investigation · investigation_provider_slackFor cs_escalate_sla_breach, use task-scoped slack discovery and exact record reads to inspect team-slack-intake.json (thread-1ad259843687a8ecd26c), operations-slack-thread.json (thread-93e2889f8cd34dde3af3). The remaining 0 listed slack records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"count": 100, "query": "\"DB24-SERVICEBENCH-375-CS_ESCALATE_SLA_BREACH\"", "sort": "timestamp", "sort_dir": "asc"}, "tool": "slack.search_messages"}, {"args": {"channel": "#support-escalations", "limit": 100, "ts": "1724937600.060243"}, "tool": "slack.conversations_replies"}, {"args": {"channel": "#support-escalations", "limit": 100, "ts": "1724937600.789045"}, "tool": "slack.conversations_replies"}].
15
authority · authority_effective_revisionUse the effective 2026.3 operating control for cs_escalate_sla_breach, corroborated by the approval and Drive revision records. Enforced by current-control PDF, Drive record, and policy revision identity.
16
authority · authority_retired_conflictTreat the retained 2025.4 note for cs_escalate_sla_breach as a competing historical hypothesis, not as permission or current authority. Enforced by current-versus-retired revision comparison.
17
correlation · correlation_cross_system_identityCorrelate the employee request, controlled workbook, system exports, Slack discussion, Drive record, and approval email for cs_escalate_sla_breach by case and immutable record IDs. Enforced by cross-file case IDs and exact world selectors.
18
correlation · correlation_live_over_snapshotReconcile the released snapshot for cs_escalate_sla_breach to successful live MCP reads before relying on either source; the files alone do not establish the final outcome. Enforced by asset digests plus exact live trace.
19
correlation · correlation_count_detail_relationshipFor cs_escalate_sla_breach, reconcile broad counts with detail records and foreign-key relationships so duplicates and similarly named distractors are not collapsed. Enforced by search/count/get selector intersection.
20
identity · identity_asset_lineageBind every cited cs_escalate_sla_breach file to its case-specific asset-manifest digest and reject evidence copied from another task, even when its filename and business vocabulary look plausible. Enforced by globally unique asset SHA-256 census and manifest membership.
21
investigation · investigation_structured_exportsReconcile the cs_escalate_sla_breach structured exports (escalations-system-export.csv, kb_articles-system-export.csv, sla_policies-system-export.csv, support_customers-system-export.csv, ticket_messages-system-export.csv, tickets-system-export.csv) with the live records they represent; no single export is sufficient when related tables disagree. Enforced by exact table selectors, live-state snapshot, and parser-validated CSV records.
22
investigation · investigation_controlled_workbookUse the parser-validated review-calendar workbook for cs_escalate_sla_breach to test timing and capacity assumptions against current records, while treating workbook conclusions as evidence rather than authorization. Enforced by XLSX parser gate, workbook digest, and exact temporal selectors.
23
correlation · correlation_communication_authority_splitFor cs_escalate_sla_breach, distinguish what the source request, operations Slack thread, and former-owner message say from what the signed approval and current control actually authorize. Enforced by communication-record correlation plus current-authority and approval-scope checks.
24
correlation · correlation_provider_live_lineageJoin the task-scoped Gmail, Drive, Sheets, and Slack records for cs_escalate_sla_breach to the exact live-domain selectors by case id, provider id, immutable row id, and effective revision; a same-name file or row from another task is not equivalent evidence. Enforced by provider asset ids and SHA-256 manifest joined to argument-aware domain selectors.
25
alternatives · alternatives_blast_radiusCompare the complete table-level blast radius of the supported cs_escalate_sla_breach branch with the hold and broad-cleanup alternatives; reject any option that changes an unapproved neighbor even if its headline answer is correct. Enforced by all-table before/after digests and unauthorized-write control.
26
decision · decision_citation_lineageCarry the selected cs_escalate_sla_breach decision into the result row with the exact current evidence IDs used to derive it, so a correct-looking answer backed by stale or cross-task evidence fails. Enforced by exact result evidence set and wrong-evidence control.
27
alternatives · alternatives_supportedKeep the bounded evidence-supported outcome for cs_escalate_sla_breach available only if effective authority and current records agree. Enforced by exact final-state and result contract.
28
alternatives · alternatives_holdRetain a hold/escalate branch for cs_escalate_sla_breach if controlling records conflict or authority is absent; difficulty alone is not a reason to hold. Enforced by wrong-decision negative control.
29
alternatives · alternatives_broadReject a broad neighboring-record cleanup for cs_escalate_sla_breach, even when similar names or stale notes make it superficially plausible. Enforced by unauthorized-write negative control.
30
alternatives · alternatives_no_changeEvaluate a no-domain-change outcome for cs_escalate_sla_breach: retain it only when the exact live predicates do not authorize a mutation; otherwise it is an incomplete state-only or shortcut branch. Enforced by task-specific source VCode plus no-op and state-only controls.
31
decision · decision_supported_branchDerive the selected cs_escalate_sla_breach branch from the intersection of authority, identity, relationship, timing, and live-state evidence—not from the prompt's requested outcome. Enforced by source VCode, exact state digests, and task-result row.
32
decision · decision_task_specific_causal_pathThe deterministic reasoning for cs_escalate_sla_breach is task-specific: It is now minute 180. Ticket 9201 belongs to an enterprise-plan customer and still has no first response. Look up the first_response_minutes of the SLA policy for that customer's plan and work out how many minutes past the SLA deadline the response now is (deadline = opened_minute + first_response_minutes). Then set ticket 9201's priority to 'urgent' and status to 'escalated', create an escalation on ticket 9201 with reason 'sla_first_response_breach', tier 'tier2' and status 'open', and post a holding reply: a ticket message on 9201 from author 'agent' at sent_minute 180 whose body states the overdue amount as 'N minutes'. The model must establish each predicate from the provider evidence and live domain records, then carry only the supported branch into state. Enforced by exact source VCode, authored selectors, before/after state, and adversarial branch controls.
33
procedure · procedure_successful_evidenceEvery exact contextual selector required for cs_escalate_sla_breach must succeed; a failed or differently filtered call is not equivalent evidence. Enforced by argument-aware successful trace.
34
procedure · procedure_provider_discovery_before_detailFor cs_escalate_sla_breach, complete the task-scoped Gmail, Drive, and Slack discovery calls before relying on exact message, file, spreadsheet, or thread records; the verifier accepts equivalent causal ordering but requires every exact discovery and detail selector before state changes. Enforced by provider list/search and exact-get trace selectors before first mutation.
35
procedure · procedure_read_before_writeComplete all 51 causal reads for cs_escalate_sla_breach before the first domain mutation or persisted completion result. Enforced by trace sequence bounds.
36
procedure · procedure_result_after_workPersist the exact cs_escalate_sla_breach result only after any supported domain transition has completed. Enforced by source-mutation-before-submit sequence check.
37
procedure · procedure_post_write_readbackReopen the exact persisted cs_escalate_sla_breach result after submission so verification observes stored state rather than trusting the write response. Enforced by exact get_task_result selector after submit.
38
state · state_escalationsProduce only the supported cs_escalate_sla_breach transition in escalations: added escalation_id=['9402']; removed escalation_id=none; updated none. The complete final table must match digest ef2a1affcc7a7dee96185b5d95f580e04a49d075c9ed1ee92756e0fd4a159396. Enforced by exact complete-table digest and source VCode.
39
state · state_ticket_messagesProduce only the supported cs_escalate_sla_breach transition in ticket_messages: added message_id=['9303']; removed message_id=none; updated none. The complete final table must match digest 5e658685c61ac4b7c0491326dbd1155f18edd84402ad624a53a88bf08f142452. Enforced by exact complete-table digest and source VCode.
40
state · state_ticketsProduce only the supported cs_escalate_sla_breach transition in tickets: added ticket_id=none; removed ticket_id=none; updated ["ticket_id=9201 (priority:'normal'→'urgent', status:'open'→'escalated')"]. The complete final table must match digest 587b28bb6dcc5f217986218d0be77c203664b7ed883b523f8d0b510a632f505f. Enforced by exact complete-table digest and source VCode.
41
state · state_authorized_transition_01The cs_escalate_sla_breach state transition with exact arguments {"priority": "urgent", "status": "escalated", "ticket_id": 9201} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
42
state · state_authorized_transition_02The cs_escalate_sla_breach state transition with exact arguments {"reason": "sla_first_response_breach", "status": "open", "ticket_id": 9201, "tier": "tier2"} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
43
state · state_authorized_transition_03The cs_escalate_sla_breach state transition with exact arguments {"author": "agent", "body": "Apologies \u2014 our first response is 50 minutes past the enterprise SLA. An engineer is now on this.", "channel": "email", "sent_minute": 180, "ticket_id": 9201} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
44
state · state_authorized_transition_04The cs_escalate_sla_breach state transition with exact arguments {"channel": "#support-escalations", "client_msg_id": "0f7a4141f8cacfb225ecdd1030ec4b9d", "text": "Escalate ticket 9201, open the tier-2 SLA escalation, and tell the customer the response is 50 minutes overdue. I reconciled 96 candidate rows across 5 live record families, excluded 92 neighbor rows, and retained 4 immutable task records. The selected outcome is scheduled for 2026-09-05 (+0 days versus the 2026-09-05 need date), costs $2475, fits with 5 internal hours to spare, and is authorized by APR-3d89e9865c44e97d. The broader neighbor remediation requires additional approval and was not executed. Exact answer: 50."} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
45
state · state_source_outcomeAll 2 task-specific executable outcome assertions for cs_escalate_sla_breach must hold against the final sandbox state. Enforced by checked-in deterministic VCode.
46
state · state_result_rowCreate exactly one task-result row for cs_escalate_sla_breach containing the supported decision, exact summary, evidence IDs, causal workpaper, and exact answer when applicable. Enforced by persisted task_results state.
47
answer · answer_result_submittedReturn an audit-ready cs_escalate_sla_breach conclusion in the persisted result row rather than only narrating intended work. Enforced by exact task-result state verifier.
48
answer · answer_evidence_citationsCite the current policy, source request, controlled workbook, and effective source document for cs_escalate_sla_breach; stale authority cannot substitute. Enforced by exact required evidence set.
49
answer · answer_exact_valueReturn the exact task-supported scalar for cs_escalate_sla_breach, derived from the live records rather than copied from an asset. Enforced by deterministic submitted-answer equality.
50
containment · containment_write_scopeUse only the cs_escalate_sla_breach authorized domain mutation set and the task-result handoff; no neighboring table or record may change. Enforced by allowed-write trace plus all-table digests.
51
containment · containment_no_rejected_mutationNo state-changing request in cs_escalate_sla_breach may fail and then be hidden by a later correct-looking final answer. Enforced by successful mutation trace.
52
identity · identity_H01_scope_identityLocate cs_escalate_sla_breach through case DB24-SERVICEBENCH-375-CS_ESCALATE_SLA_BREACH and the exact immutable target map {"escalations": [], "sla_policies": [9101], "support_customers": [9001], "ticket_messages": [9301], "tickets": [9201]}. Enforced by exact workpaper.scope plus successful argument-aware search/count/get evidence.
53
decision · decision_H02_requirement_derivationApply the controlled task rule from operating-policy.md#task-specific-decision-control and derive all 5 required record families before choosing an outcome. Enforced by exact workpaper.requirement and D02.
54
investigation · investigation_H03_availability_reconciliationReconcile 96 observed candidate rows, 92 excluded neighbors, and 4 usable targets. Enforced by exact workpaper.reconciliation and D03-D05.
55
decision · decision_H04_gap_or_boundNet 5 required families against 5 covered families into exact gap 0. Enforced by exact workpaper.gapOrBound and D06.
56
authority · authority_H05_external_constraintApply independent approval APR-3d89e9865c44e97d and its $2700 cost boundary. Enforced by exact workpaper.externalConstraint corroborated by approval email and Drive record.
57
investigation · investigation_H06_internal_constraintApply the 2026-09-05 business need, 16 available hours, and $225 hourly rate from the controlled calendar. Enforced by exact workpaper.internalConstraint and D07-D10.
58
alternatives · alternatives_H07_qualified_optionsGrade all three task-specific alternatives with exact outcome, date, signed variance, incremental cost, authority status, and one selected branch. Enforced by exact workpaper.options equality and wrong-reasoning control.
59
decision · decision_H08_recommendationSelect evidence-backed-scoped-outcome with exact outcome 'Escalate ticket 9201, open the tier-2 SLA escalation, and tell the customer the response is 50 minutes overdue.'. Enforced by exact workpaper.recommendation and D13.
60
decision · decision_H09_control_comparisonCompare outcome 2026-09-05 to need 2026-09-05 into signed variance 0 and status ON_OR_BEFORE_NEED. Enforced by independently recomputed exact workpaper.controlComparison and D11.
61
authority · authority_H10_authority_applicationApply authority APR-3d89e9865c44e97d to the selected immutable scope and reject broad-neighbor-remediation. Enforced by exact workpaper.authorityApplication and D12.
62
state · state_H11_controlled_executionExecute exactly 3 authorized domain writes across 3 changed tables, perform 3 exact-record provider readbacks, then reopen the persisted result. Enforced by exact writes, complete-table digests, provider-native exact-record readbacks, persisted result, and result readback trace.
63
answer · answer_H12_stakeholder_communicationPost the exact stakeholder handoff to #support-escalations carrying the selected outcome, timing, cost, capacity constraint, and authority. Enforced by exact slack.chat_postMessage call and persisted stakeholder-message row.
64
answer · answer_H13_exact_intermediatesPersist all 13 dependent derivations and every intermediate requirement, reconciliation, constraint, option, authority, state, and answer field exactly. Enforced by exact causal-workpaper equality; wrong-reasoning negative control.
65
containment · containment_visible_hidden_partitionThe 28 files visible for cs_escalate_sla_breach contain evidence only; verifier digests, the oracle trajectory, and precomputed outcomes remain outside the asset room. Enforced by release-tree partition and leakage gate.

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in v3 causal investigation replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: cs_escalate_sla_breach
Steps: 60
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

gmail.messages.listtool success

Arguments

{
  "maxResults": 100,
  "q": "\"DB24-SERVICEBENCH-375-CS_ESCALATE_SLA_BREACH\"",
  "userId": "me"
}

Result

{
  "messages": [
    {
      "id": "msg-e03774d4fb6a840b7773",
      "threadId": "msg-e03774d4fb6a840b7773"
    },
    {
      "id": "msg-6871a5c44fab7d8021d3",
      "threadId": "msg-6871a5c44fab7d8021d3"
    },
    {
      "id": "msg-dc90a4c698690fb4b5f5",
      "threadId": "msg-dc90a4c698690fb4b5f5"
    }
  ],
  "resultSizeEstimate": 3
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-e03774d4fb6a840b7773",
  "userId": "me"
}

Result

{
  "id": "msg-e03774d4fb6a840b7773",
  "name": "source-request.eml",
  "mimeType": "message/rfc822",
  "sha256": "5951b74cb9c034052b867d39d2a492d02d5577d0a097421f1284c5af65ed7075",
  "bytes": 878
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-6871a5c44fab7d8021d3",
  "userId": "me"
}

Result

{
  "id": "msg-6871a5c44fab7d8021d3",
  "name": "former-owner-suggestion.eml",
  "mimeType": "message/rfc822",
  "sha256": "578b55ccbc53c999a8543856da023bb2037300ffa87a9843e5e95e203b7eb9ee",
  "bytes": 619
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-dc90a4c698690fb4b5f5",
  "userId": "me"
}

Result

{
  "id": "msg-dc90a4c698690fb4b5f5",
  "name": "approval-scope.eml",
  "mimeType": "message/rfc822",
  "sha256": "70e44dce74bde646a18a5853c130d4d673b5dff52d9c513491cadb6435287327",
  "bytes": 824
}

google_drive.files.listtool success

Arguments

{
  "fields": "files(id,name,mimeType,modifiedTime,version)",
  "pageSize": 1000,
  "q": "fullText contains 'DB24-SERVICEBENCH-375-CS_ESCALATE_SLA_BREACH'"
}

Result

{
  "files": [
    {
      "id": "file-cc93d194ef55489378e9",
      "name": "operating-policy.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-f73e51a20b918a55986e",
      "name": "customer-service-policy.pdf",
      "mimeType": "application/pdf",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "sheet-2d36b0b7ee8ac04aee9a",
      "name": "sla-and-queue-review.xlsx",
      "mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-1eed20baeac829904ba6",
      "name": "approval-decision.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-ca3ba70ad52bf7cd34ab",
      "name": "drive-source-index.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-312229dd966ec37c6bda",
      "name": "provider-contracts.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-a0fa18cac1ad466d9ac2",
      "name": "drive-approval-record.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-e382c605326d17ec3009",
      "name": "authority-revision-2026-3.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-e52576f8afeae220e8ec",
      "name": "authority-revision-2025-4.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-7432da3fe489d642c616",
      "name": "control-standard-2025-4.pdf",
      "mimeType": "application/pdf",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "sheet-b9879849ffa8f571cfe9",
      "name": "review-calendar.xlsx",
      "mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-24660208b49359388709",
      "name": "record-lineage.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-3f8acf3d17d593074143",
      "name": "system-audit.log",
      "mimeType": "text/plain",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-54512f4a7d32f3c0a4b3",
      "name": "live-snapshot.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-3813207038483c30aa13",
      "name": "evidence-status.yaml",
      "mimeType": "application/yaml",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-e1b2673a8dd940ec0bdb",
      "name": "custodian-notes.txt",
      "mimeType": "text/plain",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-3a7111786ef3f5819250",
      "name": "support_customers-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-a311ca3a4bb10b2140d5",
      "name": "tickets-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-f4bcafe9a879e28209b2",
      "name": "sla_policies-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-c70b5d1aa0c53ff57280",
      "name": "ticket_messages-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-b9a585a1dea79eb6d724",
      "name": "escalations-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-463a367ae757a7c5eafd",
      "name": "kb_articles-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-538d71d2ca41522f1e08",
      "name": "asset-manifest.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    }
  ]
}