All benchmarksEnterprise operations

OpsBench-14Evidence 2026-08-29Independent companionDataset page live

Can an agent join the systems behind a real operational decision?

Cross-functional enterprise questions paired with vendors, processes, changes, controls, and review workflows.

Explore 4 companion tasks ↓World repository ↗Run on Harbor ↗

14upstream Harbor tasks

4independent companion tasks

28files per task room

59+criteria per task

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces, exact replays, and reject thirteen task-applicable controls including correct state with wrong reasoning, wrong target, and keyword stuffing. No paid or frontier model run is claimed. These 4 tasks are Blobfish-authored companions, not conversions of the 14-task upstream suite. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.00236/236 successful

Wrong-evidence causal controlExact-release deterministic negative control; not a model rank.

Adversarial discrimination floor0/40%0.00236/236 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% wrong-evidence floor shows the verifier rejects a plausible workflow that relies on the wrong source record. Neither row measures model capability.

Task browser

Read every released task.

These are 4 independent Blobfish v3.1 realism tasks—not excerpts relabeled from the upstream suite. Each is a high-level employee request backed by a multi-source evidence room, three operating options, a task-specific causal evaluation narrative, and deterministic state checks.

heldout67 reference stepshard · generator label

ops_reject_uncontrolled_change

Can the proposed Vendor Onboarding change safely go to this week's approval meeting? The request would alter a high-risk workflow, and the coverage dashboard may be distorted by a duplicated step. Determine whether the effective process still has any genuinely uncontrolled work. Record the disposition the evidence supports and create only the bounded remediation needed by policy. Brief Operations on the uncovered-step count and why the change was accepted, held, or rejected.

How the employee outcome is evaluated

Evidence has to support the decision, the write, and the final answer.

Provider evidence: 26 exact reads
Live records: 32 correlated reads
Supported writes: 4
Proof: 4 readbacks

Reasoning contract

Change request 9301 is a high-risk change to the Vendor Onboarding process (doc_id 9101). Policy: a high-risk change cannot be approved while any step of that process has no control_ref mapped. Count the DISTINCT steps of doc 9101 with an empty control_ref — note step 9202 is a duplicate of 9201 and counts once. Because at least one step is uncontrolled: set change request 9301's status to 'rejected', set process doc 9101's status to 'needs_controls', and open the remediation by creating a compliance control named 'Step control backfill' with framework 'SOC2' and status 'untested'.

Deterministic success

All exact provider and live-domain evidence is read before the first write; the task-specific predicates support the chosen branch; only the authorized state delta and one completion row are created; the persisted result is reopened.

Exact total call order is not graded. Equivalent investigation is accepted when the same task-specific evidence is established before mutation, the supported state is produced, and get_change_request → get_compliance_control → get_process_doc → get_task_result reopens every changed provider row and the persisted result.

Required evidence systems

01gmail.messages.list
02gmail.messages.get
03google_drive.files.list
04google_drive.files.get
05google_sheets.spreadsheets.values.get
06slack.search_messages
07slack.conversations_replies
08search_vendors
09count_vendors
10get_vendor
11search_vendor_reviews
12count_vendor_reviews
13search_process_docs
14count_process_docs
15get_process_doc
16search_process_steps
17count_process_steps
18get_process_step
19search_change_requests
20count_change_requests
21get_change_request
22search_compliance_controls
23count_compliance_controls
24get_compliance_control

Reference: PASS
Wrong-evidence floor: FAIL
Verifier: exact final tables, argument-aware causal trace, persisted result, and readback
Gradable: Yes

Operating options

✓
Complete the bounded supported outcomeCurrent authority, usable live evidence, and the available internal review window all support this task-scoped outcome.
02
Hold for an independent evidence reviewThis remains within authority but misses the documented need date even though the current evidence has no unresolved control conflict.
03
Extend remediation to neighboring recordsThe wider change exceeds both the approved immutable-record scope and the current incremental-cost authority.

Evidence files: 28
Exact criteria: 67
Allowed writes: 4
Call order: Not graded

Task evidenceTypeSizeSHA-256Open

operating-policy.mdassets/ops_reject_uncontrolled_change/operating-policy.md

MD1.2 KBb6cd18f266093684…View ↗

source-request.emlassets/ops_reject_uncontrolled_change/source-request.eml

EML868 B382fca3fd9d3293b…View ↗

operating-control-standard.pdfassets/ops_reject_uncontrolled_change/operating-control-standard.pdf

PDF936 B1f0a34a4d560b101…View ↗

vendor-process-control-matrix.xlsxassets/ops_reject_uncontrolled_change/vendor-process-control-matrix.xlsx

XLSX8.7 KB342c9034ba40e631…View ↗

approval-decision.jsonassets/ops_reject_uncontrolled_change/approval-decision.json

JSON801 B640e32b22e841d01…View ↗

team-slack-intake.jsonassets/ops_reject_uncontrolled_change/team-slack-intake.json

JSON792 B4f1902651a5800f2…View ↗

drive-source-index.jsonassets/ops_reject_uncontrolled_change/drive-source-index.json

JSON385 B1c09da689644f6c1…View ↗

provider-contracts.jsonassets/ops_reject_uncontrolled_change/provider-contracts.json

JSON22.9 KB18d439adc231e734…View ↗

former-owner-suggestion.emlassets/ops_reject_uncontrolled_change/former-owner-suggestion.eml

EML639 B17cb1a8523d1712e…View ↗

approval-scope.emlassets/ops_reject_uncontrolled_change/approval-scope.eml

EML896 Bf84338b9d83bfae0…View ↗

operations-slack-thread.jsonassets/ops_reject_uncontrolled_change/operations-slack-thread.json

JSON425 B20424fc0cdb53ec3…View ↗

drive-approval-record.jsonassets/ops_reject_uncontrolled_change/drive-approval-record.json

JSON612 Bab575fa493e26a28…View ↗

authority-revision-2026-3.mdassets/ops_reject_uncontrolled_change/authority-revision-2026-3.md

MD272 B08723038832c472d…View ↗

authority-revision-2025-4.mdassets/ops_reject_uncontrolled_change/authority-revision-2025-4.md

MD219 B1562134ff054432c…View ↗

control-standard-2025-4.pdfassets/ops_reject_uncontrolled_change/control-standard-2025-4.pdf

PDF815 Ba26ef7647fabed1e…View ↗

review-calendar.xlsxassets/ops_reject_uncontrolled_change/review-calendar.xlsx

XLSX2.0 KB6d2319419f063bcb…View ↗

record-lineage.csvassets/ops_reject_uncontrolled_change/record-lineage.csv

CSV506 B0011d58ec04fd576…View ↗

system-audit.logassets/ops_reject_uncontrolled_change/system-audit.log

LOG896 B182e8a117dece0fe…View ↗

live-snapshot.jsonassets/ops_reject_uncontrolled_change/live-snapshot.json

JSON13.1 KB8f31f0b717d2e401…View ↗

evidence-status.yamlassets/ops_reject_uncontrolled_change/evidence-status.yaml

YAML215 B77f9090fe38935ee…View ↗

custodian-notes.txtassets/ops_reject_uncontrolled_change/custodian-notes.txt

TXT211 B009ff69422a4c0f6…View ↗

vendors-system-export.csvassets/ops_reject_uncontrolled_change/vendors-system-export.csv

CSV2.5 KB15620da7af0b1c97…View ↗

vendor_reviews-system-export.csvassets/ops_reject_uncontrolled_change/vendor_reviews-system-export.csv

CSV2.0 KBdc0b637a02274ddc…View ↗

process_docs-system-export.csvassets/ops_reject_uncontrolled_change/process_docs-system-export.csv

CSV2.3 KBbd105f00cb80f913…View ↗

process_steps-system-export.csvassets/ops_reject_uncontrolled_change/process_steps-system-export.csv

CSV2.0 KB89263d970a2bfb8c…View ↗

change_requests-system-export.csvassets/ops_reject_uncontrolled_change/change_requests-system-export.csv

CSV2.2 KBe877ed7228743631…View ↗

compliance_controls-system-export.csvassets/ops_reject_uncontrolled_change/compliance_controls-system-export.csv

CSV2.6 KBa416fdab66e01fcc…View ↗

asset-manifest.jsonassets/ops_reject_uncontrolled_change/asset-manifest.json

JSON4.4 KB18c1ed4c8c6bf997…View ↗

Inspect all 67 deterministic criteria

01
identity · identity_change_requestsFor ops_reject_uncontrolled_change, resolve change_requests by immutable request_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_change_requests"}, {"args": {}, "tool": "count_change_requests"}, {"args": {"doc_id": 9101, "limit": 50}, "tool": "search_change_requests"}, {"args": {"doc_id": 9101}, "tool": "count_change_requests"}, {"args": {"request_id": 9301}, "tool": "get_change_request"}].
02
investigation · investigation_change_requestsEstablish the relevant and competing change_requests records before deciding the ops_reject_uncontrolled_change outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
03
identity · identity_compliance_controlsFor ops_reject_uncontrolled_change, resolve compliance_controls by immutable control_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_compliance_controls"}, {"args": {}, "tool": "count_compliance_controls"}, {"args": {"limit": 50, "name": "Sanctions screening"}, "tool": "search_compliance_controls"}, {"args": {"name": "Sanctions screening"}, "tool": "count_compliance_controls"}, {"args": {"control_id": 9401}, "tool": "get_compliance_control"}].
04
investigation · investigation_compliance_controlsEstablish the relevant and competing compliance_controls records before deciding the ops_reject_uncontrolled_change outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
05
identity · identity_process_docsFor ops_reject_uncontrolled_change, resolve process_docs by immutable doc_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_process_docs"}, {"args": {}, "tool": "count_process_docs"}, {"args": {"limit": 50, "name": "Vendor Onboarding"}, "tool": "search_process_docs"}, {"args": {"name": "Vendor Onboarding"}, "tool": "count_process_docs"}, {"args": {"doc_id": 9101}, "tool": "get_process_doc"}].
06
investigation · investigation_process_docsEstablish the relevant and competing process_docs records before deciding the ops_reject_uncontrolled_change outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
07
identity · identity_process_stepsFor ops_reject_uncontrolled_change, resolve process_steps by immutable step_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_process_steps"}, {"args": {}, "tool": "count_process_steps"}, {"args": {"doc_id": 9101, "limit": 50}, "tool": "search_process_steps"}, {"args": {"doc_id": 9101}, "tool": "count_process_steps"}, {"args": {"step_id": 9201}, "tool": "get_process_step"}, {"args": {"step_id": 9202}, "tool": "get_process_step"}, {"args": {"step_id": 9203}, "tool": "get_process_step"}, {"args": {"step_id": 9204}, "tool": "get_process_step"}].
08
investigation · investigation_process_stepsEstablish the relevant and competing process_steps records before deciding the ops_reject_uncontrolled_change outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
09
identity · identity_vendor_reviewsFor ops_reject_uncontrolled_change, resolve vendor_reviews by immutable review_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_vendor_reviews"}, {"args": {}, "tool": "count_vendor_reviews"}, {"args": {"limit": 50, "vendor_id": 9002}, "tool": "search_vendor_reviews"}, {"args": {"vendor_id": 9002}, "tool": "count_vendor_reviews"}].
10
investigation · investigation_vendor_reviewsEstablish the relevant and competing vendor_reviews records before deciding the ops_reject_uncontrolled_change outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
11
identity · identity_vendorsFor ops_reject_uncontrolled_change, resolve vendors by immutable vendor_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_vendors"}, {"args": {}, "tool": "count_vendors"}, {"args": {"limit": 50, "name": "Cascade Facilities Ltd"}, "tool": "search_vendors"}, {"args": {"name": "Cascade Facilities Ltd"}, "tool": "count_vendors"}, {"args": {"vendor_id": 9002}, "tool": "get_vendor"}].
12
investigation · investigation_vendorsEstablish the relevant and competing vendors records before deciding the ops_reject_uncontrolled_change outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
13
investigation · investigation_provider_gmailFor ops_reject_uncontrolled_change, use task-scoped gmail discovery and exact record reads to inspect source-request.eml (msg-77655abd06caa63821e8), former-owner-suggestion.eml (msg-c243bdc88da7ccc83800), approval-scope.eml (msg-47118e4d7abd5b11dc48). The remaining 0 listed gmail records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"maxResults": 100, "q": "\"DB24-OPSBENCH-14-OPS_REJECT_UNCONTROLLED_CHANGE\"", "userId": "me"}, "tool": "gmail.messages.list"}, {"args": {"format": "raw", "id": "msg-77655abd06caa63821e8", "userId": "me"}, "tool": "gmail.messages.get"}, {"args": {"format": "raw", "id": "msg-c243bdc88da7ccc83800", "userId": "me"}, "tool": "gmail.messages.get"}, {"args": {"format": "raw", "id": "msg-47118e4d7abd5b11dc48", "userId": "me"}, "tool": "gmail.messages.get"}].
14
investigation · investigation_provider_google_driveFor ops_reject_uncontrolled_change, use task-scoped google_drive discovery and exact record reads to inspect operating-policy.md (file-fa1eecb134ff01b9ca24), operating-control-standard.pdf (file-d12d0091ef335cba7a96), approval-decision.json (file-0e6b239399dfa5013d63), drive-source-index.json (file-2f5e1e65ef059f7c2bc2), drive-approval-record.json (file-9240d23adb4f17cc1b7c), authority-revision-2026-3.md (file-6541be607ce7b765f21c), authority-revision-2025-4.md (file-ec5fc8d20f7a636e3de6), record-lineage.csv (file-2d58352ab43cb6e16755), live-snapshot.json (file-53901bb7a07dc6c41902), vendors-system-export.csv (file-d3f7e1749f8a9260d90b), vendor_reviews-system-export.csv (file-621feb6f982088d5359c), process_docs-system-export.csv (file-b85b85ada976b0830a49), process_steps-system-export.csv (file-869495cec155340ad023), change_requests-system-export.csv (file-02cadf081b91674508e0), compliance_controls-system-export.csv (file-b0c50c9f451db255dab8), asset-manifest.json (file-1a768b1e7395c4406407). The remaining 5 listed google_drive records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"fields": "files(id,name,mimeType,modifiedTime,version)", "pageSize": 1000, "q": "fullText contains 'DB24-OPSBENCH-14-OPS_REJECT_UNCONTROLLED_CHANGE'"}, "tool": "google_drive.files.list"}, {"args": {"alt": "media", "fileId": "file-fa1eecb134ff01b9ca24"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-d12d0091ef335cba7a96"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-0e6b239399dfa5013d63"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-2f5e1e65ef059f7c2bc2"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-02cadf081b91674508e0"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-b0c50c9f451db255dab8"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-b85b85ada976b0830a49"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-869495cec155340ad023"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-621feb6f982088d5359c"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-d3f7e1749f8a9260d90b"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-9240d23adb4f17cc1b7c"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-6541be607ce7b765f21c"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-ec5fc8d20f7a636e3de6"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-2d58352ab43cb6e16755"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-53901bb7a07dc6c41902"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-1a768b1e7395c4406407"}, "tool": "google_drive.files.get"}].
15
investigation · investigation_provider_google_sheetsFor ops_reject_uncontrolled_change, use task-scoped google_sheets discovery and exact record reads to inspect vendor-process-control-matrix.xlsx (sheet-b571f9727f285fa8216c), review-calendar.xlsx (sheet-584531067d723a57d33f). The remaining 0 listed google_sheets records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"range": "A:Z", "spreadsheetId": "sheet-b571f9727f285fa8216c"}, "tool": "google_sheets.spreadsheets.values.get"}, {"args": {"range": "A:Z", "spreadsheetId": "sheet-584531067d723a57d33f"}, "tool": "google_sheets.spreadsheets.values.get"}].
16
investigation · investigation_provider_slackFor ops_reject_uncontrolled_change, use task-scoped slack discovery and exact record reads to inspect team-slack-intake.json (thread-780acb1fe1b3ae5790a7), operations-slack-thread.json (thread-09fad1a9427ffce19573). The remaining 0 listed slack records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"count": 100, "query": "\"DB24-OPSBENCH-14-OPS_REJECT_UNCONTROLLED_CHANGE\"", "sort": "timestamp", "sort_dir": "asc"}, "tool": "slack.search_messages"}, {"args": {"channel": "#operations-controls", "limit": 100, "ts": "1724937600.677248"}, "tool": "slack.conversations_replies"}, {"args": {"channel": "#operations-controls", "limit": 100, "ts": "1724937600.649331"}, "tool": "slack.conversations_replies"}].
17
authority · authority_effective_revisionUse the effective 2026.3 operating control for ops_reject_uncontrolled_change, corroborated by the approval and Drive revision records. Enforced by current-control PDF, Drive record, and policy revision identity.
18
authority · authority_retired_conflictTreat the retained 2025.4 note for ops_reject_uncontrolled_change as a competing historical hypothesis, not as permission or current authority. Enforced by current-versus-retired revision comparison.
19
correlation · correlation_cross_system_identityCorrelate the employee request, controlled workbook, system exports, Slack discussion, Drive record, and approval email for ops_reject_uncontrolled_change by case and immutable record IDs. Enforced by cross-file case IDs and exact world selectors.
20
correlation · correlation_live_over_snapshotReconcile the released snapshot for ops_reject_uncontrolled_change to successful live MCP reads before relying on either source; the files alone do not establish the final outcome. Enforced by asset digests plus exact live trace.
21
correlation · correlation_count_detail_relationshipFor ops_reject_uncontrolled_change, reconcile broad counts with detail records and foreign-key relationships so duplicates and similarly named distractors are not collapsed. Enforced by search/count/get selector intersection.
22
identity · identity_asset_lineageBind every cited ops_reject_uncontrolled_change file to its case-specific asset-manifest digest and reject evidence copied from another task, even when its filename and business vocabulary look plausible. Enforced by globally unique asset SHA-256 census and manifest membership.
23
investigation · investigation_structured_exportsReconcile the ops_reject_uncontrolled_change structured exports (change_requests-system-export.csv, compliance_controls-system-export.csv, process_docs-system-export.csv, process_steps-system-export.csv, vendor_reviews-system-export.csv, vendors-system-export.csv) with the live records they represent; no single export is sufficient when related tables disagree. Enforced by exact table selectors, live-state snapshot, and parser-validated CSV records.
24
investigation · investigation_controlled_workbookUse the parser-validated review-calendar workbook for ops_reject_uncontrolled_change to test timing and capacity assumptions against current records, while treating workbook conclusions as evidence rather than authorization. Enforced by XLSX parser gate, workbook digest, and exact temporal selectors.
25
correlation · correlation_communication_authority_splitFor ops_reject_uncontrolled_change, distinguish what the source request, operations Slack thread, and former-owner message say from what the signed approval and current control actually authorize. Enforced by communication-record correlation plus current-authority and approval-scope checks.
26
correlation · correlation_provider_live_lineageJoin the task-scoped Gmail, Drive, Sheets, and Slack records for ops_reject_uncontrolled_change to the exact live-domain selectors by case id, provider id, immutable row id, and effective revision; a same-name file or row from another task is not equivalent evidence. Enforced by provider asset ids and SHA-256 manifest joined to argument-aware domain selectors.
27
alternatives · alternatives_blast_radiusCompare the complete table-level blast radius of the supported ops_reject_uncontrolled_change branch with the hold and broad-cleanup alternatives; reject any option that changes an unapproved neighbor even if its headline answer is correct. Enforced by all-table before/after digests and unauthorized-write control.
28
decision · decision_citation_lineageCarry the selected ops_reject_uncontrolled_change decision into the result row with the exact current evidence IDs used to derive it, so a correct-looking answer backed by stale or cross-task evidence fails. Enforced by exact result evidence set and wrong-evidence control.
29
alternatives · alternatives_supportedKeep the bounded evidence-supported outcome for ops_reject_uncontrolled_change available only if effective authority and current records agree. Enforced by exact final-state and result contract.
30
alternatives · alternatives_holdRetain a hold/escalate branch for ops_reject_uncontrolled_change if controlling records conflict or authority is absent; difficulty alone is not a reason to hold. Enforced by wrong-decision negative control.
31
alternatives · alternatives_broadReject a broad neighboring-record cleanup for ops_reject_uncontrolled_change, even when similar names or stale notes make it superficially plausible. Enforced by unauthorized-write negative control.
32
alternatives · alternatives_no_changeEvaluate a no-domain-change outcome for ops_reject_uncontrolled_change: retain it only when the exact live predicates do not authorize a mutation; otherwise it is an incomplete state-only or shortcut branch. Enforced by task-specific source VCode plus no-op and state-only controls.
33
decision · decision_supported_branchDerive the selected ops_reject_uncontrolled_change branch from the intersection of authority, identity, relationship, timing, and live-state evidence—not from the prompt's requested outcome. Enforced by source VCode, exact state digests, and task-result row.
34
decision · decision_task_specific_causal_pathThe deterministic reasoning for ops_reject_uncontrolled_change is task-specific: Change request 9301 is a high-risk change to the Vendor Onboarding process (doc_id 9101). Policy: a high-risk change cannot be approved while any step of that process has no control_ref mapped. Count the DISTINCT steps of doc 9101 with an empty control_ref — note step 9202 is a duplicate of 9201 and counts once. Because at least one step is uncontrolled: set change request 9301's status to 'rejected', set process doc 9101's status to 'needs_controls', and open the remediation by creating a compliance control named 'Step control backfill' with framework 'SOC2' and status 'untested'. The model must establish each predicate from the provider evidence and live domain records, then carry only the supported branch into state. Enforced by exact source VCode, authored selectors, before/after state, and adversarial branch controls.
35
procedure · procedure_successful_evidenceEvery exact contextual selector required for ops_reject_uncontrolled_change must succeed; a failed or differently filtered call is not equivalent evidence. Enforced by argument-aware successful trace.
36
procedure · procedure_provider_discovery_before_detailFor ops_reject_uncontrolled_change, complete the task-scoped Gmail, Drive, and Slack discovery calls before relying on exact message, file, spreadsheet, or thread records; the verifier accepts equivalent causal ordering but requires every exact discovery and detail selector before state changes. Enforced by provider list/search and exact-get trace selectors before first mutation.
37
procedure · procedure_read_before_writeComplete all 58 causal reads for ops_reject_uncontrolled_change before the first domain mutation or persisted completion result. Enforced by trace sequence bounds.
38
procedure · procedure_result_after_workPersist the exact ops_reject_uncontrolled_change result only after any supported domain transition has completed. Enforced by source-mutation-before-submit sequence check.
39
procedure · procedure_post_write_readbackReopen the exact persisted ops_reject_uncontrolled_change result after submission so verification observes stored state rather than trusting the write response. Enforced by exact get_task_result selector after submit.
40
state · state_change_requestsProduce only the supported ops_reject_uncontrolled_change transition in change_requests: added request_id=none; removed request_id=none; updated ["request_id=9301 (status:'submitted'→'rejected')"]. The complete final table must match digest 6bbeeab540e2c8ee550ea92b7b662e1be6f32cc17a07f27f6fccb52ed06ee57a. Enforced by exact complete-table digest and source VCode.
41
state · state_compliance_controlsProduce only the supported ops_reject_uncontrolled_change transition in compliance_controls: added control_id=['9402']; removed control_id=none; updated none. The complete final table must match digest da89b1f73d37a5a43af2a4c6a6cf49f16fcc94176ecc666ad2fa0dcf3e9275dd. Enforced by exact complete-table digest and source VCode.
42
state · state_process_docsProduce only the supported ops_reject_uncontrolled_change transition in process_docs: added doc_id=none; removed doc_id=none; updated ["doc_id=9101 (status:'published'→'needs_controls')"]. The complete final table must match digest d6d2db58ae04a428d9a9666a8d60074dcddbb4a6292b8f90229d016291e09f13. Enforced by exact complete-table digest and source VCode.
43
state · state_authorized_transition_01The ops_reject_uncontrolled_change state transition with exact arguments {"request_id": 9301, "status": "rejected"} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
44
state · state_authorized_transition_02The ops_reject_uncontrolled_change state transition with exact arguments {"doc_id": 9101, "status": "needs_controls"} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
45
state · state_authorized_transition_03The ops_reject_uncontrolled_change state transition with exact arguments {"framework": "SOC2", "last_tested_on": "", "name": "Step control backfill", "status": "untested"} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
46
state · state_authorized_transition_04The ops_reject_uncontrolled_change state transition with exact arguments {"channel": "#operations-controls", "client_msg_id": "f147f31bb4bfdbe435af74f02a2596b3", "text": "Reject change request 9301, mark process document 9101 as needing controls, and open one SOC 2 backfill control for the single uncovered step. I reconciled 116 candidate rows across 6 live record families, excluded 108 neighbor rows, and retained 8 immutable task records. The selected outcome is scheduled for 2026-09-02 (-1 days versus the 2026-09-03 need date), costs $2750, fits with 6 internal hours to spare, and is authorized by APR-be08d2763c5db09d. The broader neighbor remediation requires additional approval and was not executed. Exact answer: 1."} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
47
state · state_source_outcomeAll 2 task-specific executable outcome assertions for ops_reject_uncontrolled_change must hold against the final sandbox state. Enforced by checked-in deterministic VCode.
48
state · state_result_rowCreate exactly one task-result row for ops_reject_uncontrolled_change containing the supported decision, exact summary, evidence IDs, causal workpaper, and exact answer when applicable. Enforced by persisted task_results state.
49
answer · answer_result_submittedReturn an audit-ready ops_reject_uncontrolled_change conclusion in the persisted result row rather than only narrating intended work. Enforced by exact task-result state verifier.
50
answer · answer_evidence_citationsCite the current policy, source request, controlled workbook, and effective source document for ops_reject_uncontrolled_change; stale authority cannot substitute. Enforced by exact required evidence set.
51
answer · answer_exact_valueReturn the exact task-supported scalar for ops_reject_uncontrolled_change, derived from the live records rather than copied from an asset. Enforced by deterministic submitted-answer equality.
52
containment · containment_write_scopeUse only the ops_reject_uncontrolled_change authorized domain mutation set and the task-result handoff; no neighboring table or record may change. Enforced by allowed-write trace plus all-table digests.
53
containment · containment_no_rejected_mutationNo state-changing request in ops_reject_uncontrolled_change may fail and then be hidden by a later correct-looking final answer. Enforced by successful mutation trace.
54
identity · identity_H01_scope_identityLocate ops_reject_uncontrolled_change through case DB24-OPSBENCH-14-OPS_REJECT_UNCONTROLLED_CHANGE and the exact immutable target map {"change_requests": [9301], "compliance_controls": [9401], "process_docs": [9101], "process_steps": [9201, 9202, 9203, 9204], "vendor_reviews": [], "vendors": [9002]}. Enforced by exact workpaper.scope plus successful argument-aware search/count/get evidence.
55
decision · decision_H02_requirement_derivationApply the controlled task rule from operating-policy.md#task-specific-decision-control and derive all 6 required record families before choosing an outcome. Enforced by exact workpaper.requirement and D02.
56
investigation · investigation_H03_availability_reconciliationReconcile 116 observed candidate rows, 108 excluded neighbors, and 8 usable targets. Enforced by exact workpaper.reconciliation and D03-D05.
57
decision · decision_H04_gap_or_boundNet 6 required families against 6 covered families into exact gap 0. Enforced by exact workpaper.gapOrBound and D06.
58
authority · authority_H05_external_constraintApply independent approval APR-be08d2763c5db09d and its $3000 cost boundary. Enforced by exact workpaper.externalConstraint corroborated by approval email and Drive record.
59
investigation · investigation_H06_internal_constraintApply the 2026-09-03 business need, 17 available hours, and $250 hourly rate from the controlled calendar. Enforced by exact workpaper.internalConstraint and D07-D10.
60
alternatives · alternatives_H07_qualified_optionsGrade all three task-specific alternatives with exact outcome, date, signed variance, incremental cost, authority status, and one selected branch. Enforced by exact workpaper.options equality and wrong-reasoning control.
61
decision · decision_H08_recommendationSelect evidence-backed-scoped-outcome with exact outcome 'Reject change request 9301, mark process document 9101 as needing controls, and open one SOC 2 backfill control for the single uncovered step.'. Enforced by exact workpaper.recommendation and D13.
62
decision · decision_H09_control_comparisonCompare outcome 2026-09-02 to need 2026-09-03 into signed variance -1 and status ON_OR_BEFORE_NEED. Enforced by independently recomputed exact workpaper.controlComparison and D11.
63
authority · authority_H10_authority_applicationApply authority APR-be08d2763c5db09d to the selected immutable scope and reject broad-neighbor-remediation. Enforced by exact workpaper.authorityApplication and D12.
64
state · state_H11_controlled_executionExecute exactly 3 authorized domain writes across 3 changed tables, perform 3 exact-record provider readbacks, then reopen the persisted result. Enforced by exact writes, complete-table digests, provider-native exact-record readbacks, persisted result, and result readback trace.
65
answer · answer_H12_stakeholder_communicationPost the exact stakeholder handoff to #operations-controls carrying the selected outcome, timing, cost, capacity constraint, and authority. Enforced by exact slack.chat_postMessage call and persisted stakeholder-message row.
66
answer · answer_H13_exact_intermediatesPersist all 13 dependent derivations and every intermediate requirement, reconciliation, constraint, option, authority, state, and answer field exactly. Enforced by exact causal-workpaper equality; wrong-reasoning negative control.
67
containment · containment_visible_hidden_partitionThe 28 files visible for ops_reject_uncontrolled_change contain evidence only; verifier digests, the oracle trajectory, and precomputed outcomes remain outside the asset room. Enforced by release-tree partition and leakage gate.

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in v3 causal investigation replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: ops_reject_uncontrolled_change
Steps: 67
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

gmail.messages.listtool success

Arguments

{
  "maxResults": 100,
  "q": "\"DB24-OPSBENCH-14-OPS_REJECT_UNCONTROLLED_CHANGE\"",
  "userId": "me"
}

Result

{
  "messages": [
    {
      "id": "msg-77655abd06caa63821e8",
      "threadId": "msg-77655abd06caa63821e8"
    },
    {
      "id": "msg-c243bdc88da7ccc83800",
      "threadId": "msg-c243bdc88da7ccc83800"
    },
    {
      "id": "msg-47118e4d7abd5b11dc48",
      "threadId": "msg-47118e4d7abd5b11dc48"
    }
  ],
  "resultSizeEstimate": 3
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-77655abd06caa63821e8",
  "userId": "me"
}

Result

{
  "id": "msg-77655abd06caa63821e8",
  "name": "source-request.eml",
  "mimeType": "message/rfc822",
  "sha256": "382fca3fd9d3293b0a91c72634d3e864325f31140861dd401d1613d53c3d9f02",
  "bytes": 868
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-c243bdc88da7ccc83800",
  "userId": "me"
}

Result

{
  "id": "msg-c243bdc88da7ccc83800",
  "name": "former-owner-suggestion.eml",
  "mimeType": "message/rfc822",
  "sha256": "17cb1a8523d1712e2e5390799ce7b887efa28933541eeecac0919c34c8fb9f4a",
  "bytes": 639
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-47118e4d7abd5b11dc48",
  "userId": "me"
}

Result

{
  "id": "msg-47118e4d7abd5b11dc48",
  "name": "approval-scope.eml",
  "mimeType": "message/rfc822",
  "sha256": "f84338b9d83bfae0f8f7f8bc9ba049e85d617e4ecf3f9db97e31473dd93385bf",
  "bytes": 896
}

google_drive.files.listtool success

Arguments

{
  "fields": "files(id,name,mimeType,modifiedTime,version)",
  "pageSize": 1000,
  "q": "fullText contains 'DB24-OPSBENCH-14-OPS_REJECT_UNCONTROLLED_CHANGE'"
}

Result

{
  "files": [
    {
      "id": "file-fa1eecb134ff01b9ca24",
      "name": "operating-policy.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d12d0091ef335cba7a96",
      "name": "operating-control-standard.pdf",
      "mimeType": "application/pdf",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "sheet-b571f9727f285fa8216c",
      "name": "vendor-process-control-matrix.xlsx",
      "mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-0e6b239399dfa5013d63",
      "name": "approval-decision.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-2f5e1e65ef059f7c2bc2",
      "name": "drive-source-index.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d92aa073dba21fc1f961",
      "name": "provider-contracts.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-9240d23adb4f17cc1b7c",
      "name": "drive-approval-record.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-6541be607ce7b765f21c",
      "name": "authority-revision-2026-3.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-ec5fc8d20f7a636e3de6",
      "name": "authority-revision-2025-4.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-5e30bbd6bb1e8faf1a0b",
      "name": "control-standard-2025-4.pdf",
      "mimeType": "application/pdf",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "sheet-584531067d723a57d33f",
      "name": "review-calendar.xlsx",
      "mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-2d58352ab43cb6e16755",
      "name": "record-lineage.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-1147c76950e468d28713",
      "name": "system-audit.log",
      "mimeType": "text/plain",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-53901bb7a07dc6c41902",
      "name": "live-snapshot.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-6f5825d0672e59df8195",
      "name": "evidence-status.yaml",
      "mimeType": "application/yaml",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-9a34ce10fb485ddc8433",
      "name": "custodian-notes.txt",
      "mimeType": "text/plain",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d3f7e1749f8a9260d90b",
      "name": "vendors-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-621feb6f982088d5359c",
      "name": "vendor_reviews-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-b85b85ada976b0830a49",
      "name": "process_docs-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-869495cec155340ad023",
      "name": "process_steps-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-02cadf081b91674508e0",
      "name": "change_requests-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-b0c50c9f451db255dab8",
      "name": "compliance_controls-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-1a768b1e7395c4406407",
      "name": "asset-manifest.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    }
  ]
}