All benchmarksSoftware engineering

RepoBench-500Evidence 2026-08-29Independent companionDataset page live

Can an agent resolve the issue in the repository—not just propose a patch?

Expert-verified GitHub issues paired with a service, repository, pull-request, incident, ADR, and standup world.

Explore 4 companion tasks ↓World repository ↗Run on Harbor ↗

500upstream Harbor tasks

4independent companion tasks

28files per task room

58+criteria per task

Baseline leaderboard

Solvable, discriminating, and honest about what ran.

All companion tasks are solvable by their reference traces, exact replays, and reject thirteen task-applicable controls including correct state with wrong reasoning, wrong target, and keyword stuffing. No paid or frontier model run is claimed. These 4 tasks are Blobfish-authored companions, not conversions of the 500-task upstream suite. The first two rows are harness controls; the model leaderboard remains open until a version-pinned submission exists.

EntryRoleTasksPass rateMean rewardTool calls

Reference solverDeterministic trace replay; not a model rank.

Solvability ceiling4/4100%1.00225/225 successful

Wrong-evidence causal controlExact-release deterministic negative control; not a model rank.

Adversarial discrimination floor0/40%0.00225/225 successful

Frontier model submissionsAgent, model revision, trials, costs, and full traces required.

Model leaderboard———Awaiting first run

Why these controls matter

A 100% reference ceiling shows the tasks are executable, not all failing. A 0% wrong-evidence floor shows the verifier rejects a plausible workflow that relies on the wrong source record. Neither row measures model capability.

Task browser

Read every released task.

These are 4 independent Blobfish v3.1 realism tasks—not excerpts relabeled from the upstream suite. Each is a high-level employee request backed by a multi-source evidence room, three operating options, a task-specific causal evaluation narrative, and deterministic state checks.

train66 reference stepshard · generator label

eng_resolve_ingest_incident

The ingest queue is healthy again, but the engineering manager does not want the incident closed from a Slack message alone. Can you establish what actually happened, how long customer-impacting mitigation took, and what follow-up belongs with the owning service? If the recorded timeline supports resolution, close the incident and leave the team the durable architecture decision and standup handoff it needs. Otherwise keep it open and state what evidence is missing.

How the employee outcome is evaluated

Evidence has to support the decision, the write, and the final answer.

Provider evidence: 26 exact reads
Live records: 31 correlated reads
Supported writes: 4
Proof: 4 readbacks

Reasoning contract

Incident 9401 on the ingest service was mitigated at minute 175. Close it out: set incident 9401's resolved_minute to 175 and status to 'resolved', work out how long it lasted (resolved_minute minus detected_minute), then record the follow-up decision by creating an ADR on service 9001 titled 'Rate-limit the ingest queue' with status 'accepted', decided_on '2026-07-21' and a decision that states the incident duration as 'N minutes'; finally post a standup note on service 9001 by author 'Rowan Novak' for day_label 'tuesday' whose body references incident 9401.

Deterministic success

All exact provider and live-domain evidence is read before the first write; the task-specific predicates support the chosen branch; only the authorized state delta and one completion row are created; the persisted result is reopened.

Exact total call order is not graded. Equivalent investigation is accepted when the same task-specific evidence is established before mutation, the supported state is produced, and get_adr_record → get_incident → get_standup_note → get_task_result reopens every changed provider row and the persisted result.

Required evidence systems

01gmail.messages.list
02gmail.messages.get
03google_drive.files.list
04google_drive.files.get
05google_sheets.spreadsheets.values.get
06slack.search_messages
07slack.conversations_replies
08search_services
09count_services
10get_service
11search_repositories
12count_repositories
13get_repository
14search_pull_requests
15count_pull_requests
16get_pull_request
17search_incidents
18count_incidents
19get_incident
20search_adr_records
21count_adr_records
22search_standup_notes
23count_standup_notes
24get_standup_note

Reference: PASS
Wrong-evidence floor: FAIL
Verifier: exact final tables, argument-aware causal trace, persisted result, and readback
Gradable: Yes

Operating options

✓
Complete the bounded supported outcomeCurrent authority, usable live evidence, and the available internal review window all support this task-scoped outcome.
02
Hold for an independent evidence reviewThis remains within authority but misses the documented need date even though the current evidence has no unresolved control conflict.
03
Extend remediation to neighboring recordsThe wider change exceeds both the approved immutable-record scope and the current incremental-cost authority.

Evidence files: 28
Exact criteria: 67
Allowed writes: 4
Call order: Not graded

Task evidenceTypeSizeSHA-256Open

operating-policy.mdassets/eng_resolve_ingest_incident/operating-policy.md

MD1.1 KB7dcd7a7ab44d10a7…View ↗

source-request.emlassets/eng_resolve_ingest_incident/source-request.eml

EML848 B5c6ece8b95f913ce…View ↗

release-and-incident-policy.pdfassets/eng_resolve_ingest_incident/release-and-incident-policy.pdf

PDF936 B52e826f77ff4ec56…View ↗

service-release-review.xlsxassets/eng_resolve_ingest_incident/service-release-review.xlsx

XLSX9.0 KB1969072c01f94b87…View ↗

approval-decision.jsonassets/eng_resolve_ingest_incident/approval-decision.json

JSON771 Bdb6f652a351751cb…View ↗

team-slack-intake.jsonassets/eng_resolve_ingest_incident/team-slack-intake.json

JSON785 B711d247f708ba3a8…View ↗

drive-source-index.jsonassets/eng_resolve_ingest_incident/drive-source-index.json

JSON381 Be25668fbaea06c25…View ↗

provider-contracts.jsonassets/eng_resolve_ingest_incident/provider-contracts.json

JSON24.0 KB1b15993ce973e0eb…View ↗

former-owner-suggestion.emlassets/eng_resolve_ingest_incident/former-owner-suggestion.eml

EML629 Baab31cd04a7c0461…View ↗

approval-scope.emlassets/eng_resolve_ingest_incident/approval-scope.eml

EML866 B2ebb3c628e622f7b…View ↗

operations-slack-thread.jsonassets/eng_resolve_ingest_incident/operations-slack-thread.json

JSON431 B055d0613798ca3d9…View ↗

drive-approval-record.jsonassets/eng_resolve_ingest_incident/drive-approval-record.json

JSON587 Bd408671e9f2b2f3f…View ↗

authority-revision-2026-3.mdassets/eng_resolve_ingest_incident/authority-revision-2026-3.md

MD275 B9b524ddebc3e53a8…View ↗

authority-revision-2025-4.mdassets/eng_resolve_ingest_incident/authority-revision-2025-4.md

MD220 Ba9725283b0ed87ce…View ↗

control-standard-2025-4.pdfassets/eng_resolve_ingest_incident/control-standard-2025-4.pdf

PDF816 Beeb946f91caf7fb3…View ↗

review-calendar.xlsxassets/eng_resolve_ingest_incident/review-calendar.xlsx

XLSX2.1 KBb891cd6a9e2975fc…View ↗

record-lineage.csvassets/eng_resolve_ingest_incident/record-lineage.csv

CSV504 B94e0b18dc4240aa1…View ↗

system-audit.logassets/eng_resolve_ingest_incident/system-audit.log

LOG876 Bfbefb014aa8000d2…View ↗

live-snapshot.jsonassets/eng_resolve_ingest_incident/live-snapshot.json

JSON14.3 KB33c0f997d1c39dc8…View ↗

evidence-status.yamlassets/eng_resolve_ingest_incident/evidence-status.yaml

YAML218 Bf38895c3fcb85d45…View ↗

custodian-notes.txtassets/eng_resolve_ingest_incident/custodian-notes.txt

TXT210 B181d9107dfcb99fb…View ↗

services-system-export.csvassets/eng_resolve_ingest_incident/services-system-export.csv

CSV2.4 KB0487485fa8aedd7b…View ↗

repositories-system-export.csvassets/eng_resolve_ingest_incident/repositories-system-export.csv

CSV1.9 KBa3fb83613bbd0443…View ↗

pull_requests-system-export.csvassets/eng_resolve_ingest_incident/pull_requests-system-export.csv

CSV2.4 KB4e9069e02b87d363…View ↗

incidents-system-export.csvassets/eng_resolve_ingest_incident/incidents-system-export.csv

CSV1.8 KB2156da23a52899c2…View ↗

adr_records-system-export.csvassets/eng_resolve_ingest_incident/adr_records-system-export.csv

CSV2.4 KBbd2584a658471a0b…View ↗

standup_notes-system-export.csvassets/eng_resolve_ingest_incident/standup_notes-system-export.csv

CSV4.8 KBc3e2e77d45eb75e8…View ↗

asset-manifest.jsonassets/eng_resolve_ingest_incident/asset-manifest.json

JSON4.4 KBb0555bbfbfa364ed…View ↗

Inspect all 67 deterministic criteria

01
identity · identity_adr_recordsFor eng_resolve_ingest_incident, resolve adr_records by immutable adr_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_adr_records"}, {"args": {}, "tool": "count_adr_records"}, {"args": {"limit": 50, "service_id": 9001}, "tool": "search_adr_records"}, {"args": {"service_id": 9001}, "tool": "count_adr_records"}].
02
investigation · investigation_adr_recordsEstablish the relevant and competing adr_records records before deciding the eng_resolve_ingest_incident outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
03
identity · identity_incidentsFor eng_resolve_ingest_incident, resolve incidents by immutable incident_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_incidents"}, {"args": {}, "tool": "count_incidents"}, {"args": {"limit": 50, "service_id": 9001}, "tool": "search_incidents"}, {"args": {"service_id": 9001}, "tool": "count_incidents"}, {"args": {"incident_id": 9401}, "tool": "get_incident"}].
04
investigation · investigation_incidentsEstablish the relevant and competing incidents records before deciding the eng_resolve_ingest_incident outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
05
identity · identity_pull_requestsFor eng_resolve_ingest_incident, resolve pull_requests by immutable pr_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_pull_requests"}, {"args": {}, "tool": "count_pull_requests"}, {"args": {"limit": 50, "repo_id": 9101}, "tool": "search_pull_requests"}, {"args": {"repo_id": 9101}, "tool": "count_pull_requests"}, {"args": {"pr_id": 9201}, "tool": "get_pull_request"}, {"args": {"pr_id": 9203}, "tool": "get_pull_request"}].
06
investigation · investigation_pull_requestsEstablish the relevant and competing pull_requests records before deciding the eng_resolve_ingest_incident outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
07
identity · identity_repositoriesFor eng_resolve_ingest_incident, resolve repositories by immutable repo_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_repositories"}, {"args": {}, "tool": "count_repositories"}, {"args": {"limit": 50, "service_id": 9001}, "tool": "search_repositories"}, {"args": {"service_id": 9001}, "tool": "count_repositories"}, {"args": {"repo_id": 9101}, "tool": "get_repository"}].
08
investigation · investigation_repositoriesEstablish the relevant and competing repositories records before deciding the eng_resolve_ingest_incident outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
09
identity · identity_servicesFor eng_resolve_ingest_incident, resolve services by immutable service_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_services"}, {"args": {}, "tool": "count_services"}, {"args": {"limit": 50, "name": "ingest"}, "tool": "search_services"}, {"args": {"name": "ingest"}, "tool": "count_services"}, {"args": {"service_id": 9001}, "tool": "get_service"}].
10
investigation · investigation_servicesEstablish the relevant and competing services records before deciding the eng_resolve_ingest_incident outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
11
identity · identity_standup_notesFor eng_resolve_ingest_incident, resolve standup_notes by immutable note_id and explicit relationships; matching display text is insufficient. Enforced by exact argument-aware selectors: [{"args": {"limit": 50}, "tool": "search_standup_notes"}, {"args": {}, "tool": "count_standup_notes"}, {"args": {"limit": 50, "service_id": 9001}, "tool": "search_standup_notes"}, {"args": {"service_id": 9001}, "tool": "count_standup_notes"}, {"args": {"note_id": 9501}, "tool": "get_standup_note"}, {"args": {"note_id": 9502}, "tool": "get_standup_note"}].
12
investigation · investigation_standup_notesEstablish the relevant and competing standup_notes records before deciding the eng_resolve_ingest_incident outcome; broad search, count, and detail evidence must agree. Enforced by successful exact MCP requests plus isolated seed.
13
investigation · investigation_provider_gmailFor eng_resolve_ingest_incident, use task-scoped gmail discovery and exact record reads to inspect source-request.eml (msg-0de03618b283a830ec95), former-owner-suggestion.eml (msg-d6245eb43680afa6f1c5), approval-scope.eml (msg-0dd7adecd243b0023705). The remaining 0 listed gmail records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"maxResults": 100, "q": "\"DB24-REPOBENCH-500-ENG_RESOLVE_INGEST_INCIDENT\"", "userId": "me"}, "tool": "gmail.messages.list"}, {"args": {"format": "raw", "id": "msg-0de03618b283a830ec95", "userId": "me"}, "tool": "gmail.messages.get"}, {"args": {"format": "raw", "id": "msg-d6245eb43680afa6f1c5", "userId": "me"}, "tool": "gmail.messages.get"}, {"args": {"format": "raw", "id": "msg-0dd7adecd243b0023705", "userId": "me"}, "tool": "gmail.messages.get"}].
14
investigation · investigation_provider_google_driveFor eng_resolve_ingest_incident, use task-scoped google_drive discovery and exact record reads to inspect operating-policy.md (file-331bd1be7b018b99dfa5), release-and-incident-policy.pdf (file-ca128c4fcd06048c6027), approval-decision.json (file-c9cf71aec9d28ed354d6), drive-source-index.json (file-a81dca6bd5b9d4d8e15c), drive-approval-record.json (file-963635f4f45eba606f03), authority-revision-2026-3.md (file-96d52b14cba35284b785), authority-revision-2025-4.md (file-2a97bf094d3ad2103535), record-lineage.csv (file-50cc0f62a279cd4615f6), live-snapshot.json (file-03624662d241f53905f2), services-system-export.csv (file-d89c017cda0078e375dd), repositories-system-export.csv (file-51e7d6d3356a584ef7c4), pull_requests-system-export.csv (file-05fc1d83f0887b69344c), incidents-system-export.csv (file-d085754be49bb71b428f), adr_records-system-export.csv (file-cceb953503f8072d9986), standup_notes-system-export.csv (file-aa59ebf413bb275d4aa6), asset-manifest.json (file-9840ca1cc587f366f7b5). The remaining 5 listed google_drive records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"fields": "files(id,name,mimeType,modifiedTime,version)", "pageSize": 1000, "q": "fullText contains 'DB24-REPOBENCH-500-ENG_RESOLVE_INGEST_INCIDENT'"}, "tool": "google_drive.files.list"}, {"args": {"alt": "media", "fileId": "file-331bd1be7b018b99dfa5"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-ca128c4fcd06048c6027"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-c9cf71aec9d28ed354d6"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-a81dca6bd5b9d4d8e15c"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-cceb953503f8072d9986"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-d085754be49bb71b428f"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-05fc1d83f0887b69344c"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-51e7d6d3356a584ef7c4"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-d89c017cda0078e375dd"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-aa59ebf413bb275d4aa6"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-963635f4f45eba606f03"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-96d52b14cba35284b785"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-2a97bf094d3ad2103535"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-50cc0f62a279cd4615f6"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-03624662d241f53905f2"}, "tool": "google_drive.files.get"}, {"args": {"alt": "media", "fileId": "file-9840ca1cc587f366f7b5"}, "tool": "google_drive.files.get"}].
15
investigation · investigation_provider_google_sheetsFor eng_resolve_ingest_incident, use task-scoped google_sheets discovery and exact record reads to inspect service-release-review.xlsx (sheet-8c63793e4ae952ec0fc9), review-calendar.xlsx (sheet-ea67636b6d3c3d1d77ae). The remaining 0 listed google_sheets records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"range": "A:Z", "spreadsheetId": "sheet-8c63793e4ae952ec0fc9"}, "tool": "google_sheets.spreadsheets.values.get"}, {"args": {"range": "A:Z", "spreadsheetId": "sheet-ea67636b6d3c3d1d77ae"}, "tool": "google_sheets.spreadsheets.values.get"}].
16
investigation · investigation_provider_slackFor eng_resolve_ingest_incident, use task-scoped slack discovery and exact record reads to inspect team-slack-intake.json (thread-05e91a9a2228a65239cb), operations-slack-thread.json (thread-a14373551b42e98541e7). The remaining 0 listed slack records remain available for open-ended investigation but are not falsely claimed as inspected. The communication or file content is evidence; it does not precompute the business outcome. Enforced by exact documented provider selectors: [{"args": {"count": 100, "query": "\"DB24-REPOBENCH-500-ENG_RESOLVE_INGEST_INCIDENT\"", "sort": "timestamp", "sort_dir": "asc"}, "tool": "slack.search_messages"}, {"args": {"channel": "#engineering-operations", "limit": 100, "ts": "1724937600.501067"}, "tool": "slack.conversations_replies"}, {"args": {"channel": "#engineering-operations", "limit": 100, "ts": "1724937600.818398"}, "tool": "slack.conversations_replies"}].
17
authority · authority_effective_revisionUse the effective 2026.3 operating control for eng_resolve_ingest_incident, corroborated by the approval and Drive revision records. Enforced by current-control PDF, Drive record, and policy revision identity.
18
authority · authority_retired_conflictTreat the retained 2025.4 note for eng_resolve_ingest_incident as a competing historical hypothesis, not as permission or current authority. Enforced by current-versus-retired revision comparison.
19
correlation · correlation_cross_system_identityCorrelate the employee request, controlled workbook, system exports, Slack discussion, Drive record, and approval email for eng_resolve_ingest_incident by case and immutable record IDs. Enforced by cross-file case IDs and exact world selectors.
20
correlation · correlation_live_over_snapshotReconcile the released snapshot for eng_resolve_ingest_incident to successful live MCP reads before relying on either source; the files alone do not establish the final outcome. Enforced by asset digests plus exact live trace.
21
correlation · correlation_count_detail_relationshipFor eng_resolve_ingest_incident, reconcile broad counts with detail records and foreign-key relationships so duplicates and similarly named distractors are not collapsed. Enforced by search/count/get selector intersection.
22
identity · identity_asset_lineageBind every cited eng_resolve_ingest_incident file to its case-specific asset-manifest digest and reject evidence copied from another task, even when its filename and business vocabulary look plausible. Enforced by globally unique asset SHA-256 census and manifest membership.
23
investigation · investigation_structured_exportsReconcile the eng_resolve_ingest_incident structured exports (adr_records-system-export.csv, incidents-system-export.csv, pull_requests-system-export.csv, repositories-system-export.csv, services-system-export.csv, standup_notes-system-export.csv) with the live records they represent; no single export is sufficient when related tables disagree. Enforced by exact table selectors, live-state snapshot, and parser-validated CSV records.
24
investigation · investigation_controlled_workbookUse the parser-validated review-calendar workbook for eng_resolve_ingest_incident to test timing and capacity assumptions against current records, while treating workbook conclusions as evidence rather than authorization. Enforced by XLSX parser gate, workbook digest, and exact temporal selectors.
25
correlation · correlation_communication_authority_splitFor eng_resolve_ingest_incident, distinguish what the source request, operations Slack thread, and former-owner message say from what the signed approval and current control actually authorize. Enforced by communication-record correlation plus current-authority and approval-scope checks.
26
correlation · correlation_provider_live_lineageJoin the task-scoped Gmail, Drive, Sheets, and Slack records for eng_resolve_ingest_incident to the exact live-domain selectors by case id, provider id, immutable row id, and effective revision; a same-name file or row from another task is not equivalent evidence. Enforced by provider asset ids and SHA-256 manifest joined to argument-aware domain selectors.
27
alternatives · alternatives_blast_radiusCompare the complete table-level blast radius of the supported eng_resolve_ingest_incident branch with the hold and broad-cleanup alternatives; reject any option that changes an unapproved neighbor even if its headline answer is correct. Enforced by all-table before/after digests and unauthorized-write control.
28
decision · decision_citation_lineageCarry the selected eng_resolve_ingest_incident decision into the result row with the exact current evidence IDs used to derive it, so a correct-looking answer backed by stale or cross-task evidence fails. Enforced by exact result evidence set and wrong-evidence control.
29
alternatives · alternatives_supportedKeep the bounded evidence-supported outcome for eng_resolve_ingest_incident available only if effective authority and current records agree. Enforced by exact final-state and result contract.
30
alternatives · alternatives_holdRetain a hold/escalate branch for eng_resolve_ingest_incident if controlling records conflict or authority is absent; difficulty alone is not a reason to hold. Enforced by wrong-decision negative control.
31
alternatives · alternatives_broadReject a broad neighboring-record cleanup for eng_resolve_ingest_incident, even when similar names or stale notes make it superficially plausible. Enforced by unauthorized-write negative control.
32
alternatives · alternatives_no_changeEvaluate a no-domain-change outcome for eng_resolve_ingest_incident: retain it only when the exact live predicates do not authorize a mutation; otherwise it is an incomplete state-only or shortcut branch. Enforced by task-specific source VCode plus no-op and state-only controls.
33
decision · decision_supported_branchDerive the selected eng_resolve_ingest_incident branch from the intersection of authority, identity, relationship, timing, and live-state evidence—not from the prompt's requested outcome. Enforced by source VCode, exact state digests, and task-result row.
34
decision · decision_task_specific_causal_pathThe deterministic reasoning for eng_resolve_ingest_incident is task-specific: Incident 9401 on the ingest service was mitigated at minute 175. Close it out: set incident 9401's resolved_minute to 175 and status to 'resolved', work out how long it lasted (resolved_minute minus detected_minute), then record the follow-up decision by creating an ADR on service 9001 titled 'Rate-limit the ingest queue' with status 'accepted', decided_on '2026-07-21' and a decision that states the incident duration as 'N minutes'; finally post a standup note on service 9001 by author 'Rowan Novak' for day_label 'tuesday' whose body references incident 9401. The model must establish each predicate from the provider evidence and live domain records, then carry only the supported branch into state. Enforced by exact source VCode, authored selectors, before/after state, and adversarial branch controls.
35
procedure · procedure_successful_evidenceEvery exact contextual selector required for eng_resolve_ingest_incident must succeed; a failed or differently filtered call is not equivalent evidence. Enforced by argument-aware successful trace.
36
procedure · procedure_provider_discovery_before_detailFor eng_resolve_ingest_incident, complete the task-scoped Gmail, Drive, and Slack discovery calls before relying on exact message, file, spreadsheet, or thread records; the verifier accepts equivalent causal ordering but requires every exact discovery and detail selector before state changes. Enforced by provider list/search and exact-get trace selectors before first mutation.
37
procedure · procedure_read_before_writeComplete all 57 causal reads for eng_resolve_ingest_incident before the first domain mutation or persisted completion result. Enforced by trace sequence bounds.
38
procedure · procedure_result_after_workPersist the exact eng_resolve_ingest_incident result only after any supported domain transition has completed. Enforced by source-mutation-before-submit sequence check.
39
procedure · procedure_post_write_readbackReopen the exact persisted eng_resolve_ingest_incident result after submission so verification observes stored state rather than trusting the write response. Enforced by exact get_task_result selector after submit.
40
state · state_adr_recordsProduce only the supported eng_resolve_ingest_incident transition in adr_records: added adr_id=['9302']; removed adr_id=none; updated none. The complete final table must match digest 7215e434310d669024856b5976810f9a2db33f9d5c2d7515d98f65315c48424e. Enforced by exact complete-table digest and source VCode.
41
state · state_incidentsProduce only the supported eng_resolve_ingest_incident transition in incidents: added incident_id=none; removed incident_id=none; updated ["incident_id=9401 (resolved_minute:None→175, status:'open'→'resolved')"]. The complete final table must match digest 880a07035d44d7061218290e7e70f01371515814ef377819a63c8b75d4120fff. Enforced by exact complete-table digest and source VCode.
42
state · state_standup_notesProduce only the supported eng_resolve_ingest_incident transition in standup_notes: added note_id=['9503']; removed note_id=none; updated none. The complete final table must match digest 2689ceae82e58a2816d9ce66a61aebabd4c5c4b28512015552c3aa3a0dd33728. Enforced by exact complete-table digest and source VCode.
43
state · state_authorized_transition_01The eng_resolve_ingest_incident state transition with exact arguments {"incident_id": 9401, "resolved_minute": 175, "status": "resolved"} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
44
state · state_authorized_transition_02The eng_resolve_ingest_incident state transition with exact arguments {"decided_on": "2026-07-21", "decision": "Incident 9401 lasted 75 minutes; rate-limit producers to keep consumers from stalling.", "service_id": 9001, "status": "accepted", "title": "Rate-limit the ingest queue"} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
45
state · state_authorized_transition_03The eng_resolve_ingest_incident state transition with exact arguments {"author": "Rowan Novak", "body": "Incident 9401 resolved; writing the rate-limit ADR follow-up.", "day_label": "tuesday", "service_id": 9001} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
46
state · state_authorized_transition_04The eng_resolve_ingest_incident state transition with exact arguments {"channel": "#engineering-operations", "client_msg_id": "1bd879dd01f01ffc28e1d0c7bed69585", "text": "Resolve incident 9401 at minute 175, record its 75-minute duration in the accepted rate-limit ADR, and post the durable standup handoff. I reconciled 115 candidate rows across 6 live record families, excluded 108 neighbor rows, and retained 7 immutable task records. The selected outcome is scheduled for 2026-09-03 (-1 days versus the 2026-09-04 need date), costs $2200, fits with 6 internal hours to spare, and is authorized by APR-2b951675bc7be5da. The broader neighbor remediation requires additional approval and was not executed. Exact answer: 75."} must succeed and be reflected in the final audited tables. Enforced by successful exact MCP trace plus complete-table digest.
47
state · state_source_outcomeAll 2 task-specific executable outcome assertions for eng_resolve_ingest_incident must hold against the final sandbox state. Enforced by checked-in deterministic VCode.
48
state · state_result_rowCreate exactly one task-result row for eng_resolve_ingest_incident containing the supported decision, exact summary, evidence IDs, causal workpaper, and exact answer when applicable. Enforced by persisted task_results state.
49
answer · answer_result_submittedReturn an audit-ready eng_resolve_ingest_incident conclusion in the persisted result row rather than only narrating intended work. Enforced by exact task-result state verifier.
50
answer · answer_evidence_citationsCite the current policy, source request, controlled workbook, and effective source document for eng_resolve_ingest_incident; stale authority cannot substitute. Enforced by exact required evidence set.
51
answer · answer_exact_valueReturn the exact task-supported scalar for eng_resolve_ingest_incident, derived from the live records rather than copied from an asset. Enforced by deterministic submitted-answer equality.
52
containment · containment_write_scopeUse only the eng_resolve_ingest_incident authorized domain mutation set and the task-result handoff; no neighboring table or record may change. Enforced by allowed-write trace plus all-table digests.
53
containment · containment_no_rejected_mutationNo state-changing request in eng_resolve_ingest_incident may fail and then be hidden by a later correct-looking final answer. Enforced by successful mutation trace.
54
identity · identity_H01_scope_identityLocate eng_resolve_ingest_incident through case DB24-REPOBENCH-500-ENG_RESOLVE_INGEST_INCIDENT and the exact immutable target map {"adr_records": [], "incidents": [9401], "pull_requests": [9201, 9203], "repositories": [9101], "services": [9001], "standup_notes": [9501, 9502]}. Enforced by exact workpaper.scope plus successful argument-aware search/count/get evidence.
55
decision · decision_H02_requirement_derivationApply the controlled task rule from operating-policy.md#task-specific-decision-control and derive all 6 required record families before choosing an outcome. Enforced by exact workpaper.requirement and D02.
56
investigation · investigation_H03_availability_reconciliationReconcile 115 observed candidate rows, 108 excluded neighbors, and 7 usable targets. Enforced by exact workpaper.reconciliation and D03-D05.
57
decision · decision_H04_gap_or_boundNet 6 required families against 6 covered families into exact gap 0. Enforced by exact workpaper.gapOrBound and D06.
58
authority · authority_H05_external_constraintApply independent approval APR-2b951675bc7be5da and its $2400 cost boundary. Enforced by exact workpaper.externalConstraint corroborated by approval email and Drive record.
59
investigation · investigation_H06_internal_constraintApply the 2026-09-04 business need, 17 available hours, and $200 hourly rate from the controlled calendar. Enforced by exact workpaper.internalConstraint and D07-D10.
60
alternatives · alternatives_H07_qualified_optionsGrade all three task-specific alternatives with exact outcome, date, signed variance, incremental cost, authority status, and one selected branch. Enforced by exact workpaper.options equality and wrong-reasoning control.
61
decision · decision_H08_recommendationSelect evidence-backed-scoped-outcome with exact outcome 'Resolve incident 9401 at minute 175, record its 75-minute duration in the accepted rate-limit ADR, and post the durable standup handoff.'. Enforced by exact workpaper.recommendation and D13.
62
decision · decision_H09_control_comparisonCompare outcome 2026-09-03 to need 2026-09-04 into signed variance -1 and status ON_OR_BEFORE_NEED. Enforced by independently recomputed exact workpaper.controlComparison and D11.
63
authority · authority_H10_authority_applicationApply authority APR-2b951675bc7be5da to the selected immutable scope and reject broad-neighbor-remediation. Enforced by exact workpaper.authorityApplication and D12.
64
state · state_H11_controlled_executionExecute exactly 3 authorized domain writes across 3 changed tables, perform 3 exact-record provider readbacks, then reopen the persisted result. Enforced by exact writes, complete-table digests, provider-native exact-record readbacks, persisted result, and result readback trace.
65
answer · answer_H12_stakeholder_communicationPost the exact stakeholder handoff to #engineering-operations carrying the selected outcome, timing, cost, capacity constraint, and authority. Enforced by exact slack.chat_postMessage call and persisted stakeholder-message row.
66
answer · answer_H13_exact_intermediatesPersist all 13 dependent derivations and every intermediate requirement, reconciliation, constraint, option, authority, state, and answer field exactly. Enforced by exact causal-workpaper equality; wrong-reasoning negative control.
67
containment · containment_visible_hidden_partitionThe 28 files visible for eng_resolve_ingest_incident contain evidence only; verifier digests, the oracle trajectory, and precomputed outcomes remain outside the asset room. Enforced by release-tree partition and leakage gate.

Generator label only; no empirical model difficulty claim is published.

Episode trace

Follow the tools, outputs, and verdict.

Switch between a successful reference replay and a real failing floor episode. Both start from the same immutable world state; neither is a model leaderboard row.

deterministic solvability ceiling

Reference solver

A checked-in v3 causal investigation replayed against a fresh SQLite copy. This proves the task can pass; it is not a model submission.

Task: eng_resolve_ingest_incident
Steps: 66
Reward: 1.0
Status: passed

00
Task loadeduser → agent
Fresh seeded database copied into an isolated episode.

gmail.messages.listtool success

Arguments

{
  "maxResults": 100,
  "q": "\"DB24-REPOBENCH-500-ENG_RESOLVE_INGEST_INCIDENT\"",
  "userId": "me"
}

Result

{
  "messages": [
    {
      "id": "msg-0de03618b283a830ec95",
      "threadId": "msg-0de03618b283a830ec95"
    },
    {
      "id": "msg-d6245eb43680afa6f1c5",
      "threadId": "msg-d6245eb43680afa6f1c5"
    },
    {
      "id": "msg-0dd7adecd243b0023705",
      "threadId": "msg-0dd7adecd243b0023705"
    }
  ],
  "resultSizeEstimate": 3
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-0de03618b283a830ec95",
  "userId": "me"
}

Result

{
  "id": "msg-0de03618b283a830ec95",
  "name": "source-request.eml",
  "mimeType": "message/rfc822",
  "sha256": "5c6ece8b95f913cecd97dde21a464b55e9e4febbe6a3b5b75833f2bdbd25bab4",
  "bytes": 848
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-d6245eb43680afa6f1c5",
  "userId": "me"
}

Result

{
  "id": "msg-d6245eb43680afa6f1c5",
  "name": "former-owner-suggestion.eml",
  "mimeType": "message/rfc822",
  "sha256": "aab31cd04a7c04613f0577d8887c14fdff1c0550582433fcbcb1905b4a324523",
  "bytes": 629
}

gmail.messages.gettool success

Arguments

{
  "format": "raw",
  "id": "msg-0dd7adecd243b0023705",
  "userId": "me"
}

Result

{
  "id": "msg-0dd7adecd243b0023705",
  "name": "approval-scope.eml",
  "mimeType": "message/rfc822",
  "sha256": "2ebb3c628e622f7b5ebf65f2a0599d82295d8cede210645523176cecbb46aa77",
  "bytes": 866
}

google_drive.files.listtool success

Arguments

{
  "fields": "files(id,name,mimeType,modifiedTime,version)",
  "pageSize": 1000,
  "q": "fullText contains 'DB24-REPOBENCH-500-ENG_RESOLVE_INGEST_INCIDENT'"
}

Result

{
  "files": [
    {
      "id": "file-331bd1be7b018b99dfa5",
      "name": "operating-policy.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-ca128c4fcd06048c6027",
      "name": "release-and-incident-policy.pdf",
      "mimeType": "application/pdf",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "sheet-8c63793e4ae952ec0fc9",
      "name": "service-release-review.xlsx",
      "mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-c9cf71aec9d28ed354d6",
      "name": "approval-decision.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-a81dca6bd5b9d4d8e15c",
      "name": "drive-source-index.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d2757e0f09b9a9cbb9da",
      "name": "provider-contracts.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-963635f4f45eba606f03",
      "name": "drive-approval-record.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-96d52b14cba35284b785",
      "name": "authority-revision-2026-3.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-2a97bf094d3ad2103535",
      "name": "authority-revision-2025-4.md",
      "mimeType": "text/markdown",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d5fa9391e2ea9e8359e2",
      "name": "control-standard-2025-4.pdf",
      "mimeType": "application/pdf",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "sheet-ea67636b6d3c3d1d77ae",
      "name": "review-calendar.xlsx",
      "mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-50cc0f62a279cd4615f6",
      "name": "record-lineage.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-4e35946b7be125e6bad9",
      "name": "system-audit.log",
      "mimeType": "text/plain",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-03624662d241f53905f2",
      "name": "live-snapshot.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-a2e58971924732c7b753",
      "name": "evidence-status.yaml",
      "mimeType": "application/yaml",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-ccdb9adf670cb4370406",
      "name": "custodian-notes.txt",
      "mimeType": "text/plain",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d89c017cda0078e375dd",
      "name": "services-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-51e7d6d3356a584ef7c4",
      "name": "repositories-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-05fc1d83f0887b69344c",
      "name": "pull_requests-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-d085754be49bb71b428f",
      "name": "incidents-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-cceb953503f8072d9986",
      "name": "adr_records-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-aa59ebf413bb275d4aa6",
      "name": "standup_notes-system-export.csv",
      "mimeType": "text/csv",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    },
    {
      "id": "file-9840ca1cc587f366f7b5",
      "name": "asset-manifest.json",
      "mimeType": "application/json",
      "modifiedTime": "2026-08-29T09:00:00Z",
      "version": "1"
    }
  ]
}