Reference solutionExact-release deterministic replay
100 tasks100.0%
38.94 calls avg.Solvability control only; this is not a model leaderboard entry.
Blobfish ResearchDevOpsBench-100 v3.2.7Public
Can an agent run the incident, ship the fix, and prove it landed?
DevOpsBench-100 measures complete operations work over one executable company: root-cause the symptom, change the code, ride the canary, and close the incident — through 97 typed MCP tools, graded only by executable checks on the investigation, decision, and world left behind.
100DevOps tickets
97MCP tools
72world tables
55–73criteria per task
0LLM grading calls
The release gate executed 1,600 isolated runs: 100/100 oracle passes, 100/100 exact deterministic replays, and 1,400 adversarial controls with 0 false accepts. 100/100 write contracts prove authorized scope, an agent-visible destination, typed provider inputs, semantic prose, persisted-state readback, and containment. Wrong-target and keyword-stuffing attacks were rejected; hidden reference text and hidden serialization required: 0 and 0. No model has run this exact release yet; the leaderboard exposes that empty state.
Inspect the verifier contract →Measured performance
Leaderboard
Ranking
Model leaderboard
Only complete, version-pinned model runs on this exact release receive a rank.
RankAgentCoverageScoreRun
No model submissions yet — awaiting first run. The reference oracle establishes the ceiling; model rows appear once a pinned run is published.
Qualification evidence
Measured controls
Oracle and adversarial replays prove solvability and verifier discrimination. They are measured, but never ranked as model submissions.
ClassControlCoverageScoreEvidence
60%Correctness
The requested outcome is true in the final world state: the right diagnosis, code change, migration, flag, or record — with world invariants intact.
30%Deployment
The change actually landed the way operations requires: CI green, merged, deployed through canary where policy demands it, rollbacks clean.
10%Quality
Professional closure: tickets updated, incidents resolved, status published, evidence recorded, no collateral writes.
100 distinct tasks
Task browser
100 shown
dob100-001-rca-analytics-egress-blockedAIOps root-cause analysis
RCA Analytics Egress Blocked
Analytics connections are refused, not slow. The incident record describes the visible effect, but it does not establish whether the apparent owner, a dependency, or the deployed configuration is responsible. The decision affects reporting freshness, warehouse replica access, scheduled rollups, and downstream decision data. The disputed mechanism is outbound connectivity, DNS reachability, firewall policy, and dependency access. Explain which conclusion survives cross-checking, record it in the existing incident w…
- Seeded files
- 53
- Reference solution
- 34 MCP calls
Complete task prompt
60 graded criteria summarizedAnalytics connections are refused, not slow. The incident record describes the visible effect, but it does not establish whether the apparent owner, a dependency, or the deployed configuration is responsible. The decision affects reporting freshness, warehouse replica access, scheduled rollups, and downstream decision data. The disputed mechanism is outbound connectivity, DNS reachability, firewall policy, and dependency access. Explain which conclusion survives cross-checking, record it in the existing incident work item, and leave the live evidence that lets the incoming engineer reproduce it; production controls should remain unchanged. This diagnosis will decide whether analytics worker enters recovery on 2026-03-04. Before closing the incident record, give the incoming owner the earliest credible recovery date, the constraint behind it, and whether approved acceleration or an exception would materially help.How the employee outcome is evaluated
16 semantic milestonesReasoning, persisted state, and the answer must agree.
Acceptance is causal and state-based: all semantic milestones must hold over successful material reads, provider state, post-write readback, the append-only audit log, and write-scope containment.
- Strict success
- all 16 semantic milestones hold over causal tool evidence, the graded capacity-plan decision record, final world state, readback, and containment
- Ordering policy
- The reference trajectory is illustrative. Material evidence may be investigated in any valid order and equivalent query shapes may add safe optional arguments; each causal read must succeed before its dependent mutation, and persisted state must be read back afterward.
Inspect the task-specific causal milestones
- investigationinvestigation.scope
Treat “Analytics connections are refused, not slow.” as one immutable cross-system work item: correlate its Jira record to the linked GitHub evidence and exclude neighboring work that merely shares a name.
- investigationinvestigation.authority
Use OPS-CONTROL-2026.03 as the operative control and treat OPS-CONTROL-2025.11 as a conflicting historical shortcut before deciding “Analytics connections are refused, not slow.”.
- investigationinvestigation.live_state
For “Analytics connections are refused, not slow.”, interrogate the live dependency path (from analytics-worker), the infrastructure inventory, current service logs (analytics-worker), and runtime resource and process state (analytics-worker); tracker prose and seeded files alone cannot establish the current answer.
- analysisanalysis.causal_reasoning
Build the causal case for “Analytics connections are refused, not slow.”: first resolve the same work across the authoritative Jira work item, the cross-tracker link registry, and the linked GitHub issue population; then use the operating-control index and the exact control revision to select the effective rule over the retired shortcut; reconcile ownership, approval, and reported context through the case-room conversation, the PagerDuty change-event history, and the current customer-status record; finally test the competing explanation against the live dependency path (from analytics-worker), the infrastructure inventory, current service logs (analytics-worker), and runtime resource and process state (analytics-worker). All 20 exact causal facts must be present inside the 28-read investigation before they support this outcome: record a configuration error diagnosis localizing responsibility to analytics-worker and identifying pg-replica with cited evidence; move the employee work item to done. Preserve this additional control: the responsible service is analytics-worker; the root cause is a misconfig fault; the offending key is pg-replica; an investigation must not change production state.
- analysisanalysis.capacity_plan
Prove that the same outcome for “Analytics connections are refused, not slow.” is deliverable through analytics-worker, using scattered records: 2 healthy replicas per zone across 3 zones means 6 required; 7 observed less 2 reserved for inventory leaves 5 usable and a 1-replica gap. Reconcile that gap with CloudCap's 2026-03-06 standard and 2026-03-04 expedited deliveries, the published change windows, and the independent 2026-03-04 customer date.
- decisiondecision.supported_path
Choose the branch for “Analytics connections are refused, not slow.” that can legitimately produce this business state: record a configuration error diagnosis localizing responsibility to analytics-worker and identifying pg-replica with cited evidence; move the employee work item to done. Reject a stale-record shortcut, an unsupported hold, and any broader workaround.
- decisiondecision.options
Compare three concrete, linked readiness options for “Analytics connections are refused, not slow.” through analytics-worker: standard capacity finishes 2026-03-07 at USD 0; expedited capacity finishes 2026-03-05 at USD 750; releasing reserved capacity finishes 2026-03-03 at USD 1400 but requires approval beyond CHG-1. Recommend expedite_capacity, record its 2026-03-05 outcome and +1-day variance, and report LATE honestly.
- statestate.primary
Establish the exact business state for “Analytics connections are refused, not slow.”: record a configuration error diagnosis localizing responsibility to analytics-worker and identifying pg-replica with cited evidence; move the employee work item to done.
- statestate.coordination
Coordinate the linked records for “Analytics connections are refused, not slow.”: close ticket OPS-163 once reported.
- verificationverification.outcome
Prove the observable result for “Analytics connections are refused, not slow.” after the transition: reopen the employee work item and confirm the persisted result.
- verificationverification.readback
After the last guarded transition for “Analytics connections are refused, not slow.”, reopen the employee work item, then reopen the completion conversation so persistence and communication are both proven.
- executionexecution.sequence
Respect the causal order for “Analytics connections are refused, not slow.”: derive the capacity plan only after every source fact it depends on has been read successfully; record the capacity plan before stating it in the completion handoff.
- containmentcontainment.scope
Keep “Analytics connections are refused, not slow.” contained to its supported records and service boundary: fabricate no records or evidence; preserve every frozen out-of-scope record; introduce no hardcoded credentials.
- answeranswer.insights
Leave the employee-facing conclusion for “Analytics connections are refused, not slow.” with the exact supported insight: record the evidence-backed root cause for analytics connection refused; record the evidence that supports the finding; write a natural scoped completion note, not a keyword list or serialized benchmark object; state the selected option (expedite_capacity) in the scoped completion handoff; state the outcome date (2026-03-05) in the scoped completion handoff; state the approval reference (CHG-1) in the scoped completion handoff; state the timing status (LATE) in the scoped completion handoff; state the binding constraint (CloudCap expedited capacity on 2026-03-04) in the scoped completion handoff.
- executionexecution.efficiency
Complete “Analytics connections are refused, not slow.” without unsafe retries or unproductive loops: reach a conclusion within 14 tool calls; avoid repeated validation runs that make no progress.
- executionexecution.delivery
Close “Analytics connections are refused, not slow.” only after the business state and readbacks are complete: close OPS-163 only once the finding is submitted, not before.
Decision space (3 grounded options)
- standard_capacity_plan: outcome 2026-03-07, incremental cost USD 0, FEASIBLE_WITH_INFERIOR_TRADEOFF: Standard CloudCap delivery 2026-03-06 reaches the 2026-03-07 window at USD 0. Standard delivery only reaches the 2026-03-07 window, 2 day(s) after expedite_capacity, and would land 3 day(s) after the cutover; it is authorised but inferior.
- expedite_capacity: outcome 2026-03-05, incremental cost USD 750, SUPPORTED_AND_APPROVED — selected: Expedited CloudCap delivery 2026-03-04 reaches the 2026-03-05 window for USD 750. Expedited CloudCap delivery on 2026-03-04 lands in the 2026-03-05 change window, 1 day(s) after the 2026-03-04 cutover, the earliest authorised result, keeps the 2 reserved replicas untouched, and carries the documented incremental cost of USD 750.
- release_reserved_capacity: outcome 2026-03-03, incremental cost USD 1400, SEPARATE_APPROVAL_OR_POLICY_EXCEPTION_REQUIRED: Releasing the 2 replicas reserved for the inventory freeze covers the gap today but needs approval beyond CHG-1. Releasing the 2 replicas reserved for the inventory freeze would cover the 1-replica gap today, but it breaks a reservation another team depends on, needs incident-commander approval beyond CHG-1, and adds the USD 1400 re-provisioning charge.
Inspect all 60 deterministic criteria
- Investigation: resolve DOB-001 through its immutable Jira key, linked GitHub issue, and cross-tracker relationship [correctness.v4_case_identity_resolved]
- Investigation: compare current control OPS-CONTROL-2026.03 with retired control OPS-CONTROL-2025.11 before acting [correctness.v4_authority_reconciled]
- Investigation: correlate the task-specific analytics-worker provider state instead of relying on files or tracker text alone [correctness.v4_live_state_correlated]
- Analysis: the responsible service is analytics-worker [correctness.service_localized]
- Analysis: the root cause is a misconfig fault [correctness.fault_type_correct]
- Analysis: the offending key is pg-replica [correctness.offending_key_correct]
- Analysis: an investigation must not change production state [deployment.investigation_was_read_only]
- Analysis: complete all 20 materially causal evidence joins inside the larger reference investigation before changing state [correctness.v4_material_evidence_complete]
- Analysis: Preserved 2026-03-04 from the published status-page cutover notice as the control date; did not infer urgency from the ticket title. [correctness.v5_answer_business_need_date]
- Analysis: Read 2 healthy replicas per production zone from the DOB-001 change-readiness standard. [correctness.v5_answer_replicas_per_zone]
- Analysis: Read 3 production zones from the PagerDuty scale record for PD-DOB-001. [correctness.v5_answer_production_zones]
- Analysis: Derived 2 per zone x 3 zones = 6 replicas required before the analytics-worker cutover. [correctness.v5_answer_required_replicas]
- Analysis: Read 7 replicas in the analytics-worker production pool from the PagerDuty scale record. [correctness.v5_answer_observed_replicas]
- Analysis: Excluded 2 replicas reserved for the inventory freeze in Linear CAP-1. [correctness.v5_answer_reserved_replicas]
- Analysis: Calculated 7 observed - 2 reserved = 5 usable replicas. [correctness.v5_answer_usable_replicas]
- Analysis: Calculated 6 required - 5 usable = 1 replicas uncovered. [correctness.v5_answer_replica_gap]
- Analysis: Kept every capacity quantity in replicas. [correctness.v5_answer_quantity_unit]
- Analysis: Read 2026-03-06 as CloudCap's independently confirmed standard delivery date from vendor order VEND-1. [correctness.v5_answer_standard_capacity_date]
- Analysis: Read 2026-03-04 as CloudCap's independently confirmed expedited delivery date (USD 750) from vendor order VEND-1. [correctness.v5_answer_expedited_capacity_date]
- Analysis: Bound the vendor request to the 1 uncovered replicas rather than the full 6-replica requirement. [correctness.v5_answer_capacity_request_replicas]
- Analysis: Read the analytics-worker change calendar (2026-03-05, 2026-03-07, 2026-03-09, 2026-03-11) from the current operating control and identified 2026-03-05 as the next window. [correctness.v5_answer_next_change_window]
- Analysis: read the readiness standard, CloudCap order VEND-1, change approval CHG-1, the Linear reservation and the status-page cutover notice before changing state [correctness.v5_capacity_evidence_complete]
- Decision: derive and execute the task-supported branch rather than the stale-note or broad-workaround alternatives [correctness.v4_supported_path_selected]
- Decision: record the DOB-001 capacity plan once as the JSON reconciliation answer to DOB-001-capacity-plan [correctness.v5_capacity_plan_recorded]
- Decision: Calculated standard_capacity_plan outcome as 2026-03-07: the first change window on or after standard delivery. [correctness.v5_answer_standard_plan_completion]
- Decision: Calculated expedite_capacity outcome as 2026-03-05: the first change window on or after expedited delivery. [correctness.v5_answer_expedited_plan_completion]
- Decision: Calculated release_reserved_capacity outcome as 2026-03-03 and kept its separate-approval condition. [correctness.v5_answer_reserved_release_completion]
- Decision: Compared the date, cost and authority of standard_capacity_plan, expedite_capacity and release_reserved_capacity; selected expedite_capacity as the best currently authorised plan under the readiness standard. [correctness.v5_answer_recommended_option]
- Decision: Calculated 2026-03-05 as the supported outcome date for expedite_capacity. [correctness.v5_answer_recommended_outcome_date]
- Decision: Applied USD 750 as the documented incremental cost of expedite_capacity. [correctness.v5_answer_recommended_incremental_cost_usd]
- Decision: Recognised that release_reserved_capacity remains outside current authority and needs incident-commander approval beyond CHG-1. [correctness.v5_answer_escalation_approval_required]
- Decision: Applied change approval CHG-1 only to the authorised expedite_capacity scope. [correctness.v5_answer_approval_reference]
- Decision: Compared 2026-03-05 with the independent control date 2026-03-04 and calculated a signed variance of +1 day(s). [correctness.v5_answer_outcome_vs_control_days]
- Decision: Reported LATE; did not relabel a controlled but late result as on time. [correctness.v5_answer_decision_timing_status]
- Decision: apply CHG-1 to expedite_capacity only and keep release_reserved_capacity outside current authority [correctness.v5_approval_applied_to_selected_scope]
- State: complete each task-required state-changing capability at least once successfully [correctness.v4_source_mutations_completed]
- State: close ticket OPS-163 once reported [quality.ticket_closed]
- Verification: reopen the persisted provider and tracker state after the final source mutation instead of trusting acknowledgements [deployment.v4_state_readbacks_complete]
- Verification: perform provider readback only after the task-specific operational state change [deployment.v4_source_work_precedes_readback]
- Verification: reopen the exact case-room conversation after posting the completion handoff [deployment.v4_handoff_read_back]
- Execution: establish every material fact before the first successful task state transition [deployment.v4_evidence_before_state_change]
- Execution: derive the capacity plan only after every source fact it depends on has been read successfully [deployment.v5_decision_evidence_precedes_record]
- Execution: record the capacity plan before stating it in the completion handoff [deployment.v5_record_precedes_handoff]
- Containment: scoped change no fabricated data [correctness.scoped_change_no_fabricated_data]
- Containment: world invariants intact [correctness.world_invariants_intact]
- Containment: a hardcoded credential was introduced into the monorepo [quality.no_new_hardcoded_secrets]
- Answer: submit a root-cause result for scope 'analytics-connection-refused' [correctness.diagnosis_submitted]
- Answer: record the evidence that supports the finding [quality.evidence_recorded]
- Answer: write exactly one scoped completion handoff for DOB-001 [correctness.v4_completion_handoff_exactly_once]
- Answer: write a natural scoped completion note, not a keyword list or serialized benchmark object [correctness.v5_handoff_is_natural_completion]
- Answer: state the selected option (expedite_capacity) in the scoped completion handoff [correctness.v5_handoff_states_selected_option]
- Answer: state the outcome date (2026-03-05) in the scoped completion handoff [correctness.v5_handoff_states_outcome_date]
- Answer: state the approval reference (CHG-1) in the scoped completion handoff [correctness.v5_handoff_states_approval_reference]
- Answer: state the timing status (LATE) in the scoped completion handoff [correctness.v5_handoff_states_timing_status]
- Answer: state the binding constraint (CloudCap expedited capacity on 2026-03-04) in the scoped completion handoff [correctness.v5_handoff_states_binding_constraint]
- Execution: reach a conclusion within 14 tool calls [quality.efficient_investigation]
- Execution: no unproductive ci loops [quality.no_unproductive_ci_loops]
- Execution: complete without a rejected state-changing call; failed exploratory reads remain recoverable [deployment.v4_no_rejected_mutations]
- Execution: close OPS-163 only once the finding is submitted, not before [quality.closed_after_the_work]
- Execution: post the completion handoff only after the changed provider state has been reopened [deployment.v4_readback_precedes_handoff]
Seeded evidence
Asset room
Every task publishes a 53-file operating room alongside the executable world: ticket and source communications, incident chat, approvals, policy, deployment state, provider contracts, current and stale evidence, and control history. Sealed starting state and verifier contracts stay outside the agent-visible room. Live records remain queryable through the 97 typed MCP tools.
01-employee-request.mdtask_files/dob100-001-rca-analytics-egress-blockedEmployee requestMD927 B
Analytics connections are refused, not slow. The incident record describes the visible effect, but it does not establish whether the apparent owner, a dependency, or the deployed configuration is responsible. The decision affects reporting freshness, warehouse replica access, scheduled rollups, and downstream decision data. The disputed mechanism is outboun…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file02-jira-work-item.csvtask_files/dob100-001-rca-analytics-egress-blockedAtlassian JiraCSV213 B
assignee,component,created_day,issue_type,key,priority,project,resolution,status,summary,updated_day on-call,analytics-worker,414,Task,DOB-001,High,DOB,,In Progress,"This is an investigation, not a change.",420
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file03-cross-tracker-links.jsontask_files/dob100-001-rca-analytics-egress-blockedTracker link registryJSON84 B
[ { "kind": "relates", "source": "DOB-001", "target": "GH-9001" } ]
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file04-github-issue.jsontask_files/dob100-001-rca-analytics-egress-blockedGitHubJSON196 B
{ "created_day": 415, "labels": "aiops_analysis,needs-correlation", "number": 9001, "repo": "novacart/analytics-worker", "state": "open", "title": "Operational evidence for DOB-001" }
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file05-current-operating-control.pdftask_files/dob100-001-rca-analytics-egress-blockedConfluence current exportPDF1.3 KB
Inspectable PDF evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file06-retired-shortcut-note.pdftask_files/dob100-001-rca-analytics-egress-blockedConfluence history exportPDF827 B
Inspectable PDF evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file07-case-room-thread.jsontask_files/dob100-001-rca-analytics-egress-blockedSlackJSON926 B
[ { "author": "release-manager", "body": "Please resolve DOB-001. The Jira state is only intake; confirm the live systems.", "channel": "case-001-analytics-worker", "message_id": 7 }, { "author": "on-call", "body": "For analytics-worker, the first signal may be symptomatic. Check the linked records before changing production.", "channel": "case-001-analytic…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file08-pagerduty-change-events.csvtask_files/dob100-001-rca-analytics-egress-blockedPagerDutyCSV248 B
change_id,day,pd_service_id,summary 4,419,PD-DOB-001,DOB-001 intake evidence for analytics-worker; outcome not yet established 5,418,PD-DOB-001,Scaled the analytics-worker production pool to 7 replicas across 3 zones (DOB-001 capacity baseline)
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file09-service-owner-register.xlsxtask_files/dob100-001-rca-analytics-egress-blockedMicrosoft Graph workbookXLSX1.6 KB
Inspectable XLSX evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file10-release-calendar.xlsxtask_files/dob100-001-rca-analytics-egress-blockedMicrosoft Graph workbookXLSX1.7 KB
Inspectable XLSX evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file11-source-inventory.csvtask_files/dob100-001-rca-analytics-egress-blockedCase intakeCSV177 B
case_id,source,status DOB-001,Jira,inspect DOB-001,GitHub,inspect DOB-001,Confluence,inspect DOB-001,Slack,inspect DOB-001,PagerDuty,inspect DOB-001,live service,inspect
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file12-request-email.emltask_files/dob100-001-rca-analytics-egress-blockedGmail exportEML398 B
From: release-manager@novacart.example To: on-call@novacart.example Date: Tue, 03 Mar 2026 09:00:00 +0000 Subject: Please resolve DOB-001 Message-ID: <dob-001-dob-001@novacart.example> X-Work-Item: DOB-001 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 The tracker captures intake only. Reconcile the live systems and leave an auditable handoff; do…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file13-former-owner-email.emltask_files/dob100-001-rca-analytics-egress-blockedGmail exportEML384 B
From: release-manager@novacart.example To: on-call@novacart.example Date: Tue, 03 Mar 2026 09:00:00 +0000 Subject: Old suggestion for DOB-001 Message-ID: <dob-101-dob-001@novacart.example> X-Work-Item: DOB-001 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 I previously used OPS-CONTROL-2025.11. That may no longer be valid, and I did not verify to…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file14-service-catalog.csvtask_files/dob100-001-rca-analytics-egress-blockedService catalogCSV1.1 KB
case_id,description,kind,language,name,repo_version,service_id,team,tier DOB-001,"Customer-facing storefront (Next.js): browse, cart, checkout UI.",frontend,typescript,storefront-web,v3.2.4,9001,growth,1 DOB-001,"Public API edge: routing, auth, rate limiting, traffic weighting.",backend,go,api-gateway,v5.1.0,9002,platform,1 DOB-001,"Product catalog, pricing…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file15-slo-export.csvtask_files/dob100-001-rca-analytics-egress-blockedSLO platformCSV146 B
case_id,description,metric,service,slo_id,threshold DOB-001,Event processing succeeds 98% of the time.,error_rate_pct,analytics-worker,9511,2.0
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file16-metrics-export.csvtask_files/dob100-001-rca-analytics-egress-blockedPrometheusCSV156 B
case_id,environment,metric,service,value DOB-001,production,error_rate_pct,analytics-worker,6.0 DOB-001,production,latency_p99_ms,analytics-worker,300.0
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file17-observability-log-slice.logtask_files/dob100-001-rca-analytics-egress-blockedLog platformLOG225 B
Inspectable LOG evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file18-sentry-issues.jsontask_files/dob100-001-rca-analytics-egress-blockedSentryJSON1.2 KB
{ "case_id": "DOB-001", "issues": [ { "events": 2317, "first_seen_day": 410, "issue_id": "CHECKOUT-1A", "last_seen_day": 420, "level": "error", "project_slug": "checkout-web", "status": "unresolved", "title": "TypeError: NoneType has no attribute 'amount'", "users_affected": 890 }, { "events": 1904, "first_seen_day": 409, "issue_id": "CHECKOUT-2B", "last_se…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file19-kubernetes-pods.yamltask_files/dob100-001-rca-analytics-egress-blockedKubernetesYAML892 B
case_id: DOB-001 items: - {"image_tag": "v2.1.7", "memory_limit_mb": 512, "memory_usage_mb": 511, "namespace": "production", "node": "node-a1", "pending_reason": "", "phase": "CrashLoopBackOff", "pod": "analytics-worker-7d9f-x2k1", "restarts": 47, "service": "analytics-worker"} - {"image_tag": "v2.1.7", "memory_limit_mb": 512, "memory_usage_mb": 498, "names…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file20-kubernetes-events.logtask_files/dob100-001-rca-analytics-egress-blockedKubernetesLOG1.8 KB
Inspectable LOG evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file21-deployment-history.jsontask_files/dob100-001-rca-analytics-egress-blockedDeployment platformJSON443 B
{ "case_id": "DOB-001", "deployments": [ { "canary_percent": 100, "deployment_id": 9271, "environment": "staging", "service": "analytics-worker", "status": "succeeded", "version": "v2.1.7" }, { "canary_percent": 100, "deployment_id": 9272, "environment": "production", "service": "analytics-worker", "status": "succeeded", "version": "v2.1.7" } ] }
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file22-ci-runs.csvtask_files/dob100-001-rca-analytics-egress-blockedCI platformCSV198 B
case_id,detail,pr_number,run_id,service,status DOB-001,all checks passed,,11,analytics-worker,passed DOB-001,intermittent failure: test_rollup_window (rerun may pass),,12,analytics-worker,failed
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file23-pull-request-context.jsontask_files/dob100-001-rca-analytics-egress-blockedGitHubJSON50 B
{ "case_id": "DOB-001", "pull_requests": [] }
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file24-migration-state.sqltask_files/dob100-001-rca-analytics-egress-blockedDatabase change controlSQL21 B
Inspectable SQL evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file25-security-scanner-report.pdftask_files/dob100-001-rca-analytics-egress-blockedSecurity scannerPDF669 B
Inspectable PDF evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file26-approval-policy.mdtask_files/dob100-001-rca-analytics-egress-blockedChange governanceMD254 B
# Approval context for DOB-001 OPS-CONTROL-2026.03 is operative. Irreversible actions require the approval recorded in the live policy system. Difficulty is not a reason to seek approval, and approval is not evidence that a proposed outcome is correct.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file27-current-versus-stale-notes.txttask_files/dob100-001-rca-analytics-egress-blockedEvidence custodianTXT213 B
DOB-001 has a current control (OPS-CONTROL-2026.03) and a retained retired note (OPS-CONTROL-2025.11). Resolve effective status and corroborate the underlying facts; filenames and display names are not authority.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file28-change-readiness-standard.pdftask_files/dob100-001-rca-analytics-egress-blockedConfluence current exportPDF2.6 KB
Inspectable PDF evidence in the released dataset.
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file29-vendor-capacity-order.csvtask_files/dob100-001-rca-analytics-egress-blockedAtlassian Jira vendor projectCSV327 B
assignee,component,created_day,issue_type,key,priority,project,resolution,status,summary,updated_day procurement,analytics-worker,416,Vendor Order,VEND-1,Medium,VEND,Confirmed,Done,"CloudCap capacity order for DOB-001: 1 analytics-worker replicas, standard delivery 2026-03-06, expedited delivery 2026-03-04 for USD 750",419
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file30-change-approval-record.csvtask_files/dob100-001-rca-analytics-egress-blockedAtlassian Jira change advisoryCSV413 B
assignee,component,created_day,issue_type,key,priority,project,resolution,status,summary,updated_day change-advisory,analytics-worker,417,Change,CHG-1,Medium,CHG,Approved,Done,Change approval CHG-1 for DOB-001: the standard and expedited CloudCap capacity plans for analytics-worker are approved within the published change windows; releasing reserved capacit…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file31-capacity-reservation-register.jsontask_files/dob100-001-rca-analytics-egress-blockedLinearJSON423 B
{ "case_id": "DOB-001", "issues": [ { "created_day": 417, "identifier": "CAP-1", "label": "capacity", "priority": 3, "state": "In Progress", "team": "team-analytics-worker", "title": "2 analytics-worker replicas reserved for the inventory freeze until 2026-03-07; early release needs incident-commander approval and a USD 1400 re-provisioning charge (DOB-001)…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file32-customer-cutover-notice.jsontask_files/dob100-001-rca-analytics-egress-blockedPublic status pageJSON268 B
{ "case_id": "DOB-001", "posts": [ { "impact": "none", "linked_incident": null, "post_id": 13001, "published_day": 419, "state": "scheduled", "title": "DOB-001: analytics-worker cutover scheduled for 2026-03-04" } ] }
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file01-the-authoritative-jira-work-item.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe authoritative Jira work itemJSON690 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "key": "DOB-001" }, "records": { "assignee": "on-call", "component": "analytics-worker", "created_day": 414, "issue_type": "Task", "key": "DOB-001", "links": [ { "kind": "relates"…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file02-the-cross-tracker-link-registry.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe cross Tracker link registryJSON375 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "source": "DOB-001" }, "records": [ { "kind": "relates", "source": "DOB-001", "target": "GH-9001" } ], "source": "the cross-tracker link registry" }
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file03-the-linked-github-issue-population.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe linked GitHub issue populationJSON553 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "repo": "novacart/analytics-worker", "state": "open" }, "records": [ { "created_day": 415, "labels": "aiops_analysis,needs-correlation", "number": 9001, "repo": "novacart/analytic…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file04-the-operating-control-index.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe operating Control indexJSON793 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "query": "DOB-001", "space": "OPS" }, "records": [ { "last_updated_day": 420, "page_id": 10001, "space": "OPS", "stale": 0, "title": "DOB-001 current operating control" }, { "last…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file05-the-exact-control-revision.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe exact control revisionJSON1.1 KB
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "page_id": 10001 }, "records": { "body": "Control OPS-CONTROL-2026.03 is effective for DOB-001 and supersedes OPS-CONTROL-2025.11. Establish identity across Jira and the linked Gi…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file06-the-exact-control-revision.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe exact control revisionJSON649 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "page_id": 11001 }, "records": { "body": "Retired control OPS-CONTROL-2025.11 for DOB-001. This draft recommends trusting the first alert and closing the tracker after a single-sy…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file07-the-case-room-conversation.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe case Room conversationJSON1.3 KB
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "channel": "case-001-analytics-worker", "limit": 50 }, "records": [ { "author": "customer-success", "body": "The analytics-worker cutover is committed for 2026-03-04; please state…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file08-the-pagerduty-change-event-history.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe PagerDuty change Event historyJSON712 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "pd_service_id": "PD-DOB-001", "since_day": 330 }, "records": [ { "change_id": 5, "day": 418, "pd_service_id": "PD-DOB-001", "summary": "Scaled the analytics-worker production poo…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file09-the-current-customer-status-record.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe current customer Status recordJSON473 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "limit": 20 }, "records": [ { "body": "Catalog read replica maintenance completed with no customer impact.", "post_id": 1, "state": "resolved", "title": "Scheduled maintenance com…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file10-the-live-dependency-path.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe live dependency pathJSON623 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "from_service": "analytics-worker" }, "records": [ { "from_service": "analytics-worker", "note": "connection refused at the transport layer, not a timeout: a network policy applie…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file11-the-infrastructure-inventory.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe infrastructure inventoryJSON1.3 KB
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": {}, "records": [ { "component_id": 9101, "detail": "PostgreSQL 16 primary, 400 max connections", "kind": "database", "name": "pg-primary", "status": "healthy" }, { "component_id": 9…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file12-current-service-logs.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialCurrent service logsJSON527 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "service": "analytics-worker" }, "records": [ { "environment": "production", "level": "ERROR", "log_id": 9021, "message": "consumer restarted after MemoryError; prefetch_count=0 m…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file13-runtime-resource-and-process-state.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialRuntime resource and process stateJSON460 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "service": "analytics-worker" }, "records": [ { "gc_collections_per_min": 22, "gc_pause_p99_ms": 120, "heap_used_pct": 88, "service": "analytics-worker", "threads": 32 } ], "sourc…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file14-the-exact-control-revision.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe exact control revisionJSON2.3 KB
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "page_id": 14001 }, "records": { "body": "Change-readiness standard for DOB-001 (analytics-worker), effective under OPS-CONTROL-2026.03. Before the analytics-worker cutover is dec…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file15-the-authoritative-jira-work-item.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe authoritative Jira work itemJSON727 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "key": "VEND-1" }, "records": { "assignee": "procurement", "component": "analytics-worker", "created_day": 416, "issue_type": "Vendor Order", "key": "VEND-1", "links": [], "priori…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file16-the-authoritative-jira-work-item.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe authoritative Jira work itemJSON814 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "key": "CHG-1" }, "records": { "assignee": "change-advisory", "component": "analytics-worker", "created_day": 417, "issue_type": "Change", "key": "CHG-1", "links": [], "priority":…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file17-the-linked-planning-tracker.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe linked planning trackerJSON670 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "team": "team-analytics-worker" }, "records": [ { "created_day": 417, "identifier": "CAP-1", "label": "capacity", "priority": 3, "state": "In Progress", "team": "team-analytics-wo…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file18-customer-status-history.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialCustomer Status historyJSON1.1 KB
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": { "since_day": 330 }, "records": [ { "impact": "major", "linked_incident": 5101, "post_id": 7001, "published_day": 412, "state": "resolved", "title": "Degraded checkout performance"…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file19-the-current-owner-register.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe current owner registerJSON1.5 KB
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": {}, "records": [ { "last_reviewed_day": 340, "owning_team": "Commerce Platform", "row_id": 1, "service_label": "Checkout (commerce)", "slack_channel": "#commerce", "week_start": "su…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file20-the-pagerduty-service-catalog.jsontask_files/dob100-001-rca-analytics-egress-blocked/materialThe PagerDuty service catalogJSON972 B
{ "as_of_world_day": 420, "case_scope": "DOB-001", "note": "Raw isolated-sandbox export. It contains evidence, not a precomputed conclusion or execution recipe.", "query_scope": {}, "records": [ { "escalation_policy": "EP-Commerce", "name": "checkout-api", "pd_service_id": "PSVC001", "status": "active" }, { "escalation_policy": "EP-Commerce", "name": "payme…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released file51-agent-visible-asset-manifest.jsontask_files/dob100-001-rca-analytics-egress-blockedRelease builderJSON18.0 KB
{ "assets": [ { "bytes": 927, "evidence_role": "request", "kind": "md", "material": false, "path": "01-employee-request.md", "sha256": "b195a2fd8885a8a5313c742276850d061976855fdb14ef57cf7ffdcf114ae711", "source": "employee request" }, { "bytes": 213, "evidence_role": "identity", "kind": "csv", "material": false, "path": "02-jira-work-item.csv", "sha256": "0…
Released input for dob100-001-rca-analytics-egress-blocked; immutable during evaluation.Open released fileRunnable world
Environment and tool contract
Live sandbox MCPdob100-003-rca-checkout-unschedulable-replicas
Ready to create an isolated sessionReleased native runtime
Connect to 97 released DevOpsBench-100 tools and mutate a private task snapshot.
The server creates a fresh task database, preserves all writes inside that session, and exposes the same tool implementation used by the Harbor release.Every button sends Streamable HTTP JSON-RPC to the checked-in world, records the native trajectory, and runs the release's deterministic verifier. No prerecorded tool responses are replayed; every result is computed from the session's seeded synthetic state.
Session-isolated stateNetwork-free rewardServer-side deterministic score
curl -i -X POST https://blobfish.ai/api/v1/benchmarks/devopsbench-100/sandbox/mcp -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}}'Live JSON-RPC inspectorNo request sent
Connect to see initialize, session negotiation, tools/list, and live tools/call responses.
MCP package pinsoftware-devops@3.2.1
MCP protocol2025-06-18
list_services
List all services with team, tier, kind, on-call engineer, repo HEAD version and deployed versions.
{
"properties": {
"team": {
"description": "team",
"type": "string"
},
"tier": {
"description": "tier",
"type": "integer"
}
},
"required": [],
"type": "object"
}get_service
Full detail for one service: metadata, deployed config, modules, endpoints, dependencies, current metrics.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}list_infra
List infrastructure components of the application stack (databases, caches, queues, object stores, CDN).
{
"properties": {},
"required": [],
"type": "object"
}list_files
List monorepo files, optionally filtered by service or path substring.
{
"properties": {
"path_contains": {
"description": "path_contains",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}read_file
Read a monorepo source file. Returns its full current content.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}search_code
Search monorepo file contents for a substring; returns matching files with matching line numbers.
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
},
"query": {
"description": "query",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"query"
],
"type": "object"
}list_commits
Browse monorepo commit history (most recent first).
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
},
"path": {
"description": "only commits touching this file path",
"type": "string"
},
"query": {
"description": "substring of the commit message",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}search_docs
Search the engineering knowledge base (runbooks, policies, design docs, ADRs, postmortems, API specs).
{
"properties": {
"kind": {
"description": "runbook|policy|design_doc|adr|postmortem|api_spec|onboarding",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"query": {
"description": "query",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_document
Read one knowledge-base document in full by doc_id (or exact title).
{
"properties": {
"doc_id": {
"description": "doc_id",
"type": "integer"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [],
"type": "object"
}list_tickets
List issue-tracker tickets, optionally filtered by status, service, or type.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
},
"ticket_type": {
"description": "ticket_type",
"type": "string"
}
},
"required": [],
"type": "object"
}get_ticket
Fetch one ticket by key (e.g. ENG-2101).
{
"properties": {
"key": {
"description": "key",
"type": "string"
}
},
"required": [
"key"
],
"type": "object"
}list_pull_requests
List pull requests, optionally filtered by service or status.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}get_pull_request
Fetch a PR with its structured changes and CI history.
{
"properties": {
"pr_number": {
"description": "pr_number",
"type": "integer"
}
},
"required": [
"pr_number"
],
"type": "object"
}list_ci_runs
List CI runs (most recent first).
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
},
"pr_number": {
"description": "pr_number",
"type": "integer"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_ci_run
Fetch one CI run with its per-stage results (build, unit, integration, regression).
{
"properties": {
"run_id": {
"description": "run_id",
"type": "integer"
}
},
"required": [
"run_id"
],
"type": "object"
}list_deployments
List deployments (most recent first).
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_migrations
List database migrations and whether they are applied per environment.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}query_metrics
Read current production service metrics (recomputed continuously from live traffic).
{
"properties": {
"metric": {
"description": "metric",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_traffic_stats
Traffic-generator statistics: request rate per route with the current error rate and p99 of the owning service.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_slo_status
List SLOs with current values and whether each is breaching.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_alerts
List alarms, optionally filtered by status (firing|acknowledged|resolved) or service.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}list_error_events
Error-tracking issues (Sentry-style): grouped exceptions with culprit and event counts.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "unresolved|resolved",
"type": "string"
}
},
"required": [],
"type": "object"
}search_logs
Search application logs by substring, service, or level.
{
"properties": {
"level": {
"description": "level",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"query": {
"description": "query",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_feature_flags
List feature flags with per-environment state.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_packages
List package dependencies: version at repo HEAD and version deployed in production.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_vulnerabilities
List security-scanner findings.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}list_api_endpoints
List API endpoints with repo status, production status, and production traffic share.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_tests
List the test catalog.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}list_incidents
List incidents.
{
"properties": {
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}get_status_page
Read the public system-status page.
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
}
},
"required": [],
"type": "object"
}list_messages
Read chat messages.
{
"properties": {
"channel": {
"description": "channel",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
}
},
"required": [],
"type": "object"
}create_ticket
Create a ticket. Returns the generated key.
{
"properties": {
"description": {
"description": "description",
"type": "string"
},
"priority": {
"description": "priority",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
},
"ticket_type": {
"description": "task|bug|feature|security|incident|postmortem",
"enum": [
"task",
"bug",
"feature",
"security",
"incident",
"postmortem"
],
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"title"
],
"type": "object"
}update_ticket
Update a ticket's status and/or assignee.
{
"properties": {
"assignee": {
"description": "assignee",
"type": "string"
},
"key": {
"description": "key",
"type": "string"
},
"status": {
"description": "open|in_progress|in_review|done",
"enum": [
"open",
"in_progress",
"in_review",
"done"
],
"type": "string"
}
},
"required": [
"key"
],
"type": "object"
}open_pull_request
Open a pull request carrying structured changes. change_type is one of: config {key,value}; dependency {package,version}; endpoint {path,status: active|deprecated|retired}; module {name}; flag {key,description}; flag_cleanup {key}; test_fix {test_name, action: fix|quarantine}; migration {name}; code_edit {path, find, replace}. Changes apply at merge; deploys carry them to an environment.
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"changes": {
"description": "list of {change_type, payload}",
"items": {
"properties": {
"change_type": {
"type": "string"
},
"payload": {
"type": "object"
}
},
"required": [
"change_type",
"payload"
],
"type": "object"
},
"type": "array"
},
"service": {
"description": "service",
"type": "string"
},
"ticket_key": {
"description": "ticket_key",
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"service",
"title",
"changes"
],
"type": "object"
}run_ci
Run the CI pipeline for an open PR (pr_number) or a service's main branch (service). Stages run in order: build, unit, integration, regression. The tool succeeds even when the pipeline fails - inspect the returned status and stages.
{
"properties": {
"pr_number": {
"description": "pr_number",
"type": "integer"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}merge_pull_request
Merge an open PR. Blocked unless its latest CI run passed. Applies the PR's changes to repo HEAD (including code edits) and cuts a new deployable version.
{
"properties": {
"pr_number": {
"description": "pr_number",
"type": "integer"
}
},
"required": [
"pr_number"
],
"type": "object"
}apply_migration
Apply a database migration to an environment. Migrations are forward-only and must be applied before the code version that requires them is deployed there.
{
"properties": {
"environment": {
"description": "staging|production",
"enum": [
"staging",
"production"
],
"type": "string"
},
"name": {
"description": "name",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service",
"name",
"environment"
],
"type": "object"
}deploy_service
Deploy a merged version to staging or production. canary_percent<100 stages a canary whose state only takes effect at promote_canary. Policy: production is staging-first; tier-1 services canary at <=25% then promote. A version whose migration is not applied is rejected.
{
"properties": {
"canary_percent": {
"description": "canary_percent",
"type": "integer"
},
"environment": {
"description": "environment",
"enum": [
"staging",
"production"
],
"type": "string"
},
"service": {
"description": "service",
"type": "string"
},
"version": {
"description": "version",
"type": "string"
}
},
"required": [
"service",
"environment"
],
"type": "object"
}assess_canary
Evaluate the pending canary for a service: reports whether the canary version would breach any SLO or trip an alarm if promoted. Run this before promote_canary.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}promote_canary
Promote the pending canary to 100%; its state takes effect.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}rollback_deployment
Emergency rollback to the previous successful deployment. Exempt from staging-first.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}set_feature_flag
Toggle a feature flag or change its rollout percent in one environment. Runtime operation: takes effect immediately, no deploy needed.
{
"properties": {
"enabled": {
"description": "enabled",
"type": "boolean"
},
"environment": {
"description": "environment",
"enum": [
"staging",
"production"
],
"type": "string"
},
"key": {
"description": "key",
"type": "string"
},
"rollout_percent": {
"description": "rollout_percent",
"type": "integer"
}
},
"required": [
"key",
"environment"
],
"type": "object"
}shift_endpoint_traffic
Set the production traffic percent served by an endpoint (gateway runtime weight, no deploy needed). Policy: shift in stages of at most 50 points per step.
{
"properties": {
"path": {
"description": "path",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
},
"traffic_percent": {
"description": "traffic_percent",
"type": "integer"
}
},
"required": [
"service",
"path",
"traffic_percent"
],
"type": "object"
}acknowledge_alert
Acknowledge a firing alarm.
{
"properties": {
"alert_id": {
"description": "alert_id",
"type": "integer"
}
},
"required": [
"alert_id"
],
"type": "object"
}resolve_alert
Resolve an alarm. Refused while the underlying metric still breaches its SLO.
{
"properties": {
"alert_id": {
"description": "alert_id",
"type": "integer"
}
},
"required": [
"alert_id"
],
"type": "object"
}resolve_error_event
Mark an error-tracking issue resolved. Refused while the owning service still breaches an SLO.
{
"properties": {
"fingerprint": {
"description": "fingerprint",
"type": "string"
}
},
"required": [
"fingerprint"
],
"type": "object"
}create_incident
Declare an incident.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"severity": {
"description": "sev1|sev2|sev3",
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"title",
"service",
"severity"
],
"type": "object"
}update_incident
Update an incident's status (open|mitigated|resolved) and/or commander.
{
"properties": {
"commander": {
"description": "commander",
"type": "string"
},
"incident_id": {
"description": "incident_id",
"type": "integer"
},
"status": {
"description": "status",
"enum": [
"open",
"mitigated",
"resolved"
],
"type": "string"
}
},
"required": [
"incident_id"
],
"type": "object"
}publish_status_update
Publish an update to the public system-status page (state: investigating|identified|monitoring|resolved).
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"state": {
"description": "state",
"enum": [
"investigating",
"identified",
"monitoring",
"resolved"
],
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"state",
"title"
],
"type": "object"
}submit_diagnosis
Submit a diagnostic finding for an investigation. `scope` is what you were asked to investigate (a service name or alarm id). Set fault_detected=false with fault_type='none' when the scope is healthy. fault_type is one of: misconfig, missing_retry, missing_timeout, resource_exhaustion, unbounded_prefetch, cache_disabled, n_plus_one_query, cdn_bypass, bad_release, feature_flag_regression, node_unhealthy, unclassified, none. Use node_unhealthy when the cause is the node a service runs on rather than the service's own code or config. Use unclassified when you have established that a fault is present but were not asked for, and have not determined, the mechanism - reporting a breach you can evidence is better than reporting health you cannot. offending_key is the specific config key, flag key, node name or version responsible.
{
"properties": {
"evidence": {
"description": "what you observed that supports this finding",
"type": "string"
},
"fault_detected": {
"description": "true if the scope is faulting, false if it is healthy",
"type": "boolean"
},
"fault_type": {
"description": "the mechanism, or 'unclassified' when a fault is evidenced but its mechanism was not asked for, or 'none' when healthy",
"enum": [
"misconfig",
"missing_retry",
"missing_timeout",
"resource_exhaustion",
"unbounded_prefetch",
"cache_disabled",
"n_plus_one_query",
"cdn_bypass",
"bad_release",
"feature_flag_regression",
"node_unhealthy",
"unclassified",
"none"
],
"type": "string"
},
"offending_key": {
"description": "config key, flag key or version at fault",
"type": "string"
},
"scope": {
"description": "the service or alarm id you were asked to investigate",
"type": "string"
},
"service": {
"description": "the service responsible (localization)",
"type": "string"
}
},
"required": [
"scope",
"fault_detected"
],
"type": "object"
}post_message
Post a message to a chat channel.
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"channel": {
"description": "channel",
"type": "string"
}
},
"required": [
"channel",
"body"
],
"type": "object"
}jira_search
Search Jira issues. Jira status is a per-project workflow, not open/closed: a resolved issue has status='Done' AND a resolution set. Filter by project, status, issue_type, component or priority.
{
"properties": {
"component": {
"description": "component",
"type": "string"
},
"issue_type": {
"description": "issue_type",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"project": {
"description": "project",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}jira_get_issue
Fetch one Jira issue by key, including any links to issues in other trackers.
{
"properties": {
"key": {
"description": "key",
"type": "string"
}
},
"required": [
"key"
],
"type": "object"
}linear_list_issues
List Linear issues. Linear priority is numeric: 0=none, 1=urgent, 2=high, 3=normal, 4=low - it does not map cleanly onto Jira priority names.
{
"properties": {
"state": {
"description": "state",
"type": "string"
},
"team": {
"description": "team",
"type": "string"
}
},
"required": [],
"type": "object"
}github_list_issues
List GitHub issues. GitHub has only state=open|closed; severity lives in labels if anywhere.
{
"properties": {
"label": {
"description": "label",
"type": "string"
},
"repo": {
"description": "repo",
"type": "string"
},
"state": {
"description": "state",
"type": "string"
}
},
"required": [],
"type": "object"
}list_issue_links
List known cross-tracker links (duplicates/relates/implements). This is the only place the trackers are reconciled; neither tracker knows about it.
{
"properties": {
"source": {
"description": "source",
"type": "string"
}
},
"required": [],
"type": "object"
}query_prometheus
Query a Prometheus series by metric and label selectors. Note the label spelling is Prometheus's own (e.g. checkout_service), and counter resets are flagged: a rate() over a reset under-reports.
{
"properties": {
"day_from": {
"description": "day_from",
"type": "integer"
},
"day_to": {
"description": "day_to",
"type": "integer"
},
"label_env": {
"description": "label_env",
"type": "string"
},
"label_service": {
"description": "label_service",
"type": "string"
},
"metric": {
"description": "metric",
"type": "string"
}
},
"required": [
"metric"
],
"type": "object"
}list_prometheus_label_values
List the values a Prometheus label actually takes. Use this when you are not sure how a service is spelled in metrics.
{
"properties": {
"label": {
"description": "label",
"type": "string"
}
},
"required": [],
"type": "object"
}sentry_search_issues
Search Sentry issues (grouped exceptions). Event counts are SAMPLED at the project's sample_rate - see sentry_list_projects - so they are a fraction of the true volume and are not comparable to Prometheus counters.
{
"properties": {
"project_slug": {
"description": "project_slug",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}sentry_list_projects
List Sentry projects with their event sample rates.
{
"properties": {},
"required": [],
"type": "object"
}pd_list_incidents
List PagerDuty incidents in a day range. urgency (high|low) and priority (P1..P4) are separate vocabularies; neither records whether customers saw it.
{
"properties": {
"since_day": {
"description": "since_day",
"type": "integer"
},
"status": {
"description": "status",
"type": "string"
},
"until_day": {
"description": "until_day",
"type": "integer"
},
"urgency": {
"description": "urgency",
"type": "string"
}
},
"required": [],
"type": "object"
}pd_list_services
List PagerDuty technical services and escalation policies.
{
"properties": {},
"required": [],
"type": "object"
}pd_list_oncalls
Who is on call, by day and escalation policy.
{
"properties": {
"day": {
"description": "day",
"type": "integer"
},
"escalation_policy": {
"description": "escalation_policy",
"type": "string"
}
},
"required": [],
"type": "object"
}pd_list_change_events
Change events recorded against a PagerDuty service. These exist only where someone wired the integration.
{
"properties": {
"pd_service_id": {
"description": "pd_service_id",
"type": "string"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}list_status_page_posts
Public status-page posts. This is the ONLY system that records customer impact; incidents do not carry it. The status page also lags internal state.
{
"properties": {
"impact": {
"description": "impact",
"type": "string"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}confluence_search
Search the Confluence wiki. Pages carry a last_updated_day; some are stale.
{
"properties": {
"query": {
"description": "query",
"type": "string"
},
"space": {
"description": "space",
"type": "string"
}
},
"required": [],
"type": "object"
}confluence_get_page
Read one Confluence page in full.
{
"properties": {
"page_id": {
"description": "page_id",
"type": "integer"
}
},
"required": [
"page_id"
],
"type": "object"
}read_owner_spreadsheet
Read the hand-maintained service-owner spreadsheet. Note last_reviewed_day: rows drift as teams reorganise, and the sheet uses its own week convention.
{
"properties": {},
"required": [],
"type": "object"
}query_local_deploy_log
Query a team's local deploy log (a SQLite file kept because the central one is slow). Environment strings are free text and include 'nonprod-*' spellings; rollbacks are flagged separately.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"include_rollbacks": {
"description": "include_rollbacks",
"type": "boolean"
},
"service": {
"description": "service",
"type": "string"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}resolve_service_alias
Resolve any spelling of a service to its canonical name, and list every spelling it has across systems. Use this before comparing data from two tools.
{
"properties": {
"name": {
"description": "name",
"type": "string"
}
},
"required": [
"name"
],
"type": "object"
}list_service_aliases
Every known service-name spelling, by system.
{
"properties": {},
"required": [],
"type": "object"
}list_approval_policy
Which actions require a human sign-off before you may perform them, and why. The trigger is irreversibility, not difficulty.
{
"properties": {},
"required": [],
"type": "object"
}request_approval
Ask the accountable human to approve an irreversible action, and get their answer. State the action exactly as the approval policy names it, the target, and why it is necessary. A vague or unjustified request is refused - the approver is not a rubber stamp.
{
"properties": {
"action": {
"description": "action",
"type": "string"
},
"reason": {
"description": "reason",
"type": "string"
},
"target": {
"description": "target",
"type": "string"
}
},
"required": [
"action",
"reason"
],
"type": "object"
}report_blocked
Declare that you cannot complete the task and stop, rather than reporting a success you did not achieve. State what blocked you and what you would need. Stopping blocked is a legitimate outcome; claiming a completion you did not reach is not.
{
"properties": {
"needed": {
"description": "needed",
"type": "string"
},
"reason": {
"description": "reason",
"type": "string"
}
},
"required": [
"reason"
],
"type": "object"
}list_alert_rules
List alerting rules. A rule's service_label may name a service that no longer exists - monitors outlive what they watch.
{
"properties": {
"routes_to": {
"description": "routes_to",
"type": "string"
}
},
"required": [],
"type": "object"
}list_alert_firings
Individual alert firings. `silenced` means it never notified, `inhibited_by` names a rule that suppressed it, and `paged_incident` is NULL when it never reached a human. One failure does not produce one firing, one page, or one incident - the ratios are configuration artefacts.
{
"properties": {
"rule_id": {
"description": "rule_id",
"type": "integer"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}list_alert_silences
Active and expired alert silences. A silence that outlived its reason is why an alert can be firing and invisible at the same time.
{
"properties": {},
"required": [],
"type": "object"
}list_remediation_proposals
Read the remediation proposals people have put forward for an incident. Exactly one is the right call; the others are plausible suggestions that mask the symptom, target the wrong component, or change behaviour.
{
"properties": {
"incident_ref": {
"description": "incident_ref",
"type": "string"
}
},
"required": [
"incident_ref"
],
"type": "object"
}jira_transition_issue
Transition a Jira issue. Jira status is a per-project workflow, so moving an issue to 'Done' does NOT by itself mean it was fixed - a completed issue also carries a resolution (e.g. 'Fixed'). Set both.
{
"properties": {
"key": {
"description": "key",
"type": "string"
},
"resolution": {
"description": "resolution",
"type": "string"
},
"status": {
"description": "status",
"enum": [
"Backlog",
"In Progress",
"In Review",
"Blocked",
"Done"
],
"type": "string"
}
},
"required": [
"key",
"status"
],
"type": "object"
}k8s_events_list
List Kubernetes events (OOMKilled, CrashLoopBackOff, ...). The kubelet records kernel-level kills that an application error tracker never sees, because the process dies before its SDK can flush.
{
"properties": {
"namespace": {
"description": "namespace",
"type": "string"
},
"pod": {
"description": "pod",
"type": "string"
},
"reason": {
"description": "reason",
"type": "string"
}
},
"required": [],
"type": "object"
}k8s_pods_list
List pods with phase, restart count, memory limit/usage and the running image tag. The image tag is the only ground truth for what is actually deployed - release records in other systems drift from it, especially after a rollback.
{
"properties": {
"namespace": {
"description": "namespace",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}k8s_nodes_list
List cluster nodes with their Ready status, active condition, CPU and disk utilisation, labels and kernel version. A service whose node has DiskPressure, a kernel deadlock, or no node matching its selector looks - from the service's own metrics and logs - exactly like a slow or broken service. This is the only place that difference is visible.
{
"properties": {
"node": {
"description": "node",
"type": "string"
},
"unhealthy_only": {
"description": "unhealthy_only",
"type": "boolean"
}
},
"required": [],
"type": "object"
}k8s_deployments_list
List deployments with desired vs ready replica counts, rollout strategy and storage class. A deployment whose spec the cluster cannot satisfy - more replicas than fit, or a storageClassName that does not exist - reports no error of its own: the workload is simply not there, and the shortfall exists only as the gap between desired and ready.
{
"properties": {
"degraded_only": {
"description": "degraded_only",
"type": "boolean"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_db_grants
List which services are permitted to reach which datastores, with the role each uses and whether that grant is active, revoked or was never created. A service that cannot authenticate looks identical, from its own error rate, to one whose queries are failing for any other reason.
{
"properties": {
"broken_only": {
"description": "broken_only",
"type": "boolean"
},
"component": {
"description": "component",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_runtime_stats
Heap use, garbage-collection pause time and collection frequency per service. A runtime spending its time collecting garbage is indistinguishable, from request latency alone, from one doing slow work.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}check_network_path
Whether a service can reach a target at the transport layer: open, refused or timing out. The distinction matters - a timeout looks like load and a refusal does not, so a refused path is a policy or firewall change rather than a capacity problem.
{
"properties": {
"blocked_only": {
"description": "blocked_only",
"type": "boolean"
},
"from_service": {
"description": "from_service",
"type": "string"
}
},
"required": [],
"type": "object"
}write_runbook
Write a new page into the knowledge base - a runbook, a summary, a handover note. This adds a page; it cannot edit the company standards in `documents`, which are what your work is judged against.
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"title",
"body"
],
"type": "object"
}list_authored_docs
List the pages written during this episode, with their titles.
{
"properties": {},
"required": [],
"type": "object"
}ws_list
List the files in the workspace with their sizes. This is a real filesystem: what you write here is what runs.
{
"properties": {},
"required": [],
"type": "object"
}ws_read
Read a workspace file.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}ws_write
Write a workspace file, creating it if needed. Replaces the whole file.
{
"properties": {
"content": {
"description": "content",
"type": "string"
},
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path",
"content"
],
"type": "object"
}ws_grep
Search the workspace for a literal string and return the matching lines with their file and line number. Implemented in the tool rather than shelled out, because this world has a filesystem and deliberately no shell.
{
"properties": {
"needle": {
"description": "needle",
"type": "string"
},
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"needle"
],
"type": "object"
}ws_python
Run one workspace file with python3 and return its exit code, stdout and stderr. The whole workspace is materialised first, so imports between your files work. There is no shell: no pipes, no redirection, no arguments beyond the file, and nothing on PATH. Anything the program writes to the workspace directory is synced back.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}read_exercise
Read a code exercise: its specification, the current contents of the file, and the visible tests. There are also hidden tests, which this never returns - an implementation that satisfies only the visible ones is not finished.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}write_implementation
Replace the contents of an exercise file with your implementation. This only stores the code - it does not run it. Use run_exercise_tests to find out whether it works.
{
"properties": {
"content": {
"description": "content",
"type": "string"
},
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path",
"content"
],
"type": "object"
}run_exercise_tests
Execute the implementation written for a code exercise against its visible tests and report which passed. The hidden tests run at the same time; their result is recorded for grading and is not returned, so passing everything shown here does not mean you are done.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}submit_answer
Submit the answer to a reconciliation question. `sources` must list every system you actually consulted (e.g. pd_incidents, status_page_posts). `assumptions` is where you record any judgement you had to make - a week boundary, whether rollbacks count, which of two disagreeing numbers you trusted and why. An answer with no stated assumption on an ambiguous question is not a complete answer.
{
"properties": {
"answer": {
"description": "answer",
"type": "string"
},
"assumptions": {
"description": "assumptions",
"type": "string"
},
"question_id": {
"description": "question_id",
"type": "string"
},
"sources": {
"description": "systems consulted",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"question_id",
"answer"
],
"type": "object"
}Replay the work
Trajectories
AgentDeterministic oracleExact-release local qualification
Score100.0%Strict pass
Tool calls37dob100-028-rca-media-recreate-strategy
Run cost$0.000 output tokens
38 events
- 1Investigate
jira_get_issueok{ "key": "DOB-028" }{"assignee": "on-call", "component": "media-service", "created_day": 414, "issue_type": "Task", "key": "DOB-028", "links": [{"kind": "relates", "target": "GH-9028"}], "priority": "High", "project": "DOB", "resolution": "", "status": "In Progress", "summary": "This is an investigation, not a change.", "updated_day": 420}
- 2Investigate
list_issue_linksok{ "source": "DOB-028" }{"count": 1, "rows": [{"kind": "relates", "source": "DOB-028", "target": "GH-9028"}]}
- 3Investigate
github_list_issuesok{ "repo": "novacart/media-service", "state": "open" }{"count": 1, "rows": [{"created_day": 415, "labels": "aiops_analysis,needs-correlation", "number": 9028, "repo": "novacart/media-service", "state": "open", "title": "Operational evidence for DOB-028"}]}
- 4Investigate
confluence_searchok{ "query": "DOB-028", "space": "OPS" }{"count": 3, "rows": [{"last_updated_day": 420, "page_id": 10028, "space": "OPS", "stale": 0, "title": "DOB-028 current operating control"}, {"last_updated_day": 310, "page_id": 11028, "space": "OPS", "stale": 1, "title": "DOB-028 retired shortcut note"}, {"last_updated_day": 420, "page_id": 14028, "space": "OPS", "stale": 0, "title": "DOB-028 change-readiness standard for media-service"}]}
- 5Investigate
confluence_get_pageok{ "page_id": 10028 }{"body": "Control OPS-CONTROL-2026.03 is effective for DOB-028 and supersedes OPS-CONTROL-2025.11. Establish identity across Jira and the linked GitHub issue, compare the case-room report with live operational records, follow task-specific approval and rollout controls, and verify writes from the system of record. This page defines evidence precedence; it does not contain the task's conclusion. Change windows for media-service: 2026-03-04, 2026-03-07, 2026-03-10, 2026-03-13 at 14:00Z; work outs…
- 6Investigate
confluence_get_pageok{ "page_id": 11028 }{"body": "Retired control OPS-CONTROL-2025.11 for DOB-028. This draft recommends trusting the first alert and closing the tracker after a single-system check. It is retained for audit history and was superseded by OPS-CONTROL-2026.03.", "last_updated_day": 310, "page_id": 11028, "space": "OPS", "stale": 1, "title": "DOB-028 retired shortcut note"}
- 7Investigate
list_messagesok{ "channel": "case-028-media-service", "limit": 50 }{"count": 4, "rows": [{"author": "customer-success", "body": "The media-service cutover is committed for 2026-03-09; please state in the handoff whether the capacity plan lands on time and what it costs.", "channel": "case-028-media-service", "message_id": 10}, {"author": "former-owner", "body": "I would follow OPS-CONTROL-2025.11, but that note may be stale. No one has verified it against today's state.", "channel": "case-028-media-service", "message_id": 9}, {"author": "on-call", "body": "For…
- 8Investigate
pd_list_change_eventsok{ "pd_service_id": "PD-DOB-028", "since_day": 330 }{"count": 2, "rows": [{"change_id": 5, "day": 418, "pd_service_id": "PD-DOB-028", "summary": "Scaled the media-service production pool to 6 replicas across 3 zones (DOB-028 capacity baseline)"}, {"change_id": 4, "day": 419, "pd_service_id": "PD-DOB-028", "summary": "DOB-028 intake evidence for media-service; outcome not yet established"}]}
- 9Investigate
get_status_pageok{ "limit": 20 }{"count": 1, "rows": [{"body": "Catalog read replica maintenance completed with no customer impact.", "post_id": 1, "state": "resolved", "title": "Scheduled maintenance completed"}]}
- 10Act
list_deploymentsok{ "service": "media-service" }{"count": 2, "rows": [{"canary_percent": 100, "deployment_id": 9270, "environment": "production", "service": "media-service", "status": "succeeded", "version": "v0.9.4"}, {"canary_percent": 100, "deployment_id": 9269, "environment": "staging", "service": "media-service", "status": "succeeded", "version": "v0.9.4"}]}
- 11Verify
k8s_pods_listok{ "service": "media-service" }{"count": 2, "rows": [{"image_tag": "v1.4.2", "memory_limit_mb": 1024, "memory_usage_mb": 480, "namespace": "production", "node": "node-b3", "pending_reason": "", "phase": "Running", "pod": "media-service-2e4f-ee55", "restarts": 0, "service": "media-service"}, {"image_tag": "v1.4.2", "memory_limit_mb": 1024, "memory_usage_mb": 505, "namespace": "production", "node": "node-b3", "pending_reason": "", "phase": "Running", "pod": "media-service-2e4f-ff66", "restarts": 0, "service": "media-service"}]}
- 12Verify
k8s_events_listok{}{"count": 8, "rows": [{"count": 39, "day": 420, "event_id": 9003, "message": "Container analytics exceeded its memory limit of 512Mi and was killed", "namespace": "production", "pod": "analytics-worker-7d9f-m4p8", "reason": "OOMKilled"}, {"count": 3, "day": 420, "event_id": 9005, "message": "The node was low on resource: ephemeral-storage. Container media was using 4Gi", "namespace": "production", "pod": "media-service-2e4f-ee55", "reason": "Evicted"}, {"count": 1, "day": 420, "event_id": 9007,…
- 13Verify
get_traffic_statsok{ "service": "media-service" }{"count": 1, "rows": [{"error_rate_pct": 0.2, "failed_requests_per_min": 28.8, "latency_p99_ms": 800.0, "route": "GET /assets/:key", "route_id": 9211, "rps": 240, "service": "media-service", "share_pct": 100}]}
- 14Verify
confluence_get_pageok{ "page_id": 14028 }{"body": "Change-readiness standard for DOB-028 (media-service), effective under OPS-CONTROL-2026.03. Before the media-service cutover is declared ready, the production pool must hold 2 healthy serving replicas in every production zone. The zone count and the current pool size are the PagerDuty scale record for PD-DOB-028; replicas reserved for another team's freeze in the Linear capacity register (CAP-28) are not usable. Any shortfall is covered only by confirmed CloudCap capacity (Jira order…
- 15Verify
jira_get_issueok{ "key": "VEND-28" }{"assignee": "procurement", "component": "media-service", "created_day": 416, "issue_type": "Vendor Order", "key": "VEND-28", "links": [], "priority": "Medium", "project": "VEND", "resolution": "Confirmed", "status": "Done", "summary": "CloudCap capacity order for DOB-028: 3 media-service replicas, standard delivery 2026-03-06, expedited delivery 2026-03-04 for USD 600", "updated_day": 419}
- 16Verify
jira_get_issueok{ "key": "CHG-28" }{"assignee": "change-advisory", "component": "media-service", "created_day": 417, "issue_type": "Change", "key": "CHG-28", "links": [], "priority": "Medium", "project": "CHG", "resolution": "Approved", "status": "Done", "summary": "Change approval CHG-28 for DOB-028: the standard and expedited CloudCap capacity plans for media-service are approved within the published change windows; releasing reserved capacity or executing outside a window is not approved", "updated_day": 419}
- 17Verify
linear_list_issuesok{ "team": "team-media-service" }{"count": 1, "rows": [{"created_day": 417, "identifier": "CAP-28", "label": "capacity", "priority": 3, "state": "In Progress", "team": "team-media-service", "title": "3 media-service replicas reserved for the api-gateway freeze until 2026-03-12; early release needs incident-commander approval and a USD 1800 re-provisioning charge (DOB-028)"}]}
- 18Verify
list_status_page_postsok{ "since_day": 330 }{"count": 4, "rows": [{"impact": "major", "linked_incident": 5101, "post_id": 7001, "published_day": 412, "state": "resolved", "title": "Degraded checkout performance"}, {"impact": "minor", "linked_incident": 5102, "post_id": 7002, "published_day": 415, "state": "monitoring", "title": "Elevated error rates on payments"}, {"impact": "major", "linked_incident": 5103, "post_id": 7003, "published_day": 417, "state": "resolved", "title": "API latency affecting some customers"}, {"impact": "none", "l…
- 19Verify
read_owner_spreadsheetok{}{"count": 6, "rows": [{"last_reviewed_day": 340, "owning_team": "Commerce Platform", "row_id": 1, "service_label": "Checkout (commerce)", "slack_channel": "#commerce", "week_start": "sunday"}, {"last_reviewed_day": 340, "owning_team": "Commerce Platform", "row_id": 2, "service_label": "Payments (commerce)", "slack_channel": "#commerce", "week_start": "sunday"}, {"last_reviewed_day": 210, "owning_team": "Discovery Squad", "row_id": 3, "service_label": "Search (growth)", "slack_channel": "#growth…
- 20Verify
pd_list_servicesok{}{"count": 5, "rows": [{"escalation_policy": "EP-Commerce", "name": "checkout-api", "pd_service_id": "PSVC001", "status": "active"}, {"escalation_policy": "EP-Commerce", "name": "payments-api", "pd_service_id": "PSVC002", "status": "active"}, {"escalation_policy": "EP-Platform", "name": "edge-gateway", "pd_service_id": "PSVC003", "status": "active"}, {"escalation_policy": "EP-Growth", "name": "search-svc", "pd_service_id": "PSVC004", "status": "active"}, {"escalation_policy": "EP-Commerce", "nam…
- 21Verify
jira_searchok{ "project": "DOB" }{"count": 1, "rows": [{"assignee": "on-call", "component": "media-service", "created_day": 414, "issue_type": "Task", "key": "DOB-028", "priority": "High", "project": "DOB", "resolution": "", "status": "In Progress", "summary": "This is an investigation, not a change.", "updated_day": 420}]}
- 22Verify
get_runtime_statsok{ "service": "media-service" }{"count": 1, "rows": [{"gc_collections_per_min": 5, "gc_pause_p99_ms": 19, "heap_used_pct": 44, "service": "media-service", "threads": 56}]}
- 23Verify
list_commitsok{ "service": "media-service" }{"count": 20, "rows": [{"additions": 9, "author": "Sam Whitfield", "day": 396, "deletions": 3, "files": "src/media/assets.py", "message": "media-service: origin egress up 6x week over week", "service": "media-service", "sha": "4a30cb8"}, {"additions": 8, "author": "Jordan Blake", "day": 376, "deletions": 11, "files": "src/media/assets.py", "message": "media-service: serve reads from origin while the CDN migration settles", "service": "media-service", "sha": "f38b0d7"}, {"additions": 14, "author…
Methodology
How the numbers are made
One executable world, not one hundred stubs
Every task runs against NovaCart, a mid-size e-commerce SaaS: 72 SQLite tables, 1,451 seeded rows, a 38-file monorepo with 417 commits, 30 knowledge-base documents, and 97 MCP tools spanning a first-party engineering stack (tickets, PRs, CI, deployments, canaries, migrations, feature flags, metrics, alerts, incidents, chat) plus deliberately disagreeing vendor-shaped surfaces — Jira, Linear, GitHub Issues, Prometheus, Sentry, PagerDuty, Confluence, spreadsheets — and Kubernetes.
High-level work, not prescribed tool recipes
All 100 prompts are distinct employee requests. The agent must discover the relevant systems, correlate records, choose among three documented operating options, and decide what to change. Reference solutions span 32–67 successful calls, and exact call order is not graded.
Deep, inspectable evidence rooms
Every task publishes at least 53 task-scoped files covering operating policy, source communications, structured exports, approvals, provider contracts, current and stale source records, and control history. Only agent-visible context files appear in the asset room; sealed starting state and verifier contracts remain evaluation artifacts.
Specific deterministic outcome grading
Each task publishes 55–73 concrete criteria across investigation, correlation, decision quality, state transition, exact answer, and write-scope containment. Read-before-write and task-scoped mutation controls are enforced without an LLM judge or exact-sequence rubric.
Qualification evidence
The exact 3.2.7 release passed 100/100 oracle replays and 100/100 deterministic replays. Its 1400 applicable negative-control executions produced 0 false accepts.
Leaderboard honesty
Only a complete, version-pinned run over all 100 tasks may appear as a model row. Older, partial, or source-world runs remain disclosed in release reports but are not ranked here.
Qualification controls — excluded from the leaderboard
- Reference solution: 100.0% across 100 tasks — Solvability control only; this is not a model leaderboard entry.
Run your model
Score your own model on the DevOpsBench-100 sandbox.
Scores the public sandbox task with your model — verified by the sandbox grader, not a leaderboard entry.
Claude, ChatGPT, Grok, DeepSeek, or a checkpoint of your own behind an OpenAI-compatible endpoint. The run drives the same isolated MCP sandbox the Live MCP console uses; the score is the sandbox grader's verdict on the state your model left behind. Run the one public task inline, or a suite of the hosted frozen tasks as a background job and publish it below the official leaderboard. The full 100-task suite runs on Harbor with the command at the bottom. How runs, suites, and community runs work →
Full 100-task suite on Harbor
harbor run -d blobfishai/devopsbench-100 -a <agent> -m anthropic/claude-sonnet-5<agent> with your agent scaffold. A hosted run above scores the frozen tasks this host carries; a leaderboard row needs all 100 tasks on this release with traces.Community runs
Self-runs on DevOpsBench-100 — below the official leaderboard, never merged into it.
self-run · graded by the benchmark's own verifier · not the frozen official suite unless suite:full
Community runs are self-runs: a caller's model, graded by this benchmark's own verifier over the hosted frozen tasks the run covered. They are listed below the official leaderboard and never merged into it. The Coverage column counts the official tasks each run itself covered, out of the 100-task release; “hosted” tasks are the frozen tasks this host carries, which caps what any run here can cover. Only a full suite over every official task reads official suite. How to run and publish one →
Loading community runs…
Run it yourself
The benchmark is public. The world is executable.
Download the dataset, replay the oracle trajectories, run the standalone verifiers, and submit the first exact, pinned model run.