devopsbench / Benchmark deep dive
Compare the outcomes, understand the gaps, and inspect the work behind every score.
All 30 selected attempts ran. Native verification completed for nine ChatGPT attempts, two Claude attempts and four DeepSeek attempts; all 15 failed. Fourteen attempts timed out and one ChatGPT attempt ended with an execution error. Those 15 unfinished attempts retain unknown native verdicts. The 0/10 task-success figures include unsuccessful execution; they do not mean ten native-test failures for each model.
The same ten tasks include eight from the released training split and two from the heldout split. Each attempt had a 600-second total window for environment setup, agent execution and native verification, with at most 100 tool calls. The source permits 2400 seconds for the agent. These single-attempt measurements do not establish full 100-task results or outcomes under the source budget. Controller receipts show that distributed subscription account waiting finished before each task window began.
The retained trajectories contain 2419 tool calls: 860 for ChatGPT, 662 for Claude and 897 for DeepSeek. Every attempt has its original task archive. The 15 completed native verifications expose their individual assertions and partial rewards separately from pass/fail. Timed-out and errored attempts keep their recorded actions and missing verdicts; no retry or replacement result is substituted.
Scores use the same selected tasks and the report’s declared primary evaluator.
| Model / agent | Success | Execution errors | Mean latency | Cost / task |
|---|---|---|---|---|
| ChatGPT subscriptiongpt-6-astra | 0.0%0/10 selected tasks | 1 | 467.9s | Not reported |
| Claude subscriptionclaude-sonnet-5 | 0.0%0/10 selected tasks | 8 | 568.4s | Not reported |
| DeepSeekdeepseek-v4-pro | 0.0%0/10 selected tasks | 6 | 573.9s | Not reported |
Pilot coverage: 10/100 source tasks. Small fixed selections do not establish a full-benchmark ranking. Missing grades remain visible.
Execution failure and an incorrect result are separate outcomes.
passed=0.0; pass threshold=1.0
TimeoutError:
passed=0.0; pass threshold=1.0
Where the tasks differ
Task success within each recorded task family, hardest first. Small selections make single tasks decisive; read the counts with the rate.
| Task family | Tasks | ChatGPT subscriptiongpt-6-astra | Claude subscriptionclaude-sonnet-5 | DeepSeekdeepseek-v4-pro | Field average | Best |
|---|---|---|---|---|---|---|
| aiops_detection | 1 | 0.0%0/1 | 0.0%0/1 · 1 err | 0.0%0/1 | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
| aiops_localization | 1 | 0.0%0/1 | 0.0%0/1 · 1 err | 0.0%0/1 · 1 err | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
| cross_system | 3 | 0.0%0/3 | 0.0%0/3 · 1 err | 0.0%0/3 · 2 err | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
| error_rate_reduction | 1 | 0.0%0/1 | 0.0%0/1 · 1 err | 0.0%0/1 · 1 err | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
| latency_optimization | 1 | 0.0%0/1 | 0.0%0/1 · 1 err | 0.0%0/1 | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
| multi_service_rollout | 2 | 0.0%0/2 | 0.0%0/2 · 2 err | 0.0%0/2 · 1 err | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
| reconciliation | 1 | 0.0%0/1 · 1 err | 0.0%0/1 · 1 err | 0.0%0/1 · 1 err | 0.0% | 0.0%ChatGPT subscription, Claude subscription, DeepSeek |
Failure modes
Labels come from each recorded attempt and never change its verdict. Execution errors and timeouts point at infrastructure or budgets; graded failures point at the answer.
dob100-014-localize-checkout-latencydob100-035-port-close-backlog-issuesdob100-049-payments-retrydob100-062-w6-copy-13dob100-085-notification-templatesdob100-089-rcn-customer-facing-incidents
dob100-010-detect-status-page-recurrencedob100-036-port-close-blocked-issuesdob100-076-gateway-pool-reusedob100-081-backorders
Open an outcome to inspect the instructions, trajectory and grading evidence for that attempt.
Primary evaluation · 10 shared tasks · at most one attempt per model
| Task | ChatGPT subscriptiongpt-6-astra | Claude subscriptionclaude-sonnet-5 | DeepSeekdeepseek-v4-pro |
|---|---|---|---|
| dob100-010-detect-status-page-recurrenceaiops_detection | Fail ↗ | Timeout ↗ | Fail ↗ |
| dob100-014-localize-checkout-latencyaiops_localization | Fail ↗ | Timeout ↗ | Timeout ↗ |
| dob100-035-port-close-backlog-issuescross_system | Fail ↗ | Timeout ↗ | Timeout ↗ |
| dob100-036-port-close-blocked-issuescross_system | Fail ↗ | Fail ↗ | Fail ↗ |
| dob100-049-payments-retryerror_rate_reduction | Fail ↗ | Timeout ↗ | Timeout ↗ |
| dob100-062-w6-copy-13cross_system | Fail ↗ | Fail ↗ | Timeout ↗ |
| dob100-076-gateway-pool-reuselatency_optimization | Fail ↗ | Timeout ↗ | Fail ↗ |
| dob100-081-backordersmulti_service_rollout | Fail ↗ | Timeout ↗ | Fail ↗ |
| dob100-085-notification-templatesmulti_service_rollout | Fail ↗ | Timeout ↗ | Timeout ↗ |
| dob100-089-rcn-customer-facing-incidentsreconciliation | Execution error ↗ | Timeout ↗ | Timeout ↗ |
Not attempted means no solver run began. Errors and timeouts describe execution. Ungraded means the completed attempt has no primary verdict. Each stays distinct from a graded failure.
Sample tasks
Each card opens the recorded trajectory: the original instruction, every tool action, the final response and the grader’s reasoning.
Customers still cannot check out, and the status page says we fixed it. The current monitoring views do not yet establish whether this is an active objective breach, a stale signal…
Open trajectory →Timeoutdob100-010-detect-status-page-recurrenceClaude subscription · aiops_detection · 602.7s · 71 actionsCustomers still cannot check out, and the status page says we fixed it. The current monitoring views do not yet establish whether this is an active objective breach, a stale signal…
Open trajectory →Faildob100-010-detect-status-page-recurrenceDeepSeek · aiops_detection · 559.0s · 100 actionsCustomers still cannot check out, and the status page says we fixed it. The current monitoring views do not yet establish whether this is an active objective breach, a stale signal…
Open trajectory →The work behind the score
Original instructions, observed actions, and the verdict for each attempt.
gpt-6-astra · 369.6s · 100 recorded actions
Customers still cannot check out, and the status page says we fixed it. The current monitoring views do not yet establish whether this is an active objective breach, a stale signal, or a healthy service being blamed for a downstream symptom. The next owner needs this settled across cart conversion, order creation, idempotency, and the customer's final purchase step. The disputed mechanism is a supposedly resolved incident recurring behind stale customer-status communication. Find the narrowest explanation that fits every material signal and leave the incident record ready for independent verification. This diagnosis will decide whether checkout enters recovery on 2026-03-06. Say whether the date holds against current on-call commitments and change windows, what a sanctioned acceleration costs, and which faster route still lacks authority.
Recorded benchmark evaluator · devopsbench-100:3.2.7:original-verifier
| correctness.diagnosis_submitted: submit a detection result for scope 'status-7001-recurrence' | Not met |
| correctness.detection_correct: status-7001-recurrence is breaching its SLO: fault_detected must be true | Not met |
| correctness.service_identified: the service behind the symptom is checkout | Not met |
| deployment.investigation_was_read_only: satisfied | Met |
| quality.evidence_recorded: record the evidence that supports the finding | Not met |
| quality.efficient_investigation: reach a conclusion within 12 tool calls | Not met |
| quality.ticket_closed: close ticket OPS-131 once reported | Not met |
| quality.closed_after_the_work: close OPS-131 only once the finding is submitted, not before | Not met |
| correctness.scoped_change_no_fabricated_data: satisfied | Met |
| correctness.world_invariants_intact: satisfied | Met |
| quality.no_new_hardcoded_secrets: satisfied | Met |
| quality.no_unproductive_ci_loops: satisfied | Met |
| correctness.v4_case_identity_resolved: resolve DOB-010 through its immutable Jira key, linked GitHub issue, and cross-tracker relationship | Not met |
| correctness.v4_authority_reconciled: compare current control OPS-CONTROL-2026.03 with retired control OPS-CONTROL-2025.11 before acting | Not met |
| correctness.v4_live_state_correlated: correlate the task-specific checkout provider state instead of relying on files or tracker text alone | Not met |
| correctness.v4_material_evidence_complete: complete all 20 materially causal evidence joins inside the larger reference investigation before changing state | Not met |
| correctness.v5_answer_business_need_date: Preserved 2026-03-06 from the published status-page cutover notice as the control date; did not infer urgency from the ticket title. | Not met |
| correctness.v5_answer_replicas_per_zone: Read 2 healthy replicas per production zone from the DOB-010 change-readiness standard. | Not met |
| correctness.v5_answer_production_zones: Read 3 production zones from the PagerDuty scale record for PD-DOB-010. | Not met |
| correctness.v5_answer_required_replicas: Derived 2 per zone x 3 zones = 6 replicas required before the checkout cutover. | Not met |
| correctness.v5_answer_observed_replicas: Read 6 replicas in the checkout production pool from the PagerDuty scale record. | Not met |
| correctness.v5_answer_reserved_replicas: Excluded 3 replicas reserved for the inventory freeze in Linear CAP-10. | Not met |
| correctness.v5_answer_usable_replicas: Calculated 6 observed - 3 reserved = 3 usable replicas. | Not met |
| correctness.v5_answer_replica_gap: Calculated 6 required - 3 usable = 3 replicas uncovered. | Not met |
| correctness.v5_answer_quantity_unit: Kept every capacity quantity in replicas. | Not met |
| correctness.v5_answer_standard_capacity_date: Read 2026-03-06 as CloudCap's independently confirmed standard delivery date from vendor order VEND-10. | Not met |
| correctness.v5_answer_expedited_capacity_date: Read 2026-03-04 as CloudCap's independently confirmed expedited delivery date (USD 1050) from vendor order VEND-10. | Not met |
| correctness.v5_answer_capacity_request_replicas: Bound the vendor request to the 3 uncovered replicas rather than the full 6-replica requirement. | Not met |
| correctness.v5_answer_next_change_window: Read the checkout change calendar (2026-03-04, 2026-03-06, 2026-03-08, 2026-03-10) from the current operating control and identified 2026-03-04 as the next window. | Not met |
| correctness.v5_capacity_evidence_complete: read the readiness standard, CloudCap order VEND-10, change approval CHG-10, the Linear reservation and the status-page cutover notice before changing state | Not met |
| correctness.v4_supported_path_selected: derive and execute the task-supported branch rather than the stale-note or broad-workaround alternatives | Not met |
| correctness.v5_capacity_plan_recorded: record the DOB-010 capacity plan once as the JSON reconciliation answer to DOB-010-capacity-plan | Not met |
| correctness.v5_answer_standard_plan_completion: Calculated standard_capacity_plan outcome as 2026-03-06: the first change window on or after standard delivery. | Not met |
| correctness.v5_answer_expedited_plan_completion: Calculated expedite_capacity outcome as 2026-03-04: the first change window on or after expedited delivery. | Not met |
| correctness.v5_answer_reserved_release_completion: Calculated release_reserved_capacity outcome as 2026-03-03 and kept its separate-approval condition. | Not met |
| correctness.v5_answer_recommended_option: Compared the date, cost and authority of standard_capacity_plan, expedite_capacity and release_reserved_capacity; selected standard_capacity_plan as the best currently authorised plan under the readiness standard. | Not met |
| correctness.v5_answer_recommended_outcome_date: Calculated 2026-03-06 as the supported outcome date for standard_capacity_plan. | Not met |
| correctness.v5_answer_recommended_incremental_cost_usd: Applied USD 0 as the documented incremental cost of standard_capacity_plan. | Not met |
| correctness.v5_answer_escalation_approval_required: Recognised that release_reserved_capacity remains outside current authority and needs incident-commander approval beyond CHG-10. | Not met |
| correctness.v5_answer_approval_reference: Applied change approval CHG-10 only to the authorised standard_capacity_plan scope. | Not met |
| correctness.v5_answer_outcome_vs_control_days: Compared 2026-03-06 with the independent control date 2026-03-06 and calculated a signed variance of +0 day(s). | Not met |
| correctness.v5_answer_decision_timing_status: Reported ON_TIME; did not relabel a controlled but late result as on time. | Not met |
| correctness.v5_approval_applied_to_selected_scope: apply CHG-10 to standard_capacity_plan only and keep release_reserved_capacity outside current authority | Not met |
| correctness.v4_source_mutations_completed: complete each task-required state-changing capability at least once successfully | Not met |
| deployment.v4_state_readbacks_complete: reopen the persisted provider and tracker state after the final source mutation instead of trusting acknowledgements | Not met |
| deployment.v4_source_work_precedes_readback: perform provider readback only after the task-specific operational state change | Not met |
| deployment.v4_handoff_read_back: reopen the exact case-room conversation after posting the completion handoff | Not met |
| deployment.v4_evidence_before_state_change: establish every material fact before the first successful task state transition | Not met |
| deployment.v5_decision_evidence_precedes_record: derive the capacity plan only after every source fact it depends on has been read successfully | Not met |
| deployment.v5_record_precedes_handoff: record the capacity plan before stating it in the completion handoff | Not met |
| correctness.v4_completion_handoff_exactly_once: write exactly one scoped completion handoff for DOB-010 | Not met |
| correctness.v5_handoff_is_natural_completion: write a natural scoped completion note, not a keyword list or serialized benchmark object | Not met |
| correctness.v5_handoff_states_selected_option: state the selected option (standard_capacity_plan) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_outcome_date: state the outcome date (2026-03-06) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_approval_reference: state the approval reference (CHG-10) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_timing_status: state the timing status (ON_TIME) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_binding_constraint: state the binding constraint (CloudCap standard capacity on 2026-03-06) in the scoped completion handoff | Not met |
| deployment.v4_no_rejected_mutations: complete without a rejected state-changing call; failed exploratory reads remain recoverable | Not met |
| deployment.v4_readback_precedes_handoff: post the completion handoff only after the changed provider state has been reopened | Not met |
passed=0.0; pass threshold=1.0 assertion: 3 verifier(s) failed (PC score=0.5733): correctness/diagnosis_submitted - submit a detection result for scope 'status-7001-recurrence'; correctness/detection_correct - status-7001-recurrence is breaching its SLO: fault_detected must be true; correctness/service_identified - the service behind the symptom is checkout | quality also flagged: quality/evidence_recorded, quality/efficient_investigation, quality/ticket_closed, quality/closed_after_the_work Partial native reward: 0.11. The recorded native verdict determines task success.
Per-model report
The same evaluation from a single model’s perspective: headline metrics, results by task family, and the deliverables for this run.
| Task family | Tasks | Task success | Completion | Mean latency |
|---|---|---|---|---|
| aiops_detection | 1 | 0.0%0/1 | 1/1 | 369.6s |
| aiops_localization | 1 | 0.0%0/1 | 1/1 | 553.9s |
| cross_system | 3 | 0.0%0/3 | 3/3 | 481.3s |
| error_rate_reduction | 1 | 0.0%0/1 | 1/1 | 533.9s |
| latency_optimization | 1 | 0.0%0/1 | 1/1 | 553.0s |
| multi_service_rollout | 2 | 0.0%0/2 | 2/2 | 501.0s |
| reconciliation | 1 | 0.0%0/1 | 0/1 | 222.3s |
Rows sort by task success. A category with one task is decided by that task; read the count with the rate. Derived CSV and Markdown deliverables are computed from the verified published report.
Reproducibility
Ten matching tasks from the exact published revision 8 (2 source-heldout, 8 source-train). Original main/world images and original deterministic state verifiers. Pilot time/command budgets are recorded per run. Source: https://blobfish.ai/benchmarks/devopsbench-100. Registered source archive SHA-256: 9fc52450e746c2de454a1662a3053f9afa6616ec9f5feea6ca31012ce871e27f. Selected tasks: dob100-010-detect-status-page-recurrence, dob100-014-localize-checkout-latency, dob100-035-port-close-backlog-issues, dob100-036-port-close-blocked-issues, dob100-049-payments-retry, dob100-062-w6-copy-13, dob100-076-gateway-pool-reuse, dob100-081-backorders, dob100-085-notification-templates, dob100-089-rcn-customer-facing-incidents. ChatGPT subscription: harbor-codex-subscription; observed chatgpt_subscription / 0.154.0 / max. Claude subscription: harbor-claude-subscription; observed claude_subscription / 2.1.273 / max. DeepSeek: harbor-deepseek; observed deepseek_api / max. 2 concurrent task attempts; 600 seconds and 100 tool calls per task; 10800 seconds per run. Same-provider task queues within this run are outside task clocks. Latency includes environment setup, agent execution and native verification. Subscription cost is unmeasured. Full traces and pinned task/world images are retained in raw_outputs.jsonl. Controller timing receipts verify the complete task window for all 30 attempts. Distributed subscription account waits finished before their task clocks started; queue time is recorded separately. Every trusted solver was submitted within its task deadline and retired before releasing its account. The source declares a 2400-second agent budget. This pilot caps setup, solver execution and native verification together at 600 seconds. Eight selected tasks come from the training split and two from the heldout split of the released source. These measurements describe this shorter-budget, single-attempt ten-task pilot. Individual native assertions are available for 15 graded task attempts in the task explorer and native_verifier_details.json, with original archive and verifier-file checksums. Partial rewards are separate from the recorded pass/fail verdict.