Executable checksFail
Recorded benchmark evaluator · devopsbench-100:3.2.7:original-verifier
| correctness.module_live_inventory: backorder_queue must be live in production inventory | Not met |
| quality.pr_linked_inventory: a merged pull request linked to ENG-2603 must add backorder_queue to inventory | Not met |
| correctness.ci_stages_inventory: the inventory pull request must have passed all four CI stages | Not met |
| correctness.migration_applied_inventory: migration 0034_backorders must be applied in production | Not met |
| deployment.staging_first#1: satisfied | Met |
| correctness.module_live_checkout: backorder_flow must be live in production checkout | Not met |
| quality.pr_linked_checkout: a merged pull request linked to ENG-2603 must add backorder_flow to checkout | Not met |
| correctness.ci_stages_checkout: the checkout pull request must have passed all four CI stages | Not met |
| deployment.staging_first#2: satisfied | Met |
| deployment.canary_then_promote#1: checkout is tier 1: canary at <=25%, assess, then promote_canary | Not met |
| correctness.module_live_storefront-web: backorder_banner must be live in production storefront-web | Not met |
| quality.pr_linked_storefront-web: a merged pull request linked to ENG-2603 must add backorder_banner to storefront-web | Not met |
| correctness.ci_stages_storefront-web: the storefront-web pull request must have passed all four CI stages | Not met |
| deployment.staging_first#3: satisfied | Met |
| deployment.canary_then_promote#2: storefront-web is tier 1: canary at <=25%, assess, then promote_canary | Not met |
| deployment.rollout_order: production rollout order must be inventory then checkout then storefront-web | Not met |
| deployment.no_alarming_deploys: satisfied | Met |
| quality.ticket_closed: ticket ENG-2603 must be closed | Not met |
| quality.closed_after_the_work: close ENG-2603 only once the work is actually done, not before | Not met |
| correctness.scoped_change_no_fabricated_data: satisfied | Met |
| correctness.world_invariants_intact: satisfied | Met |
| quality.no_new_hardcoded_secrets: satisfied | Met |
| quality.no_unproductive_ci_loops: satisfied | Met |
| correctness.v4_case_identity_resolved: resolve DOB-081 through its immutable Jira key, linked GitHub issue, and cross-tracker relationship | Not met |
| correctness.v4_authority_reconciled: compare current control OPS-CONTROL-2026.03 with retired control OPS-CONTROL-2025.11 before acting | Not met |
| correctness.v4_live_state_correlated: correlate the task-specific storefront-web provider state instead of relying on files or tracker text alone | Not met |
| correctness.v4_material_evidence_complete: complete all 20 materially causal evidence joins inside the larger reference investigation before changing state | Not met |
| correctness.v5_answer_business_need_date: Preserved 2026-03-08 from the published status-page cutover notice as the control date; did not infer urgency from the ticket title. | Not met |
| correctness.v5_answer_replicas_per_zone: Read 2 healthy replicas per production zone from the DOB-081 change-readiness standard. | Not met |
| correctness.v5_answer_production_zones: Read 2 production zones from the PagerDuty scale record for PD-DOB-081. | Not met |
| correctness.v5_answer_required_replicas: Derived 2 per zone x 2 zones = 4 replicas required before the storefront-web cutover. | Not met |
| correctness.v5_answer_observed_replicas: Read 5 replicas in the storefront-web production pool from the PagerDuty scale record. | Not met |
| correctness.v5_answer_reserved_replicas: Excluded 3 replicas reserved for the checkout freeze in Linear CAP-81. | Not met |
| correctness.v5_answer_usable_replicas: Calculated 5 observed - 3 reserved = 2 usable replicas. | Not met |
| correctness.v5_answer_replica_gap: Calculated 4 required - 2 usable = 2 replicas uncovered. | Not met |
| correctness.v5_answer_quantity_unit: Kept every capacity quantity in replicas. | Not met |
| correctness.v5_answer_standard_capacity_date: Read 2026-03-05 as CloudCap's independently confirmed standard delivery date from vendor order VEND-81. | Not met |
| correctness.v5_answer_expedited_capacity_date: Read 2026-03-04 as CloudCap's independently confirmed expedited delivery date (USD 1200) from vendor order VEND-81. | Not met |
| correctness.v5_answer_capacity_request_replicas: Bound the vendor request to the 2 uncovered replicas rather than the full 4-replica requirement. | Not met |
| correctness.v5_answer_next_change_window: Read the storefront-web change calendar (2026-03-05, 2026-03-07, 2026-03-09, 2026-03-11) from the current operating control and identified 2026-03-05 as the next window. | Not met |
| correctness.v5_capacity_evidence_complete: read the readiness standard, CloudCap order VEND-81, change approval CHG-81, the Linear reservation and the status-page cutover notice before changing state | Not met |
| correctness.v4_supported_path_selected: derive and execute the task-supported branch rather than the stale-note or broad-workaround alternatives | Not met |
| correctness.v5_capacity_plan_recorded: record the DOB-081 capacity plan once as the JSON reconciliation answer to DOB-081-capacity-plan | Not met |
| correctness.v5_answer_standard_plan_completion: Calculated standard_capacity_plan outcome as 2026-03-05: the first change window on or after standard delivery. | Not met |
| correctness.v5_answer_expedited_plan_completion: Calculated expedite_capacity outcome as 2026-03-05: the first change window on or after expedited delivery. | Not met |
| correctness.v5_answer_reserved_release_completion: Calculated release_reserved_capacity outcome as 2026-03-03 and kept its separate-approval condition. | Not met |
| correctness.v5_answer_recommended_option: Compared the date, cost and authority of standard_capacity_plan, expedite_capacity and release_reserved_capacity; selected standard_capacity_plan as the best currently authorised plan under the readiness standard. | Not met |
| correctness.v5_answer_recommended_outcome_date: Calculated 2026-03-05 as the supported outcome date for standard_capacity_plan. | Not met |
| correctness.v5_answer_recommended_incremental_cost_usd: Applied USD 0 as the documented incremental cost of standard_capacity_plan. | Not met |
| correctness.v5_answer_escalation_approval_required: Recognised that release_reserved_capacity remains outside current authority and needs incident-commander approval beyond CHG-81. | Not met |
| correctness.v5_answer_approval_reference: Applied change approval CHG-81 only to the authorised standard_capacity_plan scope. | Not met |
| correctness.v5_answer_outcome_vs_control_days: Compared 2026-03-05 with the independent control date 2026-03-08 and calculated a signed variance of -3 day(s). | Not met |
| correctness.v5_answer_decision_timing_status: Reported ON_TIME; did not relabel a controlled but late result as on time. | Not met |
| correctness.v5_approval_applied_to_selected_scope: apply CHG-81 to standard_capacity_plan only and keep release_reserved_capacity outside current authority | Not met |
| correctness.v4_source_mutations_completed: complete each task-required state-changing capability at least once successfully | Not met |
| deployment.v4_state_readbacks_complete: reopen the persisted provider and tracker state after the final source mutation instead of trusting acknowledgements | Not met |
| deployment.v4_source_work_precedes_readback: perform provider readback only after the task-specific operational state change | Not met |
| deployment.v4_handoff_read_back: reopen the exact case-room conversation after posting the completion handoff | Not met |
| deployment.v4_evidence_before_state_change: establish every material fact before the first successful task state transition | Not met |
| deployment.v5_decision_evidence_precedes_record: derive the capacity plan only after every source fact it depends on has been read successfully | Not met |
| deployment.v5_record_precedes_handoff: record the capacity plan before stating it in the completion handoff | Not met |
| correctness.v4_completion_handoff_exactly_once: write exactly one scoped completion handoff for DOB-081 | Not met |
| correctness.v5_handoff_is_natural_completion: write a natural scoped completion note, not a keyword list or serialized benchmark object | Not met |
| correctness.v5_handoff_states_selected_option: state the selected option (standard_capacity_plan) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_outcome_date: state the outcome date (2026-03-05) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_approval_reference: state the approval reference (CHG-81) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_timing_status: state the timing status (ON_TIME) in the scoped completion handoff | Not met |
| correctness.v5_handoff_states_binding_constraint: state the binding constraint (CloudCap standard capacity on 2026-03-05) in the scoped completion handoff | Not met |
| deployment.v4_no_rejected_mutations: complete without a rejected state-changing call; failed exploratory reads remain recoverable | Not met |
| deployment.v4_readback_precedes_handoff: post the completion handoff only after the changed provider state has been reopened | Not met |
passed=0.0; pass threshold=1.0 assertion: 10 verifier(s) failed (PC score=0.3333): correctness/module_live_inventory - backorder_queue must be live in production inventory; correctness/ci_stages_inventory - the inventory pull request must have passed all four CI stages; correctness/migration_applied_inventory - migration 0034_backorders must be applied in production; correctness/module_live_checkout - backorder_flow must be live in production checkout; correctness/ci_stages_checkout - the checkout pull request must have passed all four CI stages; deployment/canary_then_promote - checkout is tier 1: canary at <=25%, assess, then promote_canary; correctness/module_live_storefront-web - backorder_banner must be live in production storefront-web; correctness/ci_stages_storefront-web - the storefront-web pull request must have passed all four CI stages; deployment/canary_then_promote - storefront-web is tier 1: canary at <=25%, assess, then promote_canary; deployment/rollout_order - production rollout order must be inventory then checkout then storefront-web | quality also flagged: quality/pr_linked_inventory, quality/pr_linked_checkout, quality/pr_linked_storefront-web, quality/ticket_closed, quality/closed_after_the_work Partial native reward: 0.119. The recorded native verdict determines task success.