Skip to benchmark
Blobfish ResearchDevOpsBench-100 v1.0.0Public

Can an agent run the incident, ship the fix, and prove it landed?

DevOpsBench-100 measures complete operations work over one executable company: root-cause the symptom, change the code, ride the canary, and close the incident — through 97 real MCP tools, graded only by executable checks on the world left behind.

100DevOps tickets
97MCP tools
72world tables
1,420deterministic checks
0LLM grading calls
Executed release

The release gate executed 371 qualification runs: 100/100 oracle replays at reward 1.0, 100 byte-identical deterministic replays, and 171 adversarial control executions with zero false accepts. No model has run the released dataset yet — the leaderboard says so.

Inspect the verifier contract →

Measured performance

Leaderboard

REF
Reference oracleDeterministic reference solver (release qualification)
100 tasks
100.0%
13 calls avg.Solvability ceiling from the qualification run; not a model submission.
No model submissions yet — awaiting first run. The reference oracle establishes the ceiling; model rows appear once a pinned run is published.
60%Correctness

The requested outcome is true in the final world state: the right diagnosis, code change, migration, flag, or record — with world invariants intact.

30%Deployment

The change actually landed the way operations requires: CI green, merged, deployed through canary where policy demands it, rollbacks clean.

10%Quality

Professional closure: tickets updated, incidents resolved, status published, evidence recorded, no collateral writes.

100 distinct tasks

Task browser

100 shown
dob100-001-rca-analytics-egress-blockedAIOps root-cause analysis

RCA Analytics Egress Blocked

This is an investigation, not a change. Do NOT modify production state: no pull requests, deploys, flag toggles, migrations or resolutions. Investigate with the read tools and report what you find with submit_diagnosis(...). OPS-163 — Root cause: ana

Seeded files
0
Reference path
9 MCP calls

Complete task prompt

13 graded criteria summarized
This is an investigation, not a change. Do NOT modify production state: no pull requests, deploys, flag toggles, migrations or resolutions. Investigate with the read tools and report what you find with submit_diagnosis(...).

OPS-163 — Root cause: analytics connections are refused, not slow

Get to the bottom of analytics-connection-refused: not just which service, but the mechanism and the exact setting responsible. The service's own code is not necessarily where this lives — confirm it against the state of the system rather than inferring it from the symptom alone. Submit your finding under scope 'analytics-connection-refused' with the responsible service, the fault type, the offending key, and your evidence.
Rubric summary
  • correctness: diagnosis_submitted
  • correctness: service_localized
  • correctness: fault_type_correct
  • correctness: offending_key_correct
  • correctness: scoped_change_no_fabricated_data
  • correctness: world_invariants_intact
  • deployment: investigation_was_read_only
  • quality: evidence_recorded
  • quality: efficient_investigation
  • quality: ticket_closed
  • quality: closed_after_the_work
  • quality: no_new_hardcoded_secrets
  • quality: no_unproductive_ci_loops

Seeded evidence

Asset room

DevOpsBench keeps its evidence inside the executable world: the 30 knowledge-base documents, the 38-file monorepo with 417 commits, tickets, metrics, and logs live in the world database and are read through the 97 MCP tools.

No per-task files ship with this release — every document, code file, and record is inside the executable world and is read through its MCP tools. Download the world to browse the evidence directly.

Runnable world

Environment and tool contract

list_services

List all services with team, tier, kind, on-call engineer, repo HEAD version and deployed versions.

{
  "properties": {
    "team": {
      "description": "team",
      "type": "string"
    },
    "tier": {
      "description": "tier",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
get_service

Full detail for one service: metadata, deployed config, modules, endpoints, dependencies, current metrics.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [
    "service"
  ],
  "type": "object"
}
list_infra

List infrastructure components of the application stack (databases, caches, queues, object stores, CDN).

{
  "properties": {},
  "required": [],
  "type": "object"
}
list_files

List monorepo files, optionally filtered by service or path substring.

{
  "properties": {
    "path_contains": {
      "description": "path_contains",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
read_file

Read a monorepo source file. Returns its full current content.

{
  "properties": {
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
search_code

Search monorepo file contents for a substring; returns matching files with matching line numbers.

{
  "properties": {
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "query": {
      "description": "query",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [
    "query"
  ],
  "type": "object"
}
list_commits

Browse monorepo commit history (most recent first).

{
  "properties": {
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "path": {
      "description": "only commits touching this file path",
      "type": "string"
    },
    "query": {
      "description": "substring of the commit message",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
search_docs

Search the engineering knowledge base (runbooks, policies, design docs, ADRs, postmortems, API specs).

{
  "properties": {
    "kind": {
      "description": "runbook|policy|design_doc|adr|postmortem|api_spec|onboarding",
      "type": "string"
    },
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "query": {
      "description": "query",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_document

Read one knowledge-base document in full by doc_id (or exact title).

{
  "properties": {
    "doc_id": {
      "description": "doc_id",
      "type": "integer"
    },
    "title": {
      "description": "title",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_tickets

List issue-tracker tickets, optionally filtered by status, service, or type.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    },
    "ticket_type": {
      "description": "ticket_type",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_ticket

Fetch one ticket by key (e.g. ENG-2101).

{
  "properties": {
    "key": {
      "description": "key",
      "type": "string"
    }
  },
  "required": [
    "key"
  ],
  "type": "object"
}
list_pull_requests

List pull requests, optionally filtered by service or status.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_pull_request

Fetch a PR with its structured changes and CI history.

{
  "properties": {
    "pr_number": {
      "description": "pr_number",
      "type": "integer"
    }
  },
  "required": [
    "pr_number"
  ],
  "type": "object"
}
list_ci_runs

List CI runs (most recent first).

{
  "properties": {
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "pr_number": {
      "description": "pr_number",
      "type": "integer"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_ci_run

Fetch one CI run with its per-stage results (build, unit, integration, regression).

{
  "properties": {
    "run_id": {
      "description": "run_id",
      "type": "integer"
    }
  },
  "required": [
    "run_id"
  ],
  "type": "object"
}
list_deployments

List deployments (most recent first).

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_migrations

List database migrations and whether they are applied per environment.

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
query_metrics

Read current production service metrics (recomputed continuously from live traffic).

{
  "properties": {
    "metric": {
      "description": "metric",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_traffic_stats

Traffic-generator statistics: request rate per route with the current error rate and p99 of the owning service.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_slo_status

List SLOs with current values and whether each is breaching.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_alerts

List alarms, optionally filtered by status (firing|acknowledged|resolved) or service.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_error_events

Error-tracking issues (Sentry-style): grouped exceptions with culprit and event counts.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "status": {
      "description": "unresolved|resolved",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
search_logs

Search application logs by substring, service, or level.

{
  "properties": {
    "level": {
      "description": "level",
      "type": "string"
    },
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "query": {
      "description": "query",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_feature_flags

List feature flags with per-environment state.

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_packages

List package dependencies: version at repo HEAD and version deployed in production.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_vulnerabilities

List security-scanner findings.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_api_endpoints

List API endpoints with repo status, production status, and production traffic share.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_tests

List the test catalog.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_incidents

List incidents.

{
  "properties": {
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_status_page

Read the public system-status page.

{
  "properties": {
    "limit": {
      "description": "limit",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
list_messages

Read chat messages.

{
  "properties": {
    "channel": {
      "description": "channel",
      "type": "string"
    },
    "limit": {
      "description": "limit",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
create_ticket

Create a ticket. Returns the generated key.

{
  "properties": {
    "description": {
      "description": "description",
      "type": "string"
    },
    "priority": {
      "description": "priority",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    },
    "ticket_type": {
      "description": "task|bug|feature|security|incident|postmortem",
      "enum": [
        "task",
        "bug",
        "feature",
        "security",
        "incident",
        "postmortem"
      ],
      "type": "string"
    },
    "title": {
      "description": "title",
      "type": "string"
    }
  },
  "required": [
    "title"
  ],
  "type": "object"
}
update_ticket

Update a ticket's status and/or assignee.

{
  "properties": {
    "assignee": {
      "description": "assignee",
      "type": "string"
    },
    "key": {
      "description": "key",
      "type": "string"
    },
    "status": {
      "description": "open|in_progress|in_review|done",
      "enum": [
        "open",
        "in_progress",
        "in_review",
        "done"
      ],
      "type": "string"
    }
  },
  "required": [
    "key"
  ],
  "type": "object"
}
open_pull_request

Open a pull request carrying structured changes. change_type is one of: config {key,value}; dependency {package,version}; endpoint {path,status: active|deprecated|retired}; module {name}; flag {key,description}; flag_cleanup {key}; test_fix {test_name, action: fix|quarantine}; migration {name}; code_edit {path, find, replace}. Changes apply at merge; deploys carry them to an environment.

{
  "properties": {
    "body": {
      "description": "body",
      "type": "string"
    },
    "changes": {
      "description": "list of {change_type, payload}",
      "items": {
        "properties": {
          "change_type": {
            "type": "string"
          },
          "payload": {
            "type": "object"
          }
        },
        "required": [
          "change_type",
          "payload"
        ],
        "type": "object"
      },
      "type": "array"
    },
    "service": {
      "description": "service",
      "type": "string"
    },
    "ticket_key": {
      "description": "ticket_key",
      "type": "string"
    },
    "title": {
      "description": "title",
      "type": "string"
    }
  },
  "required": [
    "service",
    "title",
    "changes"
  ],
  "type": "object"
}
run_ci

Run the CI pipeline for an open PR (pr_number) or a service's main branch (service). Stages run in order: build, unit, integration, regression. The tool succeeds even when the pipeline fails - inspect the returned status and stages.

{
  "properties": {
    "pr_number": {
      "description": "pr_number",
      "type": "integer"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
merge_pull_request

Merge an open PR. Blocked unless its latest CI run passed. Applies the PR's changes to repo HEAD (including code edits) and cuts a new deployable version.

{
  "properties": {
    "pr_number": {
      "description": "pr_number",
      "type": "integer"
    }
  },
  "required": [
    "pr_number"
  ],
  "type": "object"
}
apply_migration

Apply a database migration to an environment. Migrations are forward-only and must be applied before the code version that requires them is deployed there.

{
  "properties": {
    "environment": {
      "description": "staging|production",
      "enum": [
        "staging",
        "production"
      ],
      "type": "string"
    },
    "name": {
      "description": "name",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [
    "service",
    "name",
    "environment"
  ],
  "type": "object"
}
deploy_service

Deploy a merged version to staging or production. canary_percent<100 stages a canary whose state only takes effect at promote_canary. Policy: production is staging-first; tier-1 services canary at <=25% then promote. A version whose migration is not applied is rejected.

{
  "properties": {
    "canary_percent": {
      "description": "canary_percent",
      "type": "integer"
    },
    "environment": {
      "description": "environment",
      "enum": [
        "staging",
        "production"
      ],
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    },
    "version": {
      "description": "version",
      "type": "string"
    }
  },
  "required": [
    "service",
    "environment"
  ],
  "type": "object"
}
assess_canary

Evaluate the pending canary for a service: reports whether the canary version would breach any SLO or trip an alarm if promoted. Run this before promote_canary.

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [
    "service"
  ],
  "type": "object"
}
promote_canary

Promote the pending canary to 100%; its state takes effect.

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [
    "service"
  ],
  "type": "object"
}
rollback_deployment

Emergency rollback to the previous successful deployment. Exempt from staging-first.

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [
    "service"
  ],
  "type": "object"
}
set_feature_flag

Toggle a feature flag or change its rollout percent in one environment. Runtime operation: takes effect immediately, no deploy needed.

{
  "properties": {
    "enabled": {
      "description": "enabled",
      "type": "boolean"
    },
    "environment": {
      "description": "environment",
      "enum": [
        "staging",
        "production"
      ],
      "type": "string"
    },
    "key": {
      "description": "key",
      "type": "string"
    },
    "rollout_percent": {
      "description": "rollout_percent",
      "type": "integer"
    }
  },
  "required": [
    "key",
    "environment"
  ],
  "type": "object"
}
shift_endpoint_traffic

Set the production traffic percent served by an endpoint (gateway runtime weight, no deploy needed). Policy: shift in stages of at most 50 points per step.

{
  "properties": {
    "path": {
      "description": "path",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    },
    "traffic_percent": {
      "description": "traffic_percent",
      "type": "integer"
    }
  },
  "required": [
    "service",
    "path",
    "traffic_percent"
  ],
  "type": "object"
}
acknowledge_alert

Acknowledge a firing alarm.

{
  "properties": {
    "alert_id": {
      "description": "alert_id",
      "type": "integer"
    }
  },
  "required": [
    "alert_id"
  ],
  "type": "object"
}
resolve_alert

Resolve an alarm. Refused while the underlying metric still breaches its SLO.

{
  "properties": {
    "alert_id": {
      "description": "alert_id",
      "type": "integer"
    }
  },
  "required": [
    "alert_id"
  ],
  "type": "object"
}
resolve_error_event

Mark an error-tracking issue resolved. Refused while the owning service still breaches an SLO.

{
  "properties": {
    "fingerprint": {
      "description": "fingerprint",
      "type": "string"
    }
  },
  "required": [
    "fingerprint"
  ],
  "type": "object"
}
create_incident

Declare an incident.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    },
    "severity": {
      "description": "sev1|sev2|sev3",
      "type": "string"
    },
    "title": {
      "description": "title",
      "type": "string"
    }
  },
  "required": [
    "title",
    "service",
    "severity"
  ],
  "type": "object"
}
update_incident

Update an incident's status (open|mitigated|resolved) and/or commander.

{
  "properties": {
    "commander": {
      "description": "commander",
      "type": "string"
    },
    "incident_id": {
      "description": "incident_id",
      "type": "integer"
    },
    "status": {
      "description": "status",
      "enum": [
        "open",
        "mitigated",
        "resolved"
      ],
      "type": "string"
    }
  },
  "required": [
    "incident_id"
  ],
  "type": "object"
}
publish_status_update

Publish an update to the public system-status page (state: investigating|identified|monitoring|resolved).

{
  "properties": {
    "body": {
      "description": "body",
      "type": "string"
    },
    "state": {
      "description": "state",
      "enum": [
        "investigating",
        "identified",
        "monitoring",
        "resolved"
      ],
      "type": "string"
    },
    "title": {
      "description": "title",
      "type": "string"
    }
  },
  "required": [
    "state",
    "title"
  ],
  "type": "object"
}
submit_diagnosis

Submit a diagnostic finding for an investigation. `scope` is what you were asked to investigate (a service name or alarm id). Set fault_detected=false with fault_type='none' when the scope is healthy. fault_type is one of: misconfig, missing_retry, missing_timeout, resource_exhaustion, unbounded_prefetch, cache_disabled, n_plus_one_query, cdn_bypass, bad_release, feature_flag_regression, node_unhealthy, unclassified, none. Use node_unhealthy when the cause is the node a service runs on rather than the service's own code or config. Use unclassified when you have established that a fault is present but were not asked for, and have not determined, the mechanism - reporting a breach you can evidence is better than reporting health you cannot. offending_key is the specific config key, flag key, node name or version responsible.

{
  "properties": {
    "evidence": {
      "description": "what you observed that supports this finding",
      "type": "string"
    },
    "fault_detected": {
      "description": "true if the scope is faulting, false if it is healthy",
      "type": "boolean"
    },
    "fault_type": {
      "description": "the mechanism, or 'unclassified' when a fault is evidenced but its mechanism was not asked for, or 'none' when healthy",
      "enum": [
        "misconfig",
        "missing_retry",
        "missing_timeout",
        "resource_exhaustion",
        "unbounded_prefetch",
        "cache_disabled",
        "n_plus_one_query",
        "cdn_bypass",
        "bad_release",
        "feature_flag_regression",
        "node_unhealthy",
        "unclassified",
        "none"
      ],
      "type": "string"
    },
    "offending_key": {
      "description": "config key, flag key or version at fault",
      "type": "string"
    },
    "scope": {
      "description": "the service or alarm id you were asked to investigate",
      "type": "string"
    },
    "service": {
      "description": "the service responsible (localization)",
      "type": "string"
    }
  },
  "required": [
    "scope",
    "fault_detected"
  ],
  "type": "object"
}
post_message

Post a message to a chat channel.

{
  "properties": {
    "body": {
      "description": "body",
      "type": "string"
    },
    "channel": {
      "description": "channel",
      "type": "string"
    }
  },
  "required": [
    "channel",
    "body"
  ],
  "type": "object"
}
jira_search

Search Jira issues. Jira status is a per-project workflow, not open/closed: a resolved issue has status='Done' AND a resolution set. Filter by project, status, issue_type, component or priority.

{
  "properties": {
    "component": {
      "description": "component",
      "type": "string"
    },
    "issue_type": {
      "description": "issue_type",
      "type": "string"
    },
    "limit": {
      "description": "limit",
      "type": "integer"
    },
    "project": {
      "description": "project",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
jira_get_issue

Fetch one Jira issue by key, including any links to issues in other trackers.

{
  "properties": {
    "key": {
      "description": "key",
      "type": "string"
    }
  },
  "required": [
    "key"
  ],
  "type": "object"
}
linear_list_issues

List Linear issues. Linear priority is numeric: 0=none, 1=urgent, 2=high, 3=normal, 4=low - it does not map cleanly onto Jira priority names.

{
  "properties": {
    "state": {
      "description": "state",
      "type": "string"
    },
    "team": {
      "description": "team",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
github_list_issues

List GitHub issues. GitHub has only state=open|closed; severity lives in labels if anywhere.

{
  "properties": {
    "label": {
      "description": "label",
      "type": "string"
    },
    "repo": {
      "description": "repo",
      "type": "string"
    },
    "state": {
      "description": "state",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_issue_links

List known cross-tracker links (duplicates/relates/implements). This is the only place the trackers are reconciled; neither tracker knows about it.

{
  "properties": {
    "source": {
      "description": "source",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
query_prometheus

Query a Prometheus series by metric and label selectors. Note the label spelling is Prometheus's own (e.g. checkout_service), and counter resets are flagged: a rate() over a reset under-reports.

{
  "properties": {
    "day_from": {
      "description": "day_from",
      "type": "integer"
    },
    "day_to": {
      "description": "day_to",
      "type": "integer"
    },
    "label_env": {
      "description": "label_env",
      "type": "string"
    },
    "label_service": {
      "description": "label_service",
      "type": "string"
    },
    "metric": {
      "description": "metric",
      "type": "string"
    }
  },
  "required": [
    "metric"
  ],
  "type": "object"
}
list_prometheus_label_values

List the values a Prometheus label actually takes. Use this when you are not sure how a service is spelled in metrics.

{
  "properties": {
    "label": {
      "description": "label",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
sentry_search_issues

Search Sentry issues (grouped exceptions). Event counts are SAMPLED at the project's sample_rate - see sentry_list_projects - so they are a fraction of the true volume and are not comparable to Prometheus counters.

{
  "properties": {
    "project_slug": {
      "description": "project_slug",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
sentry_list_projects

List Sentry projects with their event sample rates.

{
  "properties": {},
  "required": [],
  "type": "object"
}
pd_list_incidents

List PagerDuty incidents in a day range. urgency (high|low) and priority (P1..P4) are separate vocabularies; neither records whether customers saw it.

{
  "properties": {
    "since_day": {
      "description": "since_day",
      "type": "integer"
    },
    "status": {
      "description": "status",
      "type": "string"
    },
    "until_day": {
      "description": "until_day",
      "type": "integer"
    },
    "urgency": {
      "description": "urgency",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
pd_list_services

List PagerDuty technical services and escalation policies.

{
  "properties": {},
  "required": [],
  "type": "object"
}
pd_list_oncalls

Who is on call, by day and escalation policy.

{
  "properties": {
    "day": {
      "description": "day",
      "type": "integer"
    },
    "escalation_policy": {
      "description": "escalation_policy",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
pd_list_change_events

Change events recorded against a PagerDuty service. These exist only where someone wired the integration.

{
  "properties": {
    "pd_service_id": {
      "description": "pd_service_id",
      "type": "string"
    },
    "since_day": {
      "description": "since_day",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
list_status_page_posts

Public status-page posts. This is the ONLY system that records customer impact; incidents do not carry it. The status page also lags internal state.

{
  "properties": {
    "impact": {
      "description": "impact",
      "type": "string"
    },
    "since_day": {
      "description": "since_day",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
confluence_search

Search the Confluence wiki. Pages carry a last_updated_day; some are stale.

{
  "properties": {
    "query": {
      "description": "query",
      "type": "string"
    },
    "space": {
      "description": "space",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
confluence_get_page

Read one Confluence page in full.

{
  "properties": {
    "page_id": {
      "description": "page_id",
      "type": "integer"
    }
  },
  "required": [
    "page_id"
  ],
  "type": "object"
}
read_owner_spreadsheet

Read the hand-maintained service-owner spreadsheet. Note last_reviewed_day: rows drift as teams reorganise, and the sheet uses its own week convention.

{
  "properties": {},
  "required": [],
  "type": "object"
}
query_local_deploy_log

Query a team's local deploy log (a SQLite file kept because the central one is slow). Environment strings are free text and include 'nonprod-*' spellings; rollbacks are flagged separately.

{
  "properties": {
    "environment": {
      "description": "environment",
      "type": "string"
    },
    "include_rollbacks": {
      "description": "include_rollbacks",
      "type": "boolean"
    },
    "service": {
      "description": "service",
      "type": "string"
    },
    "since_day": {
      "description": "since_day",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
resolve_service_alias

Resolve any spelling of a service to its canonical name, and list every spelling it has across systems. Use this before comparing data from two tools.

{
  "properties": {
    "name": {
      "description": "name",
      "type": "string"
    }
  },
  "required": [
    "name"
  ],
  "type": "object"
}
list_service_aliases

Every known service-name spelling, by system.

{
  "properties": {},
  "required": [],
  "type": "object"
}
list_approval_policy

Which actions require a human sign-off before you may perform them, and why. The trigger is irreversibility, not difficulty.

{
  "properties": {},
  "required": [],
  "type": "object"
}
request_approval

Ask the accountable human to approve an irreversible action, and get their answer. State the action exactly as the approval policy names it, the target, and why it is necessary. A vague or unjustified request is refused - the approver is not a rubber stamp.

{
  "properties": {
    "action": {
      "description": "action",
      "type": "string"
    },
    "reason": {
      "description": "reason",
      "type": "string"
    },
    "target": {
      "description": "target",
      "type": "string"
    }
  },
  "required": [
    "action",
    "reason"
  ],
  "type": "object"
}
report_blocked

Declare that you cannot complete the task and stop, rather than reporting a success you did not achieve. State what blocked you and what you would need. Stopping blocked is a legitimate outcome; claiming a completion you did not reach is not.

{
  "properties": {
    "needed": {
      "description": "needed",
      "type": "string"
    },
    "reason": {
      "description": "reason",
      "type": "string"
    }
  },
  "required": [
    "reason"
  ],
  "type": "object"
}
list_alert_rules

List alerting rules. A rule's service_label may name a service that no longer exists - monitors outlive what they watch.

{
  "properties": {
    "routes_to": {
      "description": "routes_to",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_alert_firings

Individual alert firings. `silenced` means it never notified, `inhibited_by` names a rule that suppressed it, and `paged_incident` is NULL when it never reached a human. One failure does not produce one firing, one page, or one incident - the ratios are configuration artefacts.

{
  "properties": {
    "rule_id": {
      "description": "rule_id",
      "type": "integer"
    },
    "since_day": {
      "description": "since_day",
      "type": "integer"
    }
  },
  "required": [],
  "type": "object"
}
list_alert_silences

Active and expired alert silences. A silence that outlived its reason is why an alert can be firing and invisible at the same time.

{
  "properties": {},
  "required": [],
  "type": "object"
}
list_remediation_proposals

Read the remediation proposals people have put forward for an incident. Exactly one is the right call; the others are plausible suggestions that mask the symptom, target the wrong component, or change behaviour.

{
  "properties": {
    "incident_ref": {
      "description": "incident_ref",
      "type": "string"
    }
  },
  "required": [
    "incident_ref"
  ],
  "type": "object"
}
jira_transition_issue

Transition a Jira issue. Jira status is a per-project workflow, so moving an issue to 'Done' does NOT by itself mean it was fixed - a completed issue also carries a resolution (e.g. 'Fixed'). Set both.

{
  "properties": {
    "key": {
      "description": "key",
      "type": "string"
    },
    "resolution": {
      "description": "resolution",
      "type": "string"
    },
    "status": {
      "description": "status",
      "enum": [
        "Backlog",
        "In Progress",
        "In Review",
        "Blocked",
        "Done"
      ],
      "type": "string"
    }
  },
  "required": [
    "key",
    "status"
  ],
  "type": "object"
}
k8s_events_list

List Kubernetes events (OOMKilled, CrashLoopBackOff, ...). The kubelet records kernel-level kills that an application error tracker never sees, because the process dies before its SDK can flush.

{
  "properties": {
    "namespace": {
      "description": "namespace",
      "type": "string"
    },
    "pod": {
      "description": "pod",
      "type": "string"
    },
    "reason": {
      "description": "reason",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
k8s_pods_list

List pods with phase, restart count, memory limit/usage and the running image tag. The image tag is the only ground truth for what is actually deployed - release records in other systems drift from it, especially after a rollback.

{
  "properties": {
    "namespace": {
      "description": "namespace",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
k8s_nodes_list

List cluster nodes with their Ready status, active condition, CPU and disk utilisation, labels and kernel version. A service whose node has DiskPressure, a kernel deadlock, or no node matching its selector looks - from the service's own metrics and logs - exactly like a slow or broken service. This is the only place that difference is visible.

{
  "properties": {
    "node": {
      "description": "node",
      "type": "string"
    },
    "unhealthy_only": {
      "description": "unhealthy_only",
      "type": "boolean"
    }
  },
  "required": [],
  "type": "object"
}
k8s_deployments_list

List deployments with desired vs ready replica counts, rollout strategy and storage class. A deployment whose spec the cluster cannot satisfy - more replicas than fit, or a storageClassName that does not exist - reports no error of its own: the workload is simply not there, and the shortfall exists only as the gap between desired and ready.

{
  "properties": {
    "degraded_only": {
      "description": "degraded_only",
      "type": "boolean"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
list_db_grants

List which services are permitted to reach which datastores, with the role each uses and whether that grant is active, revoked or was never created. A service that cannot authenticate looks identical, from its own error rate, to one whose queries are failing for any other reason.

{
  "properties": {
    "broken_only": {
      "description": "broken_only",
      "type": "boolean"
    },
    "component": {
      "description": "component",
      "type": "string"
    },
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
get_runtime_stats

Heap use, garbage-collection pause time and collection frequency per service. A runtime spending its time collecting garbage is indistinguishable, from request latency alone, from one doing slow work.

{
  "properties": {
    "service": {
      "description": "service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
check_network_path

Whether a service can reach a target at the transport layer: open, refused or timing out. The distinction matters - a timeout looks like load and a refusal does not, so a refused path is a policy or firewall change rather than a capacity problem.

{
  "properties": {
    "blocked_only": {
      "description": "blocked_only",
      "type": "boolean"
    },
    "from_service": {
      "description": "from_service",
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}
write_runbook

Write a new page into the knowledge base - a runbook, a summary, a handover note. This adds a page; it cannot edit the company standards in `documents`, which are what your work is judged against.

{
  "properties": {
    "body": {
      "description": "body",
      "type": "string"
    },
    "title": {
      "description": "title",
      "type": "string"
    }
  },
  "required": [
    "title",
    "body"
  ],
  "type": "object"
}
list_authored_docs

List the pages written during this episode, with their titles.

{
  "properties": {},
  "required": [],
  "type": "object"
}
ws_list

List the files in the workspace with their sizes. This is a real filesystem: what you write here is what runs.

{
  "properties": {},
  "required": [],
  "type": "object"
}
ws_read

Read a workspace file.

{
  "properties": {
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
ws_write

Write a workspace file, creating it if needed. Replaces the whole file.

{
  "properties": {
    "content": {
      "description": "content",
      "type": "string"
    },
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path",
    "content"
  ],
  "type": "object"
}
ws_grep

Search the workspace for a literal string and return the matching lines with their file and line number. Implemented in the tool rather than shelled out, because this world has a filesystem and deliberately no shell.

{
  "properties": {
    "needle": {
      "description": "needle",
      "type": "string"
    },
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "needle"
  ],
  "type": "object"
}
ws_python

Run one workspace file with python3 and return its exit code, stdout and stderr. The whole workspace is materialised first, so imports between your files work. There is no shell: no pipes, no redirection, no arguments beyond the file, and nothing on PATH. Anything the program writes to the workspace directory is synced back.

{
  "properties": {
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
read_exercise

Read a code exercise: its specification, the current contents of the file, and the visible tests. There are also hidden tests, which this never returns - an implementation that satisfies only the visible ones is not finished.

{
  "properties": {
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
write_implementation

Replace the contents of an exercise file with your implementation. This only stores the code - it does not run it. Use run_exercise_tests to find out whether it works.

{
  "properties": {
    "content": {
      "description": "content",
      "type": "string"
    },
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path",
    "content"
  ],
  "type": "object"
}
run_exercise_tests

Execute the implementation written for a code exercise against its visible tests and report which passed. The hidden tests run at the same time; their result is recorded for grading and is not returned, so passing everything shown here does not mean you are done.

{
  "properties": {
    "path": {
      "description": "path",
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
submit_answer

Submit the answer to a reconciliation question. `sources` must list every system you actually consulted (e.g. pd_incidents, status_page_posts). `assumptions` is where you record any judgement you had to make - a week boundary, whether rollbacks count, which of two disagreeing numbers you trusted and why. An answer with no stated assumption on an ambiguous question is not a complete answer.

{
  "properties": {
    "answer": {
      "description": "answer",
      "type": "string"
    },
    "assumptions": {
      "description": "assumptions",
      "type": "string"
    },
    "question_id": {
      "description": "question_id",
      "type": "string"
    },
    "sources": {
      "description": "systems consulted",
      "items": {
        "type": "string"
      },
      "type": "array"
    }
  },
  "required": [
    "question_id",
    "answer"
  ],
  "type": "object"
}

Replay the work

Trajectories

AgentReference oracleDeterministic reference solver (release qualification)
ScoreReference trace
Tool calls13dob100-098-retire-debug-endpoint
Run cost
14 events
  1. 1
    Investigateget_ticketok
    {
      "key": "SEC-905"
    }

    {"assignee": "", "description": "The unauthenticated /internal/debug endpoint is still reachable in production.", "key": "SEC-905", "priority": "high", "service": "api-gateway", "status": "open", "ticket_id": 9128, "title": "Retire the exposed /internal/debug

  2. 2
    Investigatesearch_docsok
    {
      "query": "deployment"
    }

    {"count": 10, "rows": [{"author": "Priya Nair", "day": 268, "doc_id": 9601, "kind": "policy", "service": "", "title": "Deployment policy"}, {"author": "Diego Ramos", "day": 254, "doc_id": 9602, "kind": "policy", "service": "", "title": "Database migration poli

  3. 3
    Investigateget_documentok
    {
      "title": "Deployment policy"
    }

    {"author": "Priya Nair", "body": "# Deployment policy\n\nThis policy is binding for every NovaCart service. It is enforced partly by\ntooling and partly by review; deviations are treated as incidents.\n\n## Staging first, always\n\nEvery production deploy must

  4. 4
    Investigatelist_api_endpointsok
    {
      "service": "api-gateway"
    }

    {"count": 16, "rows": [{"path": "/internal/debug", "production_status": "active", "production_traffic_percent": 0, "repo_status": "active", "service": "api-gateway"}, {"path": "/internal/metrics", "production_status": "active", "production_traffic_percent": 0,

  5. 5
    Changeopen_pull_requestok
    {
      "body": "Removes the unauthenticated debug surface. SEC-905.",
      "changes": [
        {
          "change_type": "endpoint",
          "payload": {
            "path": "/internal/debug",
            "status": "retired"
          }
        }
      ],
      "service": "api-gateway",
      "ticket_key": "SEC-905",
      "title": "api-gateway: retire /internal/debug"
    }

    {"next": "run_ci(pr_number=9203) then merge_pull_request(pr_number=9203)", "ok": true, "pr_number": 9203, "service": "api-gateway", "status": "open"}

  6. 6
    Changerun_ciok
    {
      "pr_number": 9203
    }

    {"detail": "intermittent failure: test_upstream_timeout (rerun may pass)", "ok": true, "pr_number": 9203, "run_id": 13, "service": "api-gateway", "stages": [{"detail": "compiled and packaged", "stage": "build", "status": "passed"}, {"detail": "unit suite green

  7. 7
    Changerun_ciok
    {
      "pr_number": 9203
    }

    {"detail": "all stages passed", "ok": true, "pr_number": 9203, "run_id": 14, "service": "api-gateway", "stages": [{"detail": "compiled and packaged", "stage": "build", "status": "passed"}, {"detail": "unit suite green", "stage": "unit", "status": "passed"}, {"

  8. 8
    Changemerge_pull_requestok
    {
      "pr_number": 9203
    }

    {"merged_version": "v5.1.1", "next": "deploy_service(service=..., environment=\"staging\")", "ok": true, "pr_number": 9203, "service": "api-gateway"}

  9. 9
    Shipdeploy_serviceok
    {
      "environment": "staging",
      "service": "api-gateway"
    }

    {"applied": true, "canary_percent": 100, "deployment_id": 9273, "environment": "staging", "ok": true, "service": "api-gateway", "status": "succeeded", "version": "v5.1.1"}

  10. 10
    Shipdeploy_serviceok
    {
      "canary_percent": 25,
      "environment": "production",
      "service": "api-gateway"
    }

    {"applied": false, "canary_percent": 25, "deployment_id": 9274, "environment": "production", "next": "assess_canary(service=...) then promote_canary(service=...)", "ok": true, "service": "api-gateway", "status": "canary", "version": "v5.1.1"}

  11. 11
    Shipassess_canaryok
    {
      "service": "api-gateway"
    }

    {"deployment_id": 9274, "detail": "no new SLO breach detected in the canary population", "environment": "production", "next": "promote_canary", "ok": true, "service": "api-gateway", "verdict": "healthy", "version": "v5.1.1"}

  12. 12
    Shippromote_canaryok
    {
      "service": "api-gateway"
    }

    {"deployment_id": 9274, "environment": "production", "ok": true, "service": "api-gateway", "status": "succeeded", "version": "v5.1.1"}

  13. 13
    Observeupdate_ticketok
    {
      "key": "SEC-905",
      "status": "done"
    }

    {"assignee": "", "key": "SEC-905", "ok": true, "status": "done"}

  14. Agent

    Verifier verdict: passed · reward 1

Methodology

How the numbers are made

01

One executable world, not one hundred stubs

Every task runs against NovaCart, a mid-size e-commerce SaaS: 72 SQLite tables, 1,451 seeded rows, a 38-file monorepo with 417 commits, 30 knowledge-base documents, and 97 MCP tools spanning a first-party engineering stack (tickets, PRs, CI, deployments, canaries, migrations, feature flags, metrics, alerts, incidents, chat) plus deliberately disagreeing vendor-shaped surfaces — Jira, Linear, GitHub Issues, Prometheus, Sentry, PagerDuty, Confluence, spreadsheets — and Kubernetes.

02

Outcome-only tickets

Prompts are tickets: a symptom and a definition of done. Company policy — canary requirements, approval gates, incident procedure — lives in the world's knowledge base, not in the prompt. Reference trajectories run 4–34 tool calls (median 13, 1,318 total) across 19 task families, with difficulty measured at 38 expert / 31 hard / 29 medium / 2 easy.

03

Deterministic vcode verification

Each task ships an executable verifier that checks the final world state and the append-only audit log, with anti-forgery table pins. Across the release that is 1,420 deterministic checks (min 8 / median 13 / max 24 per task) scored 0.6 correctness / 0.3 deployment / 0.1 quality, exactly as the shipped verifier scripts weight them. No LLM judge, no network, no clock in the reward path.

04

Qualification evidence

The release gate executed 371 runs: 100 oracle replays all passing with reward 1.0, 100 byte-identical deterministic replays, and 171 adversarial control executions (naive, pristine-world, shortcut, wrong-source) with zero false accepts. Full per-task evidence ships in reports/qualification.json.

05

What the leaderboard shows

The reference row is a solvability proof, not a model score. A pre-release deepseek-v4-pro run over the full 187-task source world exists, but it covers only 95 of the 100 released tasks, so it is not shown as a leaderboard row; model rows appear once a pinned run over the released dataset is published.

Run it yourself

The benchmark is public. The world is executable.

Download the dataset, replay the oracle trajectories, run the standalone verifiers, and submit the first pinned model run.