Can an agent run the incident, ship the fix, and prove it landed?
DevOpsBench-100 measures complete operations work over one executable company: root-cause the symptom, change the code, ride the canary, and close the incident — through 97 real MCP tools, graded only by executable checks on the world left behind.
The release gate executed 371 qualification runs: 100/100 oracle replays at reward 1.0, 100 byte-identical deterministic replays, and 171 adversarial control executions with zero false accepts. No model has run the released dataset yet — the leaderboard says so.
Inspect the verifier contract →Measured performance
Leaderboard
The requested outcome is true in the final world state: the right diagnosis, code change, migration, flag, or record — with world invariants intact.
The change actually landed the way operations requires: CI green, merged, deployed through canary where policy demands it, rollbacks clean.
Professional closure: tickets updated, incidents resolved, status published, evidence recorded, no collateral writes.
100 distinct tasks
Task browser
RCA Analytics Egress Blocked
This is an investigation, not a change. Do NOT modify production state: no pull requests, deploys, flag toggles, migrations or resolutions. Investigate with the read tools and report what you find with submit_diagnosis(...). OPS-163 — Root cause: ana
- Seeded files
- 0
- Reference path
- 9 MCP calls
Complete task prompt
13 graded criteria summarizedThis is an investigation, not a change. Do NOT modify production state: no pull requests, deploys, flag toggles, migrations or resolutions. Investigate with the read tools and report what you find with submit_diagnosis(...).
OPS-163 — Root cause: analytics connections are refused, not slow
Get to the bottom of analytics-connection-refused: not just which service, but the mechanism and the exact setting responsible. The service's own code is not necessarily where this lives — confirm it against the state of the system rather than inferring it from the symptom alone. Submit your finding under scope 'analytics-connection-refused' with the responsible service, the fault type, the offending key, and your evidence.Rubric summary
- correctness: diagnosis_submitted
- correctness: service_localized
- correctness: fault_type_correct
- correctness: offending_key_correct
- correctness: scoped_change_no_fabricated_data
- correctness: world_invariants_intact
- deployment: investigation_was_read_only
- quality: evidence_recorded
- quality: efficient_investigation
- quality: ticket_closed
- quality: closed_after_the_work
- quality: no_new_hardcoded_secrets
- quality: no_unproductive_ci_loops
Seeded evidence
Asset room
DevOpsBench keeps its evidence inside the executable world: the 30 knowledge-base documents, the 38-file monorepo with 417 commits, tickets, metrics, and logs live in the world database and are read through the 97 MCP tools.
Runnable world
Environment and tool contract
list_services
List all services with team, tier, kind, on-call engineer, repo HEAD version and deployed versions.
{
"properties": {
"team": {
"description": "team",
"type": "string"
},
"tier": {
"description": "tier",
"type": "integer"
}
},
"required": [],
"type": "object"
}get_service
Full detail for one service: metadata, deployed config, modules, endpoints, dependencies, current metrics.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}list_infra
List infrastructure components of the application stack (databases, caches, queues, object stores, CDN).
{
"properties": {},
"required": [],
"type": "object"
}list_files
List monorepo files, optionally filtered by service or path substring.
{
"properties": {
"path_contains": {
"description": "path_contains",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}read_file
Read a monorepo source file. Returns its full current content.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}search_code
Search monorepo file contents for a substring; returns matching files with matching line numbers.
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
},
"query": {
"description": "query",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"query"
],
"type": "object"
}list_commits
Browse monorepo commit history (most recent first).
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
},
"path": {
"description": "only commits touching this file path",
"type": "string"
},
"query": {
"description": "substring of the commit message",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}search_docs
Search the engineering knowledge base (runbooks, policies, design docs, ADRs, postmortems, API specs).
{
"properties": {
"kind": {
"description": "runbook|policy|design_doc|adr|postmortem|api_spec|onboarding",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"query": {
"description": "query",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_document
Read one knowledge-base document in full by doc_id (or exact title).
{
"properties": {
"doc_id": {
"description": "doc_id",
"type": "integer"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [],
"type": "object"
}list_tickets
List issue-tracker tickets, optionally filtered by status, service, or type.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
},
"ticket_type": {
"description": "ticket_type",
"type": "string"
}
},
"required": [],
"type": "object"
}get_ticket
Fetch one ticket by key (e.g. ENG-2101).
{
"properties": {
"key": {
"description": "key",
"type": "string"
}
},
"required": [
"key"
],
"type": "object"
}list_pull_requests
List pull requests, optionally filtered by service or status.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}get_pull_request
Fetch a PR with its structured changes and CI history.
{
"properties": {
"pr_number": {
"description": "pr_number",
"type": "integer"
}
},
"required": [
"pr_number"
],
"type": "object"
}list_ci_runs
List CI runs (most recent first).
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
},
"pr_number": {
"description": "pr_number",
"type": "integer"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_ci_run
Fetch one CI run with its per-stage results (build, unit, integration, regression).
{
"properties": {
"run_id": {
"description": "run_id",
"type": "integer"
}
},
"required": [
"run_id"
],
"type": "object"
}list_deployments
List deployments (most recent first).
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_migrations
List database migrations and whether they are applied per environment.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}query_metrics
Read current production service metrics (recomputed continuously from live traffic).
{
"properties": {
"metric": {
"description": "metric",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_traffic_stats
Traffic-generator statistics: request rate per route with the current error rate and p99 of the owning service.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_slo_status
List SLOs with current values and whether each is breaching.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_alerts
List alarms, optionally filtered by status (firing|acknowledged|resolved) or service.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}list_error_events
Error-tracking issues (Sentry-style): grouped exceptions with culprit and event counts.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "unresolved|resolved",
"type": "string"
}
},
"required": [],
"type": "object"
}search_logs
Search application logs by substring, service, or level.
{
"properties": {
"level": {
"description": "level",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"query": {
"description": "query",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_feature_flags
List feature flags with per-environment state.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_packages
List package dependencies: version at repo HEAD and version deployed in production.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_vulnerabilities
List security-scanner findings.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}list_api_endpoints
List API endpoints with repo status, production status, and production traffic share.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_tests
List the test catalog.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}list_incidents
List incidents.
{
"properties": {
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}get_status_page
Read the public system-status page.
{
"properties": {
"limit": {
"description": "limit",
"type": "integer"
}
},
"required": [],
"type": "object"
}list_messages
Read chat messages.
{
"properties": {
"channel": {
"description": "channel",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
}
},
"required": [],
"type": "object"
}create_ticket
Create a ticket. Returns the generated key.
{
"properties": {
"description": {
"description": "description",
"type": "string"
},
"priority": {
"description": "priority",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
},
"ticket_type": {
"description": "task|bug|feature|security|incident|postmortem",
"enum": [
"task",
"bug",
"feature",
"security",
"incident",
"postmortem"
],
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"title"
],
"type": "object"
}update_ticket
Update a ticket's status and/or assignee.
{
"properties": {
"assignee": {
"description": "assignee",
"type": "string"
},
"key": {
"description": "key",
"type": "string"
},
"status": {
"description": "open|in_progress|in_review|done",
"enum": [
"open",
"in_progress",
"in_review",
"done"
],
"type": "string"
}
},
"required": [
"key"
],
"type": "object"
}open_pull_request
Open a pull request carrying structured changes. change_type is one of: config {key,value}; dependency {package,version}; endpoint {path,status: active|deprecated|retired}; module {name}; flag {key,description}; flag_cleanup {key}; test_fix {test_name, action: fix|quarantine}; migration {name}; code_edit {path, find, replace}. Changes apply at merge; deploys carry them to an environment.
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"changes": {
"description": "list of {change_type, payload}",
"items": {
"properties": {
"change_type": {
"type": "string"
},
"payload": {
"type": "object"
}
},
"required": [
"change_type",
"payload"
],
"type": "object"
},
"type": "array"
},
"service": {
"description": "service",
"type": "string"
},
"ticket_key": {
"description": "ticket_key",
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"service",
"title",
"changes"
],
"type": "object"
}run_ci
Run the CI pipeline for an open PR (pr_number) or a service's main branch (service). Stages run in order: build, unit, integration, regression. The tool succeeds even when the pipeline fails - inspect the returned status and stages.
{
"properties": {
"pr_number": {
"description": "pr_number",
"type": "integer"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}merge_pull_request
Merge an open PR. Blocked unless its latest CI run passed. Applies the PR's changes to repo HEAD (including code edits) and cuts a new deployable version.
{
"properties": {
"pr_number": {
"description": "pr_number",
"type": "integer"
}
},
"required": [
"pr_number"
],
"type": "object"
}apply_migration
Apply a database migration to an environment. Migrations are forward-only and must be applied before the code version that requires them is deployed there.
{
"properties": {
"environment": {
"description": "staging|production",
"enum": [
"staging",
"production"
],
"type": "string"
},
"name": {
"description": "name",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service",
"name",
"environment"
],
"type": "object"
}deploy_service
Deploy a merged version to staging or production. canary_percent<100 stages a canary whose state only takes effect at promote_canary. Policy: production is staging-first; tier-1 services canary at <=25% then promote. A version whose migration is not applied is rejected.
{
"properties": {
"canary_percent": {
"description": "canary_percent",
"type": "integer"
},
"environment": {
"description": "environment",
"enum": [
"staging",
"production"
],
"type": "string"
},
"service": {
"description": "service",
"type": "string"
},
"version": {
"description": "version",
"type": "string"
}
},
"required": [
"service",
"environment"
],
"type": "object"
}assess_canary
Evaluate the pending canary for a service: reports whether the canary version would breach any SLO or trip an alarm if promoted. Run this before promote_canary.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}promote_canary
Promote the pending canary to 100%; its state takes effect.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}rollback_deployment
Emergency rollback to the previous successful deployment. Exempt from staging-first.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [
"service"
],
"type": "object"
}set_feature_flag
Toggle a feature flag or change its rollout percent in one environment. Runtime operation: takes effect immediately, no deploy needed.
{
"properties": {
"enabled": {
"description": "enabled",
"type": "boolean"
},
"environment": {
"description": "environment",
"enum": [
"staging",
"production"
],
"type": "string"
},
"key": {
"description": "key",
"type": "string"
},
"rollout_percent": {
"description": "rollout_percent",
"type": "integer"
}
},
"required": [
"key",
"environment"
],
"type": "object"
}shift_endpoint_traffic
Set the production traffic percent served by an endpoint (gateway runtime weight, no deploy needed). Policy: shift in stages of at most 50 points per step.
{
"properties": {
"path": {
"description": "path",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
},
"traffic_percent": {
"description": "traffic_percent",
"type": "integer"
}
},
"required": [
"service",
"path",
"traffic_percent"
],
"type": "object"
}acknowledge_alert
Acknowledge a firing alarm.
{
"properties": {
"alert_id": {
"description": "alert_id",
"type": "integer"
}
},
"required": [
"alert_id"
],
"type": "object"
}resolve_alert
Resolve an alarm. Refused while the underlying metric still breaches its SLO.
{
"properties": {
"alert_id": {
"description": "alert_id",
"type": "integer"
}
},
"required": [
"alert_id"
],
"type": "object"
}resolve_error_event
Mark an error-tracking issue resolved. Refused while the owning service still breaches an SLO.
{
"properties": {
"fingerprint": {
"description": "fingerprint",
"type": "string"
}
},
"required": [
"fingerprint"
],
"type": "object"
}create_incident
Declare an incident.
{
"properties": {
"service": {
"description": "service",
"type": "string"
},
"severity": {
"description": "sev1|sev2|sev3",
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"title",
"service",
"severity"
],
"type": "object"
}update_incident
Update an incident's status (open|mitigated|resolved) and/or commander.
{
"properties": {
"commander": {
"description": "commander",
"type": "string"
},
"incident_id": {
"description": "incident_id",
"type": "integer"
},
"status": {
"description": "status",
"enum": [
"open",
"mitigated",
"resolved"
],
"type": "string"
}
},
"required": [
"incident_id"
],
"type": "object"
}publish_status_update
Publish an update to the public system-status page (state: investigating|identified|monitoring|resolved).
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"state": {
"description": "state",
"enum": [
"investigating",
"identified",
"monitoring",
"resolved"
],
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"state",
"title"
],
"type": "object"
}submit_diagnosis
Submit a diagnostic finding for an investigation. `scope` is what you were asked to investigate (a service name or alarm id). Set fault_detected=false with fault_type='none' when the scope is healthy. fault_type is one of: misconfig, missing_retry, missing_timeout, resource_exhaustion, unbounded_prefetch, cache_disabled, n_plus_one_query, cdn_bypass, bad_release, feature_flag_regression, node_unhealthy, unclassified, none. Use node_unhealthy when the cause is the node a service runs on rather than the service's own code or config. Use unclassified when you have established that a fault is present but were not asked for, and have not determined, the mechanism - reporting a breach you can evidence is better than reporting health you cannot. offending_key is the specific config key, flag key, node name or version responsible.
{
"properties": {
"evidence": {
"description": "what you observed that supports this finding",
"type": "string"
},
"fault_detected": {
"description": "true if the scope is faulting, false if it is healthy",
"type": "boolean"
},
"fault_type": {
"description": "the mechanism, or 'unclassified' when a fault is evidenced but its mechanism was not asked for, or 'none' when healthy",
"enum": [
"misconfig",
"missing_retry",
"missing_timeout",
"resource_exhaustion",
"unbounded_prefetch",
"cache_disabled",
"n_plus_one_query",
"cdn_bypass",
"bad_release",
"feature_flag_regression",
"node_unhealthy",
"unclassified",
"none"
],
"type": "string"
},
"offending_key": {
"description": "config key, flag key or version at fault",
"type": "string"
},
"scope": {
"description": "the service or alarm id you were asked to investigate",
"type": "string"
},
"service": {
"description": "the service responsible (localization)",
"type": "string"
}
},
"required": [
"scope",
"fault_detected"
],
"type": "object"
}post_message
Post a message to a chat channel.
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"channel": {
"description": "channel",
"type": "string"
}
},
"required": [
"channel",
"body"
],
"type": "object"
}jira_search
Search Jira issues. Jira status is a per-project workflow, not open/closed: a resolved issue has status='Done' AND a resolution set. Filter by project, status, issue_type, component or priority.
{
"properties": {
"component": {
"description": "component",
"type": "string"
},
"issue_type": {
"description": "issue_type",
"type": "string"
},
"limit": {
"description": "limit",
"type": "integer"
},
"project": {
"description": "project",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}jira_get_issue
Fetch one Jira issue by key, including any links to issues in other trackers.
{
"properties": {
"key": {
"description": "key",
"type": "string"
}
},
"required": [
"key"
],
"type": "object"
}linear_list_issues
List Linear issues. Linear priority is numeric: 0=none, 1=urgent, 2=high, 3=normal, 4=low - it does not map cleanly onto Jira priority names.
{
"properties": {
"state": {
"description": "state",
"type": "string"
},
"team": {
"description": "team",
"type": "string"
}
},
"required": [],
"type": "object"
}github_list_issues
List GitHub issues. GitHub has only state=open|closed; severity lives in labels if anywhere.
{
"properties": {
"label": {
"description": "label",
"type": "string"
},
"repo": {
"description": "repo",
"type": "string"
},
"state": {
"description": "state",
"type": "string"
}
},
"required": [],
"type": "object"
}list_issue_links
List known cross-tracker links (duplicates/relates/implements). This is the only place the trackers are reconciled; neither tracker knows about it.
{
"properties": {
"source": {
"description": "source",
"type": "string"
}
},
"required": [],
"type": "object"
}query_prometheus
Query a Prometheus series by metric and label selectors. Note the label spelling is Prometheus's own (e.g. checkout_service), and counter resets are flagged: a rate() over a reset under-reports.
{
"properties": {
"day_from": {
"description": "day_from",
"type": "integer"
},
"day_to": {
"description": "day_to",
"type": "integer"
},
"label_env": {
"description": "label_env",
"type": "string"
},
"label_service": {
"description": "label_service",
"type": "string"
},
"metric": {
"description": "metric",
"type": "string"
}
},
"required": [
"metric"
],
"type": "object"
}list_prometheus_label_values
List the values a Prometheus label actually takes. Use this when you are not sure how a service is spelled in metrics.
{
"properties": {
"label": {
"description": "label",
"type": "string"
}
},
"required": [],
"type": "object"
}sentry_search_issues
Search Sentry issues (grouped exceptions). Event counts are SAMPLED at the project's sample_rate - see sentry_list_projects - so they are a fraction of the true volume and are not comparable to Prometheus counters.
{
"properties": {
"project_slug": {
"description": "project_slug",
"type": "string"
},
"status": {
"description": "status",
"type": "string"
}
},
"required": [],
"type": "object"
}sentry_list_projects
List Sentry projects with their event sample rates.
{
"properties": {},
"required": [],
"type": "object"
}pd_list_incidents
List PagerDuty incidents in a day range. urgency (high|low) and priority (P1..P4) are separate vocabularies; neither records whether customers saw it.
{
"properties": {
"since_day": {
"description": "since_day",
"type": "integer"
},
"status": {
"description": "status",
"type": "string"
},
"until_day": {
"description": "until_day",
"type": "integer"
},
"urgency": {
"description": "urgency",
"type": "string"
}
},
"required": [],
"type": "object"
}pd_list_services
List PagerDuty technical services and escalation policies.
{
"properties": {},
"required": [],
"type": "object"
}pd_list_oncalls
Who is on call, by day and escalation policy.
{
"properties": {
"day": {
"description": "day",
"type": "integer"
},
"escalation_policy": {
"description": "escalation_policy",
"type": "string"
}
},
"required": [],
"type": "object"
}pd_list_change_events
Change events recorded against a PagerDuty service. These exist only where someone wired the integration.
{
"properties": {
"pd_service_id": {
"description": "pd_service_id",
"type": "string"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}list_status_page_posts
Public status-page posts. This is the ONLY system that records customer impact; incidents do not carry it. The status page also lags internal state.
{
"properties": {
"impact": {
"description": "impact",
"type": "string"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}confluence_search
Search the Confluence wiki. Pages carry a last_updated_day; some are stale.
{
"properties": {
"query": {
"description": "query",
"type": "string"
},
"space": {
"description": "space",
"type": "string"
}
},
"required": [],
"type": "object"
}confluence_get_page
Read one Confluence page in full.
{
"properties": {
"page_id": {
"description": "page_id",
"type": "integer"
}
},
"required": [
"page_id"
],
"type": "object"
}read_owner_spreadsheet
Read the hand-maintained service-owner spreadsheet. Note last_reviewed_day: rows drift as teams reorganise, and the sheet uses its own week convention.
{
"properties": {},
"required": [],
"type": "object"
}query_local_deploy_log
Query a team's local deploy log (a SQLite file kept because the central one is slow). Environment strings are free text and include 'nonprod-*' spellings; rollbacks are flagged separately.
{
"properties": {
"environment": {
"description": "environment",
"type": "string"
},
"include_rollbacks": {
"description": "include_rollbacks",
"type": "boolean"
},
"service": {
"description": "service",
"type": "string"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}resolve_service_alias
Resolve any spelling of a service to its canonical name, and list every spelling it has across systems. Use this before comparing data from two tools.
{
"properties": {
"name": {
"description": "name",
"type": "string"
}
},
"required": [
"name"
],
"type": "object"
}list_service_aliases
Every known service-name spelling, by system.
{
"properties": {},
"required": [],
"type": "object"
}list_approval_policy
Which actions require a human sign-off before you may perform them, and why. The trigger is irreversibility, not difficulty.
{
"properties": {},
"required": [],
"type": "object"
}request_approval
Ask the accountable human to approve an irreversible action, and get their answer. State the action exactly as the approval policy names it, the target, and why it is necessary. A vague or unjustified request is refused - the approver is not a rubber stamp.
{
"properties": {
"action": {
"description": "action",
"type": "string"
},
"reason": {
"description": "reason",
"type": "string"
},
"target": {
"description": "target",
"type": "string"
}
},
"required": [
"action",
"reason"
],
"type": "object"
}report_blocked
Declare that you cannot complete the task and stop, rather than reporting a success you did not achieve. State what blocked you and what you would need. Stopping blocked is a legitimate outcome; claiming a completion you did not reach is not.
{
"properties": {
"needed": {
"description": "needed",
"type": "string"
},
"reason": {
"description": "reason",
"type": "string"
}
},
"required": [
"reason"
],
"type": "object"
}list_alert_rules
List alerting rules. A rule's service_label may name a service that no longer exists - monitors outlive what they watch.
{
"properties": {
"routes_to": {
"description": "routes_to",
"type": "string"
}
},
"required": [],
"type": "object"
}list_alert_firings
Individual alert firings. `silenced` means it never notified, `inhibited_by` names a rule that suppressed it, and `paged_incident` is NULL when it never reached a human. One failure does not produce one firing, one page, or one incident - the ratios are configuration artefacts.
{
"properties": {
"rule_id": {
"description": "rule_id",
"type": "integer"
},
"since_day": {
"description": "since_day",
"type": "integer"
}
},
"required": [],
"type": "object"
}list_alert_silences
Active and expired alert silences. A silence that outlived its reason is why an alert can be firing and invisible at the same time.
{
"properties": {},
"required": [],
"type": "object"
}list_remediation_proposals
Read the remediation proposals people have put forward for an incident. Exactly one is the right call; the others are plausible suggestions that mask the symptom, target the wrong component, or change behaviour.
{
"properties": {
"incident_ref": {
"description": "incident_ref",
"type": "string"
}
},
"required": [
"incident_ref"
],
"type": "object"
}jira_transition_issue
Transition a Jira issue. Jira status is a per-project workflow, so moving an issue to 'Done' does NOT by itself mean it was fixed - a completed issue also carries a resolution (e.g. 'Fixed'). Set both.
{
"properties": {
"key": {
"description": "key",
"type": "string"
},
"resolution": {
"description": "resolution",
"type": "string"
},
"status": {
"description": "status",
"enum": [
"Backlog",
"In Progress",
"In Review",
"Blocked",
"Done"
],
"type": "string"
}
},
"required": [
"key",
"status"
],
"type": "object"
}k8s_events_list
List Kubernetes events (OOMKilled, CrashLoopBackOff, ...). The kubelet records kernel-level kills that an application error tracker never sees, because the process dies before its SDK can flush.
{
"properties": {
"namespace": {
"description": "namespace",
"type": "string"
},
"pod": {
"description": "pod",
"type": "string"
},
"reason": {
"description": "reason",
"type": "string"
}
},
"required": [],
"type": "object"
}k8s_pods_list
List pods with phase, restart count, memory limit/usage and the running image tag. The image tag is the only ground truth for what is actually deployed - release records in other systems drift from it, especially after a rollback.
{
"properties": {
"namespace": {
"description": "namespace",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}k8s_nodes_list
List cluster nodes with their Ready status, active condition, CPU and disk utilisation, labels and kernel version. A service whose node has DiskPressure, a kernel deadlock, or no node matching its selector looks - from the service's own metrics and logs - exactly like a slow or broken service. This is the only place that difference is visible.
{
"properties": {
"node": {
"description": "node",
"type": "string"
},
"unhealthy_only": {
"description": "unhealthy_only",
"type": "boolean"
}
},
"required": [],
"type": "object"
}k8s_deployments_list
List deployments with desired vs ready replica counts, rollout strategy and storage class. A deployment whose spec the cluster cannot satisfy - more replicas than fit, or a storageClassName that does not exist - reports no error of its own: the workload is simply not there, and the shortfall exists only as the gap between desired and ready.
{
"properties": {
"degraded_only": {
"description": "degraded_only",
"type": "boolean"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}list_db_grants
List which services are permitted to reach which datastores, with the role each uses and whether that grant is active, revoked or was never created. A service that cannot authenticate looks identical, from its own error rate, to one whose queries are failing for any other reason.
{
"properties": {
"broken_only": {
"description": "broken_only",
"type": "boolean"
},
"component": {
"description": "component",
"type": "string"
},
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}get_runtime_stats
Heap use, garbage-collection pause time and collection frequency per service. A runtime spending its time collecting garbage is indistinguishable, from request latency alone, from one doing slow work.
{
"properties": {
"service": {
"description": "service",
"type": "string"
}
},
"required": [],
"type": "object"
}check_network_path
Whether a service can reach a target at the transport layer: open, refused or timing out. The distinction matters - a timeout looks like load and a refusal does not, so a refused path is a policy or firewall change rather than a capacity problem.
{
"properties": {
"blocked_only": {
"description": "blocked_only",
"type": "boolean"
},
"from_service": {
"description": "from_service",
"type": "string"
}
},
"required": [],
"type": "object"
}write_runbook
Write a new page into the knowledge base - a runbook, a summary, a handover note. This adds a page; it cannot edit the company standards in `documents`, which are what your work is judged against.
{
"properties": {
"body": {
"description": "body",
"type": "string"
},
"title": {
"description": "title",
"type": "string"
}
},
"required": [
"title",
"body"
],
"type": "object"
}list_authored_docs
List the pages written during this episode, with their titles.
{
"properties": {},
"required": [],
"type": "object"
}ws_list
List the files in the workspace with their sizes. This is a real filesystem: what you write here is what runs.
{
"properties": {},
"required": [],
"type": "object"
}ws_read
Read a workspace file.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}ws_write
Write a workspace file, creating it if needed. Replaces the whole file.
{
"properties": {
"content": {
"description": "content",
"type": "string"
},
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path",
"content"
],
"type": "object"
}ws_grep
Search the workspace for a literal string and return the matching lines with their file and line number. Implemented in the tool rather than shelled out, because this world has a filesystem and deliberately no shell.
{
"properties": {
"needle": {
"description": "needle",
"type": "string"
},
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"needle"
],
"type": "object"
}ws_python
Run one workspace file with python3 and return its exit code, stdout and stderr. The whole workspace is materialised first, so imports between your files work. There is no shell: no pipes, no redirection, no arguments beyond the file, and nothing on PATH. Anything the program writes to the workspace directory is synced back.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}read_exercise
Read a code exercise: its specification, the current contents of the file, and the visible tests. There are also hidden tests, which this never returns - an implementation that satisfies only the visible ones is not finished.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}write_implementation
Replace the contents of an exercise file with your implementation. This only stores the code - it does not run it. Use run_exercise_tests to find out whether it works.
{
"properties": {
"content": {
"description": "content",
"type": "string"
},
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path",
"content"
],
"type": "object"
}run_exercise_tests
Execute the implementation written for a code exercise against its visible tests and report which passed. The hidden tests run at the same time; their result is recorded for grading and is not returned, so passing everything shown here does not mean you are done.
{
"properties": {
"path": {
"description": "path",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}submit_answer
Submit the answer to a reconciliation question. `sources` must list every system you actually consulted (e.g. pd_incidents, status_page_posts). `assumptions` is where you record any judgement you had to make - a week boundary, whether rollbacks count, which of two disagreeing numbers you trusted and why. An answer with no stated assumption on an ambiguous question is not a complete answer.
{
"properties": {
"answer": {
"description": "answer",
"type": "string"
},
"assumptions": {
"description": "assumptions",
"type": "string"
},
"question_id": {
"description": "question_id",
"type": "string"
},
"sources": {
"description": "systems consulted",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"question_id",
"answer"
],
"type": "object"
}Replay the work
Trajectories
- 1Investigate
get_ticketok{ "key": "SEC-905" }{"assignee": "", "description": "The unauthenticated /internal/debug endpoint is still reachable in production.", "key": "SEC-905", "priority": "high", "service": "api-gateway", "status": "open", "ticket_id": 9128, "title": "Retire the exposed /internal/debug
- 2Investigate
search_docsok{ "query": "deployment" }{"count": 10, "rows": [{"author": "Priya Nair", "day": 268, "doc_id": 9601, "kind": "policy", "service": "", "title": "Deployment policy"}, {"author": "Diego Ramos", "day": 254, "doc_id": 9602, "kind": "policy", "service": "", "title": "Database migration poli
- 3Investigate
get_documentok{ "title": "Deployment policy" }{"author": "Priya Nair", "body": "# Deployment policy\n\nThis policy is binding for every NovaCart service. It is enforced partly by\ntooling and partly by review; deviations are treated as incidents.\n\n## Staging first, always\n\nEvery production deploy must
- 4Investigate
list_api_endpointsok{ "service": "api-gateway" }{"count": 16, "rows": [{"path": "/internal/debug", "production_status": "active", "production_traffic_percent": 0, "repo_status": "active", "service": "api-gateway"}, {"path": "/internal/metrics", "production_status": "active", "production_traffic_percent": 0,
- 5Change
open_pull_requestok{ "body": "Removes the unauthenticated debug surface. SEC-905.", "changes": [ { "change_type": "endpoint", "payload": { "path": "/internal/debug", "status": "retired" } } ], "service": "api-gateway", "ticket_key": "SEC-905", "title": "api-gateway: retire /internal/debug" }{"next": "run_ci(pr_number=9203) then merge_pull_request(pr_number=9203)", "ok": true, "pr_number": 9203, "service": "api-gateway", "status": "open"}
- 6Change
run_ciok{ "pr_number": 9203 }{"detail": "intermittent failure: test_upstream_timeout (rerun may pass)", "ok": true, "pr_number": 9203, "run_id": 13, "service": "api-gateway", "stages": [{"detail": "compiled and packaged", "stage": "build", "status": "passed"}, {"detail": "unit suite green
- 7Change
run_ciok{ "pr_number": 9203 }{"detail": "all stages passed", "ok": true, "pr_number": 9203, "run_id": 14, "service": "api-gateway", "stages": [{"detail": "compiled and packaged", "stage": "build", "status": "passed"}, {"detail": "unit suite green", "stage": "unit", "status": "passed"}, {"
- 8Change
merge_pull_requestok{ "pr_number": 9203 }{"merged_version": "v5.1.1", "next": "deploy_service(service=..., environment=\"staging\")", "ok": true, "pr_number": 9203, "service": "api-gateway"}
- 9Ship
deploy_serviceok{ "environment": "staging", "service": "api-gateway" }{"applied": true, "canary_percent": 100, "deployment_id": 9273, "environment": "staging", "ok": true, "service": "api-gateway", "status": "succeeded", "version": "v5.1.1"}
- 10Ship
deploy_serviceok{ "canary_percent": 25, "environment": "production", "service": "api-gateway" }{"applied": false, "canary_percent": 25, "deployment_id": 9274, "environment": "production", "next": "assess_canary(service=...) then promote_canary(service=...)", "ok": true, "service": "api-gateway", "status": "canary", "version": "v5.1.1"}
- 11Ship
assess_canaryok{ "service": "api-gateway" }{"deployment_id": 9274, "detail": "no new SLO breach detected in the canary population", "environment": "production", "next": "promote_canary", "ok": true, "service": "api-gateway", "verdict": "healthy", "version": "v5.1.1"}
- 12Ship
promote_canaryok{ "service": "api-gateway" }{"deployment_id": 9274, "environment": "production", "ok": true, "service": "api-gateway", "status": "succeeded", "version": "v5.1.1"}
- 13Observe
update_ticketok{ "key": "SEC-905", "status": "done" }{"assignee": "", "key": "SEC-905", "ok": true, "status": "done"}
Methodology
How the numbers are made
One executable world, not one hundred stubs
Every task runs against NovaCart, a mid-size e-commerce SaaS: 72 SQLite tables, 1,451 seeded rows, a 38-file monorepo with 417 commits, 30 knowledge-base documents, and 97 MCP tools spanning a first-party engineering stack (tickets, PRs, CI, deployments, canaries, migrations, feature flags, metrics, alerts, incidents, chat) plus deliberately disagreeing vendor-shaped surfaces — Jira, Linear, GitHub Issues, Prometheus, Sentry, PagerDuty, Confluence, spreadsheets — and Kubernetes.
Outcome-only tickets
Prompts are tickets: a symptom and a definition of done. Company policy — canary requirements, approval gates, incident procedure — lives in the world's knowledge base, not in the prompt. Reference trajectories run 4–34 tool calls (median 13, 1,318 total) across 19 task families, with difficulty measured at 38 expert / 31 hard / 29 medium / 2 easy.
Deterministic vcode verification
Each task ships an executable verifier that checks the final world state and the append-only audit log, with anti-forgery table pins. Across the release that is 1,420 deterministic checks (min 8 / median 13 / max 24 per task) scored 0.6 correctness / 0.3 deployment / 0.1 quality, exactly as the shipped verifier scripts weight them. No LLM judge, no network, no clock in the reward path.
Qualification evidence
The release gate executed 371 runs: 100 oracle replays all passing with reward 1.0, 100 byte-identical deterministic replays, and 171 adversarial control executions (naive, pristine-world, shortcut, wrong-source) with zero false accepts. Full per-task evidence ships in reports/qualification.json.
What the leaderboard shows
The reference row is a solvability proof, not a model score. A pre-release deepseek-v4-pro run over the full 187-task source world exists, but it covers only 95 of the 100 released tasks, so it is not shown as a leaderboard row; model rows appear once a pinned run over the released dataset is published.
Run it yourself
The benchmark is public. The world is executable.
Download the dataset, replay the oracle trajectories, run the standalone verifiers, and submit the first pinned model run.