SRE World
206 incident tasks cut from real recordings of a broken online shop. The agent gets a complaint and the shop's records, and must say what broke and when.
The task
“users can't finish buying things”, reported at 11:02 AM. The agent starts at 11:52 AM; nothing recorded after that moment exists for it.
It hands in two files: answer.json, naming each cause and when it started and ended, and report.md, the incident report with its evidence. Here the answer is deploy:quote:2.2.0-build.5932, 2:52 to 3:07 PM UTC: a shipping-quote release that made every 10-item order fail at checkout until it was rolled back.
The files
sre-eb3da7dd72de1bf3/
instruction.md the complaint, the clock, the tools, the answer format
task.toml 1-hour limit, 2 CPUs, 2 GB
environment/ the agent's shell and the on-call desk (an MCP server)
tests/ verify.py and the answer key
solution/ the reference answer, which scores 1.0
worlds/variant-s6/world.db
the recording all of a world's tasks share: traces, metrics,
logs, alerts, releases, code historyEach world is a recording of the OpenTelemetry Astronomy Shop running with simulated shoppers while we ship real buggy releases, stop its cache or flood it. Each task cuts the recording at its own start time. The agent reads it through 20 read-only tools (search logs and traces, query metrics, list releases, read commits) or through the shop's own Grafana.
What is tested
Finding the cause, not describing symptoms. A script scores each answer: the exact release (commit or version) or outage counts for 70%, and its start and end times for 30%. The right service or kind of fault earns partial credit, and every extra cause costs points. There is no AI judge.
Each task turns some of these levers:
- Ticket detail: specific, an area, or vague
- When the agent starts: 15 minutes to a day after the complaint
- How the time is given: exact, a range, or just the day
- Several causes at once, and silent bugs that log no error
- Services rewritten in other languages, or the whole shop renamed
- Grafana only, without release markers
Results
On 8 hard tasks (six an inexpensive model failed in screening, two with Grafana only), one or two tries per model:
| Model | Average score | Named every real cause |
|---|---|---|
| Claude Opus 5.5 | 0.74 | 57% |
| Claude Fable 5.1 | 0.69 | 71% |
| GPT-6 Astra | 0.54 | 83% |
| GPT-6.1 Sol | 0.45 | 77% |
- One task beat every model: a vague “staff are reporting business issues” with a silent bug, a loud bug and a later outage. The best score was 0.50.
- An outage that began after the complaint, but before the agent started, fooled three of the four models.
- Claude Opus 5.5 never blamed an extra incident but missed more causes. The GPT-6 models found more causes and also blamed unrelated ones, the same way on every try.
- One miss is our fault: a ticket says “payments” where the answer key counts every checkout problem. Every model skipped the same cause, and we are fixing the wording.