Skip to SRE World
Blobfish AI
BenchmarksTalk to usOpen app
Blobfish ResearchResearch previewBuilt on ORCA-bench

SRE World

A factory that turns a real online shop into graded “what broke?” tests for AI agents. Every incident really happens: buggy code ships, databases stop, shoppers pile in, and everything the shop records becomes the evidence.

Inspect the tasks ↓Watch an agent investigate
How it worksWhat makes it harderStatus and limits
47 minof recorded shop
2 of 3incidents showed up
45generated tasks
20investigation tools
19commits of readable history

How the factory works

ORCA-bench asks an agent what broke in an online shop, using the shop's records. Its problems are built by hand from on/off fault switches. SRE World makes them automatically, with real causes.

  1. 1 · Describe

    Pick a scenario

    “A bad checkout release on a busy evening.” Written by hand or by a model, then checked.

  2. 2 · Break

    Break the real shop

    Ship buggy code, stop a database, starve a service or flood it with shoppers, on a schedule.

  3. 3 · Record

    Keep every record

    Logs, metrics, request timelines, alerts, releases, pull requests and the code's git history.

  4. 4 · Ask

    Write the tickets

    Customer complaints at three levels of detail, investigated minutes or a day later, each with its answer key.

  5. 5 · Check

    Quality checks

    Every bug must show up, nothing may give the answer away, no two tasks may ask the same question.

  6. 6 · Test

    Run the agent

    The agent investigates through 20 read-only tools, cut off at its current time, and is graded by a plain script.

One incident, start to finish

On Tue Apr 21, 1:00 PM EDT, a release to the product catalog added an ID check that rejects every product whose ID starts with a digit. Twelve minutes later it was rolled back. This is the shop's own error rate, recorded while it happened.

0.60.30release 1:05rolled back 1:171:001:051:101:151:201:25 PM
Product-catalog errors per second. Each point averages the previous three minutes, so the curve trails the release slightly.
  1. 1:15 PM

    “Users say product pages are broken.”

    The ticket the agent gets.

  2. Step 1

    Lists recent releases: product catalog shipped 2.2.0-d04195e at 1:05 PM.

  3. Step 2

    Finds failing requests in product-catalog: Invalid product id: 2ZYFJ3GM2N.

  4. Step 3

    Opens the change d04195e: a new check accepts only ^[A-Z][A-Z0-9]{9}$.

  5. Answer

    “The 1:05 PM product-catalog release.” Graded automatically.

Inspect the generated tasks

Each incident becomes many tasks: the ticket is specific, vague or very vague, investigated 15 minutes to a day later, with the time given exactly or loosely. These 45 come from one 47-minute recording with two real incidents and one change that caused no harm, which counts as a red herring.

45 of 45 tasks. Open a task to see its answer key and the incidents a careful responder should rule in or out.

  1. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported at 1:30 PM today ET · investigated 15 min later · exact time · one cause

    Agent's current time: Tue Apr 21, 1:35 PM EDT. Task ID: 8bc40205b7dcf576

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
  2. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported at 1:15 PM today ET · investigated 15 min later · exact time · one cause

    Agent's current time: Tue Apr 21, 1:20 PM EDT. Task ID: 7748265b166c7be9

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is one of the affected products, and its product page returns HTTP 500 with no product details.
  3. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported at 1:30 PM today ET · investigated 1 h later · exact time · one cause

    Agent's current time: Tue Apr 21, 2:20 PM EDT. Task ID: e071d3adc013001e

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  4. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported between 1:00 PM and 2:00 PM today ET · investigated 1 h later · time range · one cause

    Agent's current time: Tue Apr 21, 2:20 PM EDT. Task ID: 2c850cb4bb1a75ba

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  5. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported at 1:15 PM today ET · investigated 1 h later · exact time · one cause

    Agent's current time: Tue Apr 21, 2:05 PM EDT. Task ID: 2825e986e53aff0f

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  6. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported between 12:45 PM and 1:45 PM today ET · investigated 1 h later · time range · one cause

    Agent's current time: Tue Apr 21, 2:05 PM EDT. Task ID: b4792521a26b685b

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  7. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported at 1:30 PM today ET · investigated 8 h later · exact time · one cause

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: b54b0f655703768e

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  8. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported sometime in the afternoon today · investigated 8 h later · part of the day · one cause

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 42aa6078c2b61ab8

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  9. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported between 1:00 PM and 2:00 PM today ET · investigated 8 h later · time range · one cause

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 213bbc57af5efc1c

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  10. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported between 1:50 PM and 2:50 PM today ET · investigated 8 h later · time range · one cause

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 1f1bda61f47e79a5

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  11. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported at 1:15 PM today ET · investigated 8 h later · exact time · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: acc0777cb417d560

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  12. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported sometime in the afternoon today · investigated 8 h later · part of the day · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: 111904a91cef678d

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  13. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported between 12:45 PM and 1:45 PM today ET · investigated 8 h later · time range · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: 47de5dd0202de14b

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  14. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported between 1:35 PM and 2:35 PM today ET · investigated 8 h later · time range · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: d63a5dd4e78ed25d

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  15. Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Reported sometime yesterday · investigated 1 day later · which day · one cause

    Agent's current time: Wed Apr 22, 1:20 PM EDT. Task ID: 4fd44d178d5d993b

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Rule out catalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
    • Could explain the ticket payment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  16. Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Reported sometime yesterday · investigated 1 day later · which day · one cause

    Agent's current time: Wed Apr 22, 1:05 PM EDT. Task ID: 51214ea1f82c9386

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
    • Rule out payment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  17. Medium“people can't finish buying things”Reported at 1:30 PM today ET · investigated 15 min later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 1:35 PM EDT. Task ID: ed8bb757083c4d75

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
  18. Medium“users say product pages are broken”Reported at 1:15 PM today ET · investigated 15 min later · exact time · one cause

    Agent's current time: Tue Apr 21, 1:20 PM EDT. Task ID: f65026650e4a3457

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no details, which shoppers could describe as product pages being broken.
  19. Medium“people can't finish buying things”Reported at 1:30 PM today ET · investigated 1 h later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 2:20 PM EDT. Task ID: edf8a8b3999e09c2

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  20. Medium“people can't finish buying things”Reported between 1:00 PM and 2:00 PM today ET · investigated 1 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 2:20 PM EDT. Task ID: 2ef3b686c86f1279

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  21. Medium“users say product pages are broken”Reported at 1:15 PM today ET · investigated 1 h later · exact time · one cause

    Agent's current time: Tue Apr 21, 2:05 PM EDT. Task ID: c63637c0a1802d37

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  22. Medium“users say product pages are broken”Reported between 12:45 PM and 1:45 PM today ET · investigated 1 h later · time range · one cause

    Agent's current time: Tue Apr 21, 2:05 PM EDT. Task ID: ad6733d8e9557ad6

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  23. Medium“people can't finish buying things”Reported at 1:30 PM today ET · investigated 8 h later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 8058b11828c9ef8b

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  24. Medium“people can't finish buying things”Reported sometime in the afternoon today · investigated 8 h later · part of the day · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: c8ece5ace3bff3a5

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  25. Medium“people can't finish buying things”Reported between 1:00 PM and 2:00 PM today ET · investigated 8 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: a734d98f33189e4a

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  26. Medium“people can't finish buying things”Reported between 1:50 PM and 2:50 PM today ET · investigated 8 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 00947eb5a7cdf599

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  27. Medium“users say product pages are broken”Reported at 1:15 PM today ET · investigated 8 h later · exact time · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: 2625531a83528078

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  28. Medium“users say product pages are broken”Reported sometime in the afternoon today · investigated 8 h later · part of the day · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: aaea263669b1185c

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  29. Medium“users say product pages are broken”Reported between 12:45 PM and 1:45 PM today ET · investigated 8 h later · time range · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: 71403f580c88e8a9

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  30. Medium“users say product pages are broken”Reported between 1:35 PM and 2:35 PM today ET · investigated 8 h later · time range · one cause

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: b3146dacc444ae05

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  31. Medium“people can't finish buying things”Reported sometime yesterday · investigated 1 day later · which day · 2 causes

    Agent's current time: Wed Apr 22, 1:20 PM EDT. Task ID: 57392705096e3872

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
    • Could explain the ticket payment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  32. Medium“users say product pages are broken”Reported sometime yesterday · investigated 1 day later · which day · one cause

    Agent's current time: Wed Apr 22, 1:05 PM EDT. Task ID: 1c39582e318161e3

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
    • Rule out payment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  33. Hard“users are reporting site issues”Reported at 1:15 PM today ET · investigated 15 min later · exact time · one cause

    Agent's current time: Tue Apr 21, 1:20 PM EDT. Task ID: 1d5b5dab4195b9d4

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Affected product pages, recommendations, cart, and checkout can return errors, which shoppers could reasonably report as site issues.
  34. Hard“users are reporting site issues”Reported at 1:30 PM today ET · investigated 15 min later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 1:35 PM EDT. Task ID: ca103bd2d4ee5b0e

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
  35. Hard“users are reporting site issues”Reported at 1:15 PM today ET · investigated 1 h later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 2:05 PM EDT. Task ID: dcf29da07b51e0e1

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  36. Hard“users are reporting site issues”Reported between 12:45 PM and 1:45 PM today ET · investigated 1 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 2:05 PM EDT. Task ID: 57edd238732db0c4

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  37. Hard“users are reporting site issues”Reported at 1:30 PM today ET · investigated 1 h later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 2:20 PM EDT. Task ID: 972b473f8893daee

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  38. Hard“users are reporting site issues”Reported between 1:00 PM and 2:00 PM today ET · investigated 1 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 2:20 PM EDT. Task ID: ad671d08c213d08d

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  39. Hard“users are reporting site issues”Reported at 1:15 PM today ET · investigated 8 h later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: e2502758efb3db14

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  40. Hard“users are reporting site issues”Reported sometime in the afternoon today · investigated 8 h later · part of the day · 2 causes

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: c332d860c7776532

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  41. Hard“users are reporting site issues”Reported between 12:45 PM and 1:45 PM today ET · investigated 8 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: ca3f05ed7cb4a391

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  42. Hard“users are reporting site issues”Reported between 1:35 PM and 2:35 PM today ET · investigated 8 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 9:05 PM EDT. Task ID: 7b11a0bb459437d4

    Answer key

    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  43. Hard“users are reporting site issues”Reported at 1:30 PM today ET · investigated 8 h later · exact time · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 054d5ef172a5bdc7

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  44. Hard“users are reporting site issues”Reported between 1:00 PM and 2:00 PM today ET · investigated 8 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 049fd451eeebac8e

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm
  45. Hard“users are reporting site issues”Reported between 1:50 PM and 2:50 PM today ET · investigated 8 h later · time range · 2 causes

    Agent's current time: Tue Apr 21, 9:20 PM EDT. Task ID: 4f677e5d03c76ea9

    Answer key

    • deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
    • deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM

    Incidents in the window

    • Could explain the ticket catalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
    • Could explain the ticket payment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
    • Rule out reco-cache: ran without effect: its probe did not fire and telemetry shows no harm

Rubrics: what the grader knows

Every incident gets a rubric: what changed, why it breaks, and the symptoms the recording actually shows, each compared with a clean stretch before it. It is used to judge the written incident report; the answer itself is scored by a plain script.

catalog-sku

A code change to product-catalog ("catalog: reject malformed product IDs early") was deployed at 17:05:00 UTC and rolled back at 17:17:00 UTC. Product pages for the 6 of 10 products whose IDs start with a digit return HTTP 500. Adding those items to the cart, checking out with them, and recommendations that include them also fail.

How it breaks

What changed. A code change to product-catalog ("catalog: reject malformed product IDs early") was deployed at 17:05:00 UTC and rolled back at 17:17:00 UTC.

The change. Priya Raman merged "catalog: reject malformed product IDs early" (commit: "catalog: validate product IDs before querying the database"), touching src/product-catalog/main.go. Bug class: over-strict input validation.

Why it breaks. GetProduct now rejects IDs that do not start with a letter (InvalidArgument); the frontend turns the gRPC error into a 500. Promise.all over GetProduct calls rejects when any recommended ID is invalid. GET /api/cart looks up every item with GetProduct. Checkout looks up every item's price with GetProduct and fails on the first invalid ID.

How shoppers are affected. - product page (GET /api/products/[productId], http_5xx, every time): Product pages for the six products whose IDs start with a digit (Solar System Color Imager, Eclipsmart Travel Refractor Telescope, Roof Binoculars, Starsense Explorer Refractor Telescope, Solar Filter, Optical Tube Assembly) return HTTP 500 and show no product details. - recommendations (GET /api/recommendations, http_5xx, intermittently): The "You May Also Like" section usually fails to load, because the frontend looks up each recommended product and most recommendation sets include an affected one. - cart (GET /api/cart, http_5xx, every time): The cart page returns HTTP 500 once an affected product is in the cart. - checkout (POST /api/checkout, http_5xx, every time): Clicking "Place Order" fails with HTTP 500 when the cart holds an affected product.

Observed in telemetry. Between 17:05:12 and 17:16:58 UTC, 202 traces fail first in product-catalog (absent from the baseline). Call chain: product-catalog:oteldemo.ProductCatalogService/GetProduct → frontend:oteldemo.ProductCatalogService/GetProduct → frontend:executing api route (pages) /api/products/[productId]/index → frontend:GET /api/products/[productId]/index → frontend:GET /api/products/{productId} → frontend-proxy:router frontend egress → frontend-proxy:GET → load-generator:GET. Smoking gun: product-catalog:oteldemo.ProductCatalogService/GetProduct span status: Invalid product id: 9SIQT8TOJO

Symptoms found in the recording

metrics

  • traces_span_metrics_calls.
  • traces_span_metrics_duration_milliseconds.
  • rpc_server_request_size_bytes.
  • app_frontend_requests.
  • rpc_client_duration_milliseconds.

logs

  • GET /api/products/{productId} → 500. Between 17:05:12 and 17:16:58 UTC, 204 frontend-proxy access-log lines matched (none in the baseline window).
  • GET /api/recommendations → 500. Between 17:05:07 and 17:16:54 UTC, 54 frontend-proxy access-log lines matched (none in the baseline window).
  • POST /api/checkout → 500. Between 17:05:31 and 17:16:48 UTC, 34 frontend-proxy access-log lines matched (none in the baseline window).

frontend

  • GET /api/products/[productId]. Product pages for the six products whose IDs start with a digit (Solar System Color Imager, Eclipsmart Travel Refractor Telescope, Roof Binoculars, Starsense Explorer Refractor Telescope, Solar Filter, Optical Tube Assembly) return HTTP 500 and show no product details.
  • GET /api/recommendations. The "You May Also Like" section usually fails to load, because the frontend looks up each recommended product and most recommendation sets include an affected one.
  • GET /api/cart. The cart page returns HTTP 500 once an affected product is in the cart.
  • POST /api/checkout. Clicking "Place Order" fails with HTTP 500 when the cart holds an affected product.

traces

  • product-catalog oteldemo.ProductCatalogService/GetProduct failures reaching load-generator:GET. Between 17:05:12 and 17:16:58 UTC, 202 traces fail first in product-catalog (absent from the baseline).
  • product-catalog oteldemo.ProductCatalogService/GetProduct failures reaching load-generator:GET. Between 17:05:09 and 17:16:54 UTC, 54 traces fail first in product-catalog (absent from the baseline).
  • product-catalog oteldemo.ProductCatalogService/GetProduct failures reaching load-generator:POST. Between 17:05:31 and 17:16:48 UTC, 34 traces fail first in product-catalog (absent from the baseline).

payment-risk

A code change to payment ("payment: add a risk-review hold for high-value orders") was deployed at 17:20:00 UTC and rolled back at 17:32:00 UTC. Checkout fails for orders above 1,000 currency units with "Transaction declined: amount held for risk review", which covers every order with the $3,599 optical tube assembly and many multi-item orders.

How it breaks

What changed. A code change to payment ("payment: add a risk-review hold for high-value orders") was deployed at 17:20:00 UTC and rolled back at 17:32:00 UTC.

The change. Marcus Lee merged "payment: add a risk-review hold for high-value orders" (commit: "payment: hold high-value charges for risk review"), touching src/payment/charge.js. Bug class: business-rule threshold set too low.

Why it breaks. payment's new risk-review limit throws for large charges; checkout returns Internal and the frontend a 500.

How shoppers are affected. - checkout (POST /api/checkout, http_5xx, every time): Placing an order above 1,000 currency units fails with "Transaction declined: amount held for risk review"; the shopper stays on the cart page and is not charged.

Observed in telemetry. Between 17:20:47 and 17:31:58 UTC, 22 traces fail first in payment (absent from the baseline). Call chain: payment:oteldemo.PaymentService/Charge → checkout:oteldemo.PaymentService/Charge → checkout:oteldemo.CheckoutService/PlaceOrder → frontend:oteldemo.CheckoutService/PlaceOrder → frontend:executing api route (pages) /api/checkout → frontend:POST /api/checkout → frontend:POST /api/checkout → frontend-proxy:router frontend egress → frontend-proxy:POST → load-generator:POST. Smoking gun: Nearest error message on the chain: checkout:oteldemo.PaymentService/Charge: Transaction declined: amount held for risk review (2189 USD).

Symptoms found in the recording

metrics

  • traces_span_metrics_calls.
  • traces_span_metrics_duration_milliseconds.
  • v8js_memory_heap_space_physical_size_bytes.
  • app_frontend_requests.
  • rpc_client_duration_milliseconds.

logs

  • POST /api/checkout → 500. Between 17:20:47 and 17:31:58 UTC, 22 frontend-proxy access-log lines matched (none in the baseline window).
  • payment: Transaction declined: amount held for risk review ({n} USD). (WARN). Between 17:20:47 and 17:31:26 UTC, 16 payment log records matched (none in the baseline window).
  • payment: Transaction declined: amount held for risk review ({n} CAD). (WARN). Between 17:25:20 and 17:31:58 UTC, 6 payment log records matched (none in the baseline window).

frontend

  • POST /api/checkout. Placing an order above 1,000 currency units fails with "Transaction declined: amount held for risk review"; the shopper stays on the cart page and is not charged.

traces

  • payment oteldemo.PaymentService/Charge failures reaching load-generator:POST. Between 17:20:47 and 17:31:58 UTC, 22 traces fail first in payment (absent from the baseline).

Watch an agent investigate

openai/gpt-6-luna, an inexpensive model, ran inside the packaged task with Harbor's terminal agent. It had only the ticket and the investigation tools. It scored 1.0 in 3.6 minutes, using 228,901 input and 10,394 output tokens ($0.01).

  1. sre-desk list
  2. sre-desk get_context
  3. sre-desk schema list_services
  4. sre-desk schema service_map
  5. sre-desk schema list_metrics
  6. sre-desk schema list_alerts
  7. sre-desk schema get_dashboard
  8. sre-desk schema list_deploys
  9. sre-desk schema list_flag_changes
  10. sre-desk schema list_pod_events
  11. sre-desk schema list_change_records
  12. sre-desk list_services --args '{}'
  13. sre-desk service_map --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z"}'
  14. sre-desk list_alerts --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  15. sre-desk list_deploys --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  16. sre-desk list_flag_changes --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  17. sre-desk list_pod_events --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  18. sre-desk list_change_records --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  19. sre-desk schema search_logs
  20. sre-desk schema search_traces
  21. sre-desk schema query_metrics
  22. sre-desk list_services --args '{}'
  23. sre-desk service_map --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z"}'
  24. sre-desk list_alerts --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  25. sre-desk list_deploys --args '{"start":"2026-04-21T16:14:14Z","end":"2026-04-21T18:05:00Z","limit":100}'
  26. sre-desk schema search_logs
  27. sre-desk schema search_traces
  28. sre-desk schema query_metrics
  29. sre-desk get_dashboard --args '{"service":"product-catalog","start":"2026-04-21T16:45:00Z","end":"2026-04-21T17:35:00Z","step":"1m"}'
  30. sre-desk get_dashboard --args '{"service":"frontend","start":"2026-04-21T16:45:00Z","end":"2026-04-21T17:35:00Z","step":"1m"}'
  31. sre-desk get_dashboard --args '{"service":"recommendation","start":"2026-04-21T16:45:00Z","end":"2026-04-21T17:35:00Z","step":"1m"}'
  32. sre-desk list_alerts --args '{"start":"2026-04-21T16:45:00Z","end":"2026-04-21T17:35:00Z","limit":100}'
  33. sre-desk schema search_logs
  34. sre-desk schema search_traces
  35. sre-desk schema get_trace
  36. sre-desk schema query_metrics
  37. sre-desk list_alerts --args '{"start":"2026-04-21T16:45:00Z","end":"2026-04-21T17:35:00Z","limit":30}'
  38. sre-desk schema search_traces
  39. sre-desk schema search_logs
  40. sre-desk schema get_trace
  41. sre-desk search_traces --args '{"service":"product-catalog","status":"error","start":"2026-04-21T17:00:00Z","end":"2026-04-21T17:25:00Z","limit":10}'
  42. sre-desk search_traces --args '{"service":"frontend","status":"error","start":"2026-04-21T17:00:00Z","end":"2026-04-21T17:25:00Z","limit":10}'
  43. sre-desk search_logs --args '{"service":"product-catalog","start":"2026-04-21T17:00:00Z","end":"2026-04-21T17:25:00Z","limit":20}'
  44. sre-desk search_traces --args '{"service":"product-catalog","status":"error","start":"2026-04-21T17:05:00Z","end":"2026-04-21T17:17:00Z","limit":5}'
  45. sre-desk schema get_trace
  46. sre-desk schema git_show
  47. sre-desk schema search_code
  48. sre-desk get_trace --args '{"trace_id":"66ca0759f9bad258db0bff08f3b4ba50"}'
  49. sre-desk git_show --args '{"commit":"d04195e6d66c096309fbf79547eb0f29cf6c806f"}'
  50. sre-desk git_show --args '{"sha":"d04195e6d66c096309fbf79547eb0f29cf6c806f"}'
  51. sre-desk list_files --args '{"path":"src/product-catalog"}'
  52. sre-desk search_traces --args '{"service":"product-catalog","operation":"*GetProduct*","status":"error","start":"2026-04-21T17:05:00Z","end":"2026-04-21T17:17:00Z","limit":10}'
  53. sre-desk search_traces --args '{"service":"product-catalog","operation":"*GetProduct*","status":"error","start":"2026-04-21T17:17:00Z","end":"2026-04-21T17:35:00Z","limit":5}'
  54. sre-desk list_deploys --args '{"service":"product-catalog","start":"2026-04-21T16:55:00Z","end":"2026-04-21T17:30:00Z","limit":20}'
  55. sre-desk schema git_show
  56. sre-desk git_show --args '{"sha":"d04195e6d66c096309fbf79547eb0f29cf6c806f","path":"src/product-catalog/main.go","max_lines":120}'
  57. sre-desk search_logs --args '{"service":"product-catalog","severity":"ERROR","start":"2026-04-21T17:05:00Z","end":"2026-04-21T17:17:00Z","limit":10}'
  58. sre-desk search_traces --args '{"service":"product-catalog","operation":"*GetProduct*","status":"error","start":"2026-04-21T17:17:00Z","end":"2026-04-21T17:35:00Z","limit":3}'
  59. sre-desk get_trace --args '{"trace_id":"1804c00feb46a832ff6b2bfa7cd28c85"}'
  60. sre-desk search_traces --args '{"service":"product-catalog","operation":"*GetProduct*","status":"error","start":"2026-04-21T17:20:00Z","end":"2026-04-21T17:47:00Z","limit":10}'
  61. sre-desk search_traces --args '{"service":"frontend","status":"error","start":"2026-04-21T17:17:00Z","end":"2026-04-21T17:47:00Z","limit":5}'
  62. sre-desk list_flag_changes --args '{"start":"2026-04-21T17:00:00Z","end":"2026-04-21T17:30:00Z","limit":20}'
  63. sre-desk list_deploys --args '{"start":"2026-04-21T17:00:00Z","end":"2026-04-21T17:30:00Z","limit":20}'
  64. sre-desk search_traces --args '{"service":"product-catalog","operation":"*GetProduct*","status":"error","start":"2026-04-21T17:20:00Z","end":"2026-04-21T17:47:00Z","limit":10}'
  65. sre-desk list_deploys --args '{"service":"product-catalog","start":"2026-04-21T17:00:00Z","end":"2026-04-21T17:30:00Z","limit":20}'
  66. sre-desk search_traces --args '{"root_service":"load-generator","operation":"user_browse_product","status":"ok","start":"2026-04-21T17:20:00Z","end":"2026-04-21T17:30:00Z","limit":3}'
  67. sre-desk search_traces --args '{"service":"product-catalog","operation":"*GetProduct*","status":"error","start":"2026-04-21T17:20:00Z","end":"2026-04-21T17:47:00Z","limit":5}'
  68. cat > /app/report.md <<'EOF'
  69. cat /app/answer.json && printf '\n--- report ---\n' && sed -n '1,240p' /app/report.md

Its answer

{
  "root_causes": [
    {
      "root_cause": "deploy:product-catalog:d04195e6d66c096309fbf79547eb0f29cf6c806f",
      "start_utc": "2026-04-21T17:05:00Z",
      "end_utc": "2026-04-21T17:17:00Z"
    }
  ]
}

Its incident report (opening)

Product-page requests failed when product-catalog rejected valid SKUs beginning with a digit. The cause was bad deploy d04195e6d66c096309fbf79547eb0f29cf6c806f (version 2.2.0-d04195e), active from 2026-04-21 17:05 UTC until rollback at 17:17 UTC. The deploy added a validator that required the first character to be a letter even though the code comment describes SKUs as ten uppercase alphanumerics. Evidence: list_deploys records the deploy at 17:05 and successful rollback to 2.2.0 at 17:17; git_show for that SHA shows ^[A-Z][A-Z0-9]{9}$; trace 66ca0759f9bad258db0bff08f3b4ba50 shows a product flow failing GetProduct with Invalid product id: 2ZYFJ3GM2N. A user_browse_product trace, 1804c00feb46a832ff6b2bfa7cd28c85, at 17:17:02 also rejected 9SIQT8TOJO during rollback propagation. The focused search_traces query for errored product-catalog GetProduct spans from 17:20 to 17:47 UTC returned zero results.

  • 16:31 UTC: Commit d04195e6d66c096309fbf79547eb0f29cf6c806f, “catalog: validate product IDs before querying the database,” authored by Priya Raman. git_show shows the new regex ^[A-Z][A-Z0-9]{9}$ and early InvalidArgument return.
  • 17:05 UTC: list_deploys records successful product-catalog deployment deploy-f08fdd90, version 2.2.0-d04195e, commit d04195e…. This is the start of the cause interval.
  • By 17:16:48 UTC: Trace

What makes the tests harder

Real code bugs

Ship a buggy change instead of flipping a fault switch: 14 bugs across Go, Node, Python, PHP, TypeScript, C# and Ruby services.

Working

Code access

The agent reads the code and its git history for some services, only the running image for others.

Working

Alerts and dashboards

The alerts and golden-signal dashboard an on-call engineer sees first, some of them noisy on purpose.

Working

More traffic

Three times the shoppers, and a daily busy and quiet rhythm.

Working

Silent bugs

Wrong numbers with no errors: Canadian customers charged about 34% too much, a ledger that keeps one order in ten.

Recording now

More services

A login service, an outside payment processor with its own outages, notifications over Kafka.

Recording next

Disguise

Parts rewritten in other languages, or the whole shop renamed as an outdoor-gear store, so a model cannot recall it from training.

Recording next

Background noise

Expired logins, retries, routine releases, scanner traffic and harmless warnings, written for this benchmark.

Partly live

Fake calendar

Months of history stitched from a few hours of recording, with seams checked statistically.

Built

Difficulty tuning

Keep only the tasks that the models being ranked sometimes fail.

Ready, runs sparingly

Status and limits

  • This page shows a preview world from the early recordings. It is not a benchmark release, and its answer keys are public on purpose.
  • Worlds with the silent bugs, the extra services, rewritten services and the disguised shop are recording now. They replace this world on this page as they pass the quality checks.
  • The difficulty gate keeps only tasks that frontier models sometimes fail. It runs sparingly against Claude Fable and OpenAI Astra once the worlds are final.
  • Everything runs on standard tools: the OpenTelemetry Astronomy Shop, Docker Compose, Prometheus, OpenSearch, Jaeger and Harbor.