Blobfish ResearchResearch previewBuilt on ORCA-bench
SRE World
A factory that turns a real online shop into graded “what broke?” tests for AI agents. Every incident really happens: buggy code ships, databases stop, shoppers pile in, and everything the shop records becomes the evidence.
ORCA-bench asks an agent what broke in an online shop, using the shop's records. Its problems are built by hand from on/off fault switches. SRE World makes them automatically, with real causes.
1 · Describe
Pick a scenario
“A bad checkout release on a busy evening.” Written by hand or by a model, then checked.
2 · Break
Break the real shop
Ship buggy code, stop a database, starve a service or flood it with shoppers, on a schedule.
3 · Record
Keep every record
Logs, metrics, request timelines, alerts, releases, pull requests and the code's git history.
4 · Ask
Write the tickets
Customer complaints at three levels of detail, investigated minutes or a day later, each with its answer key.
5 · Check
Quality checks
Every bug must show up, nothing may give the answer away, no two tasks may ask the same question.
6 · Test
Run the agent
The agent investigates through 20 read-only tools, cut off at its current time, and is graded by a plain script.
One incident, start to finish
On Tue Apr 21, 1:00 PM EDT, a release to the product catalog added an ID check that rejects every product whose ID starts with a digit. Twelve minutes later it was rolled back. This is the shop's own error rate, recorded while it happened.
Product-catalog errors per second. Each point averages the previous three minutes, so the curve trails the release slightly.
1:15 PM
“Users say product pages are broken.”
The ticket the agent gets.
Step 1
Lists recent releases: product catalog shipped 2.2.0-d04195e at 1:05 PM.
Step 2
Finds failing requests in product-catalog: Invalid product id: 2ZYFJ3GM2N.
Step 3
Opens the change d04195e: a new check accepts only ^[A-Z][A-Z0-9]{9}$.
Answer
“The 1:05 PM product-catalog release.” Graded automatically.
Inspect the generated tasks
Each incident becomes many tasks: the ticket is specific, vague or very vague, investigated 15 minutes to a day later, with the time given exactly or loosely. These 137 come from 2 recordings: Astronomy Shop: two code bugs (45 tasks, 47 minutes); Cairnline Outfitters: the shop in disguise (92 tasks, 100 minutes). Pick a world to see its tickets.
137 of 137 tasks. Open a task to see its answer key and the incidents a careful responder should rule in or out.
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 15 min later · exact time · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 15 min later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is one of the affected products, and its product page returns HTTP 500 with no product details.
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 1 h later · exact time · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported between 1:00 PM and 2:00 PM today ET · investigated 1 h later · time range · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 1 h later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported between 12:45 PM and 1:45 PM today ET · investigated 1 h later · time range · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 8 h later · exact time · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported sometime in the afternoon today · investigated 8 h later · part of the day · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported between 1:00 PM and 2:00 PM today ET · investigated 8 h later · time range · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported between 1:50 PM and 2:50 PM today ET · investigated 8 h later · time range · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 8 h later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported sometime in the afternoon today · investigated 8 h later · part of the day · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported between 12:45 PM and 1:45 PM today ET · investigated 8 h later · time range · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported between 1:35 PM and 2:35 PM today ET · investigated 8 h later · time range · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“A few shoppers trying to pay for orders over $1,000 see "Transaction declined: amount held for risk review" and the order does not go through”Astronomy Shop: two code bugs · Reported sometime yesterday · investigated 1 day later · which day · one cause
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Rule outcatalog-sku: It causes HTTP 500 errors only when affected products are involved, not the amount-specific risk-review decline shoppers reported.
Could explain the ticketpayment-risk: It produces the exact risk-review decline for orders above 1,000 and prevents the order from going through.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Multiple users opening the Roof Binoculars page get an HTTP 500 error instead of the product details”Astronomy Shop: two code bugs · Reported sometime yesterday · investigated 1 day later · which day · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Roof Binoculars is explicitly among the affected products, whose product pages return HTTP 500 without showing details.
Rule outpayment-risk: This incident affects high-value order placement at checkout, not opening the Roof Binoculars product page.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 15 min later · exact time · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 15 min later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no details, which shoppers could describe as product pages being broken.
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 1 h later · exact time · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported between 1:00 PM and 2:00 PM today ET · investigated 1 h later · time range · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 1 h later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported between 12:45 PM and 1:45 PM today ET · investigated 1 h later · time range · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 8 h later · exact time · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported sometime in the afternoon today · investigated 8 h later · part of the day · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported between 1:00 PM and 2:00 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported between 1:50 PM and 2:50 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 8 h later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported sometime in the afternoon today · investigated 8 h later · part of the day · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported between 12:45 PM and 1:45 PM today ET · investigated 8 h later · time range · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported between 1:35 PM and 2:35 PM today ET · investigated 8 h later · time range · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“people can't finish buying things”Astronomy Shop: two code bugs · Reported sometime yesterday · investigated 1 day later · which day · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Shoppers with an affected product in their cart see the cart or order placement fail, preventing them from completing a purchase.
Could explain the ticketpayment-risk: Orders above 1,000 currency units are declined at checkout, so affected shoppers cannot complete their purchase.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Medium“users say product pages are broken”Astronomy Shop: two code bugs · Reported sometime yesterday · investigated 1 day later · which day · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages for six products return HTTP 500 and show no product details, which directly matches shoppers reporting broken product pages.
Rule outpayment-risk: This issue only affects placing high-value orders at checkout and does not break product pages.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 15 min later · exact time · one cause
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Affected product pages, recommendations, cart, and checkout can return errors, which shoppers could reasonably report as site issues.
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 15 min later · exact time · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 1 h later · exact time · 2 causes
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported between 12:45 PM and 1:45 PM today ET · investigated 1 h later · time range · 2 causes
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 1 h later · exact time · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported between 1:00 PM and 2:00 PM today ET · investigated 1 h later · time range · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported at 1:15 PM today ET · investigated 8 h later · exact time · 2 causes
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported sometime in the afternoon today · investigated 8 h later · part of the day · 2 causes
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported between 12:45 PM and 1:45 PM today ET · investigated 8 h later · time range · 2 causes
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported between 1:35 PM and 2:35 PM today ET · investigated 8 h later · time range · 2 causes
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported at 1:30 PM today ET · investigated 8 h later · exact time · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported between 1:00 PM and 2:00 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Hard“users are reporting site issues”Astronomy Shop: two code bugs · Reported between 1:50 PM and 2:50 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payment:payment-risk-review-limit from 1:20 PM to 1:32 PM
deploy:product-catalog:catalog-sku-validation from 1:05 PM to 1:17 PM
Incidents in the window
Could explain the ticketcatalog-sku: Product pages, recommendations, cart, and checkout can fail for affected products, which shoppers could describe as site issues.
Could explain the ticketpayment-risk: Some shoppers cannot complete checkout for orders above 1,000, which could reasonably be reported as a site issue.
Rule outreco-cache: ran without effect: its probe did not fire and telemetry shows no harm
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 10:47 AM EDT. Task ID:4e9580d971d3e2cf
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 15 min later · exact time · 2 causes
Agent's current time: Fri Apr 24, 11:40 AM EDT. Task ID:6f73539cab208c89
Answer key
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 11:20 AM EDT. Task ID:ea8f7ed067b87288
Answer key
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 10:23 AM EDT. Task ID:94edaf4e60b290f9
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 1 h later · exact time · one cause
Agent's current time: Fri Apr 24, 11:32 AM EDT. Task ID:2f6ceefcb34609ad
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported between 10:12 AM and 11:12 AM today ET · investigated 1 h later · time range · one cause
Agent's current time: Fri Apr 24, 11:32 AM EDT. Task ID:e02d5c94f792b99d
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: affects cart, checkout, not checkout in the way reported
Rule outvisa-electron: affects checkout, not checkout in the way reported
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 1 h later · exact time · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported between 11:05 AM and 12:05 PM today ET · investigated 1 h later · time range · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 1 h later · exact time · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported between 10:45 AM and 11:45 AM today ET · investigated 1 h later · time range · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 1 h later · exact time · one cause
Agent's current time: Fri Apr 24, 11:08 AM EDT. Task ID:a9640b62bf90c428
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported between 9:48 AM and 10:48 AM today ET · investigated 1 h later · time range · one cause
Agent's current time: Fri Apr 24, 11:08 AM EDT. Task ID:e718012c1494f592
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 8 h later · exact time · one cause
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · one cause
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: affects cart, checkout, not checkout in the way reported
Rule outvisa-electron: affects checkout, not checkout in the way reported
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported between 10:12 AM and 11:12 AM today ET · investigated 8 h later · time range · one cause
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: affects cart, checkout, not checkout in the way reported
Rule outvisa-electron: affects checkout, not checkout in the way reported
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported between 11:02 AM and 12:02 PM today ET · investigated 8 h later · time range · one cause
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: affects cart, checkout, not checkout in the way reported
Rule outvisa-electron: affects checkout, not checkout in the way reported
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 8 h later · exact time · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported between 11:05 AM and 12:05 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported between 11:55 AM and 12:55 PM today ET · investigated 8 h later · time range · one cause
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Rule outcart-cache: outside the reported time window
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 8 h later · exact time · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported between 10:45 AM and 11:45 AM today ET · investigated 8 h later · time range · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported between 11:35 AM and 12:35 PM today ET · investigated 8 h later · time range · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 8 h later · exact time · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: affects cart, checkout, not product page in the way reported
Rule outvisa-electron: affects checkout, not product page in the way reported
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported between 9:48 AM and 10:48 AM today ET · investigated 8 h later · time range · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported between 10:38 AM and 11:38 AM today ET · investigated 8 h later · time range · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: affects cart, checkout, not product page in the way reported
Rule outvisa-electron: affects checkout, not product page in the way reported
Easy“Customer support says customers paying in Canadian dollars were charged about a third more than their total”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · one cause
Agent's current time: Sat Apr 25, 10:32 AM EDT. Task ID:1d60f7892e6f075e
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: affects cart, checkout, not checkout in the way reported
Rule outvisa-electron: affects checkout, not checkout in the way reported
Easy“Several shoppers see "Sorry, we cannot process Visa Electron cards" at checkout for orders paid with Visa cards starting 4910 to 4919”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · 2 causes
Agent's current time: Sat Apr 25, 11:25 AM EDT. Task ID:215504eb0a9791ee
Answer key
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Easy“Some buyers click Add to Cart on the Red Flashlight and see a 500 error instead of their basket”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · one cause
Agent's current time: Sat Apr 25, 11:05 AM EDT. Task ID:194fb8141272ffab
Answer key
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Easy“The storefront's product grid comes up empty with a 500 error for a couple of customers, though direct product links still open”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · one cause
Agent's current time: Sat Apr 25, 10:08 AM EDT. Task ID:66d16e222b64f305
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: affects cart, checkout, not product page in the way reported
Rule outvisa-electron: affects checkout, not product page in the way reported
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 10:23 AM EDT. Task ID:75dab103c9636717
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 10:47 AM EDT. Task ID:329b48085bb310fc
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 15 min later · exact time · 2 causes
Agent's current time: Fri Apr 24, 11:40 AM EDT. Task ID:9e56d9d6c7ec86d1
Answer key
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 11:20 AM EDT. Task ID:fd82718d94b76a61
Answer key
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 1 h later · exact time · one cause
Agent's current time: Fri Apr 24, 11:08 AM EDT. Task ID:c19eabf493e69eef
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported between 9:48 AM and 10:48 AM today ET · investigated 1 h later · time range · one cause
Agent's current time: Fri Apr 24, 11:08 AM EDT. Task ID:32ee195ab981660f
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 1 h later · exact time · one cause
Agent's current time: Fri Apr 24, 11:32 AM EDT. Task ID:7c1ba3f0d4508731
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported between 10:12 AM and 11:12 AM today ET · investigated 1 h later · time range · 3 causes
Agent's current time: Fri Apr 24, 11:32 AM EDT. Task ID:273f09d235d47f77
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 1 h later · exact time · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported between 11:05 AM and 12:05 PM today ET · investigated 1 h later · time range · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 1 h later · exact time · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported between 10:45 AM and 11:45 AM today ET · investigated 1 h later · time range · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 8 h later · exact time · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: affects cart, checkout, not product page in the way reported
Rule outvisa-electron: affects checkout, not product page in the way reported
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported between 9:48 AM and 10:48 AM today ET · investigated 8 h later · time range · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported between 10:38 AM and 11:38 AM today ET · investigated 8 h later · time range · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: affects cart, checkout, not product page in the way reported
Rule outvisa-electron: affects checkout, not product page in the way reported
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 8 h later · exact time · one cause
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 3 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported between 10:12 AM and 11:12 AM today ET · investigated 8 h later · time range · 3 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported between 11:02 AM and 12:02 PM today ET · investigated 8 h later · time range · 3 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 8 h later · exact time · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported between 11:05 AM and 12:05 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported between 11:55 AM and 12:55 PM today ET · investigated 8 h later · time range · one cause
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Rule outcart-cache: outside the reported time window
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 8 h later · exact time · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported between 10:45 AM and 11:45 AM today ET · investigated 8 h later · time range · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported between 11:35 AM and 12:35 PM today ET · investigated 8 h later · time range · one cause
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Medium“shoppers are having trouble with product pages”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · one cause
Agent's current time: Sat Apr 25, 10:08 AM EDT. Task ID:a654aa7ebc70ff5b
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: affects product page: The home page product grid fails to load (HTTP 500 on the product list); opening a single product still works.
Rule outusd-settlement: affects checkout, not product page in the way reported
Rule outcart-cache: affects cart, checkout, not product page in the way reported
Rule outvisa-electron: affects checkout, not product page in the way reported
Medium“support staff reports problems with payments”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · 3 causes
Agent's current time: Sat Apr 25, 10:32 AM EDT. Task ID:a8253119f9342f8a
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Could explain the ticketusd-settlement: affects checkout: Customers paying in Canadian dollars are charged their CAD total in US dollars, about 34% more than they should pay; payment succeeds and nothing is logged.
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“users cannot complete checkout”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · 2 causes
Agent's current time: Sat Apr 25, 11:25 AM EDT. Task ID:87ac929ad1a6a19e
Answer key
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not checkout in the way reported
Rule outusd-settlement: affects checkout, not checkout in the way reported
Could explain the ticketcart-cache: affects checkout: Clicking "Place Order" fails with HTTP 500; the shopper stays on the cart page (checkout reads the cart).
Could explain the ticketvisa-electron: affects checkout: Orders paid with Visa cards numbered 4910-4919 are declined with "Sorry, we cannot process Visa Electron cards"; the shopper stays on the cart page.
Medium“visitors cannot add things to cart”Cairnline Outfitters: the shop in disguise · Reported sometime yesterday · investigated 1 day later · which day · one cause
Agent's current time: Sat Apr 25, 11:05 AM EDT. Task ID:7634f23379dcb54a
Answer key
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: affects product page, recommendations, not cart in the way reported
Rule outusd-settlement: affects checkout, not cart in the way reported
Could explain the ticketcart-cache: affects cart: Viewing the cart or adding an item to it fails with HTTP 500.
Rule outvisa-electron: affects checkout, not cart in the way reported
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 15 min later · exact time · 2 causes
Agent's current time: Fri Apr 24, 10:47 AM EDT. Task ID:4248fe9a41d53688
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: staff-visible incident; the report names no area
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 11:20 AM EDT. Task ID:8d56c16ba69fc7b9
Answer key
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 15 min later · exact time · one cause
Agent's current time: Fri Apr 24, 10:23 AM EDT. Task ID:e5c1f6f6a492be6a
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 15 min later · exact time · 2 causes
Agent's current time: Fri Apr 24, 11:40 AM EDT. Task ID:ed5aba724886298c
Answer key
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 1 h later · exact time · 2 causes
Agent's current time: Fri Apr 24, 11:32 AM EDT. Task ID:6aa8cff1cdfb2240
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: staff-visible incident; the report names no area
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported between 10:12 AM and 11:12 AM today ET · investigated 1 h later · time range · 4 causes
Agent's current time: Fri Apr 24, 11:32 AM EDT. Task ID:0a563b4bf0d717a7
Answer key
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: staff-visible incident; the report names no area
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Could explain the ticketcart-cache: staff-visible incident; the report names no area
Could explain the ticketvisa-electron: staff-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 1 h later · exact time · 2 causes
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 10:45 AM and 11:45 AM today ET · investigated 1 h later · time range · 3 causes
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 1 h later · exact time · one cause
Agent's current time: Fri Apr 24, 11:08 AM EDT. Task ID:8172fbe83577cdd7
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Rule outcart-cache: outside the reported time window
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 9:48 AM and 10:48 AM today ET · investigated 1 h later · time range · one cause
Agent's current time: Fri Apr 24, 11:08 AM EDT. Task ID:201c8851bc3f6d56
Answer key
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Rule outcart-cache: outside the reported time window
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 1 h later · exact time · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 11:05 AM and 12:05 PM today ET · investigated 1 h later · time range · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported at 10:42 AM today ET · investigated 8 h later · exact time · 2 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: staff-visible incident; the report names no area
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 4 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: staff-visible incident; the report names no area
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Could explain the ticketcart-cache: staff-visible incident; the report names no area
Could explain the ticketvisa-electron: staff-visible incident; the report names no area
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported between 10:12 AM and 11:12 AM today ET · investigated 8 h later · time range · 4 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: staff-visible incident; the report names no area
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Could explain the ticketcart-cache: staff-visible incident; the report names no area
Could explain the ticketvisa-electron: staff-visible incident; the report names no area
Hard“staff are reporting business issues”Cairnline Outfitters: the shop in disguise · Reported between 11:02 AM and 12:02 PM today ET · investigated 8 h later · time range · 3 causes
deploy:order-svc:checkout-usd-settlement from 10:32 AM to 10:54 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Could explain the ticketusd-settlement: staff-visible incident; the report names no area
Could explain the ticketcart-cache: staff-visible incident; the report names no area
Could explain the ticketvisa-electron: staff-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 11:15 AM today ET · investigated 8 h later · exact time · 2 causes
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 3 causes
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 10:45 AM and 11:45 AM today ET · investigated 8 h later · time range · 3 causes
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 11:35 AM and 12:35 PM today ET · investigated 8 h later · time range · 2 causes
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 10:18 AM today ET · investigated 8 h later · exact time · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 3 causes
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 9:48 AM and 10:48 AM today ET · investigated 8 h later · time range · one cause
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Rule outcart-cache: outside the reported time window
Rule outvisa-electron: outside the reported time window
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 10:38 AM and 11:38 AM today ET · investigated 8 h later · time range · 3 causes
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported at 11:35 AM today ET · investigated 8 h later · exact time · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported sometime in the morning today · investigated 8 h later · part of the day · 3 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
deploy:catalog-api:catalog-hide-discontinued from 10:08 AM to 10:23 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Could explain the tickethide-discontinued: user-visible incident; the report names no area
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 11:05 AM and 12:05 PM today ET · investigated 8 h later · time range · 2 causes
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
dependency:basket-cache:down from 11:05 AM to 11:17 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: affects checkout, not the site in the way reported
Could explain the ticketcart-cache: user-visible incident; the report names no area
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Hard“users are reporting site issues”Cairnline Outfitters: the shop in disguise · Reported between 11:55 AM and 12:55 PM today ET · investigated 8 h later · time range · one cause
deploy:payments:payment-decline-visa-electron from 11:25 AM to 11:40 AM
Incidents in the window
Rule outhide-discontinued: outside the reported time window
Rule outusd-settlement: outside the reported time window
Rule outcart-cache: outside the reported time window
Could explain the ticketvisa-electron: user-visible incident; the report names no area
Rubrics: what the grader knows
Every incident gets a rubric: what changed, why it breaks, and the symptoms the recording actually shows, each compared with a clean stretch before it. It is used to judge the written incident report; the answer itself is scored by a plain script.
catalog-sku
A code change to product-catalog ("catalog: reject malformed product IDs early") was deployed at 17:05:00 UTC and rolled back at 17:17:00 UTC. Product pages for the 6 of 10 products whose IDs start with a digit return HTTP 500. Adding those items to the cart, checking out with them, and recommendations that include them also fail.
How it breaks
What changed. A code change to product-catalog ("catalog: reject malformed product IDs early") was deployed at 17:05:00 UTC and rolled back at 17:17:00 UTC.
The change. Priya Raman merged "catalog: reject malformed product IDs early" (commit: "catalog: validate product IDs before querying the database"), touching src/product-catalog/main.go. Bug class: over-strict input validation.
Why it breaks. GetProduct now rejects IDs that do not start with a letter (InvalidArgument); the frontend turns the gRPC error into a 500. Promise.all over GetProduct calls rejects when any recommended ID is invalid. GET /api/cart looks up every item with GetProduct. Checkout looks up every item's price with GetProduct and fails on the first invalid ID.
How shoppers are affected.
- product page (GET /api/products/[productId], http_5xx, every time): Product pages for the six products whose IDs start with a digit (Solar System Color Imager, Eclipsmart Travel Refractor Telescope, Roof Binoculars, Starsense Explorer Refractor Telescope, Solar Filter, Optical Tube Assembly) return HTTP 500 and show no product details.
- recommendations (GET /api/recommendations, http_5xx, intermittently): The "You May Also Like" section usually fails to load, because the frontend looks up each recommended product and most recommendation sets include an affected one.
- cart (GET /api/cart, http_5xx, every time): The cart page returns HTTP 500 once an affected product is in the cart.
- checkout (POST /api/checkout, http_5xx, every time): Clicking "Place Order" fails with HTTP 500 when the cart holds an affected product.
Observed in telemetry. Between 17:05:12 and 17:16:58 UTC, 202 traces fail first in product-catalog (absent from the baseline). Call chain: product-catalog:oteldemo.ProductCatalogService/GetProduct → frontend:oteldemo.ProductCatalogService/GetProduct → frontend:executing api route (pages) /api/products/[productId]/index → frontend:GET /api/products/[productId]/index → frontend:GET /api/products/{productId} → frontend-proxy:router frontend egress → frontend-proxy:GET → load-generator:GET. Smoking gun: product-catalog:oteldemo.ProductCatalogService/GetProduct span status: Invalid product id: 9SIQT8TOJO
Symptoms found in the recording
metrics
traces_span_metrics_calls.
traces_span_metrics_duration_milliseconds.
rpc_server_request_size_bytes.
app_frontend_requests.
rpc_client_duration_milliseconds.
logs
GET /api/products/{productId} → 500. Between 17:05:12 and 17:16:58 UTC, 204 frontend-proxy access-log lines matched (none in the baseline window).
GET /api/recommendations → 500. Between 17:05:07 and 17:16:54 UTC, 54 frontend-proxy access-log lines matched (none in the baseline window).
POST /api/checkout → 500. Between 17:05:31 and 17:16:48 UTC, 34 frontend-proxy access-log lines matched (none in the baseline window).
frontend
GET /api/products/[productId]. Product pages for the six products whose IDs start with a digit (Solar System Color Imager, Eclipsmart Travel Refractor Telescope, Roof Binoculars, Starsense Explorer Refractor Telescope, Solar Filter, Optical Tube Assembly) return HTTP 500 and show no product details.
GET /api/recommendations. The "You May Also Like" section usually fails to load, because the frontend looks up each recommended product and most recommendation sets include an affected one.
GET /api/cart. The cart page returns HTTP 500 once an affected product is in the cart.
POST /api/checkout. Clicking "Place Order" fails with HTTP 500 when the cart holds an affected product.
traces
product-catalog oteldemo.ProductCatalogService/GetProduct failures reaching load-generator:GET. Between 17:05:12 and 17:16:58 UTC, 202 traces fail first in product-catalog (absent from the baseline).
product-catalog oteldemo.ProductCatalogService/GetProduct failures reaching load-generator:GET. Between 17:05:09 and 17:16:54 UTC, 54 traces fail first in product-catalog (absent from the baseline).
product-catalog oteldemo.ProductCatalogService/GetProduct failures reaching load-generator:POST. Between 17:05:31 and 17:16:48 UTC, 34 traces fail first in product-catalog (absent from the baseline).
payment-risk
A code change to payment ("payment: add a risk-review hold for high-value orders") was deployed at 17:20:00 UTC and rolled back at 17:32:00 UTC. Checkout fails for orders above 1,000 currency units with "Transaction declined: amount held for risk review", which covers every order with the $3,599 optical tube assembly and many multi-item orders.
How it breaks
What changed. A code change to payment ("payment: add a risk-review hold for high-value orders") was deployed at 17:20:00 UTC and rolled back at 17:32:00 UTC.
The change. Marcus Lee merged "payment: add a risk-review hold for high-value orders" (commit: "payment: hold high-value charges for risk review"), touching src/payment/charge.js. Bug class: business-rule threshold set too low.
Why it breaks. payment's new risk-review limit throws for large charges; checkout returns Internal and the frontend a 500.
How shoppers are affected.
- checkout (POST /api/checkout, http_5xx, every time): Placing an order above 1,000 currency units fails with "Transaction declined: amount held for risk review"; the shopper stays on the cart page and is not charged.
Observed in telemetry. Between 17:20:47 and 17:31:58 UTC, 22 traces fail first in payment (absent from the baseline). Call chain: payment:oteldemo.PaymentService/Charge → checkout:oteldemo.PaymentService/Charge → checkout:oteldemo.CheckoutService/PlaceOrder → frontend:oteldemo.CheckoutService/PlaceOrder → frontend:executing api route (pages) /api/checkout → frontend:POST /api/checkout → frontend:POST /api/checkout → frontend-proxy:router frontend egress → frontend-proxy:POST → load-generator:POST. Smoking gun: Nearest error message on the chain: checkout:oteldemo.PaymentService/Charge: Transaction declined: amount held for risk review (2189 USD).
Symptoms found in the recording
metrics
traces_span_metrics_calls.
traces_span_metrics_duration_milliseconds.
v8js_memory_heap_space_physical_size_bytes.
app_frontend_requests.
rpc_client_duration_milliseconds.
logs
POST /api/checkout → 500. Between 17:20:47 and 17:31:58 UTC, 22 frontend-proxy access-log lines matched (none in the baseline window).
payment: Transaction declined: amount held for risk review ({n} USD). (WARN). Between 17:20:47 and 17:31:26 UTC, 16 payment log records matched (none in the baseline window).
payment: Transaction declined: amount held for risk review ({n} CAD). (WARN). Between 17:25:20 and 17:31:58 UTC, 6 payment log records matched (none in the baseline window).
frontend
POST /api/checkout. Placing an order above 1,000 currency units fails with "Transaction declined: amount held for risk review"; the shopper stays on the cart page and is not charged.
traces
payment oteldemo.PaymentService/Charge failures reaching load-generator:POST. Between 17:20:47 and 17:31:58 UTC, 22 traces fail first in payment (absent from the baseline).
Watch an agent investigate
openai/gpt-6-luna, an inexpensive model, ran inside the packaged task with Harbor's terminal agent. It had only the ticket and the investigation tools. It scored 1.0 in 3.6 minutes, using 228,901 input and 10,394 output tokens.
Product-page requests failed when product-catalog rejected valid SKUs beginning with a digit. The cause was bad deploy d04195e6d66c096309fbf79547eb0f29cf6c806f (version 2.2.0-d04195e), active from 2026-04-21 17:05 UTC until rollback at 17:17 UTC. The deploy added a validator that required the first character to be a letter even though the code comment describes SKUs as ten uppercase alphanumerics. Evidence: list_deploys records the deploy at 17:05 and successful rollback to 2.2.0 at 17:17; git_show for that SHA shows ^[A-Z][A-Z0-9]{9}$; trace 66ca0759f9bad258db0bff08f3b4ba50 shows a product flow failing GetProduct with Invalid product id: 2ZYFJ3GM2N. A user_browse_product trace, 1804c00feb46a832ff6b2bfa7cd28c85, at 17:17:02 also rejected 9SIQT8TOJO during rollback propagation. The focused search_traces query for errored product-catalogGetProduct spans from 17:20 to 17:47 UTC returned zero results.
16:31 UTC: Commit d04195e6d66c096309fbf79547eb0f29cf6c806f, “catalog: validate product IDs before querying the database,” authored by Priya Raman. git_show shows the new regex ^[A-Z][A-Z0-9]{9}$ and early InvalidArgument return.
17:05 UTC:list_deploys records successful product-catalog deployment deploy-f08fdd90, version 2.2.0-d04195e, commit d04195e…. This is the start of the cause interval.
By 17:16:48 UTC: Trace
What makes the tests harder
Real code bugs
Ship a buggy change instead of flipping a fault switch: 14 bugs across Go, Node, Python, PHP, TypeScript, C# and Ruby services.
Working
Code access
The agent reads the code and its git history for some services, only the running image for others.
Working
Alerts and dashboards
The alerts and golden-signal dashboard an on-call engineer sees first, some of them noisy on purpose.
Working
More traffic
Three times the shoppers, and a daily busy and quiet rhythm.
Working
Silent bugs
Wrong numbers with no errors: Canadian customers charged about 34% too much, a ledger that keeps one order in ten, large orders shipped for free.
Working
More services
A login service, an outside payment processor with its own outages, notifications over Kafka.
Recorded
Disguise
Parts rewritten in other languages, or the whole shop renamed as an outdoor-gear store, so a model cannot recall it from training. Its 92 tasks are in the browser below.
Working
Background noise
Expired logins, retries, routine releases, scanner traffic and harmless warnings, written for this benchmark.
Partly live
Fake calendar
Months of history stitched from a few hours of recording, with seams checked statistically.
Built
Observability surface
How the agent sees the evidence: an MCP desk of ready-made tools, or the real Grafana, Prometheus, log search and request timelines it would use on call, with or without release markers on the dashboards, or both. Each is cut off at the agent's current time.
Working
Difficulty measurement
Measure how often each model being ranked solves each task. A release can keep every task or choose its mix of easy and hard ones.
Ready, runs sparingly
Status and limits
This page shows a preview world from the early recordings. It is not a benchmark release, and its answer keys are public on purpose.
Worlds with the silent bugs, the extra services, rewritten services and the disguised shop are recording now. They replace this world on this page as they pass the quality checks.
The difficulty gate measures every task against the models being ranked and labels it; nothing is dropped unless a release asks for a harder mix. It runs sparingly against Claude Fable and OpenAI Astra once the worlds are final.
Everything runs on standard tools: the OpenTelemetry Astronomy Shop, Docker Compose, Prometheus, OpenSearch, Jaeger and Harbor.
