Play
On call. Find what broke before the pager does it again.
Every failure here is something I fixed in production or wrote about on this site. The alerts are rebuilt from memory and my notes, the causes and the fixes are real. Same kind of evidence I had: the curl, the kubectl output, the terraform plan. Each shift deals 8 of the 12, so the next one is different.
pager · 02:44
You are on call. 8 alerts are coming.
- 01Read the alert and the evidence.
- 02Pick what actually broke. 25 seconds each.
- 03Stuck? Dig deeper for one more clue, it costs 40 points.
- 04Fast answers and streaks score more.
Keys: 1 to 4 to answer, H to dig deeper, Enter for the next alert.
Why a game
Reading about incidents does not stick.
Guessing wrong with the clock running does. Most of these took me longer than 25 seconds the first time, some took a whole night. If you run production, a few will feel familiar. If you hire for it, ask your candidates these.
- 01Edge1 alert
- 02Load balancer2 alerts
- 03Cluster2 alerts
- 04App3 alerts
- 05Data2 alerts
- 06Machine2 alerts