Skip to content

Incident scenario bank

Production incidents to work through: 502s after a deploy, slow queries, connection spikes, cache stampedes, memory leaks, and CORS, CDN and cookie bugs.

12 lessons

Data tier

  1. 1Incident: 502s after a deploy (pool exhaustion, missing FK index)Same page, two causes: the state column tells held connections from a slow new query; remove the query first, then index its foreign key.
  2. 2Incident: a query slows down as data growsSame 50 rows, ten times the reading: when the statistics are fresh, the plan is the cause, so change the index, not the statistics.
  3. 3Incident: database connection spike after pre-scalingMore pods bring more pools, not more slots: the refusals wait for the herd, and the fix is fewer connections, not a reload.
  4. 4Incident: cache stampede at 09:00Redis's counters say why the hot key was gone and how big the herd was; move its expiry past the next start, then prove it.

Runtime and async

  1. 5Incident: Node memory climbing to OOMOOMKilled is the container counting every byte: chart external beside heapUsed, find the long-lived holder in a snapshot, cut its reference.
  2. 6Incident: duplicate records from retriesA timeout is not a failure: the abandoned request finishes, its resend repeats the effect; key the intent, pass the key on, clean, then constrain.
  3. 7Incident: queue backing up silentlyA call with no deadline holds its slot without failing; read active against concurrency, page on job age, and fix it with a timeout.

Edge, cookies and browser

  1. 8Incident: CORS failing only in productionA 204 preflight can still fail: replay it with the page's Origin, read Access-Control-Allow-Origin, and fix the allow-list's value, not the policy.
  2. 9Incident: CDN serving the wrong localeA page that picks its language must be cached under that language: compare what the page reads with what the key holds.
  3. 10Incident: login loop after a cookie changeThe cookie is in the jar; Strict keeps it off every hop of a trip another site's page started.

Release quality

  1. 11Incident: flaky end-to-end suiteWhen a suite turns flaky as workers go up, read what the assertion received, give each worker its own data, then fix the wait.
  2. 12Incident: INP regression after a releaseWhen INP's processing grows after a release, count what one keystroke re-renders, keep fast-changing text local, and share only a rarely changing boolean.