Incident: cache stampede at 09:00

Redis's counters say why the hot key was gone and how big the herd was; move its expiry past the next start, then prove it.

Before you read: predict

PAGE · 09:00:07 · ExamHall GET /exams/:id/paper · 5xx 65% · Postgres CPU 96% · Redis up

ECON-201 opened at 09:00:00 for 2,400 candidates, 400 on each of 6 API instances. The paper was cached at 08:00:00 by SET exam:ECON-201:paper <json> EX 3600, and nothing has written or deleted that key since. Each instance reads it with plain cache-aside: GET, and on a miss it builds the paper and writes it back. At 09:00:00 every one of the 2,400 requests sent its GET before the first rebuilt paper was written back. The on-call engineer compares Redis's INFO stats taken at 08:59:59 with one taken at 09:00:05.

By how much did expired_keys grow?

The 30-second version

A page at a scheduled start, with Redis up and Postgres busy, is a herd on one missing key until the counters say otherwise. One expiry and a miss per reader in Redis's counters mean the TTL ran out at the start. Count rebuilds in Postgres. Before the next start, move the expiry past the exam's end and check EXPIRETIME.

Redis's counters say why the hot key was gone and how big the herd was; move its expiry past the next start, then prove it.

How it works

Step 1 of 12

Read the page's shape. Errors begin at a scheduled start, hit every instance and clear within seconds, with no deploy: suspect a herd.

Analogy: A court building's daily cause list

The listing office pins each day's list in the lobby case, stamped "valid until 10:00", and takes it down at the stamp, just as sittings open and the whole lobby turns to read it.

In the court lobby	In the incident
the list in the lobby case, stamped "valid until 10:00"	the paper key and its TTL, ending at the start (step 9)
everyone who finds the case empty queues at the listing window	each miss rebuilds (`cache-stampede-ttl-jitter`)
the office's take-down book: one line per list removed, however many people looked	`expired_keys` +1 (step 5)
a list withdrawn for a correction leaves a signed slip instead	a `DEL` in `cmdstat_del` (step 5)
the lobby door's clicker: one click per person who looked and found nothing	`keyspace_misses` per read (step 6)
people stopped at a locked lobby door are never clicked	a refused read moves no server counter (steps 3, 8)
pinning a new list "only if the case is empty" does nothing while the old one hangs	`SET … NX` over a present key (step 10)
reading the list never changes its stamp	a `GET` leaves the TTL (step 10)
the 09:55 round reads the stamp, not just that a list hangs	`EXPIRETIME` against the exam's end (step 11)

Breaks down when: a clerk notices the queue and pins a new list at once; nothing in cache-aside notices, so every reader rebuilds until one refill lands.

Contrasting case: a hot key that is present.

In real terms: after a page at a start, read expired_keys, keyspace_misses and the rebuild's calls before you change anything; then move the next start's expiry and read it back.

Examples, run for real

4.1 ExamHall EXH-2960: the console at 09:00:05

4.1 Redis 7.4.11 · SET exam:ECON-201:paper <json> EX 3600 at 08:00:00 (compressed clock) · 6 instances x 400 GETs at 09:00:00
4.1 08:59:59 PEXPIRETIME exam:ECON-201:paper = the start's millisecond: true
4.1 09:00:00, after the reads and before any write-back: TTL exam:ECON-201:paper = -2
4.1 the same moment, INFO stats since 08:59:59: expired_keys +1 · keyspace_misses +2,400 · keyspace_hits +0

The key's expiry was the start's second (step 9). One expiry, 2,400 misses and no hits: the TTL ran out and every reader missed (steps 4–6).

4.2 Ticketing MR !3260: single-flight, read in Postgres

A venue's seat-map service hit the same page at a 10:00 on-sale, 500 fans on each of 8 pods. The next on-sale, on real Redis and Postgres:

4.2 Redis 7.4.11 · PostgreSQL 17.11 · MR !3260 · 8 pods x 500 fans · the GET in front of a per-pod single-flight
4.2 INFO stats across the on-sale: keyspace_misses +4,000 · pg_stat_statements calls for the seat-map query +8
4.2 fans served: 4,000 · rebuilds per pod: 1

The misses still count every fan, because every GET still reaches Redis; the calls show the shared rebuild, one per pod (steps 7, 12).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

A venue's on-sale for event 88 opens at 10:00:00, when the waiting room releases 4,000 fans, 500 on each of 8 pods, and each pod reads the seat map with plain cache-aside. The seat map was cached at 09:00:00 by SET event:88:seatmap <json> EX 3600. At 09:05 the team adds a job that runs once at 09:55.

Which job leaves no fan's read missing at 10:00:00?

Question 2 of 7

A video service's premiere 731 starts at 21:00:00 for 3,000 viewers, 300 on each of 10 pods. Its manifest key premiere:731:manifest was written at 20:50:00 with EX 600.

All 3,000 requests call loadManifest(731) before any pod's call has settled.

Between 20:59:59 and 21:00:05, by how much do Redis's keyspace_misses and the manifest query's calls in pg_stat_statements grow?

const loadManifest = singleFlight(async (id) => {  const hit = await redis.get(`premiere:${id}:manifest`);  if (hit) return JSON.parse(hit);  const { rows } = await pool.query('SELECT rendition, url FROM manifests WHERE premiere_id = $1', [id]);  await redis.set(`premiere:${id}:manifest`, JSON.stringify(rows), { EX: 600 });  return rows;});

Question 3 of 7

A chat service's live session in room 42 starts at 20:00:00, and 1,000 members, 250 on each of 4 pods, load the room's member list room:42:members, written at 19:55:00 with EX 3600. After a password rotation at 19:58, the pods restarted with the old password, and Redis refuses them. Each pod reads through this helper:

Between 19:59:59 and 20:00:05, how many lookups do Redis's keyspace_hits and keyspace_misses count, and by how much do the member query's calls in pg_stat_statements grow?

async function cached(key, load) {  let hit = null;  try { hit = await redis.get(key); } catch { /* Redis unavailable: fall through */ }  if (hit !== null) return JSON.parse(hit);  const value = await load();  redis.set(key, JSON.stringify(value), { EX: 3600 }).catch(() => {});  return value;}

Question 4 of 7

A payments ledger's treasury dashboard caches the day's totals in ledger:totals:2026-10-06, written at 23:00:00 with EX 3600. At 00:00:00, 200 open dashboards, 50 on each of 4 pods, refresh; every one misses and runs:

SELECT currency, sum(amount_cents) FROM ledger_entries WHERE booked_on = $1 GROUP BY currency;

ledger_entries holds 400,000 entries, with an index on booked_on; each pod has a pool of 10. On a quiet day the dashboards run the statement a few times an hour. The on-call engineer compares the statement's pg_stat_statements row from 23:59:59 with the row from 00:00:05, and measures its pages (shared_blks_hit + shared_blks_read) in quiet-day calls: the pages one call reads on a quiet day.

How much did calls and the pages grow?

Question 5 of 7

It is 09:05 on the day of EXH-2960. ECON-202 starts at 10:00:00 and ends at 12:00:00; its paper key exam:ECON-202:paper was written at 09:00:00 with EX 3600. Give the one Redis command your 09:55 job sends so that the 10:00 start and the whole exam read a cached paper, and the check you run at 09:56 to prove it.

Run it: node sandbox.mjs '<your command>' prints the misses at 10:00:00 and whether the key's EXPIRETIME is after 12:00:00.

Your answer (optional, stays on this page)

Question 6 of 7

After EXH-2960 the team adds a check that runs 5 minutes before each start, sends TTL exam:<id>:paper and posts to #exam-ops when the reply is -2. Two exams start at 11:00:00: ECON-301, whose paper key was written at 10:00:00 with EX 3600, and ECON-302, whose key was written at 09:30:00 with EX 3600. Nobody writes or deletes either key afterwards.

If nobody acts on what the check posts at 10:55, which start's readers find no paper key at 11:00:00 although the check posted nothing for that exam?

Question 7 of 7

A venue's on-sale for event 91 opened at 10:00:00, when the waiting room released 4,000 fans, 500 on each of 8 pods, and the seat-map page returned 5xx for its first seconds. Each pod reads event:91:seatmap with plain cache-aside from the one Redis server the pods read (no replica), and every fan's GET reached Redis before the first rebuilt seat map was written back. The on-call engineer diffs INFO stats and INFO commandstats taken at 09:59:57 and 10:00:05: expired_keys +0 · cmdstat_del calls +1 · keyspace_misses +4,000 · keyspace_hits +0.

Which cause does this readout support?

Go deeper

Pro tips

Snapshot the counters, then diff. Save redis-cli INFO stats and INFO commandstats before the start and again after it, and read each counter's change across the same seconds, before any fix. When: steps 3–6, before any fix. Status: EXECUTED.
Read the next start's expiry as a time. EXPIRETIME exam:<id>:paper returns a Unix second; compare it with the exam's end, not with now (curriculum row's tip, extended: say where the queue is, then when it comes back). When: step 11, at T−5. Status: EXECUTED.

Source

Redis, PostgreSQL and node-postgres behaviour can change: the facts below are as of 2026-10-06. No item is keyed on a source; every key is executed. Valid as of 2026-10-06.

Redis documentation, INFO: https://redis.io/docs/latest/commands/info/ (redis-info.txt). Quoted: expired_keys "Total number of key expiration events"; keyspace_hits "Number of successful lookup of keys in the main dictionary"; keyspace_misses "Number of failed lookup of keys in the main dictionary"; evicted_keys "Number of evicted keys due to maxmemory limit"; commandstats "cmdstat_XXX:calls=XXX". Supports steps 3, 5–6.
Redis documentation, EXPIRE: https://redis.io/docs/latest/commands/expire/ (redis-expire.txt). Quoted: "Redis keys are expired in two ways: a passive way and an active way." "A key is passively expired when a client tries to access it and the key is timed out." Supports step 5: the key is removed once, by whichever way comes first.
Redis documentation, EXPIRETIME: https://redis.io/docs/latest/commands/expiretime/ (redis-expiretime.txt). Quoted: "Returns the absolute Unix timestamp (since January 1, 1970) in seconds at which the given key will expire." Supports step 11.
Redis documentation, TTL: https://redis.io/docs/latest/commands/ttl/ (redis-ttl.txt). Quoted: "The command returns -2 if the key does not exist."
Redis documentation, SET: https://redis.io/docs/latest/commands/set/ (redis-set.txt). Quoted: "Only set the key if it does not already exist." (NX); "Any previous time to live associated with the key is discarded on successful SET operation."
PostgreSQL 17 documentation, F.32 pg_stat_statements: https://www.postgresql.org/docs/17/pgstatstatements.html (pg-stat-statements.txt). Quoted: calls "Number of times the statement was executed"; shared_blks_hit "Total number of shared block cache hits by the statement".
PostgreSQL documentation, pg_locks: https://www.postgresql.org/docs/current/view-pg-locks.html (pg-locks.txt), the curriculum row's second source. Quoted: granted "True if lock is held, false if lock is awaited". Supports step 2's locked-database hypothesis, which db-connection-pooling rules out; no item here turns on it.
node-postgres, pg.Pool: https://node-postgres.com/apis/pool (node-postgres-pool.txt), the row's third source. Quoted: connectionTimeoutMillis "Number of milliseconds to wait before timing out when connecting a new client." The pool's queue and its timeouts are the record's; the pools here have no acquire timeout, so every request in a run completes.
Vattani, Chierichetti and Lowenstein, Optimal Probabilistic Cache Stampede Prevention, PVLDB 8(8), 2015: https://www.vldb.org/pvldb/vol8/p886-vattani.pdf, the row's first source. Fetched as a 6-page PDF (sources/raw/vattani.pdf, sha256 fb581a20b929…); no text was extracted (this machine has no PDF text tool), so it is quoted nowhere, supports no step and keys nothing. Early refresh belongs to the record.

Checked against the tools and docs as of 2026-10-06.

Predict first