Retries with exponential backoff and jitter

Every household bakes in the one oven, and a cook who finds it full carries her dough home and tries again later.

At the oven	In the system
the oven is only full, so another walk can work; an oven cold for repairs all month is not worth one	transient vs permanent failures (step 1)
asking again for a slot does no harm; paying the baking fee twice does	safe to repeat (step 2)
each time she is turned away, she waits twice as long before the next walk	exponential backoff (step 3)
everyone turned away at seven comes back together at ten past, then half past	waves (step 4)
the baker's rule: draw a random minute inside your wait	jitter (step 6)
she sends her son, who sends a neighbour's child, each told "try three times"	layers multiply (step 7)
a notice on the bakehouse door: repeat trips, at most one in ten	a retry budget (step 10)

In real terms: a retry is extra load. Aim it at transient failures of repeatable work, spread it in time, allow it at one place in the stack, and cap its share.

Breaks down when: cooks see the smoke from the chimney before they set out and can wait on their own. A client sees nothing until its attempt fails. And the baker can call "come back at twenty to"; a server's Retry-After is sd-overload-protection's lane.

Contrasting case: a cook who never comes back after one refusal. She adds no load, but every busy moment costs her a loaf: without retries each transient failure reaches the user.

4.1 ExamHall: the live feed restarts (EXH-1720, MR !1920)

At 09:00 on ECON-201 day the live-feed server restarted. Its 60 proctor consoles lost the stream together. On the virtual clock the server is down for 5 s, then accepts 15 reconnects a second while it warms up and answers 503 to the rest. The backoff starts at 1 s, doubling, capped at 30 s:

4.1 60 consoles · feed down 0-5 s, then 15 reconnects accepted a second · backoff 1 s doubling, cap 30 s · seed 1720
4.1 strategy       attempts in each second 0-15                        | after 5 s: busiest second | all 60 back at | attempts
4.1 none          60 60  0 60  0  0  0 60  0  0  0  0  0  0  0 45 | 60 |  61.0 s | 330
4.1 full         130 38 29 18 18 16  8  7  8  7  6  6  0  0  0  1 | 16 |  20.2 s | 294
4.1 equal        120  9 51  0 17 30 13  0  3  0 12  6  7  4  0  0 | 30 |  13.6 s | 272
4.1 decorrelated  60 24 37 15  6 18 10 12  9  8  3  1  0  0  2  0 | 18 |  14.8 s | 205

Without jitter all 60 return in the same second, again and again; each wave gets 15 in and sends 45 back to wait (steps 3–4). Every jittered strategy spreads them, so the warming server is never handed 60 at once, and all 60 are back in a third of the time or less (step 6). Full jitter also tries more often in the first second, while the server is still down. MR !1920 switched the console to full jitter.

4.2 Proof domain, catalog: 27 queries for one page view

A category page's BFF calls the search service, search calls the inventory service, and inventory queries the stock database, which fails every query. Three real local servers above a database stub:

4.2 every layer 3 attempts (page BFF 3 -> search 3 -> inventory 3): stock-database queries per page view 27 · search receives 3 · inventory receives 9
4.2 only inventory retries (page BFF 1 -> search 1 -> inventory 3):  stock-database queries per page view 3 · search receives 1 · inventory receives 1

Three attempts at each of three layers is 3 × 3 × 3 at the bottom (step 7). Letting only the layer next to the database retry leaves 3 (step 8). This chain waits between attempts not at all: waits would change when those queries arrive, not how many (step 9).