Skip to content

Failure-mode thinking: what breaks at 10x and when X is down

Walk every dependency through down, slow, wrong, repeated and ten times the load; decide each one's degraded mode early, and rehearse it.

Predict first

Before you read: predict

An online shop's checkout confirms an order in one request. Part of that request calls the loyalty service, which adds the order's points to the customer's balance. On a sale morning the loyalty service stops answering for 20 minutes, and every call to it fails at once with a refused connection.

What should checkout do with an order placed in those minutes?

Pick one

The 30-second version

A dependency can be down, slow, wrong, repeated or swamped by ten times today's traffic. Walk each one through those states before launch, and decide what the product does without it: serve older data, hide the extra, or accept the user's action and finish its write later. Security checks fail closed. Rehearse each degraded path before it is needed.

Walk every dependency through down, slow, wrong, repeated and ten times the load; decide each one's degraded mode early, and rehearse it.

How it works

Step 1 of 14

More than up or down. A dependency can be down, slow, wrong, repeated or overloaded; a "noncritical" one that never answers can stall pages.
In the kitchen
  • an airspeed indicator that reads wrong is worse than a blank one, because the crew trusts it → wrong can hurt more than down (step 1)

Analogy: An airline's minimum equipment list

Before each departure, every broken item on the aircraft is checked against a list written long before: may the flight leave without it, and on what condition?

In dispatch and on the flight deck In a design
the list is written item by item, long before any flight, never at the gate each dependency's degraded mode, decided in advance (steps 2–3)
a broken seat-back screen: the flight leaves, the seat is labelled inoperative, and the fault is logged show the rest, and record it (steps 4, 8)
a broken galley oven: cold meals are loaded instead other data (step 3)
an airspeed indicator that reads wrong is worse than a blank one, because the crew trusts it wrong can hurt more than down (step 1)
a faulty weather radar: the flight may leave only if no storms are forecast on its route a safety check fails closed (step 7)
crews practise instrument failures in regular simulator checks rehearsing the degraded path (step 12)

In real terms: before launch, each dependency gets a line saying what users see when it fails, and what tells the team it has.

Breaks down when: an aircraft's list is fixed for its type and approved in advance; a service's dependencies change each release, and several can fail at the same moment (step 9).

Contrasting case: a car with no such list. The driver learns what a dead headlight means at night, on the road.

Examples, run for real

4.1 ExamHall EXH-2190: nothing was down

EXH-2190: ECON-201's launch review planned each dependency only for being down. The 09:00 start, replayed with the canonical numbers:

4.1 ECON-201 at 09:00:00 · 2,400 candidates · 6 API instances, all up · database up · pool 20 per instance · acquire timeout 2,000 ms
4.1 paper query 300 ms (the 09:00 load): per instance 400 arrive · 140 served · 260 gave up waiting for a connection
4.1 fleet: 840 served · 1,560 failed · components down 0
4.1 paper query 20 ms (idle): per instance 400 served · 0 failed

Every component was up. One query was slow, and that was enough (steps 1–2; the cause: cache-stampede-ttl-jitter; pool mechanics: db-connection-pooling). MR !2490 adds the five-state walk to the launch review.

4.2 Checkout: delivery rates pushed ahead

A checkout prices delivery from a rate service, which is down from 10:00 to 14:00. One design asks it on every order. The other has it push its whole rate table to every checkout instance each hour, and each order reads the local copy:

4.2 24 hours · 1,000 orders an hour · the rate service is down 10:00-14:00
4.2 asked on every order: orders priced 20,000 · failed 4,000
4.2 table pushed every hour, read locally: orders priced 24,000 · failed 0 · oldest table in use 4 h 59 min · pushes that ran 20 of 24

Read locally, the 09:00 table kept pricing through the outage. The push runs every hour, so the path checkout leans on is the one it uses every day, never one saved for emergencies (step 10).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

In ExamHall's admin console, the "Release results" action calls two services before it publishes an exam's marks: the role service, which says whether this staff account may release that exam's results, and the usage-metrics service, which counts the click for the product team's dashboard. Either can be unreachable for minutes, its calls failing at once.

What should the action do while each one is unreachable?

Pick one

Question 2 of 6

A shop's product pages carry a stock badge ("Only 3 left"). Each of its 40 web servers reads the badge from a cache process running beside it, and the caches are filled from the inventory database. A bad cache-process release can stop all 40 at once, and the inventory database serves at most 2,000 reads a second against 30,000 badge reads a second at peak. The team is writing down what a page does when its server's cache process fails; product pages must keep loading.

Which design?

Pick one

Question 3 of 6

MR !2491 gives exam start a degraded mode: when the paper service is unreachable at the start, each room's proctor switches the room to the printed papers held by the exam office. The mode has run only in unit tests, and the next sitting is in three weeks.

Before then, what tells the team whether the mode works?

Pick one

Question 4 of 6

A shop's order-history page lists a customer's last 20 orders, and the page server fills in each order's delivery status before it sends the page:

The tracking service usually answers in 10 ms. On a sale evening it stays up, but it answers only after 400 ms; every server stays up too.

How much longer than usual does an order-history page take?

Code for this question
for (const order of orders) {          // the customer's last 20 orders  order.status = await tracking.status(order.id, { deadline: 1000 });   // ms}
Pick one

Question 5 of 6

A ticket site keeps three caches in one Redis node, each in front of its own database. At today's peak:

cache          peak reads/s   hit ratio   its database serves at most
event pages    500            99%         700 queries/s
seat maps      1,500          95%         1,200 queries/s
search         6,000          85%         9,000 queries/s

At peak the Redis node is lost, and its replacement starts empty.

Which part of the site goes down?

Pick one

Question 6 of 6

A payments team ships one-click refunds on Monday. On Thursday the team lead books an hour with the six engineers and the support lead, to find the ways Monday's launch could fail.

How should she run the hour?

Pick one

Go deeper

Pro tips
  1. Write the degradation table into the design doc. One row per dependency: what users see when it fails, the timeout, the switch's name and its owner. Raising it for two dependencies, unprompted, is a senior signal in a design round. When: any design review or MR that adds a remote call (steps 2, 13). Status: DOC (the curriculum's pro tip; REL05-BP01: "For each failure mode, have a way to deliver most or at least the most critical functionality of the component to callers or customers."); not run in this session.
Source

The Builders' Library article is a PDF, turned into text with pdfjs-dist (extract-pdf.mjs); the HBR article's text sits in the page's JSON-LD articleBody. Both files carry a Via line. Valid as of 2026-10-01.

Related lessons

Checked against the tools and docs as of 2026-10-01.