Change safety: blast radius, detection, the way back

Before shipping, name the blast radius, the metric already counting, the way back and the exposure limit; roll back before fixing forward.

Before you read: predict

ExamHall's candidate API runs on 20 pods. At 14:00 on a Tuesday, build 7.5 reached all of them. 7.5 ran no migration and changed no stored format, and this service's pipeline has run rollbacks before. At 14:20 the autosave failure rate is twice its usual level. The on-call engineer has found the cause, a one-line typo in 7.5, and says he can write, review and deploy the corrected line.

What is the first step?

The 30-second version

Before a production change, answer four questions. Blast radius: what does it reach if it is wrong? Detection: which metric would show it, already counted before the change? Undo: how do you get back in minutes? Roll back first; fix forward only when rollback is unsafe or slower. Exposure: which limit (flag, canary, one tenant, off-peak) actually contains it?

Before shipping, name the blast radius, the metric already counting, the way back and the exposure limit; roll back before fixing forward.

How it works

Step 1 of 12

Four questions. Answer each before the change ships. Google's SRE book finds that roughly 70% of outages follow changes to a live system.

Analogy: A city traffic engineer retiming a corridor's lights

Before a new timing plan goes into the controllers, the engineer writes the change's plan on one sheet.

On the corridor	In the change review
the junctions retimed, and the side streets whose queues back up into them	blast radius (steps 2–3)
stop-line detectors that have counted the queues for weeks, so "after" has a "before"	detection, already counting (steps 4–5)
the old plan still stored in each controller, one command from the control room	a rollback to a known-good state (step 6)
re-timing on the street during the jam, by an engineer sure of the fault	a forward fix under pressure (step 7)
a repainted turning lane that the old plan's phases no longer fit	the flip (step 8)
switching to the new plan and back on a test controller first	rehearsing the way back (step 9)
one junction first, on a Sunday morning	limited exposure (steps 10–12)

In real terms: before a change ships, write who it reaches, what would show it failing, how you get back and who meets it first.

Breaks down when: a jam is visible from the control room, while a bad build can fail without a sound; only a metric that moves shows it (step 4). Queues also clear once the old plan returns, but data written in a new format stays.

Contrasting case: a diversion set up because a water main burst is not a planned change, and it gets no review sheet. That is incident response (obs-incident-response-oncall).

Examples, run for real

4.1 ExamHall EXH-2160: the same bad build, three ways to meet it

EXH-2160: autosave build 7.5 went to every candidate API pod at 09:05 on a Monday, during ECON-201, and its failure counter shipped in the same build. MR !2460 is the review it should have had:

Question	MR !2460's answer
Blast radius	every autosave through the candidate API, for every institution
Detection	the autosave failure rate per build, its panel shipped in 7.4
Way back	roll back to 7.4, rehearsed on four staging pods
Exposure	1 of 20 pods for 15 minutes, Tuesday 14:00, outside exam windows

7.5 fails 1 autosave in 50, twice 7.4's rate; it runs 15 minutes, and each candidate online autosaves twice a minute.

4.1 EXH-2160 as shipped: 7.5 on 20 of 20 pods, Monday 09:05    candidates online 2400 · autosaves on 7.5 72000 of 72000 · failed on 7.5 1440
4.1 a 1-pod canary in the same exam window                     candidates online 2400 · autosaves on 7.5  3600 of 72000 · failed on 7.5   72
4.1 MR !2460: the 1-pod canary, Tuesday 14:00                  candidates online   60 · autosaves on 7.5    90 of  1800 · failed on 7.5    1

The canary cuts what meets 7.5 by its share of pods; moving it out of the exam window cuts it again, by the people online (steps 10, 12).

4.2 Proof domain, checkout: rolling back against fixing forward

Checkout build 4.3 fails 1 checkout in 5, and 300 checkouts arrive each minute. The alarm fires at minute 4. Rolling back to 4.2 is a 6-minute rolling update; the forward fix needs 9 minutes to write, 4 to review, 11 for CI and the same 6-minute deploy.

4.2 roll back to 4.2     deploy 4.2 6 min                                        · users safe from minute 10 · failed checkouts 600
4.2 fix forward as 4.4   write 9 min + review 4 min + CI 11 min + deploy 4.4 6 min · users safe from minute 34 · failed checkouts 2040

Both paths end with one deploy. The forward fix waits for writing, review and CI first, and users stay broken meanwhile (steps 6–7). Its deploy is also a build nobody has run in production.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

The shop's checkout service runs build 3.8 on 20 pods behind a load balancer that spreads checkouts evenly over them. Every build has counted each checkout in checkouts_total for a year. Build 3.9 changes the card-payment retry and adds a second counter, payment_failures_total. 3.9 has no defect: on any build, 1 checkout in 200 fails when the bank times out. At 14:00 3.9 goes onto 1 pod for a 15-minute canary, while 2,000 checkouts arrive each minute. metrics.count(name, filter) returns the total of the samples that match the filter. At 14:15 the gate runs once:

What does the gate do?

const rate = (build) => metrics.count('payment_failures_total', { build, since: '14:00' })  / metrics.count('checkouts_total', { build, since: '14:00' });if (rate('3.9') > 2 * rate('3.8')) rollBack('3.9'); else promote('3.9');

Question 2 of 6

ExamHall's API runs build 7.6 on 20 pods. Each night at 02:00 the results job emails every institution its graded results. It runs once a night, on whichever pod takes the job's lock first; any pod may. Build 7.7 changes how that job rounds grades. On each of the next 10 nights the team wants north-college, the pilot institution, to get 7.7's rounding, and the other 39 institutions 7.6's.

Which plan does that?

Question 3 of 6

The checkout service's build 6.0 saved each basket snapshot as plain JSON. Build 6.1 compresses each snapshot before saving it. Twenty minutes after 6.1 reached every pod, checkout errors doubled. The on-call engineer has found the cause, a mistyped VAT rate in 6.1's tax call, and a corrected build can be made, reviewed and deployed.

What is the first step?

Question 4 of 6

The shop's MR changes one file, money.js, where it edits roundPrice(). The API serves price quotes for all 40 tenants, and jobs/invoices.js runs at 01:00 each night for every tenant, storing each invoice it makes. The code that reaches money.js is below.

If the new roundPrice() is wrong, what does the change reach?

// money.js: the MR edits roundPrice() onlyexport const roundPrice = (pence) => Math.round(pence / 5) * 5;// fees.jsimport { roundPrice } from './money.js';export const withFees = (pence) => roundPrice(pence * 1.02);// api/quote.jsimport { withFees } from '../fees.js';export const quote = (req) => ({ total: withFees(req.pence) });// jobs/invoices.jsimport { withFees } from '../fees.js';export const invoices = (tenant) => tenant.orders.map((o) => ({ id: o.id, total: withFees(o.pence) }));

Question 5 of 6

A ticket shop's seat-map service runs build 4.7 on 12 servers that send each other a heartbeat every 5 seconds. Build 4.8 changes that period to 10 seconds, a change of one number. Staging runs one server per service.

Before 4.8 ships, which check shows whether rolling 4.8 back would go cleanly?

Question 6 of 6

A chat service runs build 9.3 on 20 pods, and every build answers a send with the handler below. Build 9.4 changes only encode(), and the team fears that 9.4's encode() could cut a message longer than 500 characters down to 500. About 1 message in 10 is longer than that, and on any build about 1 message in 500 ends in spaces, which trimEnd() drops, so it is stored a few characters short. 9.4 goes onto 1 pod for a 15-minute canary; the gate rolls it back if 9.4's reading is more than 1.5 times the 9.3 pods' reading.

Which reading should the gate use, so that it rolls back a 9.4 that cuts long messages and promotes one that does not?

async function send(text) {  const body = encode(text.trimEnd());  const id = await store.insert('messages', { body });  return { status: 200, id };}

Go deeper

Pro tips

Make the four answers required fields in the MR template, and let CI refuse a production deploy whose "Way back" field is empty or says only "revert". When: every change bound for production (steps 1, 6). Verify:
```
review-fields: OK   autosave MR
review-fields: FAIL results-email MR (Exposure empty; Way back says only "revert")
```

Source

The AWS article is a PDF, read through the r.jina.ai reader (its file carries Via: r.jina.ai). Valid as of 2026-10-01.

Jennifer Mace, Generic mitigations, O'Reilly Radar, 15 December 2020, the curriculum row's first source: https://www.oreilly.com/content/generic-mitigations/ (generic-mitigations.txt). Rollback "reverts your service—most commonly, your binary, but there are other options—to a known-good state"; "You don't need to fully understand your outage to use it"; "The most expensive stretch of an outage is the time when users can see it"; "Far too many think they have safe rollbacks, only to learn otherwise during an outage." Supports steps 6–7, 9, the opening question and t2.
Google SRE book, Effective Troubleshooting, the row's second source: https://sre.google/sre-book/effective-troubleshooting/ (sre-troubleshooting.txt). "Stopping the bleeding should be your first priority; you aren't helping your users if the system dies while you're root-causing." Background for steps 6–7 (mitigating before the cause belongs to obs-incident-response-oncall); no item is keyed on it.
PostgreSQL docs, CREATE INDEX, the row's third source: https://www.postgresql.org/docs/current/sql-createindex.html (pg-create-index.txt). "Normally PostgreSQL locks the table to be indexed against writes and performs the entire index build with a single scan of the table." Supports step 3 (as of 2026-10-01).
Google SRE book, Introduction: https://sre.google/sre-book/introduction/ (sre-introduction.txt). "SRE has found that roughly 70% of outages are due to changes in a live system"; progressive rollouts, detection and safe rollback "effectively minimizes the aggregate number of users and operations exposed to bad changes." Supports steps 1, 10.
Google SRE workbook, Canarying Releases: https://sre.google/workbook/canarying-releases/ (sre-canarying.txt). Patching and deploying a new version during an outage "will almost certainly prolong the user impact of the bug"; "the metric needs to be able to indicate problems in the service."
Sandeep Pokkunuri, Ensuring rollback safety during deployments, Amazon Builders' Library: https://d1.awsstatic.com/builderslibrary/pdfs/ensuring-rollback-safety-during-deployments.pdf (aws-rollback-safety.txt). "After the new version writes some compressed data, rolling back isn't an option"; rewriting old data is "backfilling", which "may require additional tooling that can run concurrently while the service is writing and reading data"; the three-stage upgrade-downgrade test; one-server test environments made "all deployments … atomic".

Checked against the tools and docs as of 2026-10-01.

Predict first

What is the first step?

The 30-second version

How it works

Analogy: A city traffic engineer retiming a corridor's lights

Examples, run for real

4.1 ExamHall EXH-2160: the same bad build, three ways to meet it

4.2 Proof domain, checkout: rolling back against fixing forward

Try it

What does the gate do?

Which plan does that?

What is the first step?

If the new roundPrice() is wrong, what does the change reach?

Before 4.8 ships, which check shows whether rolling 4.8 back would go cleanly?

Which reading should the gate use, so that it rolls back a 9.4 that cuts long messages and promotes one that does not?

Go deeper

Related lessons