Skip to content

Debugging production systematically

Symptom metric first, then what changed, then RED for services and USE for resources; test one hypothesis against the data before touching anything.

Predict first

Before you read: predict

At 10:40 a proctor reports that ExamHall's live alert feed "has lagged all morning". The feed's delivery histogram, read in 5-minute windows, gives:

window alerts delivered delivered more than 2 s late
09:45–09:50 1,210 3
09:50–09:55 1,190 2
09:55–10:00 1,205 4
10:00–10:05 1,200 3
10:05–10:10 1,195 236
10:10–10:15 1,210 241
10:15–10:20 1,190 238
10:20–10:25 1,205 244

The morning's change records:

time record entry
09:12 deploy log grading-api 4.1 rolled out to all pods
10:04 deploy log feed-gateway 2.7 rolled out to all pods
10:07 repository PR #431 "feed: batch tile updates" merged to main
10:31 config log proctor-console REFRESH_SECONDS changed from 5 to 10

Which entry should on-call look into first?

Pick one

The 30-second version

Start from the symptom's metric and date its onset. Ask what changed just before it: deploys, config, traffic. The deploy log, not main, says what runs. Read RED (rate, errors, duration) for each service and USE (utilization, saturation, errors) for each resource. Test one hypothesis at a time against that data before changing anything.

Symptom metric first, then what changed, then RED for services and USE for resources; test one hypothesis against the data before touching anything.

How it works

Step 1 of 13

Start from the symptom. Name what users see, then find the metric that shows it: which service, which requests, since when.
In the kitchen
  • the duty manager reads the lock-keepers' tally first: boats through each hour, boats turned back, minutes from bottom gate to top → the symptom's metric, and RED for the flight as a service (steps 1, 2, 6)

Analogy: A canal company's flight of six locks (inland waterways)

On a holiday weekend, boaters complain that the flight "takes all afternoon".

On the flight In production
the duty manager reads the lock-keepers' tally first: boats through each hour, boats turned back, minutes from bottom gate to top the symptom's metric, and RED for the flight as a service (steps 1, 2, 6)
she reads the works log and the notice board: a new paddle at lock 4, a booking rule, a hire firm's fleet out what changed: work, rules, traffic (steps 3, 8)
a paddle ordered for lock 4 but still in the workshop changes nothing merged is not deployed (step 4)
she walks the flight lock by lock: how long each chamber spent filling, how many boats wait below it, which gates jam USE for each lock (steps 9–11)
"lock 4's new paddle fills slowly" predicts longer fills at lock 4 and a queue below it only; she checks the tally before touching it one hypothesis, tested against data (step 12)
she changes one paddle setting and watches change one thing (step 13)

Breaks down when: a lock-keeper writes the tally by hand on the day; a service's metrics must be instrumented before the incident. And boats pass the locks in series, while a request fans out to several services at once (step 7).

Contrasting case: a boat broken down inside lock 4 is an outage, not a slow flight: clear the lock first (obs-incident-response-oncall), then investigate.

In real terms: read the metric, check the change records, read RED and USE, then test one idea against the export.

Examples, run for real

4.1 ExamHall EXH-2390: autosave after Tuesday's deploy

On Tuesday, candidates said answers saved slowly. The autosave service's RED and USE readings, the hour before and after its 8.3 rollout, and the deploy log's diff:

4.1 RED autosave, per minute, hour before -> hour after 8.3: requests 2400 -> 2410 · failed 0.2% -> 0.2% · mean duration 0.09 s -> 0.84 s
4.1 USE after 8.3: pod CPUs 41% busy, 0 waiting, 0 errors · DB pool 100% in use, 38 waiting, 0 timeouts · Postgres CPUs 35% busy, 0 waiting, 0 errors
4.1 deploy log 8.2 -> 8.3 (Tue 10:12) · diff answers/serialize.ts · queries per 25-answer autosave: 8.2 2, 8.3 26, MR !2690 2
4.1 hypothesis "the new serializer loads each answer on its own": more queries yes · pool waiting yes · Postgres waiting no · stands

RED says users wait longer while nothing fails (step 6). USE flags the database pool: every connection in use and callers queued, while the pods' CPUs and Postgres have nothing waiting (steps 9–10). The diff names one file (step 5). The hypothesis "the new serializer loads each answer on its own" predicts more queries per autosave and a queued pool with an idle database; the export shows both (step 12). MR !2690 loads the answers in one query (db-n-plus-one) and marks every rollout on the autosave graphs.

4.2 Ticketing: seat maps before an on-sale

At 18:52 a box office's seat map turned slow. Three hypotheses, each with what it predicts, against the last ten minutes' export:

4.2 cache TTL cut from 60 s to 5 s at 18:50 (config log): predicts hit share down, database queries up · observed hits 92% -> 31%, database queries 144 -> 1242 a minute · stands
4.2 database saturated: predicts queries waiting at the database · observed 0 waiting · fails
4.2 payment provider slower: predicts its duration up · observed 0.30 s -> 0.30 s · fails

Two predictions fail against the export; the cache hypothesis survives. The team sets the TTL back, one change, noted with its time, and watches the hit share recover (step 13).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

A box office's checkout request goes from edge to checkout-api. For each checkout, checkout-api calls inventory-api three times and then payments-api once, one call after another, with its own work in between. Since Thursday's release, buyers say paying for tickets is slow. Each service's RED, for the hour before the release and the hour after it:

service requests failed mean duration (sum ÷ count)
edge 36,000 → 36,100 0.1% → 0.1% 0.34 s → 1.52 s
checkout-api 36,000 → 36,100 0.1% → 0.1% 0.32 s → 1.50 s
inventory-api 108,000 → 108,300 0.0% → 0.0% 0.01 s → 0.06 s
payments-api 36,000 → 36,100 2.0% → 2.0% 0.12 s → 0.12 s

Where does most of each checkout's added time come from?

Pick one

Question 2 of 5

During Saturday's match, a video service's thumbnail workers fell behind: a new clip's thumbnail took 9 s to appear, against 2 s on a normal evening. A USE pass over one worker host, for the slow hour:

resource utilization saturation errors
CPUs 58% busy, in 5-minute averages 6 runnable threads waiting, on average 0
memory 91% in use 0 pages swapped 0 failed allocations
network interface 35% of its bandwidth 0 packets dropped 0
disk 20% busy 0 requests queued 0

Which conclusion does the pass support?

Pick one

Question 3 of 5

A shop's refund service has been slow since Wednesday afternoon. Its deploy log for the week:

when build commit
Mon 10:00 r-217 f8cdd8c
Wed 14:00 r-218 2139288

git log --oneline main, newest first, with the one file each commit changed:

commit message file
74cccd1 docs: refund FAQ docs/refunds.md
f18a39a limits: one query per refund refunds/limits.ts
2139288 fx: fetch the rate for each item refunds/fx.ts
d73f179 webhooks: add jitter refunds/webhook.ts
f8cdd8c refunds: show the reason code refunds/reasons.ts

The on-call engineer is about to read the newest refund code on main.

Which files' changes should she read first?

Pick one

Question 4 of 5

Since 13:00, a chat service's mean response time has risen from 0.06 s to 0.22 s. Its deploy and config logs have no entry since Monday. Requests per minute and mean duration (sum ÷ count) per endpoint, for the hour before 13:00 and the hour after:

endpoint requests per minute mean duration
POST /messages 6,000 → 5,100 0.04 s → 0.04 s
GET /messages 3,000 → 2,700 0.06 s → 0.06 s
GET /search 100 → 1,300 1.20 s → 1.25 s

What changed at 13:00?

Pick one

Question 5 of 5

Card payments at a box office have been slow since Monday's release, which changed how the fraud score is fetched. Payments now go through the backup processor, so nobody is waiting. On a laptop the release's build answers in 40 ms with the team's test data, and the slowdown does not appear. An engineer suspects the fraud-score call.

What should the team do first to test that suspicion?

Pick one

Go deeper

Pro tips
  1. When a Node service's memory climbs until it restarts, compare the heap after each full garbage collection across hours: a floor that keeps rising means objects are being kept alive; a floor that returns does not. When: a process restarted every few hours by its process manager, with 502s around each restart. Where the objects are kept is js-memory-leaks-gc. Verify:
    tip heap floor after a full GC, rose by more than 1 MB: keeps each payload 5 of 5 hours · drops it 0 of 5 hours
    Status: RAN. Heap sizes vary between runs, so the script prints only the direction of the floor.
Source

Valid as of 2026-10-01.

  • Brendan Gregg, The USE Method: https://www.brendangregg.com/usemethod.html (usemethod.txt). The curriculum row's first source. Quoted: "For every resource, check utilization, saturation, and errors."; utilization is "the average time that the resource was busy servicing work"; saturation "the degree to which the resource has extra work which it can't service, often queued"; "A burst of high utilization can cause saturation and performance issues, even though utilization is low when averaged over a long interval."; "Saturation: any degree of saturation can be a problem (non-zero)."; "100% utilization is usually a sign of a bottleneck (check saturation and its effect to confirm)."; memory capacity saturation is "anonymous paging or thread swapping"; "Errors: non-zero error counters are worth investigating, especially if they are still increasing while performance is poor."; thread pools: "saturation by the number of requests waiting to be serviced by the thread pool".
  • Google SRE book, Effective Troubleshooting (Chris Jones): https://sre.google/sre-book/effective-troubleshooting/ (sre-effective-troubleshooting.txt). The row's second source. Quoted: monitoring metrics "are a good place to start figuring out what's wrong"; "Recent changes to a system can be a productive place to start identifying what's going wrong."; production logging "to track new version deployments and configuration changes at all layers of the stack"; "a configuration change or a shift in the type of load served"; "We can compare the observed state of the system against our theories to find confirming or disconfirming evidence."; changes "in a systematic and documented fashion"; reproducing at will "can be difficult to do in production systems"; "latching on to causes of past problems".
  • Node.js docs, Diagnostics: Memory: https://nodejs.org/en/learn/diagnostics/memory (node-diagnostics-memory.txt). The row's third source. Quoted: "The user observes continuously increasing memory usage (can be fast or slow, over days or even weeks) then sees the process crashing and restarting by the process manager."; "Increased GC activity leads to higher CPU usage and slower response time". Supports the pro tip.
  • Grafana Labs blog, The RED Method: How to Instrument Your Services (Tom Wilkie's talk, 2018): https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/ (grafana-red-method.txt). Not a source the curriculum row names, so it keys nothing. Background for step 6: "Rate (the number of requests per second)", "Errors (the number of those requests that are failing)", "Duration (the amount of time those requests take)"; "The USE Method doesn't really apply to services".

Related lessons

Checked against the tools and docs as of 2026-10-01.