On-call load, toil and alert hygiene

Half the time for engineering. Page only what a person must act on now. Count the load and engineer the toil away.

Before you read: predict

A payments ledger's on-call was paged 46 times in last week's 14 shifts. 24 of the pages came from one rule: the reporting replica's disk more than 80% full. Each time, the on-call opened the graphs, saw that the nightly clean-up job would free the space by 04:00 as it does every night, and went back to sleep; postings and reports were never affected.

In the alert review, what should happen to this rule?

The 30-second version

Keep operational work under half of each engineer's time, and treat toil, hands-on repeated work that grows with the service, as a cost to engineer away. Page a person only when they are needed within minutes and can act; the rest becomes tickets or goes. Count the load weekly, and plan the way back when it runs over.

Half the time for engineering. Page only what a person must act on now. Count the load and engineer the toil away.

How it works

Step 1 of 13

Half for engineering. Operational work stays under 50% of each engineer's time, averaged over quarters; the rest goes to projects cutting toil or adding features.

In the kitchen

the superintendent keeps half the engineers' hours for planned improvements, not watches and dips → the cap on operational work (step 1)

Analogy: The engine room of a cargo ship at night

On many nights the engine room is left unmanned, and the duty engineer is called to it by alarm.

In the engine room	On call
the duty engineer sleeps in her cabin, and the alarm system calls her down only for faults that will not wait for morning	a page is urgent (step 8)
each alarm has a card at the control console saying what to check and what to do	a page has an action (step 9)
a bilge alarm sounds whenever the ship rolls; she finds nothing each time, and starts going down slowly	noise hides the real page (step 10)
a fuel-filter warning goes on the morning job list, not the cabin call	a ticket (step 8)
one burst pipe sets off five alarms	one page per incident (step 11)
the chief engineer logs each night's calls and reads the count every week	counting the load (step 12)
an engineer dips every tank each morning with a sounding tape and writes the levels down; a bigger ship has more tanks	toil, growing with the service (steps 2, 6)
remote tank gauges, fitted once, end the morning dips	engineering it away (step 7)
the superintendent keeps half the engineers' hours for planned improvements, not watches and dips	the cap on operational work (step 1)

Breaks down when: a ship's classification rules fix which alarms it must carry; a team chooses its own alerts, and deletes the ones that ask nothing.

Contrasting case: an engine room with an engineer on watch around the clock: nobody is ever woken, but every hour of every night is spent watching.

In real terms: page someone only when they must act now and can; count the pages; and spend at least half the time removing the work that keeps coming back.

Examples, run for real

The counts are a labelled fixture: one exam week as counted below, not data from any real team.

4.1 ExamHall EXH-4380: exam season doubles the pager

During ECON-201's exam week the proctoring on-call took 56 pages in 14 twelve-hour shifts, against 28 in a term week. Half were "video upload slow": recordings reached storage late but complete, and nothing could be done before the morning's capacity check.

4.1 EXH-4380 exam week 56 pages (a term week 28) · "video upload slow" 28, none needing action before morning · per shift median 4, most 6 · shifts over the 2-page budget 13 of 14
4.1 MR !4680 upload alerts to the morning ticket queue (28 tickets), a runbook on each of the 2 rules still paging · 28 pages · per shift median 2, most 4 · shifts over the budget 3 of 14

MR !4680 moved the upload alerts to the ticket queue and wrote a runbook for each rule still paging: proctoring stream down and grading queue stuck (steps 8–9). The weekly review counts pages per shift against a budget of 2 pages per shift (the review's choice, after the prerequisite's ceiling of two incidents); three shifts are still over it, so the review goes on (step 12).

4.2 A payments-ledger rotation's toil floor

A ledger team choosing its rotation size computes the floor (step 5):

4.2 toil floor, one primary and one secondary week per cycle · 4 people 50% · 6 people 33% · 8 people 25% · left under the 50% cap for other operational work: 0 points, 17 points, 25 points

With four people, on-call alone fills the cap; at eight there is room for interrupts and releases.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

A ticket shop's on-call week had 14 twelve-hour shifts and 9 incidents: four in shift 3, two in shift 8 and one each in shifts 1, 5 and 11. Each incident fired 3 alerts, and each alert paged: checkout errors and payment-gateway timeouts within a minute of each other, then order-queue depth about ten minutes later as the backlog grew. The weekly review flags every shift with more than 2 pages.

Had the team grouped its alerts as recommended, how many of the week's shifts would the review have flagged?

Question 2 of 7

An engineer on a payments-ledger team logged a fortnight of 80 working hours; her team counts an on-call week as toil in full. In week 1 she was the primary on-call, 40 hours. In week 2 she started the failed-settlement import by hand on each of the 5 mornings, spending 36 minutes each time starting it and checking its output, while the import itself ran for 3 hours. On Friday she spent 2 hours running, once, a repair script that corrected the bank mappings behind the failures, and the import has not failed since. The rest of week 2 went on team meetings, 4 hours, and building the ledger's new reconciliation export, 31 hours.

What share of her fortnight was toil, to the nearest percent?

Question 3 of 7

A video-streaming team lists four pieces of last month's work.

Which of them is toil?

Question 4 of 7

A chat company's team renews each business customer's single-sign-on certificate by hand once a quarter, 9 minutes each, and sets up each new business customer by hand, 30 minutes each. Last quarter started with 400 business customers and added 60: 60 hours of renewals and 30 hours of set-ups. Next quarter starts with 460 business customers, and sales expect 120 new ones.

How many hours of this work will next quarter need?

Question 5 of 7

A ticket shop's team does three jobs by hand: re-sending e-tickets that bounced (1 minute each, 400 a week), refunding orders for cancelled events (20 minutes each, 45 a week) and rebuilding a venue's seat map after the venue edits it (4 hours each, one every two weeks). One engineer can automate one of them now, taking 1 week for the re-sends, 4 weeks for the refunds or 2 weeks for the seat maps; a job stays by hand until its automation ships.

Which automation removes the most hours of hand work over the next 26 weeks?

Question 6 of 7

In an interview you say your payments-ledger team keeps operational work under half its time. The interviewer asks: "A developer team's new release tripled your pages: real posting failures, each needing a failover. Their fix needs two quarters of re-architecture, and your team is already at 70% operational work. What do you set up until it lands?" You offer one of three plans.

Which plan survives the next question?

Question 7 of 7

A chat service's on-call took 33 pages last week from five alert rules: delivery failing for over 2% of sent messages: 3 pages; each time users' messages stopped arriving and the on-call drained a broken node within 10 minutes. The API's TLS certificate expiring within 14 days: 7 pages, one a night; the on-call renewed it the following Tuesday afternoon. Push-provider latency above 3 seconds: 15 pages; users' notifications arrived up to 30 seconds late, nothing on the team's side could change it, and the provider recovered each time within 20 minutes. The presence service losing one of its six replicas: 4 pages; the orchestrator replaced the replica within a minute and users saw nothing. The nightly analytics export failing: 4 pages; the on-call re-ran it the next morning, and nothing users see reads it. A review keeps paging only the rules that ought to page.

How many of last week's 33 pages would still have paged?

Go deeper

Pro tips

Before an alert review, list each rule's pages with what the on-call did after each one, and how soon; the rule's fate follows from that column. When: the row's typical question "Which of these alerts would you delete, and why?" (steps 8–9). Verify:
```
tip ExamHall exam week, what the on-call did after each page · "video upload slow" 28 pages, acted on within the hour 0 · stream down and grading stuck 28 pages, acted on within the hour 28
```
Status: RAN.

Source

quote-check.mjs finds every passage below verbatim in its saved text. Valid as of 2026-10-15.

Google SRE book, chapter 11, "Being On-Call" (Andrea Spadaccini; O'Reilly 2016, published by Google under CC BY-NC-ND 4.0): https://sre.google/sre-book/being-on-call/ (sre-being-on-call.txt). The row's first source. Quoted: "We cap the amount of time SREs spend on purely operational work at 50%" · "When on-call, an engineer is available to perform operations on production systems within minutes, according to the paging response times agreed to by the team and the business system owners." · "Nonpaging production events, such as lower priority alerts or software releases, can also be handled and/or vetted by the on-call engineer during business hours." · "Paging alerts should be aligned with the symptoms that threaten a service's SLOs." · "All paging alerts should also be actionable." · "Low-priority alerts that bother the on-call engineer every hour (or more frequently) disrupt productivity, and the fatigue such alerts induce can also cause serious alerts to be treated with less attention than necessary." · "Let’s define an incident as a sequence of events and alerts that are related to the same root cause and would be discussed as part of the same postmortem." · "Noisy alerts that systematically generate more than one alert per incident should be tweaked to approach a 1:1 alert/incident ratio." · "Ideally, symptoms of operational overload should be measurable, so that the goals can be quantified (e.g., number of daily tickets < 5, paging events per shift < 2)." · "The SRE team and leadership are responsible for including concrete objectives in quarterly work planning in order to make sure that the workload returns to sustainable levels." · "In such cases, the SRE team should not be subject to an excessive operational load."
Google SRE book, chapter 5, "Eliminating Toil" (Vivek Rau; same book and licence): https://sre.google/sre-book/eliminating-toil/ (sre-eliminating-toil.txt). The row's second source. Quoted: "Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows." · "Overhead is often work not directly tied to running a production service" · "Administrative work not tied directly to running a service. Examples include hiring, HR paperwork, team/company meetings, bug queue hygiene, snippets, peer reviews and self-assessments, and training courses." · "Grungy work can sometimes have long-term value, and in that case, it's not toil, either." · "If you're performing a task for the first time ever, or even the second time, this work is not toil." · "Running a script may be quicker than manually executing each step in the script, but the hands-on time a human spends running that script (not the elapsed time) is still toil time." · "Handling pager alerts is toil." · "If your service remains in the same state after you have finished a task, the task was probably toil." · "If the task produced a permanent improvement in your service, it probably wasn’t toil" · "If a machine could accomplish the task just as well as a human, or the need for the task could be designed away, that task is toil." · "If the work involved in a task scales up linearly with service size, traffic volume, or user count, that task is probably toil." · "An ideally managed and designed service can grow by at least one order of magnitude with zero additional work, other than some one-time efforts to add resources." · "Our SRE organization has an advertised goal of keeping operational work (i.e., toil) below 50% of each SRE's time." · "toil tends to expand if left unchecked and can quickly fill 100% of everyone's time." · "A typical SRE has one week of primary on-call and one week of secondary on-call in each cycle" · "the lower bound on potential toil is 2/6 = 33% of an SRE's time. In an 8-person rotation, the lower bound is 2/8 = 25%." · "Every SRE needs to spend at least 50% of their time on engineering work, when averaged over a few quarters or a year."

Checked against the tools and docs as of 2026-10-15.

Predict first