Version skew: every release works with N−1

Every release must read what its predecessor wrote, and write only what its predecessor can read, for as long as both can meet.

Before you read: predict

ExamHall's release 8 changes two payloads. A grade job gains rubricVersion; release 7's API queued jobs as { attemptId, answers }. An autosave now sends answers as an array of { q, a } pairs; release 7's exam page sent an object, such as { q1: 'B' }. Release 8's code:

The rollout replaces every API pod and every worker pod between 10:00 and 10:06. At 10:20 nine things arrive:

3 retries of grade jobs that failed on a database timeout at 09:58, each scheduled to run again at 10:20;
2 autosaves from candidates who opened the exam at 09:10 and have not reloaded it;
4 autosaves from candidates who opened the exam at 10:12.

How many of the nine succeed?

// grading workerfunction grade(job) {  const rubric = rubrics[job.rubricVersion];  return score(job.answers, rubric.key);} // autosave routefunction autosave(body) {  if (!Array.isArray(body.answers)) return 422;  return save(body.attemptId, body.answers);}

The 30-second version

No release replaces its predecessor at once. During a rollout and after a rollback, while pages stay open and jobs wait, old and new code read each other's payloads, caches and stored data. So each release must read what the previous one wrote, and write what it can read. Add fields, never change one; rewrite stored data later, alone.

Every release must read what its predecessor wrote, and write only what its predecessor can read, for as long as both can meet.

How it works

Step 1 of 11

Skew is normal. Old and new code run together: pods for minutes, open pages until reloaded, queued jobs until run, mobile apps for months.

In the kitchen

old and new notes in the same tills for months → both releases running together (step 1)

Analogy: A central bank brings in a new banknote

Banks and shops update their cash machines and tills before the first new note is issued. For months old and new notes circulate together, and the old note stays legal tender until a withdrawal date announced on its own, long after the last old note was paid out.

At the central bank	In the release
old and new notes in the same tills for months	both releases running together (step 1)
an updated till takes either note	new reads old (step 3)
a till that missed the update refuses the new note	old reads new (steps 4, 6)
machines updated before the first new note is issued	reader before writer (step 7)
old notes still in drawers and tins after the last one was paid out	the window's last holders (step 8)
the withdrawal date, announced separately, months later	the irreversible step, later and alone (step 10)

Breaks down when: a shopkeeper who refuses a note asks for another; an old pod handed a new payload fails the request, or misreads it and carries on. Notes never go back to an older design; a release can be rolled back.

Contrasting case: a script that keeps no state, started fresh for each run: one release runs at a time, and it reads nothing an older one wrote.

In real terms: between two releases, every shared payload, stored value and open page must work for both, and for the old one again after a rollback.

Examples, run for real

4.1 ExamHall EXH-2740: a new field in grade jobs during the rollout

ExamHall's release 8 adds rubricVersion to grade jobs. Four API pods and four worker pods are replaced one each minute, and each job runs a minute after it is queued. Release 7's workers destructure only the fields they use. MR !3040 makes release 8 read a missing rubricVersion as r1:

4.1 4 API pods and 4 worker pods replaced one each minute, 10:01-10:04 · 24 grade jobs, each run the minute after it was queued
4.1 release 8 as drafted (rubrics[job.rubricVersion]): failed 6 · 6 release 7 jobs on release 8 workers
4.1 release 8 with a default, but release 7 workers that refused unknown fields: failed 2 · 2 release 8 jobs on release 7 workers
4.1 MR !3040 (release 8 reads a missing rubricVersion as r1; release 7 ignores it): failed 0

Each direction fails on its own: new workers on old jobs (step 3), and old workers on new jobs when they refuse unknown fields (steps 4 and 6). MR !3040 needs only a default, because release 7 already ignores what it does not use.

4.2 Chat: a rollback with release 8's events still in the stream

Chat's release 8 renamed text to body in its message.edited events. It was rolled back while five of them were still in the stream, so release 7's consumer applied them:

4.2 release 8 rolled back with 5 message.edited events still in the stream; release 7's consumer applies them
4.2 release 8 renamed text to body: edits applied 5 · messages now blank 5 · errors 0
4.2 MR !3041 (body added, text still written): edits applied 5 · messages now blank 0 · errors 0

Release 7 read text, found nothing, and saved blank messages without an error (steps 5 and 9). MR !3041 adds body and keeps writing text; dropping text waits for a later release (step 10).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

A chat service adds polls in release 8. Its API sends a poll as { "roomId": "r9", "senderName": "Ana", "kind": "poll", "question": "Lunch at 1?", "choices": ["yes", "no"] }, and a text message, as before, as { "roomId": "r9", "senderName": "Ana", "kind": "text", "text": "hi all" }. The push notifier is a separate service; it stays on release 7 while the API's three pods move to release 8:

During the rollout some messages get no push at all.

Where is the bug?

function notify(msg) {  const { roomId, senderName, kind, text } = msg;  const preview = kind === 'image' ? 'sent a photo' : text.slice(0, 60);  for (const member of rooms.get(roomId)) push(member, `${senderName}: ${preview}`);}

Question 2 of 6

A video service keeps each viewer's resume point only in Redis, under resume:<user>:<video>. Release 7 writes { "positionSec": 734 } and reads it back with entry?.positionSec ?? 0. Release 8 adds chapters; as drafted, it writes { "position": { "sec": 734, "chapter": 3 } } under the same key and reads entry.position.sec. Its four pods are replaced one at a time, every request may land on any pod, and the plan keeps a rollback to release 7 ready.

Which change to release 8 makes every resume start where the viewer last stopped, during the rollout and after a rollback?

Question 3 of 6

A ticket shop's release 9 changes the checkout request from seats: ["A1", "A2"] to seats: [{ "row": "A", "n": 1 }, { "row": "A", "n": 2 }]. Three web pods serve both the checkout page and its API, and any page load or request may go to any pod. The rollout replaces one pod at 10:02, one at 10:04 and the last at 10:06; each new pod takes over the moment its old pod stops. A checkout page sends its request at most 30 minutes after it loaded. (Round up to the minute.)

Until what time can a request in release 8's seats shape still arrive?

Question 4 of 6

A shop keeps each product as one JSON record in a shared store. Release 7 writes stock as the number of units left; release 8 writes it as { "onHand": 4, "reserved": 1 }. Each release's badge code:

Midway through the rollout, with two of the four pods on release 8, a release 8 pod saves a lamp with 4 units on hand and 1 reserved. Then four shoppers open the lamp's page, and the load balancer sends the four page loads to the four pods, one each.

What do the shoppers see?

// release 7const badge = product.stock > 0 ? `In stock (${product.stock})` : 'Sold out';// release 8const left = product.stock.onHand - product.stock.reserved;const badge = left > 0 ? `In stock (${left})` : 'Sold out';

Question 5 of 6

A chat web app ran release 8 from 09:00. At 10:00 the team rolled every server, and the page bundle, back to release 7. Release 8 had made four changes:

it saved unsent drafts in localStorage under drafts as { "r9": { "text": "see you", "mentions": ["ana"] } }, where release 7 saves { "r9": "see you" };
it stored each new message row with an extra reactions column;
it added a third theme, dim, saved in the theme cookie;
it cached each user's room list in Redis under rooms:v8:<user>.

Release 7's page-load code includes:

After 10:00 some users' chat pages crash as they load.

Which of release 8's changes causes it?

const draft = (JSON.parse(localStorage.getItem('drafts')) ?? {})[roomId]?.trim() ?? '';const { id, senderName, text } = row;const theme = ['light', 'dark'].includes(cookies.theme) ? cookies.theme : 'light';const rooms = (await redis.get(`rooms:${userId}`)) ?? (await loadRooms(userId));

Question 6 of 6

A music service sends player events through a queue from its API to a stats worker. Release 7's API sends { "track": "t1", "ms": 30000 }, the milliseconds played, and its worker adds up event.ms. Release 8 reports whole seconds: its API sends { "track": "t1", "ms": 30, "unit": "s" }, and its worker reads each event with the line below. Both will roll out together, pod by pod.

Which check, run in CI before the release, fails?

const playedMs = event.unit === 's' ? event.ms * 1000 : event.ms;

Go deeper

Pro tips

Review every MR with two questions: what if the old release reads what this writes, and what if this release reads what the old one wrote? The curriculum row's tip; it covers payloads, cache entries, events and stored client state in one pass. When: any change to a shape that crosses a release boundary (step 2). Status: RAN (Node 24.19.0).

Source

No item is keyed on a source; every key is executed. Valid as of 2026-10-03.

Hyrum Wright, Hyrum's Law: https://www.hyrumslaw.com/ (hyrums-law.txt). Quoted: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." The row's first source; the law itself is taught in judgment-api-caller-view. Supports the note after step 11.
Danilo Sato, martinfowler.com, ParallelChange (2014): https://martinfowler.com/bliki/ParallelChange.html (fowler-parallel-change.txt). Quoted: "In the expand phase you augment the interface to support both the old and the new versions." Supports steps 7 and 10.
Martin Fowler, martinfowler.com, TolerantReader (2011): https://martinfowler.com/bliki/TolerantReader.html (fowler-tolerant-reader.txt). Quoted: "only take the elements you need, ignore anything you don't"; "Adding field to an interface like this shouldn't be a breaking change for anyone - but often does break these schemes." Supports steps 5–6.
Protocol Buffers documentation, Language Guide (proto 3), Updating A Message Type: https://protobuf.dev/programming-guides/proto3/ (protobuf-proto3.txt). Quoted, as of 2026-10-03: "messages created by your new code can be parsed by your old code: old binaries simply ignore the new field when parsing"; "adding a value to a preexisting enum would be a compilation break for any code with an exhaustive switch on that enum". Supports steps 4–6.
Kubernetes documentation, Deployments, Rolling Update Deployment: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ (k8s-deployment.txt). Quoted: "Deployment ensures that only a certain number of Pods are down while they are being updated." The row's third source; rolling updates are taught in devops-deploy-strategies. Supports step 1.
Kubernetes, Version Skew Policy: https://kubernetes.io/releases/version-skew-policy/ (k8s-version-skew.txt). Quoted, as of 2026-10-03: "the newest and oldest kube-apiserver instances must be within one minor version." A published N−1 rule. Supports the title and step 1.

Checked against the tools and docs as of 2026-10-03.

Predict first