Skip to content

Test-gap analysis

A test counts only for the changes it would fail on; plant the change and run.

Predict first

Before you read: predict

MR !3401 lets a proctor reopen a submitted ExamHall attempt for extra minutes, and refuses an attempt that has already been graded. The MR's two tests are green:

// src/proctor/reopen.test.ts (after the MR)
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { reopenAttempt } from './reopen.ts';

test('reopens a submitted attempt for the extra minutes', () => {
  const a = reopenAttempt({ id: 'a1', status: 'submitted', endsAt: 0 }, 10);
  assert.deepEqual(a, { id: 'a1', status: 'open', endsAt: 600_000 });
});

test('refuses a graded attempt', () => {
  try {
    reopenAttempt({ id: 'a2', status: 'graded', endsAt: 0 }, 10);
  } catch (err) {
    assert.equal((err as Error).message, 'attempt already graded');
  }
});

A reviewer deletes the if (attempt.status === 'graded') line and runs the two tests again.

What do they report?

Code for this question
// src/proctor/reopen.ts (after the MR)import type { Attempt } from '../types.ts'; export function reopenAttempt(attempt: Attempt, minutes: number): Attempt {  if (attempt.status === 'graded') throw new Error('attempt already graded');  return { ...attempt, status: 'open', endsAt: attempt.endsAt + minutes * 60_000 };}
Pick one

The 30-second version

Review the MR's tests as code. For each new path, find the test that reaches it; then ask the mutation question: if this line were deleted or flipped, would any test fail? Plant the change and run. A test still green there checks nothing; the gap comment names the input, the expected outcome and the change it catches.

A test counts only for the changes it would fail on; plant the change and run.

How it works

Step 1 of 12

Tests are code in the diff. Read them after the change they test, with the same care: they decide what the MR proves.
In the kitchen
  • she reads the test certificate as part of the work handed over, not as paperwork → tests are code in the diff (step 1)

Analogy: A clerk of works reviewing a plumber's pressure-test certificate on a new heating extension

On the site In the review
she reads the test certificate as part of the work handed over, not as paperwork tests are code in the diff (step 1)
she asks which new runs of pipe were under pressure during the test map each new path to the test that reaches it (step 2)
water ran through the new radiator branch, but the gauge sat before its valve reaching is not checking (steps 3, 7)
she loosens one new joint a quarter turn and watches the gauge plant the change and run (step 4)
the gauge holds, so the test could never have seen that joint leak green on a planted change is a gap (step 5)
"pump ran 15 minutes" says what was done; "held 6 bar for 15 minutes" says what was found assert the outcome, not the steps (step 8)
her note names the joint, the pressure and how long it must hold a gap comment names its test (step 12)

In real terms: a test proves only what it would fail on, so you find a gap by changing the code and watching whether anything fails.

Breaks down when: loosening a real joint risks a flood, while a planted code change costs one test run and is undone at once. And a gauge reads one pressure; a test can assert many things, each pinning different lines.

Contrasting case: an MR that only renames a local variable adds no path, so there is nothing new for a test to pin; the existing suite passing is the whole check.

Examples, run for real

4.1 ExamHall EXH-3100: the auto-submit that fired a second early

EXH-3100: ExamHall's auto-submit shipped with a green test. Candidates later reported their papers taken before the clock read zero. The MR as merged:

// src/attempts/auto-submit.ts (after the MR)
export function scheduleAutoSubmit(attempt: { id: string; endsAt: number }, submit: (id: string) => void): void {
  setTimeout(() => submit(attempt.id), Math.max(attempt.endsAt - Date.now(), 0));
}
// src/attempts/auto-submit.test.ts (after the MR)
import { test, mock } from 'node:test';
import assert from 'node:assert/strict';
import { scheduleAutoSubmit } from './auto-submit.ts';

test('submits the attempt when time is up', () => {
  mock.timers.enable({ apis: ['setTimeout', 'Date'], now: 0 });
  const submit = mock.fn();
  scheduleAutoSubmit({ id: 'a1', endsAt: 60_000 }, submit);
  mock.timers.tick(60_000);
  assert.equal(submit.mock.callCount(), 1);
  mock.timers.reset();
});

Plant the off-by-one, a delay one second short, and run both the MR's test and the reviewer's, which checks the clock at 59,999 ms and at 60,000 ms:

4.1 as merged: submit fires at 60,000 ms · MR's test passes · reviewer's test passes
4.1 planted (delay 1,000 ms short): submit fires at 59,000 ms · MR's test passes · reviewer's test fails
4.1 MR !3400 adds the reviewer's test: the planted change now fails a test

The MR's test asks only whether the submit happened by 60,000 ms, so firing at 59,000 ms passes it (steps 4–5). MR !3400 adds the reviewer's test, which pins the deadline from both sides (step 11).

4.2 Proof domain, chat: a suspicion a test clears, and a gap it does not

A chat MR stops notifying muted members unless they are mentioned. Its tests: a muted member is not notified; an offline member is.

// src/notify/should-notify.ts (after the MR)
import type { Member } from '../types.ts';

export function shouldNotify(member: Member, mentioned: boolean): boolean {
  if (member.muted && !mentioned) return false;
  return !member.online;
}
4.2 as merged: 2 tests: all pass
4.2 suspicion "no test pins the mute rule" · mute line deleted: 2 tests: fails `a muted member is not notified` · already caught, not posted
4.2 gap · `&& !mentioned` dropped: 2 tests: all pass · the missing test (muted, mentioned, offline -> notified) passes on the MR and fails on the change

The reviewer suspected the mute rule had no test; the planted deletion fails one, so that comment is dropped (step 12). The mention exception is a real gap: no test sends a mention, so dropping && !mentioned stays green.

Picture it

Concept illustration for this lesson.
An illustration to help it stick. The mechanism is in the sections above.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

An MR to a team chat app lets members edit a message: only its author, and only within 15 minutes of sending it. Both of the MR's tests pass:

The reviewer plants each change below on its own and runs both tests: X deletes the author check, Y deletes the 15-minute check, and Z drops editedAt: now from the reply.

Which changes leave both tests passing?

Code for this question
// src/messages/edit.ts (after the MR)import type { EditReply, Message } from '../types.ts'; const WINDOW_MS = 15 * 60_000; export function editMessage(m: Message, userId: string, text: string, now: number): EditReply {  if (m.authorId !== userId) return { status: 403 };  if (now - m.sentAt > WINDOW_MS) return { status: 409 };  return { status: 200, message: { ...m, text, editedAt: now } };}// src/messages/edit.test.ts (after the MR)import { test } from 'node:test';import assert from 'node:assert/strict';import type { Message } from '../types.ts';import { editMessage } from './edit.ts'; const sent: Message = { id: 'm1', authorId: 'u1', text: 'hi', sentAt: 0 }; test('the author edits their message', () => {  const r = editMessage(sent, 'u1', 'hello', 60_000);  assert.deepEqual(r, { status: 200, message: { ...sent, text: 'hello', editedAt: 60_000 } });}); test('a late edit is refused', () => {  const r = editMessage(sent, 'u2', 'hello', 20 * 60_000);  assert.notEqual(r.status, 200);});
Pick one

Question 2 of 7

A video platform's player picks the highest rendition that fits 80% of the measured bandwidth. The MR's test passes:

Three later commits each change one thing on top of the MR: X replaces the body of pickRendition with return ladder.findLast((r) => budget.fits(r, kbps)) ?? ladder[0];, Y changes kbps * 0.8 to kbps * 1.2, and Z replaces the body with return ladder.find((r) => budget.fits(r, kbps)) ?? ladder[0];.

Which commits does the MR's test fail on?

Code for this question
// src/player/pick-rendition.ts (after the MR)import type { Rendition } from '../types.ts'; export const budget = {  fits(r: Rendition, kbps: number): boolean {    return r.kbps <= kbps * 0.8;  },}; export function pickRendition(ladder: Rendition[], kbps: number): Rendition {  const fitting = ladder.filter((r) => budget.fits(r, kbps));  return fitting.at(-1) ?? ladder[0];}// src/player/pick-rendition.test.ts (after the MR)import { test, mock } from 'node:test';import assert from 'node:assert/strict';import type { Rendition } from '../types.ts';import { budget, pickRendition } from './pick-rendition.ts'; const LADDER: Rendition[] = [  { id: '360p', kbps: 800 }, { id: '540p', kbps: 1_800 }, { id: '720p', kbps: 3_500 }, { id: '1080p', kbps: 6_000 },]; test('checks every rendition against the bandwidth', () => {  const fits = mock.method(budget, 'fits');  pickRendition(LADDER, 5_000);  assert.equal(fits.mock.callCount(), 4);  mock.restoreAll();});
Pick one

Question 3 of 7

A box office's MR adds holds to reserve. Its one test passes:

Each proposed test starts with seat D4 held by b1 until 600,000.

Which one passes on the MR and would fail if the if line were later deleted?

Code for this question
// src/seats/reserve.ts (after the MR)import type { Seat } from '../types.ts'; const HOLD_MS = 10 * 60_000; export function reserve(seat: Seat, buyerId: string, now: number): Seat {  if (seat.heldBy && seat.heldBy !== buyerId && now < seat.holdUntil) throw new Error('seat held');  return { ...seat, heldBy: buyerId, holdUntil: now + HOLD_MS };}// src/seats/reserve.test.ts (after the MR)import { test } from 'node:test';import assert from 'node:assert/strict';import { reserve } from './reserve.ts'; test('a free seat is held for ten minutes', () => {  const s = reserve({ id: 'D4', heldBy: null, holdUntil: 0 }, 'b1', 0);  assert.deepEqual(s, { id: 'D4', heldBy: 'b1', holdUntil: 600_000 });});
Pick one

Question 4 of 7

A payments ledger's MR settles card payments with a 1.5% fee, RATE_BP 150 in basis points, rounded to the nearest cent. Its test passes:

The reviewer plants each change below on its own and runs the test: X sets RATE_BP to 1500, and Y returns net: amountCents + fee.

Which changes make the test fail?

Code for this question
// src/fees/settle.ts (after the MR)import type { Settlement } from '../types.ts'; export const RATE_BP = 150; export function feeFor(amountCents: number): number {  return Math.round((amountCents * RATE_BP) / 10_000);} export function settle(amountCents: number): Settlement {  const fee = feeFor(amountCents);  return { gross: amountCents, fee, net: amountCents - fee };}// src/fees/settle.test.ts (after the MR)import { test } from 'node:test';import assert from 'node:assert/strict';import { feeFor, settle } from './settle.ts'; test('settles a card payment', () => {  const r = settle(20_050);  assert.deepEqual(r, { gross: 20_050, fee: feeFor(20_050), net: 20_050 - feeFor(20_050) });});
Pick one

Question 5 of 7

MR !3403 adds ExamHall's late penalty: 5 marks off per started 10 minutes late, never below 0, and 0 marks after 30 minutes. Its only test is the on-time paper:

The reviewer plants five changes, one at a time: (1) > 30 becomes >= 30; (2) Math.ceil becomes Math.floor; (3) the Math.max(…, 0) floor is removed; (4) the if (minutesLate > 30) return 0; line is deleted; (5) 5 * becomes 10 *.

Write a late-mark.test.ts that passes on the MR and fails on each of the five planted changes.

Code for this question
// src/grading/late-mark.ts (after the MR)export function lateMark(raw: number, minutesLate: number): number {  if (minutesLate <= 0) return raw;  if (minutesLate > 30) return 0;  return Math.max(raw - 5 * Math.ceil(minutesLate / 10), 0);}// src/grading/late-mark.test.ts (after the MR)import { test } from 'node:test';import assert from 'node:assert/strict';import { lateMark } from './late-mark.ts'; test('an on-time paper keeps its mark', () => {  assert.equal(lateMark(80, 0), 80);});

Question 6 of 7

A payments ledger's withdraw refuses an overdraft. Two versions of its test, X and Y, both pass on the MR:

A later commit moves the debit above the check: withdraw first runs account.balanceCents -= amountCents;, then throws InsufficientFunds when account.balanceCents < 0.

What do X and Y report on that commit?

Code for this question
// src/accounts/withdraw.ts (after the MR)import type { Account } from '../types.ts'; export class InsufficientFunds extends Error {} export function withdraw(account: Account, amountCents: number): void {  if (account.balanceCents < amountCents) throw new InsufficientFunds(`balance ${account.balanceCents}`);  account.balanceCents -= amountCents;}// src/accounts/withdraw.test.ts (after the MR)import { test } from 'node:test';import assert from 'node:assert/strict';import { InsufficientFunds, withdraw } from './withdraw.ts'; test('X: refuses an overdraft', () => {  const acct = { id: 'acc-9', balanceCents: 300 };  assert.throws(() => withdraw(acct, 500));}); test('Y: refuses an overdraft and keeps the balance', () => {  const acct = { id: 'acc-9', balanceCents: 300 };  assert.throws(() => withdraw(acct, 500), InsufficientFunds);  assert.equal(acct.balanceCents, 300);});
Pick one

Question 7 of 7

A team chat shows "u1 is typing…" and hides it 5 seconds later. The MR's test passes:

The reviewer plants each change below on its own and runs the test: X sets TYPING_MS to 500, Y sets it to 50_000, and Z deletes the setTimeout line.

Which changes make the test fail?

Code for this question
// src/presence/typing.ts (after the MR)import type { TypingView } from '../types.ts'; export const TYPING_MS = 5_000; export function showTyping(userId: string, view: TypingView): void {  view.show(userId);  setTimeout(() => view.hide(userId), TYPING_MS);}// src/presence/typing.test.ts (after the MR)import { test, mock } from 'node:test';import assert from 'node:assert/strict';import { showTyping } from './typing.ts'; test('hides the typing indicator', () => {  mock.timers.enable({ apis: ['setTimeout'] });  const hide = mock.fn();  showTyping('u1', { show: () => {}, hide });  mock.timers.runAll();  assert.equal(hide.mock.callCount(), 1);  mock.timers.reset();});
Pick one

Go deeper

Pro tip
  1. Break the expectation before you trust a new test. Change its expected value to something wrong and run it once. A test that still passes never reached its assertion, or asserts something any result satisfies. When: a new test in an MR, before you count it as covering a path (steps 3, 7). Verify (RAN, tips.mjs):
    tip `submits the attempt when time is up` (§4): passes · expected 1 submit -> 2: fails, so its assertion runs and can fail
    tip `a muted member is not notified` (§4): passes · expected false -> true: fails, so its assertion runs and can fail
    tip `nobody in an empty room is notified` (assertion in a loop over an empty list): passes · expected false -> true: still passes, so it never asserted
    Status: RAN.
Source

Every gap this lesson grades is executed; the sources support the steps and key nothing. Valid as of 2026-10-06.

  • Stryker Mutator, What is mutation testing? (the curriculum row's first source), https://stryker-mutator.io/docs/ (stryker-docs.txt). Quoted: "Mutation testing introduces changes to your code, then runs your unit tests against the changed code. It is expected that your unit tests will now fail. If they don't fail, it might indicate your tests do not sufficiently cover the code." Supports steps 4–5: this lesson asks the same question by hand, one planted change at a time; the tool and its score belong to test-coverage-vs-mutation.
  • Playwright, Best Practices (the row's second source), https://playwright.dev/docs/best-practices (playwright-best-practices.txt). Quoted: "Automated tests should verify that the application code works for the end users, and avoid relying on implementation details such as things which users will not typically use, see, or even know about such as the name of a function". Supports step 8.
  • Martin Fowler (Ham Vocke), The Practical Test Pyramid (the row's third source), https://martinfowler.com/articles/practical-test-pyramid.html (fowler-test-pyramid.txt). Quoted: "Don't reflect your internal code structure within your unit tests. Test for observable behaviour instead." Supports steps 8–9.

Checked against the tools and docs as of 2026-10-06.