Skip to content

Incident: Node memory climbing to OOM

OOMKilled is the container counting every byte: chart external beside heapUsed, find the long-lived holder in a snapshot, cut its reference.

Predict first

Before you read: predict

PAGE · ECON-201, minute 51 · a proctor gateway pod restarts; 600 proctor sockets reconnect to the other pods at once

Each gateway pod has a 1 GiB memory limit and holds about 600 proctor sockets during an exam. Since release 2026.10.02-1 the pods restart about every 50 minutes, and kubectl describe pod shows Last State: Terminated · Reason: OOMKilled · Exit Code: 137. The release's whole diff is a refactor of the connection handler:

On her laptop the engineer runs the release with 600 sockets open while 2,000 more connect, send one message and close. With a forced garbage collection before and after, she reads process.memoryUsage() each time.

Per closed socket, what has grown?

Code for this question
const alerts = new EventEmitter();             // one alert feed for the whole gateway wss.on('connection', (ws) => {  const inbox = Buffer.alloc(16 * 1024);       // this socket's read buffer  ws.on('message', (data) => data.copy(inbox));  alerts.on('alert', (alert) => ws.send(alert));});
Pick one

The 30-second version

OOMKilled, exit 137, is the container limit, which counts all resident memory, not only the V8 heap. Chart heapUsed, external and rss together: a Buffer's bytes are external, so a buffer leak climbs there while heapUsed stays flat. Diff two heap snapshots, follow a new object's retainers to its long-lived holder, and cut that reference.

OOMKilled is the container counting every byte: chart external beside heapUsed, find the long-lived holder in a snapshot, cut its reference.

How it works

Step 1 of 12

Read the kill first. OOMKilled, exit 137, is the container's memory limit. A full V8 heap logs "JavaScript heap out of memory" and aborts instead.
In the kitchen
  • the fire officer counts all stock against the lease's limit and shuts the shop → the container limit: OOMKilled, every socket dropped (steps 1–2)

Analogy: A high-street photo lab

Each customer who drops off a film gets a line in the counter book and an envelope on the back racks. Every order also adds a line to the shared "text me when the machine runs" list on the wall, naming that customer's envelope.

At the photo lab In the incident
a line in the counter book the socket's small objects in the V8 heap
the envelope of prints on the back racks the inbox Buffer's bytes, in external (step 4)
the wall list's line naming the envelope the listener on the shared emitter, holding the socket (step 7)
customers collect and leave; nobody strikes their line sockets close; the listener stays (step 8)
the owner checks the counter book's thickness, and it barely grows watching heapUsed alone (step 5)
the fire officer counts all stock against the lease's limit and shuts the shop the container limit: OOMKilled, every socket dropped (steps 1–2)
a thicker counter book --max-old-space-size (step 10)
a bigger shop a higher memory limit: the closing day moves (step 9)
striking the line when the customer leaves off on close (step 11)

Breaks down when: an owner would see the racks fill; Node shows the climb only to someone who charts external beside heapUsed.

Contrasting case: the same forgotten listener keeping each socket's last message as a string or a plain array. Now heapUsed climbs, and a --max-old-space-size below the container limit makes V8 abort first, with "JavaScript heap out of memory" and no OOMKilled.

In real terms: read the kill reason, chart the three lines, and look for the long-lived holder before you touch a limit or a flag.

Examples, run for real

4.1 ExamHall EXH-2970: the gateway on a laptop

Release 2026.10.02-1's handler on a real ws server: 600 sockets stay open while 2,000 connect, send one message and close. Then two heap snapshots 1,000 closes apart, and the hook's restart arithmetic:

4.1 the §2 handler, ws 8.22.0 · 600 sockets open, 200 warm-up and 2000 more connected, sent one message and closed
4.1 alerts.listenerCount('alert'): 2800 · wss.clients.size: 600 · MaxListenersExceededWarning printed: yes
4.1 per closed socket, after forced collections: heapUsed under +8 KiB · external about +16 KiB · arrayBuffers about +16 KiB
4.1 heap snapshot 2 minus snapshot 1, 1000 sockets closed in between: Buffer +2000 · Socket +1000 · WebSocket +1000 (each at least one per closed socket)
4.1 path | Buffer            ← inbox in system / Context | new since snapshot 1
4.1 path | system / Context  ← context in (closure) | new since snapshot 1
4.1 path | (closure)         ← [n] in Array | new since snapshot 1
4.1 path | Array             ← alert in Object | at snapshot 1
4.1 path | Object            ← _events in EventEmitter | at snapshot 1
4.1 path | EventEmitter      ← alerts in system / Context | at snapshot 1
4.1 first object going up that existed at snapshot 1: Array, the alert listener array of alerts
4.1 the hook's restart: (1024 MiB - 230 MiB after start) / ((16 + 2) KiB kept per close x 900 closes a minute) = 50.2 minutes

The listener count follows every socket ever opened, not the 600 open ones (step 8). The 16 KiB inboxes show in external while heapUsed hardly moves (steps 3–5). The snapshot path goes up from a closed socket's inbox to the alert array, the first object that existed before those sockets (step 7), and the arithmetic gives the hook's 50 minutes (step 9).

4.2 Payments ledger MR !3270: the listener removed on close

A wallet's balance feed shipped the same handler. MR !3270 names the listener and removes it when the socket closes:

wss.on('connection', (ws) => {
  const inbox = Buffer.alloc(16 * 1024);
  ws.on('message', (data) => data.copy(inbox));
  const onBalance = (balance) => ws.send(balance);
  balances.on('balance', onBalance);
  ws.on('close', () => balances.off('balance', onBalance));
});
4.2 MR !3270's handler, ws 8.22.0 · 600 sockets open, 200 warm-up and 2000 more connected, sent one message and closed
4.2 balances.listenerCount('balance'): 600 · wss.clients.size: 600 · MaxListenersExceededWarning printed: yes
4.2 per closed socket, after forced collections: heapUsed under +1 KiB · external under +1 KiB

Listeners now equal open sockets, and both lines return to their floor after each close (step 11). Node still warns about more than 10 listeners, because 600 open sockets are 600 listeners: the warning cannot tell a fan-out from a leak, and the count against open sockets can (step 12).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

Its pods have a 1 GiB memory limit and run with --max-old-space-size=512. Right after a start, rss reads 248 MiB and heapUsed 40 MiB. Per closed socket, external grows by 16 KiB and heapUsed by 3 KiB, and rss grows by exactly what those two gain. Clients close 1,000 sockets a minute.

How many minutes after a start is a pod OOMKilled, to the nearest minute?

Pick one

Question 2 of 7

A chat gateway is deployed on a VM with 8 GiB of RAM and no container limit, started with node --max-old-space-size=128 server.js. Its handler:

Overnight, 600 sockets stay open while 12,000 more connect, send one message and close.

What has happened by morning?

Code for this question
const rooms = new EventEmitter();              // one message feed for the whole gateway wss.on('connection', (ws) => {  const draft = Buffer.alloc(16 * 1024);       // this socket's unsent draft  ws.on('message', (data) => data.copy(draft));  rooms.on('message', (m) => ws.send(m));});
Pick one

Question 3 of 7

A ticketing gateway keeps one registry of the fans to tell when a seat frees up:

The engineer takes a heap snapshot, lets 1,000 fans connect, send one message and leave, and takes another. She picks one Buffer that is new in the second snapshot and reads its retainers upward:

Buffer            ← pending in system / Context
system / Context  ← context in (closure)
(closure)         ← [n] in (internal array)
(internal array)  ← table in Map
Map               ← byFan in Object
Object            ← registry in system / Context
… up to the GC roots

Which object's reference down this path is the one the fix must remove?

Code for this question
const registry = {  byFan: new Map(),                              // fan id -> how to reach that fan's open socket  add(fan, notify) { this.byFan.set(fan, notify); },  remove(fan) { this.byFan.delete(fan); },       // called by the logout route}; wss.on('connection', (ws, req) => {  const fan = new URL(req.url, 'http://gateway').searchParams.get('fan');  const pending = Buffer.alloc(16 * 1024);       // the seats this fan is choosing  ws.on('message', (data) => data.copy(pending));  registry.add(fan, (seat) => ws.send(seat));});
Pick one

Question 4 of 7

A streaming service's watch-party gateway gives each viewer a copy of the party's recent messages, for a replay button:

600 viewers stay connected while 2,000 more connect, send one message and leave.

With a forced garbage collection before and after, per closed socket, what has grown in process.memoryUsage()?

Code for this question
const party = new EventEmitter();              // one feed for the whole watch partyconst recent = Array.from({ length: 32 }, (_, i) => `line ${i} `.padEnd(512, '.'));   // the party's last 32 messages wss.on('connection', (ws) => {  const replay = recent.join('');              // this viewer's copy of the messages so far  party.on('seek', (to) => ws.send(`${to} ${replay.length}`));});
Pick one

Question 5 of 7

In the interview you are given EXH-2970's page and asked: "A Node service restarts every 50 minutes with OOMKilled.

How do you find the leak, and do you raise --max-old-space-size or the memory limit?" What do you answer, in at most 150 words?

Question 6 of 7

A payments ledger's balance gateway climbs toward its memory limit after a release. The on-call engineer replays the release's traffic on her laptop: 600 sockets stay open while 2,000 more connect, send one message and close. With a forced garbage collection before and after, she reads process.memoryUsage(). Per closed socket: heapUsed about +16 KiB; external under +8 KiB; arrayBuffers under +8 KiB.

Which cause does this readout support?

Pick one

Question 7 of 7

ExamHall's report worker renders each exam's results as CSV and keeps every rendered string for its "recent reports" page. It runs in a pod with a 256 MiB memory limit, started with node --max-old-space-size=512 worker.js, and each report is about 1 MiB of text:

Nothing else in the worker grows, and exams finish all day.

Left running, what happens to the worker?

Code for this question
const rendered = [];                             // every report rendered since the worker started function renderReport(exam) {  const csv = exam.rows.join('\n');              // this exam's results, one line per candidate  rendered.push({ exam: exam.id, csv });  return csv;}
Pick one

Go deeper

Pro tips
  1. Plot heapUsed, external and rss on one chart. When rss climbs, the other two say where: in the leak lab external grows about 16 KiB per closed socket while heapUsed stays under 8 KiB, which places the bytes outside the V8 heap (curriculum row's own tip). When: the first look at any memory page. Status: EXECUTED (rss itself is not graded).
  2. Export the listener count beside the open-socket count. alerts.listenerCount('alert') against wss.clients.size tells a leak from a large fan-out, which the MaxListeners warning cannot. When: steps 8 and 12, as a metric and an alert. Status: EXECUTED.
Source

Node.js and Kubernetes behaviour can change: the facts below are as of 2026-10-06. No item is keyed on a source; every key is executed. Valid as of 2026-10-06.

  • Node.js, Using Heap Snapshot: https://nodejs.org/en/learn/diagnostics/memory/using-heap-snapshot (node-heap-snapshot.txt), the curriculum row's first source. Quoted: "You can find a memory leak by comparing two snapshots."; "Take another heap snapshot. The difference between the two should mostly contain what was leaking."; "Look for large positive deltas and explore the references that caused them in the bottom panel." Supports step 7.
  • Node.js API, events, events.defaultMaxListeners: https://nodejs.org/api/events.html (node-events.txt), the row's second source. Quoted: "By default, a maximum of 10 listeners can be registered for any single event."; "This is not a hard limit. The EventEmitter instance will allow more listeners to be added but will output a trace warning to stderr indicating that a "possible EventEmitter memory leak" has been detected." Supports steps 8 and 12.
  • Kubernetes, Assign Memory Resources to Containers and Pods: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/ (k8s-assign-memory.txt), the row's third source. Quoted: "If the Container continues to consume memory beyond its limit, the Container is terminated."; its example output: "exitCode: 137" and "reason: OOMKilled". Supports steps 1–2.
  • Node.js API, process, process.memoryUsage(): https://nodejs.org/api/process.html (node-process.txt). Quoted: "heapTotal and heapUsed refer to V8's memory usage."; "external refers to the memory usage of C++ objects bound to JavaScript objects managed by V8."; "arrayBuffers refers to memory allocated for ArrayBuffers and SharedArrayBuffers, including all Node.js Buffers. This is also included in the external value."; "rss, Resident Set Size, is the amount of space occupied in the main memory device … for the process, including all C++ and JavaScript objects and code." Supports steps 3–5.
  • Node.js API, Command-line options, --max-old-space-size: https://nodejs.org/api/cli.html (node-cli.txt). Quoted: "Sets the max memory size of V8's old memory section." Supports step 10.
  • Node.js API, Buffer: https://nodejs.org/api/buffer.html (node-buffer.txt), saved for the Buffer and typed-array relation; no step rests on it alone.

Checked against the tools and docs as of 2026-10-06.