Skip to content

Incident: database connection spike after pre-scaling

More pods bring more pools, not more slots: the refusals wait for the herd, and the fix is fewer connections, not a reload.

Predict first

Before you read: predict

POSTMORTEM DRAFT · ECON-201 · autosaves failing after the pre-scale

The stampede postmortem's action ran at 08:50: ExamHall's attempt API went from 6 pods to 12, ahead of ECON-201 at 09:00:00. No release went out. Each pod creates its pool once, when it starts:

Postgres runs with max_connections 150 and 3 superuser-reserved slots, and the grading workers, admin and monitoring hold 18 connections all morning. From 08:40 each pod has at most 2 autosaves in flight at once; at 09:00:00 each pod gets 30 at once. The on-call is writing the timeline from the pods' error logs.

When does the first autosave get a connection error?

Code for this question
export const pool = new Pool({ max: 20 });
Pick one

The 30-second version

Pre-scaling multiplies pools, not database slots. A pool connects only when requests need it, so the scale-up looks healthy until the herd; then every pod grows at once and Postgres refuses with 53300, old pods as well as new. Cut demand to the budget; a reload cannot raise max_connections.

More pods bring more pools, not more slots: the refusals wait for the herd, and the fix is fewer connections, not a reload.

How it works

Step 1 of 13

The change is the scale-up. No release went out; the extra pods changed only how many pools may each open their max.
In the kitchen
  • a club's allowance of 20 pitches → a pod's pool max (steps 1, 4)

Analogy: A caravan site during a club rally

The council licenses the site for 150 pitches and 3 stay free for the warden and first aid. Each caravan club may take up to 20. For the summer rally the organiser doubles the clubs from 6 to 12.

At the site In the incident
the council's 150 pitches, 3 kept for the warden max_connections and the reserved slots (step 5)
a club's allowance of 20 pitches a pod's pool max (steps 1, 4)
members trickle in all week, pitching only on arrival, so Monday looks fine pools connect on demand (steps 6–7)
Saturday morning: every club's full contingent at the gate at once the herd (step 8)
the steward sends each extra van to the gate, where it is turned away, instead of waiting for a club pitch a pool below its max failing requests on refusals (step 9)
the licence can be raised only after the council re-inspects a cleared site a start-time setting and its restart (step 10)
a per-club cap written at the gate a role's CONNECTION LIMIT (step 13)

Breaks down when: a warden can tell a driver how long the wait will be; Postgres refuses at once and says nothing about later.

Contrasting case: a pool too small for its traffic. Requests queue in the app and fail with the pool's acquire timeout, while Postgres has free slots and logs no refusal (db-connection-pooling failure mode 1).

In real terms: before a scale-up, count every process's pool against max_connections minus the reserved slots, because the herd will.

Examples, run for real

4.1 ExamHall EXH-2950: the pre-scale and the 09:00 herd

The stem's server and fleet, with the 09:00 autosaves dealt to the pods in turn, then the console's two database reads:

4.1 PostgreSQL 17.11 · max_connections 150 · superuser_reserved_connections 3 · reserved_connections 0 · grading workers, admin and monitoring 18 · each pod: new Pool({ max: 20 })
4.1 08:40 · 6 pods, 2 autosaves in flight each: examhall_api sessions 12 · refused 0
4.1 08:50 · 12 pods, 2 autosaves in flight each: examhall_api sessions 24 · refused 0
4.1 09:00 · 12 pods x 30 autosaves, dealt in turn: connected 129 · refused 231 · SQLSTATE 53300
4.1 the first refusal's text: remaining connection slots are reserved for roles with the SUPERUSER attribute
4.1 pg_stat_activity, client backends by role: examhall_api 129 · examhall_jobs 17 · postgres 1
4.1 budget: 150 - 3 - 18 = 129 for the API · the pools may ask for 12 x 20 = 240

Nothing is refused while the twelve pools are small (steps 6–7). At 09:00 the API's connections stop at exactly what Postgres admits, and every request past them is refused with 53300 (steps 3, 5, 8–9).

4.2 Streaming MR !3250: pools sized to the budget

A video service pre-scales for a premiere the same way. MR !3250 sizes each pod's pool from the budget for the most pods the autoscaler may run, then the premiere's burst arrives, dealt to the pods in turn:

4.2 PostgreSQL 17.11 · max_connections 150 · superuser_reserved_connections 3 · transcoders and admin 21 · the autoscaler's most pods 14
4.2 MR !3250 · budget 150 - 3 - 21 = 126 · pool max = floor(126 / 14) = 9 · export const pool = new Pool({ max: 9 });
4.2 premiere · 14 pods x 30 play starts, dealt in turn: connected 126 · refused 0 · waiting in a pod's queue 294
4.2 once the licence checks return: served 420 of 420

With pools that fit, the requests past a pod's connections wait in its queue and are served; Postgres refuses nothing (steps 11–12).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

Two ExamHall services page at once. Each service runs its pods on its own Postgres, and every pod creates its pool with:

One pod of each logs its first failed request:

attempt-api-3: Error: timeout exceeded when trying to connect
grading-api-2: error: remaining connection slots are reserved for roles with the SUPERUSER attribute

What does each line point to, and which change does it call for?

Code for this question
export const pool = new Pool({ max: 10, connectionTimeoutMillis: 2000 });
Pick one

Question 2 of 7

At month-end a payments ledger's settlement run scales out its workers, and Postgres starts refusing them. SHOW max_connections prints 120, set earlier with ALTER SYSTEM and a restart, and the workers' role holds every slot an ordinary role may. Restarting nothing, the on-call runs:

ALTER SYSTEM SET max_connections = 240;
SELECT pg_reload_conf();

The reload returns t. Then one more settlement worker connects, as the role ledger_jobs, which is not a superuser.

What happens?

Pick one

Question 3 of 7

A box office's API runs each pod as a cluster of two workers:

// server.mjs, at the top of the module
export const pool = new Pool({ max: 10 });

Postgres has max_connections 100, 3 superuser-reserved slots and no other reserved slots, and the ticket scanners and admin hold 17 connections. At the 10:00 on-sale every worker will have more requests in flight than connections.

How many API pods can run before Postgres refuses one of them a connection?

Code for this question
// main.mjsimport cluster from 'node:cluster';if (cluster.isPrimary) {  cluster.fork();  cluster.fork();} else {  await import('./server.mjs');}
Pick one

Question 4 of 7

A chat service scales its message API from 4 pods to 8 at 20:30, ahead of a live Q&A at 21:00. Each pod creates its pool when it starts:

Postgres has max_connections 100 and 3 superuser-reserved slots, and the push workers and admin hold 15 connections. Until 21:00 each pod has at most 2 requests in flight at once. At 21:00, 25 requests per pod arrive, dealt to the pods in turn, and each keeps its connection until the Q&A's first answer is posted. At 21:00:30 the on-call searches each pod's log since 20:30 for SQLSTATE 53300.

Which pods' logs have it?

Code for this question
export const pool = new Pool({ max: 20 });
Pick one

Question 5 of 7

In the interview you are given EXH-2950's page and asked: "We pre-scaled from 6 to 12 pods before the exam, and the errors started at 09:00.

Why did adding capacity cause the outage, and what do you change?" What do you answer, in at most 150 words?

Question 6 of 7

After last month's on-sale page, a box office's DBA, restarting nothing, ran:

ALTER ROLE box_api CONNECTION LIMIT 36;

Nothing has restarted Postgres since. Postgres has max_connections 60 and 3 superuser-reserved slots. The ticket scanners, which log in as box_scan, and the on-call's console hold 12 connections all evening. The API's 4 pods log in as box_api, and each creates its pool when it starts:

At the 19:00 on-sale, 25 requests reach each pod, dealt to the pods in turn, and each keeps its connection until the sale closes. At 19:00:30, 6 more scanners log in as box_scan, one after another.

What do box_api and the 6 scanners get?

Code for this question
export const pool = new Pool({ max: 20 });
Pick one

Question 7 of 7

During a premiere, a video service's API logs in to Postgres as the role stream_api, which has no connection limit and 70 sessions open. Postgres has max_connections 100 and nothing else connected but the on-call. Restarting nothing, the on-call runs:

ALTER ROLE stream_api CONNECTION LIMIT 50;

Then 10 more stream_api sessions log in, one after another.

What happens?

Pick one

Go deeper

Pro tips
  1. Count the pods' connect errors by SQLSTATE before touching the pool. One grep per pod shows whether Postgres refused anyone, and which pods; resizing a pool before that guesses at the cause. When: the first minute of any connection-error page. Status: EXECUTED.
  2. Put the budget on one screen. SELECT usename, count(*) FROM pg_stat_activity WHERE backend_type = 'client backend' GROUP BY 1; next to SHOW max_connections; and SHOW superuser_reserved_connections; shows who holds the slots and how many are left. When: before and after any replica change. Status: EXECUTED.
Source

PostgreSQL, PgBouncer, node-postgres and Node behaviour can change: the facts below are as of 2026-10-06. No item is keyed on a source; every key is executed. Valid as of 2026-10-06.

  • PostgreSQL 17 documentation, 19.3 Connections and Authentication: https://www.postgresql.org/docs/17/runtime-config-connection.html (pg-runtime-config-connection.txt), the curriculum row's first source. Quoted: max_connections "Determines the maximum number of concurrent connections to the database server. … This parameter can only be set at server start."; reserved_connections "The default value is zero connections. … This parameter can only be set at server start."; superuser_reserved_connections "Whenever the number of active concurrent connections is at least max_connections minus superuser_reserved_connections, new connections will be accepted only for superusers." Supports steps 5 and 10.
  • PgBouncer, Features: https://www.pgbouncer.org/features.html (pgbouncer-features.txt), the row's second source. Quoted: "Transaction pooling A server connection is assigned to a client only during a transaction." and "This mode breaks a few session-based features of PostgreSQL."
  • node-postgres, pg.Pool: https://node-postgres.com/apis/pool (node-postgres-pool.txt), the row's third source. Quoted: "The pool is initially created empty and will create new clients lazily as they are needed."; on min: "currently the pool will not automatically create and connect new clients up to the min". The rule behind step 9 is read from pg-pool 3.14.0's source (index.js l.240–281: a failed connect calls _pulseQueue(), which opens a client for the next queued request), and so is the acquire timeout's text, timeout exceeded when trying to connect (l.224), which D1's run prints. Supports steps 3, 6 and 9.
  • PostgreSQL 17 documentation, ALTER SYSTEM: https://www.postgresql.org/docs/17/sql-altersystem.html (pg-alter-system.txt). Quoted: "Values set with ALTER SYSTEM will be effective after the next server configuration reload, or after the next server restart in the case of parameters that can only be changed at server start." Supports step 10.
  • PostgreSQL 17 documentation, pg_settings: https://www.postgresql.org/docs/17/view-pg-settings.html (pg-view-settings.txt). Quoted: pending_restart "true if the value has been changed in the configuration file but needs a restart; or false otherwise." Supports step 10.
  • PostgreSQL 17 documentation, CREATE ROLE: https://www.postgresql.org/docs/17/sql-createrole.html (pg-create-role.txt). Quoted: CONNECTION LIMIT "If role can log in, this specifies how many concurrent connections the role can make." and "The CONNECTION LIMIT option is only enforced approximately". Supports step 13.
  • PostgreSQL 17 documentation, Appendix A. Error Codes: https://www.postgresql.org/docs/17/errcodes-appendix.html (pg-errcodes.txt): 53300 too_many_connections. Supports steps 3 and 5.
  • Node.js documentation, Cluster: https://nodejs.org/api/cluster.html (node-cluster.txt). Quoted: "The worker processes are spawned using the child_process.fork() method". Supports step 4.

Related lessons

Checked against the tools and docs as of 2026-10-06.