Incident: queue backing up silently

A call with no deadline holds its slot without failing; read active against concurrency, page on job age, and fix it with a timeout.

Before you read: predict

PAGE · ECON-201, 10:20 · results not released · no errors anywhere

ECON-201 closed at 10:00, and its 2,400 submissions entered the grading queue at once. Results normally release by 10:05: one worker process with concurrency: 20 grades each submission in about 2.5 s, most of it spent waiting on the plagiarism vendor's API. At 10:00 the vendor began accepting connections and answering none. The handler awaits that call with no timeout, and the worker keeps BullMQ's default lock and stalled-job settings. Nothing else is graded between 10:00 and 10:20. The grading service pages on three alerts, each evaluated once a minute:

FAILED: at least one grading job failed in the last 5 minutes.
LATE: of the results released in the last 10 minutes, more than 1% came out more than 5 minutes after their submission.
AGE: the oldest waiting job is older than 10 minutes.

By 10:20, which alerts have paged?

The 30-second version

Late results and no errors: read the queue. Active stuck at the total concurrency, waiting growing, nothing completed or failed: every slot awaits a call that never answers. Live workers renew those locks, so nothing stalls, and alerts counting finished jobs stay quiet. Restarts and more slots only move the hang. Give the call a deadline; page on job age.

A call with no deadline holds its slot without failing; read active against concurrency, page on job age, and fix it with a timeout.

How it works

Step 1 of 13

Late, not failing. The page is "results late, nothing failed". A job that neither finishes nor fails moves no failure counter.

Analogy: A consulate's visa windows

Twenty officers each keep one applicant at their window while a background-check office is asked about each applicant. The check office logs every request and has stopped replying.

At the consulate	In the incident
twenty windows, one applicant each	the worker's 20 slots
numbered tickets waiting in the hall	waiting jobs
an officer waiting at the window for the check office's reply	a handler awaiting a call with no deadline (steps 4, 6)
the window lamp stays lit while an officer sits there	the live worker renewing the lock (step 5)
the hall board counts refusals and visas issued; neither moves	failure and release counts (step 7)
the oldest ticket still in the hall	the oldest waiting job's age (step 8)
guards clear the windows; the next tickets sit down at once, and the cleared applicants later wait at the head of the hall	kill, refill, stall, front of waiting (steps 9–10)
a rule: no reply in an hour, the file goes to "check not returned" and the window frees	`AbortSignal.timeout` (step 12)

Breaks down when: an officer would grow restless and phone the check office; a handler never notices it is waiting, and nothing ends the wait unless a deadline was set.

Contrasting case: a handler that never yields its event loop. Its locks are not renewed, its jobs stall and then fail, and the queue's event stream fills with stalled events: loud, not silent (step 2; async-queue-fundamentals-bullmq step 7).

In real terms: when a queue is late and nothing failed, compare active with the total concurrency before anything else, then give the awaited call a deadline.

Examples, run for real

4.1 ExamHall EXH-2990: the grading queue at 10:00

ECON-201's grading queue, scaled down: 24 submissions, one worker with concurrency: 4, and a vendor that accepts each call and answers none. The snapshot waits until the stall check has run three times and every lock has been renewed three times:

4.1 BullMQ 6.3.9 · Redis 7.4.11 · grading: 24 submissions · concurrency 4 · the vendor accepts every call and answers none · handler awaits the call with no timeout · lockDuration 2000 ms, lockRenewTime 250 ms, stalledInterval 1000 ms
4.1 after 3 stall checks, each lock renewed 3 times: getJobCounts() waiting 20 · active 4 · completed 0 · failed 0 · delayed 0
4.1 the queue's event stream: added 24 · active 4 · stalled 0 · failed 0 · completed 0
4.1 the 4 active jobs: started 4 · finished 0 · attempts made 0 · vendor: calls received 4, answered 0, still open 4
4.1 worker.close() called: still waiting after each lock is renewed 3 more times: true
4.1 same 24 submissions, the call given signal: AbortSignal.timeout(300): completed 0 · failed 24 ("The operation was aborted") · active 0 · waiting 0

Four slots, four calls with no reply, nothing completed, failed or stalled (steps 3–5), and close() still waiting (step 9). With a deadline on the call, the same jobs fail and show up (step 12).

4.2 Media streaming MR !3290: one call in ten never answers

A thumbnail queue: 60 jobs, concurrency: 4, and a frame grabber that answers every call except every tenth. MR !3290 gives the call a deadline and two attempts:

4.2 thumbnails: 60 jobs · concurrency 4 · the frame grabber answers every call at once except every 10th, which it never answers · no timeout
4.2 after 3 stall checks: calls made 40 · completed 36 · active 4 (all 4 on unanswered calls) · waiting 20 · failed 0 · stalled 0
4.2 MR !3290 · the call given signal: AbortSignal.timeout(300), attempts: 2 · every job ended (completed or failed): true · active 0 · waiting 0 · calls still held open by the frame grabber: 0

Each unanswered call kept its slot until all four were held, and the queue stopped with nothing failed (step 6). With the deadline, every job ended and no slot stayed held.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

A video platform's thumbnails queue holds 3,000 jobs at 02:00, and no more arrive. One worker process runs them with concurrency: 24; each job awaits one call to the frame-grabber service, with no timeout. Tonight the frame grabber accepts every call and answers its first 1,000; after that it never answers every 10th call it receives (the 1,010th, the 1,020th and so on). The calls it answers take 1 s.

How many of the 3,000 jobs complete?

Question 2 of 7

A chat app's push queue receives 30 jobs a minute, steadily. One worker process with concurrency: 10 runs them; each job calls the push gateway, which normally answers in 1 s, and the handler awaits the call with no timeout. For the last 10 minutes the gateway has accepted every call and answered none. The worker keeps BullMQ's default lock and stalled-job settings; jobs are added with attempts: 3 and a fixed 30 s backoff. A restart below always goes the same way: SIGTERM, whose handler awaits worker.close(); the process is killed if it is still running after a 30 s grace period; then a new process starts.

Which change makes every job that arrives between 10 and 20 minutes after the change end, completed or failed, within 3 minutes of arriving, while the gateway still answers nothing?

Question 3 of 7

A ticketing site's refund worker runs as one process with concurrency: 6. Each refund job calls the card processor's refund API with no timeout, and the processor now accepts every call and answers none. Six refund jobs are active and none is waiting. The worker sets no maxStalledCount. The on-call engineer restarts the worker twice. Each restart: SIGTERM, whose handler awaits worker.close(); the process is killed if it is still running after the grace period; then a new process starts.

After the second restart, what does the queue show for the six jobs?

Question 4 of 7

A payment app's receipts queue is late and its error rate is zero. Two worker processes, each with concurrency: 3, take its jobs; each job renders a PDF receipt and sends it through the mail API, and the deployment restarts any worker process that exits. The on-call engineer reads getJobCounts() twice, a minute apart, and the queue's event stream: first waiting 24 · active 6 · completed 0 · failed 0, then waiting 36 · active 6 · completed 0 · failed 0, and 0 stalled events.

Which hypothesis do these two readouts support?

Question 5 of 7

In the interview you are given EXH-2990's page and asked: "Results are 20 minutes late and there are no errors.

Where do you look?" What do you answer, in at most 150 words?

Your answer (optional, stays on this page)

Question 6 of 7

A payments ledger's payout batch enqueues 3,600 payouts at 00:00, and nothing else arrives. One worker process with concurrency: 20 sends them; each payout's bank call takes 4 s on a normal night. The team wants a page on the age of the oldest waiting job, checked continuously, that never fires on a normal night.

What is the smallest whole number of minutes it can use as "older than … minutes"?

Question 7 of 7

Every night at 02:00 a streaming catalogue enqueues 3,000 thumbnail jobs at once. One worker process with concurrency: 24 runs them; each job calls the frame grabber with no timeout, and the worker keeps BullMQ's default lock and stalled-job settings. Tonight the frame grabber accepts every call from 02:00 and answers none. The on-call engineer restarts the worker at 02:10 and again at 02:20; each restart is SIGTERM, whose handler awaits worker.close(), then a kill after a 30 s grace period, then a new process.

At 02:30, what has become of the first 24 jobs?

Go deeper

Pro tips

Alert on the age of the oldest waiting job. One number that catches hangs, slow consumers and dead workers (curriculum row's own tip). When: any BullMQ queue whose results someone waits for. Status: EXECUTED (model).
Compare active with the total concurrency before reading any error rate. Active pinned at the sum of every worker's concurrency, with nothing completing, means every slot is waiting on something. When: the first minute of a "late, no errors" page. Status: EXECUTED.

Source

BullMQ and Node behaviour can change: the facts below are as of 2026-10-06. No item is keyed on a source; every key is executed. Valid as of 2026-10-06.

BullMQ docs, Timeout jobs: https://docs.bullmq.io/patterns/timeout-jobs (bullmq-timeout-jobs.txt), the curriculum row's first source. Quoted: "BullMQ does not provide a specific mechanism to timeout jobs, however this can be accomplished in many cases with a custom timeout code in the worker's process function." Supports steps 5 and 12.
BullMQ docs, Stalled Jobs: https://docs.bullmq.io/guide/workers/stalled-jobs (bullmq-stalled-jobs.txt), the row's second source. Quoted: "the worker needs to periodically notify BullMQ that it is still working on the job"; "if the CPU is very busy (due to the process being very CPU intensive), the worker may not have time to renew the lock". Supports steps 2 and 5 and the contrasting case.
BullMQ docs, Graceful shutdown: https://docs.bullmq.io/guide/workers/graceful-shutdown (bullmq-graceful-shutdown.txt), the row's third source. Quoted: "it will wait for all the current jobs to be processed (or failed). This call will not timeout by itself, so you should make sure that your jobs finalize in a timely manner." Supports step 9.
BullMQ docs, Metrics: https://docs.bullmq.io/guide/metrics (bullmq-metrics.txt). Quoted: "counters of the number of jobs that have been processed either completed or failed during 1 minute intervals." Supports step 7: a job that is neither adds to no counter.
BullMQ 6.3.9 source, dist/esm/commands/moveStalledJobsToWait-9.lua (installed package): a job whose lock is missing is moved back with RPUSH, the end of the wait list a worker takes from next; past maxStalledCount it is marked "job stalled more than allowable limit". Supports steps 9–10 and the model's rule R3.
Node.js documentation, Globals, AbortSignal.timeout(delay): https://nodejs.org/api/globals.html (node-globals.txt). Quoted: "Returns a new AbortSignal which will be aborted in delay milliseconds." Supports step 12.
Google SRE book, Monitoring Distributed Systems: https://sre.google/sre-book/monitoring-distributed-systems/ (sre-monitoring.txt). Quoted: "The four golden signals of monitoring are latency, traffic, errors, and saturation." Supports step 8: saturation, measured as queued work's age.

Checked against the tools and docs as of 2026-10-06.

Predict first