Job queues and BullMQ's job lifecycle

Orders are tickets on a rail. A framer takes a ticket to the bench and hangs a "working until" card over it, stamping a new time every few minutes. A botched mount marks a box on the ticket, and the ticket goes on the "try again later" hook; all boxes marked, it goes to the rejects tray. A supervisor walks round: a card whose time has passed means the framer has gone, so the ticket goes back on the rail with a tally mark, and its boxes stay as they were.

In the workshop	In BullMQ
a ticket on the rail	a waiting job (step 2)
the "working until" card, re-stamped while the framer works	the lock and its renewal (step 3)
a botched mount: one box marked, ticket on the "later" hook	a throw spends an attempt; backoff in delayed (steps 4–5)
an expired card: ticket back on the rail, a tally mark, boxes untouched	the stalled-job check and stall counter (steps 7–8)
a framer with three clamps, three frames drying at once	concurrency: 3 (step 9)
the next framer glues the glass again after the first went home ill	at-least-once (step 11)

In real terms: a job is state in Redis plus a lock with an expiry; a handler that stops renewing the lock gets its job run again.

Breaks down when: the next framer can see the glass is already glued; BullMQ cannot see what your handler already did. Wrong inference: "the queue will skip the part that finished." It runs the whole handler again.

Contrasting case: a counter where the customer waits while the frame is made. One absent framer loses the order and the customer waited throughout: that is work done inside the request.

4.1 ExamHall: submit returns before grading

EXH-1350: at exam close, final-submit requests timed out, because the handler graded the attempt inside the request. MR !1550 adds a job and returns 202.

// after MR !1550: the request adds a job and answers at once
const job = await grading.add('grade-attempt', { attemptId }, { attempts: 3, backoff: { type: 'exponential', delay: 1000 } });
res.writeHead(202).end(JSON.stringify({ attemptId, status: 'grading' }));

4.1 before: grading still running, response sent false · after: 202 {"attemptId":"a-5521","status":"grading"} · job finished when 202 arrived false
4.1 before, once grading ends: 200 {"score":71} · after: job completed · events added waiting active completed

With grading held open, the old request still waits; the new one has answered, and the worker completes the job later (steps 1–2).

4.2 ExamHall: two failures, then a score

The scoring service fails twice, then answers; the job has attempts: 4 and exponential backoff from 100 ms.

4.2 events added waiting active delayed waiting active delayed waiting active completed
4.2 attemptsMade seen by each run 0,1,2 · final completed · attemptsMade 3 · backoff before attempts 2,3,4 (Backoffs.calculate): 100, 200, 400 ms

Each throw sends the job to delayed for its backoff, then back to waiting (steps 4–5). The last line is BullMQ's own backoff calculation, not a measured time.

4.3 Proof domain, ticketing: a seat map rendered twice

A render-seat-map worker process is killed after it rendered and before it returned.

4.3 events added waiting active waiting stalled active completed
4.3 handler starts 2 (killed run + rerun) · the rerun saw attemptsMade 0 · stalledCounter 1 · final completed

The stall check put the job back in waiting (step 7). The rerun saw no attempt spent and one stall counted (step 8). The render ran twice (step 11).

4.4 Proof domain, ticketing: a delayed release and one worker's slots

4.4 release-hold state right after add: delayed · events added delayed waiting active completed · final completed
4.4 one worker, concurrency 4, 6 jobs held on an unanswered call: active 4 · waiting 2

The worker was free, so its fetch moved the due job to waiting and took it (steps 2, 10). One worker never runs more than its concurrency (step 9).