Skip to content

Async processing & messaging

Background jobs and messaging: job lifecycles, retries and dead letters, idempotent consumers, the outbox pattern, and queues vs logs.

5 lessons

Queues and delivery guarantees

  1. 1Job queues and BullMQ's job lifecycleA queue runs each job at least once, not exactly once: a dead worker's job runs again, and that costs no attempt.
  2. 2Retries, poison messages and the dead-letter queueRetry what time can fix, fail fast what it cannot, and keep every failure where someone is paged and can replay it.
  3. 3Idempotent consumers and the exactly-once mythA consumer separates delivery identity from durable business identity.
  4. 4The transactional outbox and the dual-write problemTwo systems never commit together: write the event in the change's own transaction, and let a relay publish it, possibly twice.

Event-driven architecture

  1. 5Queue vs log vs pub/sub, BullMQ, RabbitMQ and SQS vs Kafka and JetStreamA queue forgets work once it is done; a log keeps history for every reader; pub/sub keeps nothing.