Distributed locks, leases and fencing tokens

A chat service's 4 workers run the hourly digest job. A worker runs it only after SET lock:digest <worker> NX PX 30000 answers OK, and its finally block runs DEL lock:digest. At 09:00:00 worker W1 takes the lock and at once freezes for 50 s; its lease runs out at 09:00:30. At 09:00:31 W2 tries the lock and, today, gets it. W1 wakes at 09:00:50, sends its digest and runs its finally. At 09:00:55 W3 tries the lock. The team wants W3 refused at 09:00:55, while W2 is still working.

A chat service's settings job writes a channel's message retention. A worker takes SET lock:retention <worker> NX PX 30000, then INCR lock:retention:fence for its token, and writes with UPDATE channels SET retention_days = $1, fence = $2 WHERE id = 7 AND fence <= $2, where $2 is its token. The row's fence is 32 and the counter is at 32. At 10:00:00 W1 takes the lock (token 33) and pauses; its lease runs out at 10:00:30. At 10:00:31 W2 takes the lock (token 34). Then W1 wakes and writes 30 days at 10:00:40, W2 writes 90 days at 10:00:45, and W1 writes 30 days again at 10:00:50.

You proposed a Redis lock for a chat service's channel-export job. The interviewer: "The job reads channel 7's export row in Postgres, sends the export, then marks the row done. Any worker can pause for 6 minutes. What stops a second export?" In the test run, W1 starts at 12:00:00 and pauses for 6 minutes just before it sends; W2 starts at 12:00:10 and, when refused, retries each minute. An answer that uses the lock takes SET lock:export-7 <worker> NX PX 30000 and a token from INCR lock:export-7:fence; the row's fence and the counter start at 32.

A chat service sends each channel's weekly digest from a job that any worker may run on PostgreSQL. The job runs BEGIN; SELECT pg_try_advisory_xact_lock(9), stopping with ROLLBACK if it returns false; SELECT status FROM digests WHERE channel_id = 9, stopping with COMMIT unless it is 'pending'; COMMIT; then it sends the digest and runs UPDATE digests SET status = 'sent' WHERE channel_id = 9. Channel 9's row is 'pending'. At 08:00:00 W1 starts the job and pauses for 2 minutes right after its COMMIT. At 08:00:30 W2 starts it and pauses for 2 minutes right after its status read, before its COMMIT. At 08:01:00 W3 starts it and does not pause.

A streaming service's transcode scheduler runs SELECT pg_try_advisory_lock(42) on a session it keeps open, and schedules only while it holds the lock. Scheduler A runs BEGIN and that call, which returns true at 12:00; then its process freezes with its connection still open. The server runs with default settings and no connection pooler. Scheduler B runs the same call on its own session at 12:10, 12:20 and 12:30. At 12:15 A's process wakes just long enough to run COMMIT, then freezes again. At 12:25 an operator ends A's session with pg_terminate_backend.

A ticketing site's nightly job rebuilds event 12's seat-map snapshot. A worker takes SET lock:seatmap <worker> NX PX 30000 and a token from INCR lock:seatmap:fence, reads the row, builds, and writes with UPDATE seatmaps SET status = 'ready', snapshot = $1 WHERE id = 12. The row's fence is 32 and so is the counter. W1 takes the lock at 02:00:00 (token 33) and reads status 'building'; after its last read of Redis or Postgres at 02:00:05 it freezes for 60 s before its write arrives. W2 tries the lock at 02:00:31 and, today, gets it (token 34), reads 'building' and writes at 02:00:40. At 02:00:50 an admin starts a manual rebuild, which sets the status back to 'building'. W1's write arrives at 02:01:05. The team wants W2's write to change the row and W1's not to.