The notification lease and retry+backoff engineering

The deep version of your best lease implementation — src/queues/notification.queue.ts.

The claim SQL (be able to write it)

UPDATE notification_delivery nd
SET attempts = attempts + 1,
    next_attempt_at = NOW() + INTERVAL '120 seconds'   -- the lease
FROM (
  SELECT id FROM notification_delivery
  WHERE status='pending' AND next_attempt_at <= NOW()
  ORDER BY next_attempt_at
  LIMIT 50                                  -- DRAIN_BATCH_SIZE
  FOR UPDATE SKIP LOCKED                    -- competing drains claim disjoint rows
) claimed
WHERE nd.id = claimed.id
RETURNING nd.*;

Every design element has a reason: SKIP LOCKED = parallel drains; the lease is forward-dated, not a status = crashed workers need no reaper (Day 32); ORDER BY next_attempt_at = oldest-first fairness; batch 50 = bounded send burst per 30 s drain tick.

Failure-mode walkthrough (the interview drill)

• Worker dies before sending: lease expires in ≤120 s; next drain re-claims; attempts incremented; eventually failed after MAX_ATTEMPTS=5. Cost: 2 minutes delay.
• Worker alive but slow (APNs takes >120 s): lease expires while send is in flight; another drain claims and double-sends; both record sent. Accepted: push is collapse-coalesced; and the design doc says quiet-hours defer, never drop — same philosophy: err toward delivering.
• APNs 5xx: failed(retryable) → backoff min(60s·2^(attempts-1), 1h) written as next next_attempt_at → status stays pending. Exponential with cap — no infinite growth.
• APNs 410/Unregistered/BadDeviceToken: invalid_token → terminal; the PushToken is invalidated; no retry — error classification again (Day 19).
• No token registered: skipped(no_push_token) — terminal, and not an error: "a decision, not a failure." The drain must never re-add skipped rows to the retry set — the code documents this.
• APNs not configured (isApnsConfigured()==false): every push → skipped(unconfigured) — deliberate, terminal. Consequence stated in the design: configuring APNs later pushes only new notifications, never a backlog — because the rows already went terminal. That's a real tradeoff decision documented in the code header; recite it.

The retry ladder inventory (both projects)

Note the two-tier retry in notifications: the job can fail (BullMQ retries 3×), and separately the delivery row accumulates its own attempts with its own backoff — the row is the durable retry state; the job is just the trigger. That's the outbox-style separation of state from execution, and it's why the system is resumable from any crash point.

The jitter rule (everywhere or nowhere): zlink's delay/2 + rand(delay/2) — without jitter, synchronized failures (e.g., APNs blip at a drain tick) produce synchronized retries; with it, the herd spreads. BullMQ's exponential backoff includes jitter by default. Your consumer republish path has no jitter — an improvement to name (sleep-before-republish or randomized delay header).