Queue lag, schedulers, and Postgres SKIP LOCKED work queues
Measuring backlog health, the scheduler taxonomy, and the pattern you shipped twice.
Concept → your projects
Queue lag = the age/depth of the backlog. Health metric #1 for async systems: throughput is vanity, lag is sanity. Lag measurement by architecture: broker-native (RabbitMQ management API queue depth; BullMQ queue.getJobCounts() — traverse logs counts in drain logs); DB-native for outboxes: SELECT count(*), max(now()-created_at) FROM outbox_events WHERE published_at IS NULL — depth AND age. The SRE practice to state: alert on age, not depth (10,000 fresh events may be a burst; 20 events 30 minutes old is an outage).
The scheduler taxonomy (how work gets triggered):
- Cron/ticker loops (zoth expiry 30 s, outbox relay 5 s; traverse node-cron jobs, notification drain 30 s) — simplest; jitter-free; multi-replica safety must be designed (yours: claims/CAS — Day 50).
- Delay queues — RabbitMQ TTL+DLX trickery, or BullMQ delayed jobs (traverse's quiet-hours: schedule at wake time rather than poll — the right choice per notification).
- Repeatable jobs (BullMQ repeat pattern, jobId-keyed — traverse's push drain, groq refresh) — cron semantics inside the queue, deduped by jobId.
- DAG orchestrators (Airflow/temporal-shaped) — dependency graphs with retries/checkpoints. Traverse's AI→submission queue forwarding is a hand-rolled 2-node DAG (AI worker always forwards to submission processing regardless of outcome) — say "I needed a 2-node graph, so I built a 2-node graph, not an orchestrator."
Postgres as work queue (the topic your track list names explicitly — and you've implemented it twice): the recipe — a jobs table (status/priority/attempts/claim until), claim via FOR UPDATE SKIP LOCKED, lease or status for crash safety, backoff via a next_attempt_at column, visibility via WHERE status='pending' AND next_attempt_at <= now(). Compare honestly against brokers:
| Postgres queue | Broker | |
|---|---|---|
| durability | transactional with business data (!) | separate durability story |
| throughput | ~100s–1000s jobs/s (poll-bound) | 10k–100k+ |
| fanout/replay/routing | none | broker features |
| ops | zero new services | one more stateful thing |
| visibility | SQL! | mgmt UIs/APIs |
The zoth outbox is this pattern's showcase: transactional durability (events commit with state — a broker cannot offer this), SQL visibility (backlog = one query), SKIP LOCKED claims for scale. The trade accepted: polling every 5 s adds ≤5 s latency — a bargain at its volume. When the polling-latency or throughput breaks, you graduate to a broker or CDC — the outbox schema is already the contract. That migration story is exactly Day 48's CDC discussion.
The bounded-polling trick you own: LIMIT 100 per relay tick (and drain batch 50) — the loop's own backpressure: a backlog doesn't make the loop slower or unbounded; it processes in fixed-size batches forever. Combine with SKIP LOCKED and multiple replicas drain in parallel — the queue scales linearly while each tick stays constant-time.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
How would you monitor queue health without a broker dashboard?
For the outbox, SQL is the dashboard: count and max-age of unpublished events (the partial index makes both cheap) — alert on age, since depth can legitimately spike with bursts. For BullMQ, getJobCounts per queue in the drain logs plus worker stalled/retry events. The general principle: every async system should be able to answer 'how far behind am I, in seconds?' with one query or one log line — if it can't, its failure mode is silent staleness.
When is a Postgres table a better queue than RabbitMQ?
When transactional coupling with business data matters more than throughput — my outbox is the canonical case: the event and the state change commit atomically, which no broker gives you; visibility is SQL; claims are SKIP LOCKED; and ops is zero (no new stateful service on the budget). It breaks down past ~1k jobs/s (poll-bound) or when you need routing/delayed-delivery/fanout semantics. My rule: start with the table, graduate to the broker when a measured limit appears — and note that the outbox's event-shaped schema makes that graduation a CDC pipeline, not a rewrite.
How do your schedulers stay safe across replicas?
They don't rely on being singletons: every loop's actions are claims — expiry transitions are conditional updates (losers get 0 rows), the relay claims batches with FOR UPDATE SKIP LOCKED, the drain claims with a forward-dated lease. Traverse's raw node-cron jobs are the exception and the known gap — the fix is a cron_runs claim row per job, the same primitive. Rule: schedulers may run anywhere, the database decides who wins.