Day 61 of 100·System Design Day·452 words

Queue lag, schedulers, and Postgres SKIP LOCKED work queues

Measuring backlog health, the scheduler taxonomy, and the pattern you shipped twice.

Concept → your projects

Queue lag = the age/depth of the backlog. Health metric #1 for async systems: throughput is vanity, lag is sanity. Lag measurement by architecture: broker-native (RabbitMQ management API queue depth; BullMQ queue.getJobCounts() — traverse logs counts in drain logs); DB-native for outboxes: SELECT count(*), max(now()-created_at) FROM outbox_events WHERE published_at IS NULL — depth AND age. The SRE practice to state: alert on age, not depth (10,000 fresh events may be a burst; 20 events 30 minutes old is an outage).

The scheduler taxonomy (how work gets triggered):

  1. Cron/ticker loops (zoth expiry 30 s, outbox relay 5 s; traverse node-cron jobs, notification drain 30 s) — simplest; jitter-free; multi-replica safety must be designed (yours: claims/CAS — Day 50).
  2. Delay queues — RabbitMQ TTL+DLX trickery, or BullMQ delayed jobs (traverse's quiet-hours: schedule at wake time rather than poll — the right choice per notification).
  3. Repeatable jobs (BullMQ repeat pattern, jobId-keyed — traverse's push drain, groq refresh) — cron semantics inside the queue, deduped by jobId.
  4. DAG orchestrators (Airflow/temporal-shaped) — dependency graphs with retries/checkpoints. Traverse's AI→submission queue forwarding is a hand-rolled 2-node DAG (AI worker always forwards to submission processing regardless of outcome) — say "I needed a 2-node graph, so I built a 2-node graph, not an orchestrator."

Postgres as work queue (the topic your track list names explicitly — and you've implemented it twice): the recipe — a jobs table (status/priority/attempts/claim until), claim via FOR UPDATE SKIP LOCKED, lease or status for crash safety, backoff via a next_attempt_at column, visibility via WHERE status='pending' AND next_attempt_at <= now(). Compare honestly against brokers:

Postgres queueBroker
durabilitytransactional with business data (!)separate durability story
throughput~100s–1000s jobs/s (poll-bound)10k–100k+
fanout/replay/routingnonebroker features
opszero new servicesone more stateful thing
visibilitySQL!mgmt UIs/APIs

The zoth outbox is this pattern's showcase: transactional durability (events commit with state — a broker cannot offer this), SQL visibility (backlog = one query), SKIP LOCKED claims for scale. The trade accepted: polling every 5 s adds ≤5 s latency — a bargain at its volume. When the polling-latency or throughput breaks, you graduate to a broker or CDC — the outbox schema is already the contract. That migration story is exactly Day 48's CDC discussion.

The bounded-polling trick you own: LIMIT 100 per relay tick (and drain batch 50) — the loop's own backpressure: a backlog doesn't make the loop slower or unbounded; it processes in fixed-size batches forever. Combine with SKIP LOCKED and multiple replicas drain in parallel — the queue scales linearly while each tick stays constant-time.

Bulletproof your resume

Say these answers aloud before moving on

These are the follow-ups an interviewer will actually ask about your projects.

01

How would you monitor queue health without a broker dashboard?

For the outbox, SQL is the dashboard: count and max-age of unpublished events (the partial index makes both cheap) — alert on age, since depth can legitimately spike with bursts. For BullMQ, getJobCounts per queue in the drain logs plus worker stalled/retry events. The general principle: every async system should be able to answer 'how far behind am I, in seconds?' with one query or one log line — if it can't, its failure mode is silent staleness.

02

When is a Postgres table a better queue than RabbitMQ?

When transactional coupling with business data matters more than throughput — my outbox is the canonical case: the event and the state change commit atomically, which no broker gives you; visibility is SQL; claims are SKIP LOCKED; and ops is zero (no new stateful service on the budget). It breaks down past ~1k jobs/s (poll-bound) or when you need routing/delayed-delivery/fanout semantics. My rule: start with the table, graduate to the broker when a measured limit appears — and note that the outbox's event-shaped schema makes that graduation a CDC pipeline, not a rewrite.

03

How do your schedulers stay safe across replicas?

They don't rely on being singletons: every loop's actions are claims — expiry transitions are conditional updates (losers get 0 rows), the relay claims batches with FOR UPDATE SKIP LOCKED, the drain claims with a forward-dated lease. Traverse's raw node-cron jobs are the exception and the known gap — the fix is a cron_runs claim row per job, the same primitive. Rule: schedulers may run anywhere, the database decides who wins.