Your consistency story — outbox vs BullMQ at-least-once
What fails, what's delayed, what's lost, in both systems — with the windows bounded.
Know your own system
The even-day counterpart to yesterday: the failure analysis of your two async mechanisms, side by side. This is the table to memorize.
| Failure | Zoth (outbox + RabbitMQ) | Traverse (DB + BullMQ) |
|---|---|---|
| Broker/Redis down at write time | No effect: event commits to outbox with the business write; relay holds backlog | Enqueue throws → request 500s → extension retry → 200-duplicate (Day 45 gap: derived work stranded) |
| Broker dies with in-flight events | Durable queues; transient events (DeliveryMode unset!) | AOF everysec: ≤1 s job loss; stalled-job detection re-assigns in-flight |
| Consumer/worker dies mid-job | Unacked message returns to queue (autoAck=false); retries to 5; DLQ | Stalled detection → job returned to wait → retried (attempts: 3, exp backoff) |
| Duplicate delivery | processed_events insert-on-conflict (exact) | unique constraints + P2002 catch (exact) |
| Slow consumer | Backlog visible (outbox age, queue depth) | removeOnComplete hygiene, worker concurrency 10/5, group limiters |
| Total DB loss | Events lost with it — but events were derived from DB state; rebuildable | Same: jobs reference rows by ID — DB is the truth both sides |
Zoth's one honest inconsistency to know cold: events are published with DeliveryMode unset — transient, not persistent! In a broker crash, queued-but-unconsumed events are lost. Why this is still safe: the outbox row stays in Postgres independent of broker durability… wait — no: the relay marks the batch published after a successful PublishWithContext (which only confirms broker acceptance, and with no publisher confirms, not even that — it confirms the socket write). So the actual guarantee chain: event committed to outbox → relay publishes (transient message) → broker crash before consumer sees it → event gone, outbox says published. The residual hole: transient messages + no publisher confirms = a broker crash can drop committed events. The remediations to recite: (a) set DeliveryMode: 2 (persistent) + durable queues (queues already are), (b) use publisher confirms and only mark published on confirm, (c) rely on drift detection (5 min) as the reconciler that catches any dropped grant/revoke. Say: "I know the durability chain has a transient-message gap under a broker crash; the fix is persistent delivery mode + confirms, and drift detection bounds the damage window today." — that's a bulletproof answer because it shows you understand the exact semantics of your own guarantees.
Traverse's mirrored honesty: the enqueue-after-commit gap (Day 45). Plus: jobs whose handler crashes mid-way have run partial DB writes — safe because handlers are written to be re-runnable (idempotent DB ops: upserts, conditional updates, P2002 catches). The pattern name: idempotent consumer — at-least-once delivery + idempotent effects = effectively-once outcomes. Both your systems live this identity.
What would make each stronger:
- Zoth: publisher confirms + persistent messages (small code change); or swap relay-to-broker for direct CDC (Day 33's comparison).
- Traverse: submission outbox (the notification outbox pattern already in the codebase, applied to submissions) — eliminating the enqueue gap; the
pending_processingflag + sweep is the lighter version.
The narrative paragraph
"Both systems follow the same identity: at-least-once delivery, idempotent effects, DB as the single source of truth. Zoth's outbox makes event emission transactional with state change — the broker is a performance detail. Traverse makes entity persistence transactional and treats the queue as a dispatcher — with a known commit-to-enqueue gap I'd close with the same outbox pattern traverse's notifications already use. The residual risks I can name precisely: zoth's transient DeliveryMode under a broker crash (drift detection bounds it), traverse's stranded derived work (reconciliation sweep would fix it). Knowing the holes is the qualification; pretending there are none is the disqualifier."
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Your RabbitMQ messages aren't marked persistent — defend or fix.
Fix, knowingly: the design intent was 'durability comes from the outbox,' and the relay marks published after a plain publish — so a broker crash between relay-publish and consumer-ack can drop an event whose outbox row says published. Three-part response: set DeliveryMode 2 (queues are already durable), adopt publisher confirms and mark the outbox row published only on confirm, and keep drift detection as the 5-minute reconciler for anything that still slips. The gap is bounded today and closable in an afternoon — I've scoped it.
Why is at-least-once + idempotency better than trying for exactly-once?
Exactly-once delivery between systems is a myth — you can only get effectively-once effects by making handlers re-runnable. Both my pipelines embrace it: zoth's consumer dedups via insert-on-conflict then runs idempotent connectors (revoke treats not-found as success); traverse's workers rely on unique constraints and conditional updates so a redelivered job is a no-op. The alternative — transactions spanning broker and DB — doesn't exist without XA-style pain, which buys coordination overhead to achieve what dedup already gives.
If Postgres is the truth in both systems, why do the brokers exist at all?
Decoupling and retry semantics, not durability. RabbitMQ gives zoth consumer-side retry accounting (x-retry-count), DLQ parking, prefetch backpressure, and process isolation for external API calls. BullMQ gives traverse delayed/repeatable jobs, worker concurrency, and stalled-job recovery. In both cases the queue is the scheduler and isolator; the DB is the truth. Lose the broker and you lose timeliness, not correctness — that's the property I'd defend in any design review.