RabbitMQ in zoth — the complete topology, verified
Every exchange, queue, retry header, and consumer behavior — the deep dive that makes you unfakeable.
Know your own system
The full inventory from internal/queue/ — know it like wiring you pulled yourself.
Topology (declared in publisher.go / consumer.go)
Exchange: zoth.access (TOPIC, durable) routing keys: access.granted | access.revoked | access.expired | access.denied | alert.suspicious | drift.detected Queues: emitter.access (durable) bound: granted/revoked/expired alerting.access (durable) bound: suspicious/drift [consumer exists, wiring in progress — known gap] DLQ: emitter.access.dlq (durable) — bound via the queue's DLX args: x-dead-letter-exchange: zoth.access (cycles back to emitter.access for republish retries — see below)
Publish path: the outbox relay calls publisher.Publish(routingKey, payload) — JSON envelope {event_id?, request_id, tenant_id, user_id, resource_id, action, timestamp}. DeliveryMode is unset → transient (Day 48's known gap: persistent delivery + publisher confirms is the fix). The publisher is mutex-guarded, channel cached, with a ChannelClosed error class.
Consume path (consumer.go): autoAck=false, Qos(prefetch=10), handler with 5-minute overall timeout. Failure ladder: handler error → if republish succeeds (new message with x-retry-count: N+1, original Acked) → else Nack(requeue=true) as last resort; retry-count ≥ 5 → Nack(requeue=false) → DLX routes to emitter.access.dlq — where the queue's dead-letter-exchange argument points back to zoth.access, so DLQ messages re-enter the main queue with x-death history growing (the DLQ acts as a park-and-retry ring, and inspection = read x-death count). It's quirky — say "the DLX cycles rather than parks permanently; a cleaner design parks in the DLQ for manual replay" and you sound like someone who's read their own code critically.
The behaviors you must be able to defend
- Why republish instead of Nack-requeue: Nack requeue = immediate redelivery hot-loop, no backoff, no counter (Day 19). Republish = controlled attempts + broker-native accounting. Cost: message identity changes each hop (the original's traceId persists in payload), and no jitter (Day 52's improvement).
- Why prefetch 10: credit-based flow control — the broker won't hand the consumer more than 10 unacked messages; a slow downstream (GitHub throttling) automatically throttles dispatch. Backpressure without any code (Day 23).
- Why topic exchange not direct/fanout: routing-key wildcard flexibility (
access.*binds future keys without topology changes) with per-queue selectivity (emitter gets the lifecycle events, alerting gets the telemetry ones) — fanout would duplicate, direct would freeze the key set. - Why the relay owns publishing (not the API): single choke point for publishing semantics (batching, error handling, marking published), and the API stays free of broker coupling (it just commits outbox rows). The API can even run with no broker at all.
The operational story (deployment facts)
StatefulSet, 1 replica, 1 Gi PVC — RabbitMQ's durability model: durable exchanges/queues survive restart if messages are persistent; yours are transient by omission → a broker crash drops queued-but-unconsumed events → the outbox (Postgres) is the real durability layer, drift detection is the reconciler. Know this chain cold (Day 48's answer). Probes: rabbitmq-diagnostics -q ping (startup 10×10 s, liveness 60 s, readiness 20 s). Diagnostics from within the pod (exec), not the mgmt UI (not exposed — cost/attack-surface decision).
The "why not" list (one-liners, ready)
- Quorum queues: classic durable queues chosen — quorum (Raft) needs 3 replicas to be meaningful; 1-node cluster gets nothing but overhead.
- Publisher confirms: would close the relay-marks-published-on-accept gap; scoped, not shipped.
- Mirror/quorum replication: same single-node budget story.
- Federation/shovel: multi-region — not a requirement.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Describe your RabbitMQ topology and retry semantics.
One durable topic exchange (zoth.access) with six routing keys; a durable emitter queue bound to the lifecycle keys; an alerting queue bound to telemetry keys (consumer in progress — a gap I know about). Consumption: manual acks, prefetch 10, 5-minute handler timeout. On failure: republish with incremented x-retry-count (up to 5) — requeue Nack only as a fallback — then dead-letter. The DLQ's dead-letter-exchange points back at the main exchange, so it functions as a retry ring with x-death history; a cleaner design would park permanently for manual replay, and I'd change that. The known hardening: persistent DeliveryMode + publisher confirms so relay-marked-published matches broker durability.
What happens to in-flight messages if the emitter crashes?
Unacked messages return to the queue (autoAck=false) and get redelivered to the next consumer — that's at-least-once. The redelivered event re-runs the handler; the processed_events insert-on-conflict makes the duplicate a no-op if the crash happened after side effects completed. The crash-after-partial-side-effects window is covered by idempotent connectors (revoke: not-found = success; grant: re-adding a collaborator is a no-op on GitHub's side). So crash recovery needs no special code — that's the payoff of designing every consumer to tolerate redelivery.
Why does the alerting queue exist but have no wired consumer?
Because the events (alert.suspicious, drift.detected) were designed as the platform's telemetry contract — the RAG suspicious-activity checker and drift service publish them — but the alerting binary's consumer isn't yet wired into its run loop. It's my own known gap, the same category as the unmounted rate limiter and the /audit-logs/verify route: capability built, plumbing not finished. The fix is a session-consuming goroutine in cmd/alerting identical to the emitter's, and it's on my list precisely because I audited myself.