"What changes at 3 replicas" — the scaling audit of zoth
The component-by-component audit of what breaks, what's already safe, and what needs one change.
Know your own system
The most valuable project-knowledge article for a scaling interview: audit zoth as if tripling replicas tomorrow.
| Component | At 3 replicas | Verdict |
|---|---|---|
zoth-api (HTTP) | stateless (JWT, no sessions); OTel fine | safe as-is |
| Expiry loop (×3) | all tick; conditional updates arbitrate; conflicts swallowed by design ("safe with multiple replicas" is literally the code comment) | safe as-is |
| Outbox relay (×3) | FOR UPDATE SKIP LOCKED claims disjoint batches | safe as-is |
| Migrations at boot (×3 rolling) | pg_advisory_lock(72183921) serializes | safe as-is |
| Consumer (emitter ×3) | competing consumers on one queue; prefetch 10 each; dedup via processed_events is atomic (insert-on-conflict) | safe as-is — dedup is the load-bearing wall |
| In-memory token buckets (rate limit) | per-pod limits (if it were mounted) | needs Redis |
| Audit hash chain | GetLastHash + Append not transactional — three writers fork the chain daily, not occasionally | needs the fix first — this is the one real blocker |
| zlink credential decrypt per call | 3× decrypt load — trivial CPU; key security unchanged | fine |
alerting drift ticker (×3) | detect-all drifts thrice → duplicate drift events/audit rows | needs claiming (list-integrations + SKIP LOCKED style) or single-replica by policy |
| OTel → Better Stack | 3× exporters, batched | fine |
| K8s Services | already ClusterIP round-robin | fine |
| Postgres connections | 3× pool sizes against one primary | watch pool sizing (e.g., 3×25 = 75 backends — fine) |
The one hard blocker — the audit chain fork: the hash chain's correctness today assumes GetLastHash → compute → insert is effectively serialized (single API replica makes it mostly true; the audit itself flags M-8). At 3 replicas, concurrent LogAction calls read the same prev_hash, both append, chain forks silently — VerifyChain would then detect it (that's its purpose) but you'd have an unusable audit log. Fixes, in order of weight: (1) serialize appends with a transaction + SELECT ... FOR UPDATE on a per-tenant chain-head row (or advisory lock per tenant); (2) commit GetLastHash+Append in one serializable transaction; (3) move to a per-tenant append with prev_hash = hash of (tenant, seq) and a DB sequence for seq — sequence-ordered chain, no read-modify-write race at all. Option 3 is the elegant one: audit_logs(tenant, seq) with seq from a Postgres sequence makes the chain construction single-writer by definition.
That's the interview story with teeth: "I can tell you the exact component that blocks horizontal scaling: not the state machine, not the queues — the audit hash chain's read-modify-write. Everything else was built claim-based or conditional. The fix is a sequence-ordered chain, and I've scoped it."
The alerting duplicate-drift issue is the softer one: drift detection should be claim-based per integration (the same SKIP LOCKED sweep over integrations) — a 20-line change following a pattern the codebase already uses twice.
The traverse version (quick audit)
- API ×N: safe (stateless JWT, no sessions) except in-memory rate limiters → Redis.
- Cron jobs ×N: node-cron fires on every replica → double streak-reset! Needs leader claim (a Postgres
cron_runsclaim row) or single-scheduler policy. (BullMQ repeatables are safe — jobId dedup — the raw node-cron ones aren't.) - Workers ×N: safe — competing consumers, idempotent handlers.
- Notification drain ×N: safe — the lease design is for this.
- Broadcast fanout ×N: jobId dedup + cursor resume — safe.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
If I triple zoth's replicas tomorrow, what breaks first?
The audit hash chain: GetLastHash and Append aren't transactional, so three writers fork the chain routinely — VerifyChain detects it, but the log becomes unusable. Everything else survives: the state machine is conditional-update based, the relay claims via SKIP LOCKED, the consumer dedups atomically, migrations serialize on an advisory lock. The fix is a per-tenant sequence-ordered chain (seq from a Postgres sequence, prev_hash = previous seq's hash), which makes appends single-ordered by construction — and it's the first thing I'd ship before scaling out.
Why are your background loops safe to run on every replica?
They're written as claims, not responsibilities. The expiry loop's transitions are conditional updates — any number of tickers can run, each transition happens exactly once, losers get 0-rows and swallow the conflict (the code comments say exactly this). The relay and drain use SKIP LOCKED/leases for the same property. The design rule I followed: never let correctness depend on who runs the loop, only on what the database allows.
What does traverse need before multi-replica?
Two things: move rate limiting to Redis (in-memory buckets fragment per pod), and claim the node-cron jobs (streak reset, subscription expiry, groq refresh would fire per replica — double streak-reset is the harmful one). The claim is a one-row conditional update in a cron_runs table, the same primitive as everywhere else. BullMQ repeatables already dedupe by jobId, and workers are competing consumers with idempotent handlers — the queue side was multi-replica from day one.