Availability, durability, consistency, cost — and when NOT to add infrastructure
The four-way tradeoff and the senior skill of leaving things out.
Concept → your projects
Four words get used interchangeably by juniors and precisely by seniors:
- Availability: the system answers requests. Uptime = MTBF / (MTBF + MTTR). Nines math: 99.9% = 8.77 h down/year, 99.99% = 52.6 min.
- Durability: committed data survives disasters. Postgres with WAL + replicated backups is durable; a Redis cache is not (by design).
- Consistency: do all readers see the latest write? Strong = linearizable; eventual = converges. Every queue introduces an eventual-consistency window somewhere.
- Cost: real money, monthly. The invisible NFR.
The classic tension: CAP (under partition, choose consistency or availability) and PACELC (else, choose latency or consistency). Full treatment on Day 47 — today, the framing and your projects' actual choices.
Your projects, scored honestly
| System | Availability posture | Durability posture | Consistency posture | Cost |
|---|---|---|---|---|
| Zoth api | 1 replica, restarts via probes | Postgres WAL (implied), no backups configured | strong for decisions (Postgres), eventual for provisioning (outbox→queue) | ~$30/mo |
| Zoth emitter | consumer restarts, retries to DLQ | at-least-once + dedup + idempotent revoke | eventual by design | shared |
| Traverse | 1 VPS, docker-compose restart | PITR: WAL archive to S3, 3-day Aiven sync | strong writes (Postgres first), reads via cache TTLs | ~$6/mo VPS |
Notice: traverse is more durable than zoth — it has actual backup/restore machinery (pitr.service.ts, admin endpoints, pg-backup-sync.sh), because user streak/revision data is the product. Zoth's data is demo-scale; its integrity story (hash-chained audit) is stronger than its recovery story. Both are defensible if stated as choices.
When NOT to add infrastructure — your three case studies
1. No Redis in zoth. The caching module (Day 43–46) explains why: cache-aside for reads that happen ~1/s adds a network hop, a failure mode, and an invalidation problem, for zero measurable latency win. Postgres + proper indexes was the whole answer. The general rule: cache reads that are hot, not reads that are rare.
2. No Kafka. Kafka earns its ops cost when you need replay, multiple consumer groups, or > ~100k events/s. Zoth has 5 routing keys, 1 consumer queue, and durability already provided by the transactional outbox. RabbitMQ (or even SQS) is the right-sized tool. The general rule: don't run a distributed log as a work queue.
3. No graph database (the Neo4j removal). Zoth's first design used Neo4j for the org hierarchy — replaced in 002_drop_neo4j.sql by a manager_id self-FK + recursive CTE with a level < 25 cycle cap. The graph traversal zoth actually needs (manager chain) is a 10-row recursive query on a table that will never exceed ~50k rows. The general rule: model your queries first; add a database only when the relational answer is provably too slow. Interviewers love this story because it shows you deleting your own cleverness.
And one from traverse: no WebSocket layer for realtime stats. Stats change on the order of minutes; the app polls, and push (where needed) rides APNs, not a server-held socket farm (Day 73–74).
The anti-pattern to name out loud
"Resume-driven architecture": adding Kafka + Redis + K8s + a service mesh to a 100-user app. Every component multiplies failure modes. The counter-question an interviewer will respect: "what failure mode am I buying insurance against, and what does the insurance cost?" In zoth, the honest ledger was: one AKS node buys self-healing processes for $30; RabbitMQ single replica risks ~30 s of delayed provisioning on crash — acceptable, because grants already made are not revoked by a broker crash (the DB is the truth, drift detection catches stragglers within 5 minutes).
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Is your system consistent or eventually consistent?
Layered answer: the decision layer (policy evaluation, state transitions) is strongly consistent — single Postgres, conditional updates. The provisioning layer is eventually consistent by design — approval writes an outbox row in the same transaction as the state change, a relay publishes to RabbitMQ within ~5 s, the emitter provisions within seconds-to-minutes with retries, and drift detection reconciles any stragglers every 5 minutes. So a user sees 'ACTIVE' immediately, while GitHub membership lands seconds later — and I can bound that window.
What's your single biggest availability risk in zoth and what would you do about it?
Single-replica RabbitMQ and single Postgres. For Postgres: streaming replica + automated failover (or managed PG). For RabbitMQ: quorum queues + 3 replicas, or replace the transport with Postgres-based polling since the outbox already exists — the outbox means the broker is a performance optimization, not a correctness dependency. That's the nice property of the design: broker down = backlog, not data loss.
When would you add a cache to zoth?
When a read becomes hot and expensive: policy evaluation per request hits ListByTenant + priority sort — fine at 1 rps, wasteful at 1k. The first cache would be tenant-scoped active policies with event-driven invalidation (policy write → invalidate). But today the correct answer is Postgres indexes — adding Redis would be paying cost (inconsistency window, ops) for latency nobody measured as a problem.