Cache availability, concurrency, and database fallback

A cache is a facilitation, not a dependency — until sloppy design makes it one. The failure ladder to avoid:

1. Cache down → app down (worst): reads that assume cache hits, writes that only write cache (write-back), or startup that requires Redis.
2. Cache down → slow but correct (target): every path has a DB fallback; timeouts bound the penalty.
3. Cache down → nothing at all (only if truly unused).

Your systems sit at (2) by construction — the patterns to name: read-through fallback (miss or error → DB → best-effort refill), timeout-bounded ops (traverse's 2000 ms withTimeout), readiness-gated degradation (isRedisReady() checks before ops — ops become no-ops, not errors), DB-first writes (nothing is only-cached). And on the queue side, the deliberate asymmetry: BullMQ tolerates Redis unavailability longer (retry strategy capped at 2 s, maxRetriesPerRequest: null) because jobs must survive; cache ops fail fast because they're optional. Two clients, two policies — Day 44.

Cache concurrency problems (the trio):

• Stampede (Day 43): miss-storm on expiry.
• Read-modify-write races: two writers compute from the same cached base → last-wins corruption. Fix: versioned writes (CAS via WATCH/MULTI in Redis, or optimistic versions) — or don't cache mutable composites at all (your choice: cached values are read projections, never write bases).
• Thundering invalidation: one write invalidates N keys, each next-read misses. Bounded by TTLs and Promise.allSettled fan-out (one round, not N serialized).

The "cache as availability shield" inversion: usually the DB is the fragile thing a cache protects (absorbing read load so the DB doesn't fall over). At your scale the inversion is honest: Redis is more likely to be a bother (restarts, memory caps) than Postgres — so the design treats Redis as the expendable component. The interview line: "I made the cache the most expendable dependency in the system on purpose — that's the only safe cache topology."

Zoth's answer is "no cache, and here's the check": would policy evaluation benefit? It reads active policies per request (ListByTenant + priority sort + first-match) — ~1 query per request. At 1000 rps this is 1000 queries/s of identical rows → the cache-aside + event-driven invalidation on policy writes would be the first cache. The reason it's absent: measured need. Day 43's doctrine: cache hot reads, not all reads.

The drill

"Redis is down during a submission burst in traverse": extension POSTs → API: duplicate-check query (DB), insert (DB), XP update (DB), enqueue fails? — enqueue is a cache-class op? NO — it's a queue op: BullMQ with retry-forever client will queue once Redis returns if the process buffers… actually the honest sequence: createSubmission's enqueue throws when Redis is unreachable → the controller 500s → extension retries (idempotencyKey makes it safe) → once Redis returns, the retry lands, dedup returns 200-duplicate BUT the first submission's job was never enqueued (it died with the 500)! That's a real subtle gap: the 200-duplicate path doesn't re-enqueue. The fix: enqueue-robustness — persist a pending_processing flag on the submission row and a reconciliation sweep (or route enqueue through the outbox pattern — traverse's notifications already do Postgres-outbox; submissions could too). Finding this edge yourself and articulating the fix is exactly the bulletproofing this course is for. (Also the honest current answer: BullMQ jobs are added after the DB commit, so a crash between commit and enqueue strands the row the same way — same fix.)