Day 88 of 100·Project Deep-Dive Day·453 words

DR posture of both projects, itemised

Every durability mechanism you own — plus the three gaps to fix before production.

Know your own system

The itemised audit, ready to recite.

Traverse — what exists

  1. WAL archiving: wal_level=replica, archive_mode=on, archive_timeout=300, archive_command → S3 bucket traverse-db-backups-697114252450 (us-east-1). Bounded RPO of 5 minutes, including idle periods (the timeout is the enforcement).
  2. Offsite base backups: scripts/pg-backup-sync.sh on host cron (every 3 days) syncing to Aiven (AIVEN_BACKUP_DB_URL) — a managed provider as the "different vendor, different failure domain" copy (the 3-2-1's offsite leg).
  3. Blast-radius separation: backups in a different region and with different credentials than user files (careers bucket, ap-south-1). A leaked application credential cannot read database backups.
  4. Executable recovery: pitr.service.ts — status, backup listing, snapshot, restore, rollback-safety, history, exposed as admin endpoints. Recovery is a documented, callable procedure.
  5. Redis AOF: job durability across restarts (≤1 s loss), paired with idempotent handlers.
  6. Immutable-ish data model: submissions/solves/attempts are append-mostly; the derived state (revisions, streaks) is recomputable from facts — so partial loss degrades features rather than corrupting truth.

Traverse — the three gaps (fixes ready)

  1. No immutability on the backup bucket: versioning + Object Lock (WORM) defends against deletion attacks (ransomware's first move). Bucket config, zero code.
  2. No drilled restore: the endpoints exist; a scheduled quarterly drill (restore into a scratch DB, verify invariants, record RTO) is the missing practice.
  3. Soft-delete purge not implemented: User.deletedAt + grace exists; the job that hard-deletes after grace (and cascades to caches/derivatives) doesn't — so "deleted" data persists indefinitely, which is both a compliance and a storage issue.

Zoth — what exists and what doesn't

Exists: Postgres in-cluster with a PVC (survives pod reschedule); Aiven managed PG in the prod env config (.env.prod) — so a managed instance is provisioned, which is the seed of a real story; migrations embedded (schema is reproducible from the repo); the audit chain (integrity, not durability); drift detection (reconciles external state, not data loss).

Doesn't exist: WAL archiving, base backups, restore runbook, drills, RPO/RTO definitions. Recovery = redeploy + re-seed.

The honest framing (this is the version to use): "Zoth's data is demo-scale and largely regenerable (users/groups sync from the identity provider; access requests are workflow artifacts). I chose not to spend the budget on backup infrastructure for data I can reconstruct — and I can say exactly what production would require: archive_command to Azure Blob, daily base backups, a restore runbook, and a monthly drill. Traverse, whose data is not regenerable (user history), got the full treatment. Matching durability investment to data value is the decision; the mistake would be doing neither."

Bulletproof your resume

Say these answers aloud before moving on

These are the follow-ups an interviewer will actually ask about your projects.

01

Compare the DR maturity of your two projects and justify the difference.

Traverse is genuinely production-shaped: WAL archiving with a 5-minute RPO, offsite copies to a second vendor, blast-radius separation between backup and user-file credentials, and executable restore endpoints. Zoth has none of that — single in-cluster PVC, no archiving, no drills. The justification is data value and regenerability: traverse holds irreplaceable user history (streaks, revisions, submissions), while zoth's identity data syncs from WorkID/WorkOS and its workflow artifacts are demo-scale. That's a deliberate allocation, not an oversight — and the production checklist for zoth (archive to Azure Blob, base backups, runbook, drill) is written down.

02

What's the fastest meaningful DR improvement you could make this week?

For traverse: enable versioning + Object Lock on the backup bucket — a config change that defends against the most likely destructive scenario (credential compromise deleting backups) and takes minutes. For zoth: add archive_command pushing WAL to Azure Blob Storage — it converts an undefined RPO into a bounded one with no new services, and Azure plumbing already exists in the deployment. Both are single-afternoon changes with outsized risk reduction, which is the definition of a good first fix.

03

How would you run a restore drill?

Provision a scratch Postgres, restore the latest base backup, replay WAL to a chosen timestamp, then verify: row counts per table against the source's historical counts, a business invariant (e.g. every Solve has a Submission; every ACTIVE request has an expires_at), and the application's own smoke flow (login, submit, schedule). Record wall-clock time as the measured RTO. Schedule quarterly, and treat a failed drill as a Sev-2 — because an undrilled restore is a hypothesis, and the drill is the only place you want to discover the missing WAL segment.