DR posture of both projects, itemised
Every durability mechanism you own — plus the three gaps to fix before production.
Know your own system
The itemised audit, ready to recite.
Traverse — what exists
- WAL archiving:
wal_level=replica,archive_mode=on,archive_timeout=300,archive_command→ S3 buckettraverse-db-backups-697114252450(us-east-1). Bounded RPO of 5 minutes, including idle periods (the timeout is the enforcement). - Offsite base backups:
scripts/pg-backup-sync.shon host cron (every 3 days) syncing to Aiven (AIVEN_BACKUP_DB_URL) — a managed provider as the "different vendor, different failure domain" copy (the 3-2-1's offsite leg). - Blast-radius separation: backups in a different region and with different credentials than user files (careers bucket, ap-south-1). A leaked application credential cannot read database backups.
- Executable recovery:
pitr.service.ts— status, backup listing, snapshot, restore, rollback-safety, history, exposed as admin endpoints. Recovery is a documented, callable procedure. - Redis AOF: job durability across restarts (≤1 s loss), paired with idempotent handlers.
- Immutable-ish data model: submissions/solves/attempts are append-mostly; the derived state (revisions, streaks) is recomputable from facts — so partial loss degrades features rather than corrupting truth.
Traverse — the three gaps (fixes ready)
- No immutability on the backup bucket: versioning + Object Lock (WORM) defends against deletion attacks (ransomware's first move). Bucket config, zero code.
- No drilled restore: the endpoints exist; a scheduled quarterly drill (restore into a scratch DB, verify invariants, record RTO) is the missing practice.
- Soft-delete purge not implemented:
User.deletedAt+ grace exists; the job that hard-deletes after grace (and cascades to caches/derivatives) doesn't — so "deleted" data persists indefinitely, which is both a compliance and a storage issue.
Zoth — what exists and what doesn't
Exists: Postgres in-cluster with a PVC (survives pod reschedule); Aiven managed PG in the prod env config (.env.prod) — so a managed instance is provisioned, which is the seed of a real story; migrations embedded (schema is reproducible from the repo); the audit chain (integrity, not durability); drift detection (reconciles external state, not data loss).
Doesn't exist: WAL archiving, base backups, restore runbook, drills, RPO/RTO definitions. Recovery = redeploy + re-seed.
The honest framing (this is the version to use): "Zoth's data is demo-scale and largely regenerable (users/groups sync from the identity provider; access requests are workflow artifacts). I chose not to spend the budget on backup infrastructure for data I can reconstruct — and I can say exactly what production would require: archive_command to Azure Blob, daily base backups, a restore runbook, and a monthly drill. Traverse, whose data is not regenerable (user history), got the full treatment. Matching durability investment to data value is the decision; the mistake would be doing neither."
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Compare the DR maturity of your two projects and justify the difference.
Traverse is genuinely production-shaped: WAL archiving with a 5-minute RPO, offsite copies to a second vendor, blast-radius separation between backup and user-file credentials, and executable restore endpoints. Zoth has none of that — single in-cluster PVC, no archiving, no drills. The justification is data value and regenerability: traverse holds irreplaceable user history (streaks, revisions, submissions), while zoth's identity data syncs from WorkID/WorkOS and its workflow artifacts are demo-scale. That's a deliberate allocation, not an oversight — and the production checklist for zoth (archive to Azure Blob, base backups, runbook, drill) is written down.
What's the fastest meaningful DR improvement you could make this week?
For traverse: enable versioning + Object Lock on the backup bucket — a config change that defends against the most likely destructive scenario (credential compromise deleting backups) and takes minutes. For zoth: add archive_command pushing WAL to Azure Blob Storage — it converts an undefined RPO into a bounded one with no new services, and Azure plumbing already exists in the deployment. Both are single-afternoon changes with outsized risk reduction, which is the definition of a good first fix.
How would you run a restore drill?
Provision a scratch Postgres, restore the latest base backup, replay WAL to a chosen timestamp, then verify: row counts per table against the source's historical counts, a business invariant (e.g. every Solve has a Submission; every ACTIVE request has an expires_at), and the application's own smoke flow (login, submit, schedule). Record wall-clock time as the measured RTO. Schedule quarterly, and treat a failed drill as a Sev-2 — because an undrilled restore is a hypothesis, and the drill is the only place you want to discover the missing WAL segment.