Backups, disaster recovery, retention, and deletion
RPO/RTO, the 3-2-1 rule, and what "delete" has to mean when a regulator asks.
Concept → your projects
The two numbers: RPO (recovery point objective — how much data may be lost, i.e. backup frequency/WAL cadence) and RTO (recovery time objective — how long until service returns). Everything else is engineering to hit them. Interview discipline: state both, then the mechanism that enforces each.
The 3-2-1 rule: 3 copies, 2 media/locations, 1 offsite. Modern addendum: 1 immutable/offline copy (ransomware deletes reachable backups first) — object-lock/WORM buckets or offline tapes.
Backup types: full (restore-ready, expensive), incremental/differential (cheaper, longer restore chain), continuous (WAL archiving / PITR) — the database-native version: base backup + WAL replay to any point (traverse's S3 + Aiven setup). Restore drills are the only proof: an untested backup is a hypothesis.
DR tiers (cost ladder): backup-and-restore (hours), pilot-light (core DB replicated, app deployed on demand), warm standby (scaled-down running), multi-site active-active (zero-ish downtime, highest cost + consistency complexity). Choose by RTO — most products need the middle.
Retention & deletion (the compliance half): retention policies per data class (access logs 1 y, user content until deletion + grace, backups 35 d), legal hold (must override deletion), and GDPR/DPDP-style erasure: delete means actually delete — including from backups eventually (the standard practice: erasure executes in live data immediately and expires naturally from backups within the retention window, documented as such), plus derivative cleanup (caches, search indexes, analytics rollups — the "delete everywhere" problem, which is where soft-delete habits hurt you).
Crypto-shredding (the elegant answer for backups): encrypt per-subject data with a per-user key; erasure = destroy the key → ciphertext becomes noise even in immutable backups. Requires key management discipline; it's the pattern to name when asked "how do you delete from a WORM backup?"
Your systems' posture (audit table)
| Traverse | Zoth | |
|---|---|---|
| RPO | ≤5 min (archive_timeout) | none defined (dev-scale) |
| RTO | minutes (base + WAL replay) | redeploy + seed |
| Copies | live + S3 WAL + Aiven offsite (3 locations) | in-cluster PVC only |
| Immutability | none (versioning off) | n/a |
| Restore drills | admin endpoints exist (executable) | none |
| Deletion | User soft-delete + 7-day grace; then? | hard delete + audit chain |
The honest summary sentence: "Traverse has a real DR posture — WAL archiving with a bounded 5-minute RPO, offsite copies, and executable restore endpoints — but no immutability on the backup bucket and no drilled restore in production. Zoth has no backup story at all; its recovery plan is redeploy, which is defensible only because it's a demo-scale system whose data is regenerable — and I'd say that out loud rather than let someone discover it." Volunteering the weakest link with the exact fix (object-lock on the backup bucket; archive_command to Azure Storage for zoth) is the bulletproof move.
The deletion gap to name in traverse: soft-delete + grace is a product feature; the compliance story needs a purge job that (a) hard-deletes after grace, (b) cascades to caches/analytics, and (c) documents what remains in backups. Currently the purge step is not implemented — a one-job fix with a clear spec.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
State your RPO/RTO and how you enforce them.
Traverse: RPO ≤5 minutes enforced by archive_timeout=300 forcing WAL segment switches even on an idle database, with segments shipped to a dedicated S3 bucket; RTO in the minutes range — restore a base backup, replay WAL to the target timestamp. Enforcement beyond config: restore is exposed as admin endpoints (status/backups/restore/rollback-safety), so recovery is executable rather than aspirational. Zoth: I'd say plainly that RPO/RTO aren't defined — single in-cluster Postgres with no archiving — and the fix is archive_command to Azure Blob plus a documented restore runbook.
How do you delete a user's data if backups are immutable?
Two layers: live data is hard-deleted (plus cache/analytics/derivative cleanup — the part soft-delete habits break), and backups are handled by policy, not by editing them: the deletion is documented, the backup retention window (e.g. 35 days) bounds how long the data persists off-system, and if the requirement is stricter, crypto-shredding — per-subject encryption keys destroyed at erasure time, rendering ciphertext in immutable backups useless. That's the honest answer regulators accept: bounded residual copies, declared in the retention policy.
What's wrong with an untested backup?
It's a hypothesis, not a capability. Failures hide in the chain: unreadable segments, missing base backups, wrong credentials after rotation, restore steps that only worked on someone's laptop. The only proof is a drill — restore into a scratch database, verify row counts and a business invariant, record the time. Traverse is halfway there (executable endpoints, documented scripts); neither project has a scheduled drill, which is the gap I'd close first because RTO is a lie until measured.