S3 in traverse — the complete file-lifecycle stories
WAL archive, career resumes, recon streaming: three file flows, each with a distinct design lesson.
Know your own system
Flow 1 — WAL archive (the DR backbone, Day 33/34 already covered; here the object mechanics)
archive_command pushes each completed segment to traverse-db-backups-697114252450 (us-east-1). Object key = segment filename (timestamped WAL names are naturally unique + ordered — the one place where sequential keys are good: restore reads them in order, no hot partition risk at DB archive volume). Objects are immutable → no versioning needed → lifecycle to cheaper classes possible for old segments. The pg-backup-sync.sh host-cron (3-day) adds base backups + syncs the set to Aiven. The operational honesty: retention is manual (segments accumulate; a lifecycle rule is the unimplemented polish — one bucket rule, zero code).
Flow 2 — career applications (user files done right)
Route: POST /api/careers/applications (multipart) → controller → service: PDF stored to the careers S3 bucket with key applications/{applicationId}/{filename}, metadata (status SUBMITTED, resume key, parsed role/fit) in Postgres, then a notification email to careers@ (the "notify a human" step — always make the human-fanout explicit in your story). Get/resume URL served from the row's key; admin approval flow flips status. Design points to claim: DB stores keys and truth, never blobs (Day 57's rule); separate bucket/region/credentials from backups (blast-radius isolation); size/limits validated before upload; the resume is immutable per application (new attempt = new application row + key, not overwrite — append-only thinking again).
Flow 3 — recon streaming (the engineering flex)
The extension accumulates capture bundles (network responses, DOM snapshots — potentially unbounded). Flow: POST to /api/recon — mounted before the JSON parser (Day 18) — handler streams the raw body to disk in chunks (no full buffering) to a .partial file → atomic rename when complete → per-platform quota accounting (KEEP_PER_PLATFORM — the retention knob) → a digest email is sent after the flush (batch notification instead of per-bundle spam). If the process dies mid-stream: .partial remains, never renamed → never visible to readers, safe to overwrite next attempt. That's atomic visibility — the same property multipart-complete gives S3, implemented in the small with rename(2).
The generalization (say this): three file flows, three correct storage decisions: WAL → append-only object log for recovery; resumes → user blobs with DB-owned keys; recon → local streaming with atomic visibility (the data is diagnostic, transient, quota'd — not S3-worthy). Each matched to durability needs, access patterns, and blast radius — no "everything goes to S3" reflex.
The missing piece to volunteer: S3 versioning is off (buckets are private + least-privilege IAM, but object overwrite/delete is single-keystroke). For backups, enabling versioning + Object Lock (WORM) would defend against the deletion attack (ransomware deleting backups) — the DR topic Day 87 picks up. One-line fix, real security posture.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Walk me through a recon dump end-to-end.
The extension buffers capture bundles per platform. POST /api/recon — routed before the JSON parser so the multi-MB body never hits express.json — the handler pipes the request stream to disk in chunks under a .partial file. On completion: atomic rename makes it visible, per-platform quota accounting (KEEP_PER_PLATFORM) prunes old dumps, and a digest email summarizes the flush instead of notifying per bundle. Crash mid-stream leaves a .partial that's invisible to readers and gets overwritten. The key property is atomic visibility: readers see complete dumps or nothing.
Why are resumes stored in S3 while their metadata is in Postgres?
Separation of blobs and state: files are multi-hundred-KB PDFs with per-application ownership — identity-keyed blobs, object-storage semantics; the application state (status, role, fit score, resume key) is transactional and queried — relational semantics. The DB row references the object key, uploads validate size/type first, and the bucket lives in a separate region with separate credentials from the backup bucket — so user-file access and database-backup access are independent blast radii.
What would you harden on the S3 side?
Three things, in order: versioning + Object Lock on the backup bucket (WORM defends backups against deletion attacks — ransomware's first move is deleting the backups); a lifecycle rule for WAL segments (cost discipline, zero code); and CloudWatch/EventBridge alerts on backup-bucket object deletion (anomaly signal). All three are bucket-config changes, not code — which is itself the point: object storage's operational surface is declarative; use it.