Media and files — direct-to-storage upload, image pipelines, CDNs
The standard media architecture and the security boundary that makes it work.
Concept → your projects
Direct-to-object-storage upload (the architecture to know): client asks your API for a pre-signed URL (or a scoped upload credential) → API authorizes the request, computes the object key, signs a short-lived PUT/POST policy → client uploads directly to S3/R2 → client notifies your API (or the bucket fires an event) → API records the object key in the database. Why it's the default: your API never buffers or streams megabytes (no bandwidth, no memory, no event-loop stalls — Day 7's traverse recon lesson), uploads scale with S3's capacity rather than yours, and the API stays the authorization boundary while storage does the transport.
Pre-signed URL mechanics (interview detail): a signature over method + key + expiry (+ content-length/type constraints) using the storage credential — anyone holding it can perform exactly that operation until it expires. Therefore: short TTLs (minutes), narrow scope (one key, one method, content-type/length constraints), and never include anything the client shouldn't control. Validate after upload (size, magic bytes/content sniffing — the extension is a lie) before treating the object as valid, and consider a quarantine prefix → virus scan → promote pattern for untrusted files.
Image pipelines: upload original → asynchronous resize (worker, or on-the-fly at the CDN edge) → variants (/img/{id}/w=320.webp) → cache forever (content-addressed or immutable-keyed). Formats: WebP/AVIF, responsive srcset, and cache-control immutability (max-age=31536000, immutable) with new keys per version. The key insight: immutable keys + infinite cache is the only cache strategy that doesn't require invalidation (the same discipline as content addressing).
CDN mechanics: edge PoPs cache by URL; Cache-Control/s-maxage + Vary control freshness; origin shield reduces origin load; signed URLs (CloudFront signed cookies/URLs) for private content — the CDN is also the auth boundary for media. Pull (lazy cache) vs push (pre-warm) — pull is the default; push for known-hot launches. Cache invalidation at the edge is slow/expensive → design for immutable keys instead.
File sync (your topic list): the hard version of file upload — chunking (fixed-block vs content-defined chunking/CDC via rolling hash), dedup by chunk hash, delta sync, conflict resolution (version vectors or last-write-wins with tombstones), and resumability (chunk-level acks). Dropbox-style systems: fixed 4 MB blocks + per-file manifest + content-addressed chunk store → a changed file uploads only changed chunks. Content addressing (Day 57): key by hash → dedup + integrity for free; the manifest (ordered chunk hashes + metadata) is the file's real identity.
Your systems' mapping (and the gaps to name)
- Traverse career resumes: server-side upload today (multipart through the API) → the direct-to-S3 upgrade is the standard next step: presign in the API (auth + key), upload from client, then a confirm callback validates and records. You already have the pieces (S3 client, per-bucket credentials, DB-owned keys).
- Traverse recon dumps: streamed server-side by design (the extension is a browser context; the API is a trusted intermediary and applies quotas) — a justified deviation from direct-upload, because server-side streaming enables per-platform quota accounting and digest emails. Naming why you deviated is stronger than blind pattern-following.
- Images: none — no avatar pipeline yet; the future shape is presign + async variants + immutable keys.
- zoth: no media at all; the only "files" are policy documents (text) — so no CDN story, and the honest answer says so.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Why is direct-to-storage upload the default architecture?
Because the API should be the authorization and metadata boundary, not a byte pipe: presigning is a stateless computation, and the client's upload then consumes S3 bandwidth instead of my process's memory and event loop. At 10 MB per file, server-side streaming would make my API the bottleneck and the cost center; direct upload scales with the storage provider. The tradeoff I accept: validation moves to post-upload (size, content sniffing) and I need a confirm step to record the object — both manageable, and both cheaper than proxying.
What are the security pitfalls of pre-signed URLs?
Over-broad scope (signing a prefix instead of a single key), long TTLs, missing content-type/length constraints, and trusting the client's declared metadata. My rules: one key per signature, minutes of TTL, constrain method and content-length, then validate the object after upload (real size, magic bytes — extensions lie) before recording it as valid. For untrusted files, upload to a quarantine prefix, scan, then promote — never serve directly from the upload location.
How does content-defined chunking beat fixed blocks?
Fixed 4 MB blocks shift every subsequent boundary when bytes are inserted or deleted near the start, so almost all chunks change. CDC picks boundaries by content (rolling hash hitting a target pattern), so insertions disturb only nearby chunks — meaning dramatically better dedup for edited files and for similar files across users. The costs: CPU for the rolling hash and variable chunk sizes to manage. For backups/sync of edited documents, CDC wins; for immutable uploads, fixed blocks are simpler and equally good.