Sharding, hot partitions, and consistent hashing
The partitioning theory you skipped by design — and the multi-tenancy shortcut you used instead.
Concept → your projects
Sharding = splitting one logical table across N machines, each owning a disjoint key range/hash slice. It buys write throughput and storage ceiling; it costs cross-shard queries, transactions, rebalancing pain, and join loss — the reason it's the last resort, not the first.
Decision cascade (recite in order): (1) can vertical scaling + indexes + caching handle it? (2) can read replicas handle the read load? (3) can partitioning-within-one-host (Postgres declarative partitioning — partition pruning, cheap retention) handle size? (4) only then shard across hosts.
Postgres native partitioning is the underused middle step: PARTITION BY RANGE (created_at) splits submissions into monthly partitions — queries with time predicates prune, retention = DROP TABLE (instant vs DELETE), and it's still one database (one connection string, cross-partition joins work). Your future-self note: traverse submissions/notifications and zoth audit_logs are the natural partition candidates if they ever grow — audit by month gives compliance "retention windows" for free.
Shard-key selection: pick the column that (a) appears in every hot query, (b) has high cardinality, (c) distributes evenly. In traverse that's unambiguously userId (every query is user-scoped). Cross-shard casualties: admin analytics (needs scatter-gather), the social graph (friends live on different shards —Instagram-style denormalization: write the friendship to both users' shards).
Hot partition: disproportionate load on one key slice — one celebrity user, one tenant 1000× bigger. Mitigations: key salting (append random suffix, scatter the hot key), request-level caching, dedicated pool for known-hots. Your domain's built-in risk: tenant-based partitioning (natural for zoth) makes every megatenant a hot partition by construction — which is exactly why zoth doesn't partition by tenant at this scale: per-tenant scoping in queries already gives isolation without the placement problems.
Consistent hashing (Day 21 mechanics, now in sharding context): hash shard-key → ring → owning node; node add/remove moves ~1/N keys. The rebalancing alternative — range-based with a metadata service (which range lives where) — allows controlled migration but needs the metadata service + manual splits. Kafka/Dynamo chose the former; Bigtable/Spanner lineage chose the latter.
Rebalancing data: dual-write or backfill-with-cursor (your broadcast fanout's cursorUserId pattern is a backfill cursor — same shape you'd use to migrate rows between shards), catch-up, cutover by key range.
What your systems chose, and how to say it
- Zoth: single Postgres; logical isolation (tenant_id on every query path) instead of physical partitioning. Capacity math (Day 3) says a tenant's data is KBs-to-MBs — physical isolation would be paying operational complexity for zero load reason. If enterprise reality demanded physical isolation: schema-per-tenant (pg schema search_path) or db-per-tenant at the extreme, both below sharding on the complexity ladder.
- Traverse: single Postgres;
userIdis the ready shard key the day write QPS demands it; BullMQ already spreads work across workers (the processing layer scaled before the storage layer needed to).
The senior line: "I partitioned nothing, and I can tell you exactly what would trigger each level: time-partition audit/submissions when retention pruning matters; shard traverse by userId when sustained writes exceed a single Primary; salt or cache a hot tenant when one appears. Every step has a trigger measured in numbers, not vibes."
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Why not shard by tenant for hard isolation?
It conflates two goals: security isolation and load distribution. Security isolation is already enforced logically — every query path carries tenant_id, fetches are tenant-asserted, and one bug there is a bug at any sharding level too (sharding doesn't fix query bugs, it hides them from tests). Load-wise, tenants are tiny — sharding would multiply operational complexity (cross-tenant admin queries, rebalancing, per-shard connection pools) for a workload measured in MBs. If a compliance requirement demanded physical isolation, the ladder is schema-per-tenant then db-per-tenant — both simpler than hash-sharding.
How would you migrate traverse to sharded submissions without downtime?
Dual-write phase (new writes go to old + new shard by userId-hash), backfill historical rows with a cursor batch worker (my broadcast fanout's cursorUserId pattern is exactly this), catch up by reading the old shard's WAL tail or timestamps, then switch reads per-shard and stop old writes. Each phase is independently reversible. The hard part isn't mechanics — it's dual-write consistency, which is why I'd route both writes through one code path that treats 'old store + new store' as an atomic outbox-style fanout.
What makes a shard key bad?
Low cardinality (status enum — one hot partition per value), monotonic timestamps (all writes land in the newest shard — the Bigtable hot-tablet problem; needs salting or hash), or keys that don't match query shape (sharding submissions by problemId when every query is user-scoped = every request scatter-gathers). The test I apply: 'do my top 5 queries each touch exactly one shard?' — traverse's userId passes; a 'global leaderboard' query never will, and that query must move to an aggregation layer (Day 69-71 sketches) rather than punish the OLTP shard key.