Load balancers, gateways, and consistent hashing
L4 vs L7, health-checked backends, and the hashing trick that keeps cache hits when nodes change.
Concept → your projects
A load balancer distributes traffic; the L4/L7 distinction is the first question:
- L4 (TCP/UDP): routes by connection — fast, no payload inspection, sticky by 4-tuple. Can't route by path/auth.
- L7 (HTTP): parses requests — route by path, host, headers; can retry idempotent GETs; the layer where nginx/Envoy/ALB live.
Core LB algorithms: round-robin, least-connections, least-request, random (surprisingly good), and consistent hashing — hash the client (or a key) onto a ring so the same key lands on the same node (cache affinity / sharded ownership), and when a node joins/leaves only ~1/N of keys move, versus all keys under hash % N.
Consistent hashing mechanics (interview-whiteboardable): hash nodes onto a ring (with 100–200 virtual nodes each to smooth skew); a key hashes to a position, owned by the next node clockwise; node death passes its arc to its neighbor. Traverse/zoth don't use it — know why: it solves state-partitioning, and your state (Postgres) isn't partitioned. The honest line: "I don't need it until I need a distributed cache or sharded writes — e.g., sharding traverse's submission stream by userId, or cache affinity for zoth's policy lookups across API replicas."
Health checking: passive (stop routing on failures) vs active (probe endpoints). The LB should route on readiness, not liveness — which is why your K8s Services only include pods whose readyz passes. When you add a second api replica, the K8s Service (ClusterIP) is the L4 LB — kube-proxy/iptables round-robin, health-check-free because pod readiness gates membership. That's your "we already have a load balancer" answer: Kubernetes is the LB layer.
API gateway vs LB: an LB spreads traffic; a gateway terminates concerns — auth, rate limiting, request shaping, response caching, per-route policy. Kong/Envoy/APISIX. Your systems have no gateway because each concern is already a middleware in the one service that exists. When would zoth need one? Multiple public services + per-tenant rate limiting + auth offload → gateway in front, services focus on domain. Cost: another hop, another failure mode, another thing on the $30 ledger.
The zoth networking story (as deployed)
- ClusterIP Services for api/emitter/alerting/rabbitmq — DNS names (
rabbitmq.zoth.svc.cluster.local,zoth-emitter:8080) via kube-dns; the ConfigMap wiresEMITTER_HEALTH_URLto the service name. - No ingress (cost, Day 10); prod overlay's nginx-ingress yaml exists with
ssl-redirect:false, no host rule — the documented compromise. - Aggregate health via
/pods/readyz— the API polls the other services through their Services: a poor-man's service mesh health check. Honest framing: a real mesh (Istio/Linkerd) gives mTLS, retries, and outlier detection between services — overkill for 3 services on 1 node, and CPU-prohibitive on a B2s.
Practice
"LB for a chat server": L7 needed (route by user → the server holding their WebSocket), consistent-hash on userId so reconnects find the same node; fallback: a presence/registry service (Redis) mapping user→node — which is how you'd actually do it, because ring state and connection state must agree.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
You have one api replica. What changes at three replicas?
Kubernetes Service already load-balances them (ClusterIP round-robin, readiness-gated). What I'd audit: (1) in-memory state — the token-bucket rate limiter and the per-request caches become per-pod (move limiter to Redis); (2) background loops — expiry loop is already race-safe via conditional updates, outbox relay claims via SKIP LOCKED, so both scale cleanly; (3) JWT needs no sticky sessions — stateless; (4) connection pools multiply against Postgres — watch pool sizing. The design was made horizontally-scalable by accident-of-correctness: every shared mutation is conditional or claimed.
When would you actually deploy consistent hashing?
When keys, not requests, need affinity: sharded write throughput (partition submissions by userId), distributed cache locality (each policy object lives on one cache node so invalidations are single-node), or WebSocket routing. None apply yet — single Postgres, single Redis, no sockets. I can derive it when needed: ring + virtual nodes + clockwise ownership; ~1/N keys move on topology change vs 100% under modulo.
What does an API gateway give you that your middleware doesn't?
Centralization across multiple services: one auth boundary, one rate-limit keying scheme, one place for request/response transforms and per-route policies, plus observability and retries uniform across services. With one public service, the middleware IS the gateway — adding Kong would be a second implementation of what Chi middleware already does. The trigger is service count and team count, not traffic.