Backpressure, batching, tail latency, and load shedding
The four tools for surviving more traffic than you can process — and where your systems already use them.
Concept → your projects
When arrival rate exceeds service rate, queues grow unboundedly and latency → ∞. Four responses, in escalating order:
- Backpressure — tell the sender to slow down. Bounded queues (reject when full), TCP window collapse, HTTP 429 + Retry-After, BullMQ per-queue limiters. RabbitMQ's
Qos(prefetch=10)is credit-based flow control: the broker won't hand a consumer more than 10 unacked messages — the consumer pulls its own pace. Traverse's AI queue:limiter: {max: 20, duration: 1000}— at most 20 AI jobs/second started. - Batching — amortize fixed costs (network RTT, commit fsync). Postgres
COPY/multi-row inserts; traverse's broadcast fanout inserts 500 notification rows per batch viacreateManyAndReturn(two queries per batch instead of 2N); the achievements refactor collapsed ~120 queries into 4 regardless of how many awards fire. - Tail latency — p99 is what users feel and what interviewers ask. Causes: queueing (the M/M/1 burst problem), GC pauses, lock contention, cold caches, retries. Remedies: hedged requests, timeout+shed, isolate slow dependencies (your #1 real example: moving 15 s LLM calls off the request path killed the p99, because one slow dependency owned the tail).
- Load shedding — drop work deliberately, in priority order. Priority queues (traverse:
priority: accepted ? 1 : 2— accepted solves jump the queue), 503-on-overload, dropping best-effort work (suspicious-activity events are fire-and-forget; queue overflow on them would be acceptable).
Your best narrative — the tail-latency story: "The 10 s p95 wasn't average slowness, it was the tail: a sequential LLM provider ladder where each provider gets 15 s. Averages were fine; the tail was owned by the slowest provider × retries. The queue fix didn't make work faster — it moved the tail off the user's critical path. The 202 response means the tail now lands on the worker, where concurrency 5 and a 20/s limiter bound it."
Backpressure failure you should own: traverse's rate limiters are per-process in-memory — under a flood, each process sheds independently, but there's no global signal, and nothing backpressures the extension (which retries submissions — safely, thanks to idempotency keys; duplicate POSTs return 200). The correct endgame: 429 with Retry-After honored by the extension + Redis-backed sliding window (Day 89).
The queue-lag lens (preview of Day 61)
Backpressure's health metric is lag — the age/depth of the backlog. BullMQ exposes job counts; your notification drain runs every 30 s claiming batches of 50 — lag above ~2 minutes means either slow APNs or a flood. Zoth's outbox: the relay polls 100 events every 5 s; the observable lag metric is outbox_events WHERE published_at IS NULL age. Designing lag observability before an incident is the difference between pagers and archaeology.
Practice
"Rate limiter for a public API": token bucket per key (permits bursts, O(1) memory) behind a gateway (Day 89 covers placement/keys), Redis-backed for multi-replica, 429 + Retry-After as the backpressure contract, and shedding order: drop analytics first, then best-effort features, never auth.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Where does your system apply backpressure vs where does it just absorb load?
Explicit: RabbitMQ prefetch=10 backpressures the emitter to its own capacity; BullMQ's group limiters (100/s submissions, 20/s AI) pace worker start rates; express-rate-limit sheds request floods (5/15 min on auth). Absorbed by design: the outbox absorbs broker outages unboundedly (bounded only by disk — fine because events are tiny), and the extension retries safely via idempotency keys. The one I'd flag: nothing backpressures the extension client-side — no 429-aware backoff — which is fine at 100 users and would be the first fix at 10k.
Why is p99 more important than the mean?
Because user experience and downstream timeouts are set by the tail: if 1% of requests take 10 s, every client with a 5 s timeout fails 1% of the time — the error rate is the tail, not the mean. Mechanically, means hide coordination losses: one straggler in a fan-out of 100 calls dominates when you wait for all (my readiness aggregator runs checks concurrently with a 5 s context precisely so the slowest dependency can't stretch the whole probe). Traverse's fix targeted the tail explicitly: the LLM ladder owned p95; removing it from the request path moved the tail to a place where nobody waits on it.
How do you decide what to shed first?
By value-per-request and idempotency. Shed order in my systems: analytics/telemetry events first (best-effort, e.g. suspicious-activity events are advisory), then non-critical notifications (deferrable via quiet-hours machinery), then AI enrichment (scheduling falls back to behavioral signals — the system is designed to degrade), and never: auth, persistence of submissions, or access-state changes in zoth. Load shedding is only safe if every tier you drop has a defined degraded mode — which is why the FSRS scheduler works without AI output.