Streaming percentiles, histograms, and sampling
Why p99 is a data-structure problem — and how your telemetry actually produces it.
Concept → your projects
Averages lie; percentiles don't. But percentiles are not mergeable or even storable cheaply from raw streams — you can't average two p99s. The solutions:
- Naive: store every latency → sort → exact percentile. Memory O(n), and you can't merge across instances (re-sorting the union works only if you kept everything).
- Histograms (fixed buckets): count into buckets → percentile = bucket interpolation. Cheap, mergeable (add counts), bounded error (bucket width). Prometheus/OTel's default. Choose bucket boundaries around your SLOs (50/100/250/500 ms) — that's how you make bucket error irrelevant where it matters.
- t-digest: adaptive clusters, high accuracy at the tails (p99, p999) with ~KB memory, mergeable. The choice when tail precision is the point.
- Exponential/UDD sketches: relative-error-bounded across magnitudes.
Sampling: you can't keep every trace/log at scale. Strategies: head sampling (decide at request start by rate — cheap, may miss rare errors), tail sampling (buffer the trace, decide at the end — keep all errors + a sample of successes; needs buffering, the collector does it), consistent sampling (hash-based, so the same request is sampled across services — trace integrity), and reservoir sampling for streaming examples.
Exemplars: attach a sampled raw instance (trace id) to a histogram bucket — the bridge from "p99 spiked at 14:02" to "here's the exact slow trace". OTel supports it; it's the single best observability ergonomic you can name.
What your telemetry actually does (and the honest gaps)
- Zoth: OTel SDK with a metrics reader (
sdkmetricperiodic reader) andotelhttpinstrumentation on the router + DB/queue instrumentation — so HTTP duration histograms and counts exist and export via OTLP to Better Stack. Default histogram boundaries apply (millisecond-ish buckets) — the tuning opportunity: align buckets to your own latency targets. - Traverse:
@opentelemetry/auto-instrumentations-node(HTTP, Express, Prisma, ioredis spans) + OTLP traces/logs to Better Stack;req.logcorrelated by trace context. Metrics are thinner — spans carry the durations, and Better Stack's trace analytics derive the distributions. That's a legitimate architecture (trace-derived percentiles) with one cost to name: trace sampling skews percentiles unless sampling is consistent and documented. - Both: no tail-sampling configuration, no SLO-burn-rate alerting, no exemplar wiring yet. Naming those three as the next observability investments is exactly the maturity signal interviewers want (Day 83 goes deeper on SLOs).
The percentile-pitfall list to recite (these questions get asked): p99 of p99s across instances is wrong (aggregate the histograms, then take the percentile); averages hide bimodality (two peaks = users experiencing two systems); "average latency" with 1% timeouts is nonsense (timeouts usually lower the average by truncating); and latency must be measured at the user's boundary (client-side or edge) — your server-side span misses network and queueing.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
How do you compute a correct p99 across 10 service instances?
Aggregate histograms, not percentiles: each instance exports bucket counts (OTel histogram / Prometheus bucket), you sum counts per bucket across instances, then interpolate the percentile on the merged distribution. Averaging ten p99 values is mathematically wrong — it can produce a number no user experienced. t-digest is the alternative when you need mergeable tail accuracy; fixed buckets aligned to SLO thresholds are usually enough.
What is tail sampling and when do you need it?
Sampling decided after the trace completes: the collector buffers spans and keeps all error/slow traces plus a rate-limited sample of healthy ones. You need it when head sampling (decide at request start) throws away the traces you care about — rare failures — while volume forces you to drop something. The costs: buffering memory and cross-service coordination (all spans of a trace must make the same decision, which is why consistent hash-based sampling exists as the middle path).
Your percentiles come from traces. What's the caveat?
Sampling bias: if traces are sampled at 10% with head sampling, the surviving population is representative in expectation but small enough that p999 becomes noise; and if sampling is inconsistent across services, spans go missing and durations understate. The mitigations: consistent sampling (hash on trace id so decisions agree), tail sampling (keep all errors), or — the pragmatic fix — derive SLO percentiles from metrics histograms (cheap, unsampled counts) and use traces for root-cause on the slow exemplar. That's the split I'd implement next.