Observability, SLOs, and error budgets
The three pillars, the SLO math, and how alerts should be designed — the theory behind your Popin work.
Concept → your projects
Three pillars: traces (one request's path across services — causality), metrics (aggregatable time series — trends and alerts), logs (discrete events — detail and forensics). The modern framing adds correlation: every pillar carrying the same trace/request id so you pivot from an alert (metric) → the slow trace → its logs. That correlation is the entire value proposition — three separate tools without shared context are three separate investigations.
SLI/SLO/SLA:
- SLI = the measurement (
fraction of requests < 300 ms,availability = successful/total). - SLO = the target (
99.9% over 30 days). - SLA = the contract with consequences (money/credits) — usually looser than the internal SLO.
- Error budget =
1 − SLO= the allowed failure: 99.9% over 30 days ≈ 43 minutes. The budget reframes reliability as a resource: spend it on launches (risky deploys) or preserve it (freeze). This is the single most useful cultural device in SRE — and a great interview topic because it turns "how reliable?" into "how much failure can we afford this month, and on what?"
Alerting discipline:
- Alert on symptoms (user-visible: error rate, latency) not causes (CPU high) — cause alerts are pages for problems users don't have.
- Multi-window burn-rate alerts: fast window (2% budget in 1 h → page now) + slow window (5% in 6 h → ticket) — catches both sudden outages and slow bleeds. Burn rate = how fast you're spending the budget relative to pace.
- Every alert needs a runbook and an owner; alerts without actions are noise that trains people to ignore pages.
Incident response (the loop): detect (alert) → triage (severity, impact) → mitigate (rollback first, root-cause later — mitigate before diagnosing is the counterintuitive rule) → communicate (status page, stakeholders) → resolve → blameless postmortem (timeline, contributing factors, action items with owners/dates). The output of an incident isn't punishment; it's a system change.
Distributed tracing specifics: W3C traceparent propagation across HTTP and message boundaries (your RabbitMQ payloads and OTel context), span attributes for the dimensions you'll query (tenant_id, platform, route), and sampling (Day 71). The payoff you can claim: "one trace shows API → Postgres → outbox → RabbitMQ → emitter → GitHub" — end-to-end causality across an async boundary, which is the hardest part of distributed debugging.
Map to your systems
- Zoth: OTel SDK in all three binaries (traces + metrics + logs with the slog bridge), OTLP → Better Stack,
otelhttpon the router, DB/queue instrumentation, health endpoints (/healthz,/readyz,/pods/readyz) as the aggregate signal, Better Stack uptime monitors on 3 continents at 3-minute intervals. Missing: explicit SLO definitions and burn-rate alerts (you have dashboards and uptime checks, not error budgets) — name that gap and the fix (define 2–3 SLIs: API availability, request latency, provisioning lag; alert on burn rate). - Traverse:
auto-instrumentations-node+ OTLP traces/logs,requestContextcorrelation, job counts in logs. Same gap: no formal SLOs; the health endpoints (/api/health/db) serve deploy gating rather than availability monitoring.
The resume-relevant claim (Popin): "centralized OpenTelemetry with full-stack observability via distributed tracing, unified log streaming, and real-time anomaly alerting" → in interview terms: correlation. MTTD dropped because an alert could be answered by opening the trace, not by grepping logs across services. Day 84 drills that.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
What is an error budget and how does it change decisions?
It's the inverse of the SLO: 99.9% availability over 30 days leaves ~43 minutes of allowed failure. It converts reliability from an absolute into a budget you spend deliberately — risky deploys and experiments consume it, and if you're burning faster than pace (burn rate > 1), you freeze features and fix reliability. The practical effect is that 'should we ship this risky change?' becomes 'can we afford its risk this month?', which is a decision with data instead of a debate. Neither of my projects has formalized budgets yet; defining three SLIs and burn-rate alerts is my next observability step.
Why alert on burn rate rather than a threshold?
A fixed threshold (say error rate > 1%) either pages on trivial blips or misses slow degradation. Burn rate expresses how fast you're consuming the error budget relative to the allowed pace: 14.4× burn for 1 h means you'll exhaust a 30-day budget in ~2 days — page now. A 5% budget consumption over 6 h is a ticket, not a page. Multi-window burn-rate alerting catches both sudden outages and slow bleeds with one coherent rule, and it scales the urgency to the actual consequence.
Why mitigate before diagnosing?
Because user impact compounds while you investigate. The first moves are reversible ones: roll back the deploy, flip the feature flag, shed load, fail over. Root cause can be found from the preserved artifacts (traces, logs, the rolled-back commit) after users are healthy. The cultural half matters too: postmortems are blameless and produce system changes — otherwise the next incident's mitigation is delayed by fear of blame.