Your OTel stacks and the MTTD story
Every telemetry decision in both projects, plus how to tell the Popin observability story credibly.
Know your own system
The concrete inventory — this is the material for the "tell me about observability work" question.
Zoth's telemetry (Go)
- SDK setup per binary (
internal/observability):OTEL_ENABLEDgate (telemetry is optional and fails soft — a collector outage never blocks requests), OTLP exporter → Better Stack (BETTERSTACK_SOURCE_TOKEN), resource attributes (service name, environment, version), and a composite propagator (W3C TraceContext + Baggage) so trace context flows through HTTP and through message headers. - Instrumentation:
otelhttpmiddleware on the router (server spans with route, status, duration), DB instrumentation via the driver wrapper, publisher/consumer span linking — the emitter's span continues the trace from the event's propagated context. That last detail is the crown jewel: an approval request's trace spans the API write, the outbox relay, RabbitMQ, and the GitHub API call. Say it in those words. - Logs:
slogJSON with an OTel bridge → logs exported with trace correlation, so a log line links to its span (and vice versa) in Better Stack. - Metrics: periodic reader; HTTP duration histograms, counts; the Go runtime metrics. Dashboard-able, alertable.
- Health:
/healthz(shallow),/readyz(Postgres + RabbitMQ),/pods/readyz(aggregate across services with per-component latency) — the last one is your composite SLI-in-an-endpoint.
Traverse's telemetry (Node)
auto-instrumentations-node(HTTP, Express, Prisma, ioredis spans for free),requestContextmiddleware attachingrequestId+ trace context toreq.log, OTLP → Better Stack, log redaction (sensitive fields stripped before export — the detail that separates a toy from a production pipeline), graceful OTel shutdown in the SIGTERM path (Day 8), and job-count logging for queue visibility.- The deliberate thinness: metrics come from traces rather than custom histograms — a valid choice with the caveat named (Day 71: sampling skews percentiles).
The MTTD story, credibly told
Your resume line: "Reduced MTTD from hours to under 10 minutes across all production services by implementing a centralized OpenTelemetry framework integrated with Better Stack for full-stack observability via distributed tracing, unified log streaming, and real-time anomaly alerting."
The interview expansion (this is the drill — say it out loud):
- The before state: separate log streams per service, no correlation; an incident meant grepping multiple sources and guessing which request was involved — hence hours of detection/triage.
- What "centralized" means mechanically: one SDK standard across services (same resource attributes, same propagator), one OTLP destination, and correlation keys that survive async boundaries (trace context in message headers, requestId in every log line).
- What "real-time anomaly alerting" means: Better Stack monitors/alerts on the telemetry (error rates, uptime probes) that route to a human with a link to the trace — detection becomes "open the trace and see which span failed" rather than "search logs".
- The measurement: alert-to-trace-correlation time — the metric you optimized. Detecting the alert was never the slow part; correlating it to a root cause was. That's the honest, precise framing, and it's more impressive than "we installed a tool."
The gap you should name yourself: both your projects export to a hosted backend; neither defines SLOs/burn-rate alerts, and zoth's alerting service (drift/suspicious events) is the internal equivalent of "anomaly detection" at the domain level. The complete observability story = telemetry (done) + SLOs (next) + runbooks (not written).
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Walk me through tracing an access grant end-to-end.
The API's server span (otelhttp) wraps the request; the approval handler's child spans cover the DB writes; the outbox insert carries the trace context in the row. The relay's publish span continues that context into the message headers; the emitter's consume span extracts it and continues — so one trace id covers API → Postgres → outbox → RabbitMQ → emitter → the GitHub API call. Logs from every hop carry the same trace context (slog bridge / requestContext), so an alert links to the trace and the trace links to every log line. That's what makes 'centralized observability' meaningful rather than decorative.
How did you measure MTTD improvement?
Alert-to-trace-correlation time — the interval between a monitor firing and an engineer identifying the failing span/root cause. The pre-state was hours of manual correlation across unlinked logs; post-state is minutes because the alert payload links directly to the trace, and the trace spans the async boundary. I'd be precise that the detection improvement came from monitor coverage, while the correlation improvement came from propagation across services and queue hops — two different mechanisms, often conflated in resumes.
What's missing from your observability setup?
Three things, in priority order: (1) SLOs with burn-rate alerting — I have dashboards and uptime probes, not error budgets; (2) runbooks attached to alerts, so pages carry actions not just signals; (3) tail-based sampling or metric-derived percentiles — my traverse percentiles come from sampled traces, which skews tail numbers. All three are defined work, not unknowns — which is the honest form of 'my observability is good but not finished.'