Popin Tech, part 1 — OpenTelemetry, Better Stack, and the MTTD claim

Your first resume line is the strongest and the most exposed: "Reduced mean time to detect (MTTD) production incidents from hours to under 10 minutes across all production services, as measured by alert-to-trace-correlation time, by implementing a centralized OpenTelemetry framework integrated with Better Stack for full-stack observability via distributed tracing, unified log streaming, and real-time anomaly alerting." That single sentence contains six separate things an interviewer can pull on. This article makes each one defensible.

What OpenTelemetry actually is (the layer under the buzzword)

OTel is not a product. It is three things stacked:

1. A specification — the data model for traces (a trace is a tree of spans; a span has a name, start/end, attributes, events, status, and a parent), for metrics (counters, gauges, histograms with explicit bucket boundaries), and for logs (structured records that can carry the active trace context). Plus context propagation: the W3C traceparent header (version-traceid-spanid-flags) that lets a span in service B point at its parent in service A.
2. APIs and SDKs — what your code calls (tracer.Start(ctx, "grant.revoke")) and what actually batches, samples, and exports. Language-agnostic; the Go SDK, the Node SDK, and the Python SDK all produce the same wire format.
3. The Collector — a standalone binary that receives OTLP, can batch/redact/sample/fan-out, and forwards to any backend. This is the piece that makes the framework centralized: every service exports to the same collector with the same resource attributes, and the collector decides where data goes. Swap Better Stack for something else and not one line of application code changes.

Resource attributes are the part people skip and then regret: service.name, service.version, deployment.environment, k8s.namespace.name, cloud.region. Get these consistent across every service and every signal, and "show me the traces for the payments service in staging on version 1.4.2" becomes a query instead of an archaeology project. Get them inconsistent and you have four dashboards that disagree.

How the framework was actually structured

The word centralized is doing real work in that bullet. The implementation pattern:

• One shared instrumentation package per language, imported by every service — so nobody hand-rolls a tracer, and the resource attributes, sampler, and exporter config are identical everywhere by construction. This is the difference between a framework and a pile of init() functions.
• Auto-instrumentation at the edges: HTTP server/client, database driver, Redis client, message-queue producer/consumer. These produce the spans you would never write by hand (connection pool waits, DNS, TLS handshake) and they are usually where the answer lives.
• Manual spans at business boundaries where auto-instrumentation goes blind: "evaluate policy", "provision GitHub team", "send notification". Attributes on these spans are your own domain vocabulary — tenant.id, resource.type, decision=allow.
• Unified log streaming: logs emitted through a structured logger that injects the active trace_id and span_id into every record. This is the single highest-leverage trick in the whole setup — it converts logs from a separate silo into a pivot target off a trace.
• Real-time anomaly alerting on top: alerts are defined on the metrics and log patterns the collector already receives, so alerting and tracing share the same pipeline instead of being two systems that describe the same outage differently.

What "alert-to-trace-correlation time" means, and why it is the right metric

MTTD — mean time to detect — is normally "time from incident start to a human knowing something is wrong." The resume sharpens it: alert-to-trace-correlation time, i.e. the gap between an alert firing and the responder having the trace that explains it in front of them. That is a better metric than raw MTTD for one reason: raw MTTD is dominated by "did the alert fire at all", which is monitoring coverage, whereas correlation time is dominated by context availability, which is exactly what an observability framework buys you.

The mechanism that collapses that gap has three parts:

1. Exemplars and deep links. A latency histogram bucket carries an exemplar — the trace_id of a request that landed in that bucket. Click the spike, land on the trace. No search, no guessing.
2. Alert payloads that carry identity. The alert notification includes service, environment, version, and a trace link, so the responder's first action is opening a trace, not opening a terminal.
3. Log↔trace↔metric pivotability. Because every log line has the trace ID, the flow is: metric spike → exemplar trace → slow span → the log lines emitted inside that span. Before this, the flow was: alert → SSH → grep → read code → guess.

That is the honest causal chain for "hours to under 10 minutes": you did not make incidents rarer or shorter, you removed the hunt. And that distinction is exactly the question an interviewer is probing when they ask "so what did you actually reduce?"

The parts that will get stress-tested

Overhead. Expect "what does this cost in latency?" The answer is in the sampling strategy: head sampling (e.g. 10% of traces plus 100% of errors and 100% of slow requests) keeps steady-state overhead low, and tail sampling at the collector keeps all the interesting traces while discarding the boring ones. You should be able to say that batching, sampling, and async export are what make the overhead bounded and predictable — and that auto-instrumentation on DB calls is where the budget usually goes.

Cardinality. Traces are cheap to make and expensive to query if you stuff unbounded values into attributes. user_id and trace_id are fine as span attributes; putting them in a metric label is how you kill a backend.

PII. Traces and logs are a data-export surface. Redaction belongs in the collector (an attribute/field processor), not in every service — another payoff of centralizing.

"Across all production services" is a claim about coverage. Be ready to say what coverage meant: which services, which signals, and what the adoption path looked like (shared package → auto-instrumentation → alert definitions migrated). Coverage is the unglamorous half of observability and saying so reads as senior.

Alert fatigue. The honest trade-off: real-time anomaly alerting only helps if alerts are trusted. Fewer, better alerts with trace links beat more alerts with no context — otherwise you have just moved the noise.

Why this matters for the rest of your prep

Day 84 covered SLOs and error budgets; Day 83 covered what to instrument. This internship is where those ideas came from in practice, and Zoth is where you rebuilt them from scratch (OTel → Better Stack, trace context carried through RabbitMQ message headers). The through-line to say out loud: "I learned observability in a product team where MTTD was the number, then re-implemented the same pipeline in my own system where I had to make the propagation work across a message broker instead of an in-process call stack." That sentence connects your internship, your resume, and Days 83–90 in one move.