Kubernetes networking in zoth — services, probes wiring, and the health graph

Zoth's K8s topology (deployments/k8s/base/ + overlays/prod/):

namespace: zoth
├── Deployment zoth-api        (Chi; probes: startup/liveness/readiness)
├── Deployment zoth-emitter    (consumer; liveness+readiness)
├── Deployment zoth-alerting   (scheduler; liveness+readiness)
├── StatefulSet rabbitmq-0     (1 replica, 1Gi PVC, exec diagnostics probes)
├── Service zoth-api           ClusterIP :8080
├── Service zoth-emitter       ClusterIP :8080  (only /healthz,/readyz used)
├── Service zoth-alerting      ClusterIP :8080
└── Service rabbitmq           ClusterIP :5672

Discovery is DNS. No service registry, no Consul — kube-dns resolves zoth-emitter.zoth.svc.cluster.local. The ConfigMap sets EMITTER_HEALTH_URL=http://zoth-emitter:8080/api/v1/healthz; the API's /pods/readyz controller fetches both sibling services concurrently (goroutines) with a 5 s context and aggregates: any failure → 503 with per-component status/latency/error. That endpoint is a system-level readiness check — your one-box answer to "how do you know the whole platform is up?" (Better Stack uptime monitors hit /healthz and /readyz every 3 min from us/eu/as, so the aggregate is human-visible too.)

The probe design (know it cold — it's on your resume):

The design decision to articulate: liveness is shallow on purpose. A liveness probe that checks dependencies causes restart storms — DB blips → every pod fails → restarted → fresh cold pods hammer the recovering DB. Readiness absorbs dependency health (traffic stops flowing, process stays warm). This is a classic SRE interview point and your yaml encodes it.

RabbitMQ as a StatefulSet (not Deployment): stable identity rabbitmq-0, 1 Gi PVC (Azure managed disk) surviving pod rescheduling, probes rabbitmq-diagnostics -q ping (readiness 20 s, liveness 60 s, startup 10×10 s). The emitter/alerting connect via amqp://user:pass@rabbitmq.zoth.svc.cluster.local:5672/ — constructed in CI from rabbitmq-secrets.

Image pull chain: ACR acrzothprod.azurecr.io — kubelet identity has an AcrPull role assignment (Bicep acr-pull.bicep, role GUID 7f951dda…) — no imagePullSecrets to manage. Kustomize prod overlay sets images (zoth-api:<sha>), bumps requests to 250m/256Mi, OTEL_ENABLED=true, Service stays ClusterIP.

What to say when asked "why K8s at all on one node?" — process supervision + declarative config + probe-driven self-healing + future scale path, for the cost of one YAML directory. And the counterpoint you should volunteer: compose would be simpler; the K8s investment was deliberate (resume + learning + real probe semantics).

The failure walkthrough (recite this — it is the real article)

Interviewers do not want the YAML; they want to know you understand what the probe semantics do when things go wrong. Four scenarios:

1. The API pod's database connection dies for 30 seconds. Readiness (/readyz) fails within one probe period → the pod is removed from the Service's endpoint list → no new traffic arrives. Liveness stays green (it only checks that the process answers HTTP), so the pod is not restarted: it sits warm, keeps its connection pool, and rejoins the endpoint list the moment /readyz passes. That is the entire value of shallow liveness — a restart would have discarded a warm pool and made a blip worse.

2. RabbitMQ goes down. /readyz fails on the raw TCP dial to rabbitmq.zoth.svc.cluster.local:5672, so the API leaves rotation and requests fail fast instead of being accepted and then failing at the publish step. The emitter's readiness also fails, so it stops consuming (there is nothing to consume) but stays alive to reconnect. /pods/readyz aggregates both siblings plus the broker and returns 503 with per-component status — so an external uptime monitor reports which component is down rather than "the site is broken."

3. A rolling deploy of the API. Kustomize applies a new image tag; the new pod runs startup (/healthz, up to ~50 s of budget, deliberately generous so goose migrations finish before the pod counts as booted), readiness then gates it into the Service, and only afterwards does the old pod terminate. Because startup is separate from liveness, a slow migration cannot trigger a kill loop. The gap to volunteer: there is no PodDisruptionBudget and only one replica of emitter and alerting — a node drain can therefore take the only consumer down. Zero-downtime is a property of the API alone, and only when the migration itself is backwards-compatible (expand/contract, Day 85).

4. A migration is slower than the startup budget. Ten failures × 5 s ≈ 50 s of grace before the kubelet kills the container. A longer migration loops: kill, restart, migrate from scratch, kill. The fixes are a longer startup budget, a separate migration Job that runs to completion before the Deployment rolls, or an init container — and naming the trade-off (a Job is cleaner but introduces an ordering dependency in the deploy pipeline) is the answer that lands.

What is deliberately missing, and why that is a decision

• No HPA. One node, one API replica, a workload whose peak is human-paced. An autoscaler would be a control loop that never fires. Say instead: "the scale path is replicas plus an HPA on CPU and request rate, added at the same time as a second node — because autoscaling a single node is just thrashing."
• No service mesh. Three services with DNS discovery do not need mTLS sidecars and a control plane. The cost is that service-to-service auth is implicit (same namespace, cluster network). The honest note: a mesh becomes worth it the moment you have a workload-identity compliance requirement.
• No PDB. Named above as the real gap.
• Self-hosted RabbitMQ as a StatefulSet rather than a managed broker. One replica with a 1 Gi PVC is the honest small-scale choice: stable identity, persistent queue state across reschedules, diagnostics-based probes. It is also a single point of failure with no quorum — which is exactly the trade to state: "self-hosted for cost and learning; managed or three-node the moment queue loss becomes unacceptable."