Deployment and migration safety — expand/contract, read drain

Deploy safety primitives:

• Rolling deploy: replace instances gradually, health-gated (readiness). Requires backward/forward compatibility between app versions sharing a database — the constraint everything else follows from.
• Blue/green: two full environments, flip traffic; instant rollback, double cost, and DB compatibility is still the hard part.
• Canary: route a small % of traffic to the new version, watch SLIs, then ramp. Best signal-per-risk; needs traffic splitting and good metrics.
• Feature flags: deploy code dark, enable per-tenant/percent — decouples deploy from release (the single most useful practice, and it moves the risk from infrastructure to a config flip you can reverse in seconds).

Expand/contract (the schema-change protocol, Day 35 revisited with the full cycle):

1. Expand: add the new column/table/index — nullable, additive, no reads yet. Both old and new code work.
2. Dual-write: new code writes both shapes; old code still reads old. (Or backfill via cursor batches — Day 39's migration pattern.)
3. Switch reads: flip readers to the new shape (flag-gated).
4. Contract: stop writing the old shape, then drop it in a later deploy — after no code references it.

The invariant across all four phases: every intermediate state is served correctly by both running versions. That's what makes zero-downtime possible, and it's why "just add a NOT NULL column" is a multi-deploy project at scale.

Parallel monolith read drain (your topic list's phrase — the migration pattern): when splitting a monolith (or moving a table), you run two writes (old store + new store) while reading from the old, then drain: shadow-read from the new store and compare (diffing catches divergence), then flip reads, then stop dual-writes. "Read drain" = the phase where reads gradually migrate while dual-writes keep both sides consistent; the safety net is continuous comparison. Your zoth emitter's processed_events dedup and idempotent connectors mean even the drain phase can double-execute safely — the same properties that make broker redelivery safe make migration safe.

Migration-specific hazards: long locks (ALTER TABLE), index builds (CREATE INDEX CONCURRENTLY outside transactions — Day 35), backfills that bloat WAL and pin old snapshots, and the rollback trap: a migration that dropped a column can't be rolled back by redeploying old code. Hence: never mix destructive schema change with app change in one deploy.

Your systems' positions (and honest deltas)

• Zoth: migrations embedded and applied at boot under an advisory lock → effectively "stop-the-world during deploy" (pods roll, new pod migrates before serving). Safe because it's fail-fast; not zero-downtime — a good trade at demo scale, and the upgrade path is explicit (separate migration job + expand/contract + CONCURRENTLY index builds).
• Traverse: migrate deploy at container start with exit-on-failure (same philosophy, same trade), deployed via GitLab CI → SSH → docker-compose up -d --build — a single-instance rolling deploy of one container: brief downtime during recreate, gated by a health check loop. Honest framing: "my deploys are minutes of planned unavailability with a health gate and an instant rollback (git reset + rebuild). Zero-downtime would need a second instance behind a proxy and expand/contract discipline — scoped, not built."
• Both: no feature-flag system. The cheapest version (env-var-driven code paths) is available and unused; naming it is a maturity signal.