Consensus, leader election, gossip, and node discovery
The coordination machinery you deliberately don't run — and what changes the day you need it.
Concept → your projects
The zoo of coordination (know the purposes, then why yours needs none):
- Consensus (Raft/Paxos): N nodes agree on a value despite failures; the foundation of linearizable metadata (etcd, ZooKeeper, Consul). Raft story: leader + term + log replication + majority commit; leader election on heartbeat timeout.
- Leader election: one coordinator for singleton work. Done badly (naive locks) → split-brain (two leaders, both acting).
- Gossip (SWIM): nodes periodically exchange peer state; membership + failure detection scales to thousands of nodes (Cassandra, HashiCorp memberlist).
- Metadata service / discovery: "where is X?" — etcd/Consul DNS, or K8s' built-in answer (kube-dns + endpoints API — you already run it).
- DHTs: consistent-hashing-based ownership distribution (Cassandra token ring, Chord).
Why zoth/traverse need none — the honest architectural position: every "who does this work?" question in your systems is answered by state, not coordination:
- Expiry loop on N replicas → conditional updates arbitrate; conflicts are swallowed (Day 32).
- Outbox relay on N replicas → SKIP LOCKED arbitrates.
- Notification drain on N replicas → forward-lease arbitrates.
- Consumer scaling → broker's own dispatch (prefetch, competing consumers) arbitrates — RabbitMQ is your leader for work assignment; dedup makes double-dispatch harmless.
That's the pattern to name: lease/claim-based single execution replaces leader election for work; consensus is only needed for shared mutable state with linearizable semantics (and your shared mutable state is one Postgres — which does run consensus internally, in WAL ordering + locking: "the database is my consensus service" is the soundbite).
What changes at 3 replicas / multi-node (the upgrade narrative):
- Postgres HA → Patroni (which itself uses DCS consensus — etcd — for leader election of the primary). At that point you run Raft, indirectly, for the database.
- RabbitMQ quorum queues → Raft-replicated queue metadata (vs classic mirrored) — again consensus, hidden inside a product.
- The 30-s expiry ticker stays as-is — any replica can run it; CAS updates make it safe. This is worth saying explicitly: your background loops are already multi-replica-safe by construction.
Split-brain, the concept to explain well: two nodes both believe they're leader (network partition + bad timeout). Consequences: double-granting, double-sending. Mitigations: quorum (majority must agree), fencing tokens (Day 51), leases with bounded terms. Your exposure: none today (single replicas), and the claim-based patterns that scale first.
Gossip's use case for completeness: when cluster membership itself is the problem (peer discovery without a K8s/etcd substrate) — self-managed Cassandra/Elasticsearch-style fleets. Yours: kube-dns. Done.
Practice question
"Design a system where a job must run exactly once per hour across 10 workers": (a) cron + lock service + fencing — old school; (b) your answer: a Postgres row job_runs(hour, claimed_by, lease_until) with the claim as conditional update/lease — the same primitive you've shipped twice. Full marks for recognizing the problem is a table, not a service.
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
You run no leader election anywhere. How is singleton work safe?
Because singleton work is expressed as claims on rows, not as roles: the expiry loop's conditional updates mean any number of replicas can tick and exactly one wins each transition; the outbox relay and notification drain use FOR UPDATE SKIP LOCKED and leases so replicas claim disjoint work. Leader election is for shared mutable state; my shared state is one Postgres, whose internal locking/WAL ordering is the consensus layer. I'd add real coordination only when the state itself partitions — and then it ships inside products: Patroni for Postgres, quorum queues for RabbitMQ.
What is split-brain and how do real systems prevent it?
Two nodes both acting as primary after a partition (naive timeout-based failover). Prevention: quorum — a candidate needs majority votes (Raft terms), so a minority-side 'primary' cannot commit; plus fencing tokens so the stale leader's writes are rejected by storage; plus leases that expire in bounded time. In my stack the exposure appears only if I added async-replica failover for Postgres — which is why I'd choose Patroni (quorum via DCS) over hand-rolled failover scripts.
When would you actually deploy etcd/Consul yourself?
When I'm building infrastructure, not applications: a scheduler needing distributed locks with sessions, service mesh control planes, or multi-cluster metadata outside Kubernetes. Inside K8s, discovery and config already come from the API server + DNS — adding Consul would duplicate the platform. The smell I avoid: reaching for a coordination service to solve what a conditional UPDATE or a SKIP LOCKED claim already solves with the durability I already operate.