Distributed locks, leases, fencing tokens, circuit breakers
Redlock's controversy, leases you already run, and where you'd add breakers.
Concept → your projects
Distributed lock = mutual exclusion across processes. The trap: a lock is not a correctness primitive — a holder can pause (GC, network) and act after its lease expired while someone else holds it. The fix: fencing tokens — a monotonically increasing number issued with the lock; storage rejects writes with stale tokens (the Kleppmann argument against Redlock's safety claims). Rule: locks optimize (avoid duplicate work), fences guarantee (reject stale work).
Leases are time-bounded locks — and you run two production instances:
- Traverse's notification drain: claim =
next_attempt_at = now + 120spersisted in the row (Day 32). The lease can't fence (a slow worker's APNs send can't be rejected by Postgres) — bounded instead by send-idempotency tolerance (push double-sends are acceptable, collapse-id coalesced). - The outbox claim (transaction-scoped row locks) — these are fenced by the DB transaction itself: a claim that dies rolls back, releasing work cleanly. The strongest version of the pattern: lock scope = transaction scope.
Redlock (Redis quorum locks) — know the debate: Kleppmann's critique (clock jumps break safety; no fencing) vs. antirez's defense. The position to hold: "for correctness-critical work I use the database's own locking (transaction-scoped claims — Postgres is my lock service); Redis locks would only ever guard idempotent work, where a stale holder costs a wasted attempt, not a wrong result."
Circuit breakers: stop hammering a failing dependency — closed (normal) → open after error-threshold (fail fast) → half-open (probe). Where yours would go, concretely:
- zlink → GitHub/Azure/Notion calls: the retry layer (3×, jittered) currently doesn't distinguish sustained failure — a full outage means every event burns 3 retries × backoff before DLQ. A breaker per (platform, credential) would fail-fast events to the DLQ or a retry-later queue, protecting the emitter's throughput. This is the single most natural breaker insertion point in your code — say so.
- Traverse AI ladder: provider-level breakers would skip a dead provider instead of eating its 15 s timeout per job — the ladder already tries next provider per job, but every job still pays the timeout; a shared breaker state (in-process, TTL'd) would collapse that.
- What you have instead: timeouts everywhere + the queue as shock absorber (slow deps slow the pipeline, never the API). Honest statement: at your scale, breakers are an optimization on top of timeouts+queues, and the queue already provides the availability decoupling breakers buy.
Bulkhead (sibling concept): isolate dependency capacity — different pools per dependency. Yours: two Redis clients (cache fail-fast vs queue retry-forever) is a bulkhead of connection policy; separate DB pools per service would be the next step.
The drill
"Prevent double-send in a notification system at 10 workers": claim row with lease (yours) → if sends must be exactly-once: per-notification token + provider idempotency keys (APNs apns-collapse-id, FCM message ids) — the provider is the fence. The full answer combines: lease (avoid duplicate work) + provider idempotency (reject duplicate effect).
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
Why is a distributed lock not a correctness guarantee?
Because the holder can outlive its lease: GC pause or network delay means it acts while another holder exists — two writers, lock intact. Correctness needs fencing: a monotonic token issued with the lock that the storage layer checks and rejects if stale. My systems avoid the problem structurally: the outbox claim's lock scope is the DB transaction (dying mid-claim rolls back), and the notification lease tolerates its bounded failure mode (duplicate push, coalesced). Locks optimize; fences and idempotency guarantee.
Where would you add circuit breakers in zoth?
In the emitter's zlink calls, per platform+credential: today a GitHub outage makes every event burn 3 jittered retries before DLQ — throughput collapses into backoff. A breaker trips after N consecutive failures and fail-fasts events to a retry-later path, with half-open probes to detect recovery. The queue already protects the API (decoupling), so the breaker is about emitter throughput and downstream politeness — not correctness, which retry+DLQ+drift already handle.
You use the database as your lock service. Justify against Redis/Redlock.
The work items are already rows in Postgres, so transaction-scoped claims give me: atomic lock+read+write in one place, automatic release on crash (rollback), and no separate service to operate or reason about during partitions. Redlock would add a second consensus-ish system for locks over data whose truth is in the first system — a split-brain between lock service and store, which is exactly the failure mode fencing exists to patch. If work items ever lived outside Postgres, I'd revisit — with fencing tokens, not bare locks.