Day 31 of 100·System Design Day·614 words

Locking, isolation levels, and MVCC

The machinery under every transaction you wrote — explained with your own races.

Concept → your projects

Postgres isolation is MVCC: every UPDATE writes a new row version; readers see the version snapshot valid for their transaction; old versions linger until VACUUM. Writers don't block readers, readers don't block writers — only writer-writer conflicts on the same row block.

Isolation levels (Postgres's actual behavior):

LevelDirty readNon-repeatable readPhantomWrite skew
Read Committed (default)noyesyesyes
Repeatable Readnonono*yes (mostly no, with serialization errors)
Serializablenononono (via SSI)

Write skew (two transactions each read overlapping data and write disjoint rows, violating a combined invariant) is the subtle one; SERIALIZABLE catches it via predicate locks (SSI). Your systems run Read Committed + explicit invariants in WHERE clauses — the pragmatic 90% answer.

Your real races, mapped to the theory

Race 1 — approve vs expire (zoth). Both flip status conditionally. Under MVCC: T1 (approve) locks the row, writes ACTIVE; T2 (expiry tick) blocks, re-checks WHERE status='ACTIVE' AND expires_at<now()... note the expiry loop first selects candidates, then updates. Between select and update, the approver can act — the update's WHERE re-check saves you (0 rows → "a concurrent transition beat us" → swallowed ErrConflict). Say it in these words: the select is advisory, the update's predicate is authoritative.

Race 2 — two approvals (zoth). Two managers click approve: both pass read-side authority checks (the pending approval row exists for both reads), then both fire the conditional update; one wins, the other gets 0 rows → 409. The invariant "exactly one transition" is enforced by the row itself, not by application locking.

Race 3 — duplicate submission (traverse). Extension retries: two POSTs race past the read-side dedup check (findUnique finds nothing for both) → both INSERT → the @@unique([userId, idempotencyKey]) constraint makes the DB reject the second with P2002 → mapped to 200-duplicate. Constraint as last line of defense after check-then-insert races — the standard three-layer answer: application check (fast path), unique constraint (truth), idempotent handling of the error (ergonomics).

Race 4 — notification claim (traverse). FOR UPDATE SKIP LOCKED + lease (Day 62 full treatment) — readers-writer coordination implemented as a claiming UPDATE.

FOR UPDATE semantics to recite: after a row is locked by one transaction, a competing SELECT FOR UPDATE waits (or skips, with SKIP LOCKED); a competing UPDATE waits, then re-evaluates its WHERE against the committed version — this "EvalPlanQual" re-check is precisely why your conditional updates are race-free. If the predicate no longer matches: 0 rows, no error, no write.

Deadlocks

Two transactions updating rows in opposite orders (A→B and B→A) deadlock; Postgres detects and kills one (ERROR: deadlock). Your code avoids it by updating single rows (nothing to order). The classic mitigation — consistent lock ordering — becomes real when a workflow touches multiple rows (e.g., approving while updating the approval row AND the request row: zoth's decide does approval-row update then request-row conditional update — if anywhere in the code the opposite order existed, you'd have a deadlock window; it doesn't, but knowing to check is the skill).

The SERIALIZABLE pitch (and why you didn't use it)

If your invariant spans rows ("an approver can't approve two requests that together exceed a risk budget"), Read Committed + conditional updates can't express it — you'd need SERIALIZABLE (with retry-on-40001) or explicit advisory locks. Honest zoth answer: current invariants are all single-row, so Read Committed suffices; the day a multi-row invariant appears (approval chains with quotas), SERIALIZABLE-with-retry or SELECT FOR UPDATE aggregates is the tool. Knowing when to escalate is the interview point.

Bulletproof your resume

Say these answers aloud before moving on

These are the follow-ups an interviewer will actually ask about your projects.

01

Writers don't block readers in Postgres — how, and what's the cost?

MVCC: an UPDATE writes a new tuple version tagged with the transaction's visibility; readers snapshot and see only committed versions as of their snapshot. Cost: old versions accumulate (bloat) until VACUUM reclaims them; long-running transactions pin dead tuples and cause table/index bloat — which is why long transactions are an operational hazard. Practical check on my systems: the outbox relay's claim transaction is deliberately tiny (select + mark published), so it can't pin anything; no transaction in either codebase spans an external call — that's a rule I keep: never hold a transaction across I/O.

02

Your expiry loop selects candidates then updates them. Isn't there a TOCTOU window?

Yes — and it's harmless by construction. The select (WHERE status='ACTIVE' AND expires_at<now()) only builds a candidate list; the transition itself is the conditional UPDATE with the same predicate. If an approver/revoker acted in the window, my update's WHERE re-check fails, 0 rows, I swallow the conflict and move on. The pattern is 'optimistic candidate selection + authoritative conditional write' — TOCTOU is only a bug when the read is trusted as the write's guard; mine isn't.

03

When would you reach for SERIALIZABLE?

When an invariant spans multiple rows and can't be collapsed into one conditional write. Example: 'a user may hold at most 3 ACTIVE grants' — two concurrent grants both check count=2, both insert → 4. Fixes in order of preference: (1) reshape as a single-statement invariant if possible; (2) SELECT FOR UPDATE on the shared parent (the user row) to serialize competitors; (3) SERIALIZABLE with retry-on-serialization-failure. I'd use (2) at low contention and (3) when predicates are complex — with the caveat that every path needs the same retry discipline I already have for ErrConflict.