Log-structured storage, immutability, and append-only design

Immutability buys three things: crash simplicity (an append either happened or didn't — no torn state), auditability (history is the data, not a reconstruction), concurrency (no locks on things that never change). It costs: compaction/reclamation (dead data lingers), read amplification (current state = fold over history), no un-write (errors are forever — GDPR says hello).

The spectrum, from your systems:

• True event-sourced: state = replay of events. Maximal audit, maximal read cost. (Neither project.)
• State + event log side-by-side (your actual pattern): mutable current-state tables + append-only event/audit records. Current-state queries stay fast; history stays provable. Zoth: access_requests (state) + audit_logs/outbox_events (log). Traverse: notification state + Notification-then-NotificationDelivery (intent, then per-channel ledger), Submission (event-ish) + Solve (fact).
• Copy-on-write snapshots: RingDay goal snapshots (traverse) — rows frozen at creation. Immutability as correctness for time-based queries: "the 4th's rings" is answerable identically forever.

Compaction, the tax collector: append-only growth needs reclamation — Traverse: removeOnComplete 1000/24h (BullMQ), expired friend requests retained but terminal, notification rows... retained (audit-ish value, small volume). Zoth: processed_events grows forever — the named gap (Day 42); outbox rows retained post-publish (small but unbounded). The design rule you now hold: every append-only structure declares its retention policy at birth — either TTL, count-bound, or "grows forever, documented with a size estimate."

End-to-end checksums (your topic list's item): corruption detection across the write path — Postgres does page checksums (initdb flag; know if your clusters have it — most managed ones do); your application-level version is the hash chain: hash = SHA256(prev_hash || canonical_data) — an end-to-end integrity check over semantic content, catching post-write tampering that page checksums can't (an attacker editing rows in-place leaves page checksums valid — the chain breaks anyway). That distinction — transport/page corruption vs. adversarial modification — is exactly what "end-to-end" means in the checksum sense (the principle generalizes: verify at the layer where the threat lives).

The lesson from your own two designs: zoth went more immutable (audit chain, no soft deletes, events as first-class); traverse went immutable where product semantics demanded it (snapshots, ledgers) and mutable everywhere else. Immutability per-concern, not per-system — the same rule as soft deletes (Day 35).

How LSM and B-tree actually differ (the mechanism, not the slogan)

"LSM writes fast, B-trees read fast" is true and useless without the mechanism.

A B-tree is mutable and page-oriented. A node is a fixed-size page (Postgres: 8 KB). A write locates the leaf page containing the key and modifies it in place. That is why reads are predictable — one descent, three or four levels for millions of rows — and why writes are relatively expensive: an in-place page modification must survive a crash, which Postgres guarantees with WAL. The write-ahead log is appended first (sequential, cheap); the dirty page is flushed later, possibly as a random write. Two consequences interviewers probe: write amplification (one logical row change can dirty a page, a WAL record, and the heap plus every index that references it) and HOT updates — if an update touches no indexed column and the new version fits on the same page, Postgres skips index maintenance entirely. That is a genuine performance cliff you can fall off simply by adding an index.

An LSM tree is append-only and tiered. Writes land in an in-memory memtable and in a WAL for durability. When the memtable fills it is flushed as an immutable SSTable — a sequential write, which is why LSM engines are fast on write-heavy workloads. Reads must consult the memtable and then every SSTable newest-to-oldest, which is why LSM engines carry Bloom filters per SSTable: here the Bloom filter is not a nicety, it is the read path, and it is the honest answer to "where would you use one in practice?" (Day 78). Background compaction merges SSTables, discarding overwritten values and tombstones — and that is where the cost moves: compaction is continuous background I/O, and its strategy (size-tiered vs leveled) trades write amplification against read amplification against space amplification.

The amplification triangle is the whole trade. You cannot minimize write, read, and space amplification simultaneously; you pick two and pay the third. Leveled compaction (the RocksDB default) minimizes read and space amplification at the cost of write amplification; size-tiered does the reverse. Which is why "Cassandra is faster than Postgres" is a meaningless sentence without the workload: it is faster at sustained sequential writes and worse at ad-hoc read patterns and secondary-index queries.

Deletes are where immutability shows its teeth. You cannot un-append, so a delete writes a tombstone — a marker meaning "this key died at time T" — and the value is only reclaimed when compaction reaches both the tombstone and the original record. Consequences worth being able to state: deletes cost space before they save it, a delete can appear to be silently undone by reading a replica that has not seen the tombstone yet, and a TTL is implemented as compaction-time expiry rather than a background job walking rows.

Where your two systems sit, mechanically. Postgres in both: B-tree indexes, WAL for durability, MVCC for concurrent readers (Day 32). You are paying read-friendliness and index-maintenance cost in exchange for what you actually need — transactional multi-row writes with constraints and a real query planner. But note that Traverse's removeOnComplete bound on BullMQ jobs and Zoth's unbounded processed_events table are both LSM-shaped problems living in a B-tree world: an append-only table in a B-tree store inherits all of the growth and none of the compaction. Which is exactly why the retention policy has to be declared explicitly — a partition drop, a scheduled delete, a count bound — rather than inherited from the storage engine.

Drill

"Design a view counter with audit": events table (video_id, ts) append-only → rollup table (video_id, day, count) maintained by incremental aggregation → counter reads the rollup. The pattern = your notification intent/ledger split, and Day 69's counting problem. Say "I've built this shape twice" and point at the files.