Broadcast fanout, coalescing, and approval routing as a rule engine

Traverse's broadcast (fanout-on-write, the engineered version)

broadcastService.broadcast(...): enumerate recipient user IDs → batched inserts (500 rows per createManyAndReturn) → per-recipient Notification rows → the drain delivers. The parts that make it production-shaped:

• Cursor resumability: the fanout job carries cursorUserId; crash mid-fanout → re-enqueue → resume from cursor. The job is a reducible operation, not an all-or-nothing transaction — and idempotent because the notification unique keys prevent duplicate rows on re-sweep (@@unique guards), with broadcastId marking provenance.
• Job identity: broadcast-fanout-<id> as BullMQ jobId — the scheduler can't stack duplicates.
• Coalescing: notification types declare a coalesceKey; before insert, lookup this user's unread notification with the same key → if found, updateMany bump count/fields instead of inserting a new row (index [userId, coalesceKey, readAt] exists exactly for this — Day 27's "indexes have client names"). The effect: 'friend solved 3 problems' not 'friend solved problem' ×3.

Zoth's notification semantics (smaller, but with one gem)

Zoth's in-app notifications (POST /notifications/{id}/read, read-all) are simple rows. The gem: unconfigured-capability-as-decision — traverse's APNs 'skipped(unconfigured)' (Day 52) and zoth's RAG routes mounting only when GEMINI_API_KEY exists share one philosophy: capability absence is a modeled state, not an error. Absent telemetry degrades, never crashes, never backfills — because you decided what absence means at design time.

Approval routing as rule engine (closing yesterday's loop)

The full decision pipeline with its rule sources:

request → policy.match_conditions (JSONB rules, priority-ordered) ──┐
  ├─ matched:  policy_id stamped; approval_config:                   │ rule layer
  │     approvers + auto_approve_enabled + (auto-approve track)      │
  ├─ no match: unrouted → any member decides (authority: UNROUTED) ──┘
  └─ always:   suspicious-activity signals (advisory) → alert.suspicious
               RAG advisory (risk/likelihood/ETA — advisory only)
  decision →  self-approval block (C-2/C-3) → authority check
           →  conditional transition (CAS) → audit + outbox event

The composite is the rule engine: structured rules decide routing and bounds; the RAG layer computes advisory scores; suspicious-activity heuristics (temporal anomaly, privilege escalation on HIGH/CRITICAL, velocity for first-time requesters, cross-domain pattern) add signals; the state machine enforces authority and atomicity. Each layer has a declared authority level — that declaration is what "hybrid authorization engine" on your resume actually means. Draw this diagram from memory and you've won the interview.

The coalescing race (the bug this design invites)

The coalescing flow reads before it writes: look up this user's unread notification with the same coalesceKey; if found, updateMany to bump the count; if not, insert a new row. Two broadcasts that overlap in time — or two workers draining the same fanout concurrently — can both read "no unread row" and both insert, producing exactly the duplicate the coalescing existed to prevent.

At 100 users with one fanout job this is nearly impossible to hit, which is precisely why it is worth discussing: it is a latent correctness bug that only appears under concurrency you do not have yet. Three fixes, in increasing order of rigor:

1. Make the read-and-write atomic with an upsert against a partial unique index: CREATE UNIQUE INDEX ... ON notifications (user_id, coalesce_key) WHERE read_at IS NULL. The insert path becomes INSERT ... ON CONFLICT DO UPDATE SET count = notifications.count + 1, and the race resolves inside the database rather than in application code. This is the same move as everywhere else in this course — push the invariant into a constraint and let the database arbitrate.
2. Serialize per user, by including the recipient in the job identity or processing one user's notifications in a single worker at a time. Simpler to reason about, but it caps per-user throughput for no benefit at this scale.
3. Accept the duplicate and dedup at read time, collapsing rows that share a coalesceKey in the inbox query. Cheapest to implement, most expensive to reason about, and it leaves the wrong data in the table for every future consumer to deal with.

Option 1 is the right answer, and the reason is a pattern you have now seen four times: unique constraints as the final arbiter (Day 20), tenant scoping as a structural gate (Day 40), CAS conditional updates (Day 32), and now coalescing. When the application has to remember a rule, it will eventually forget.

What changes at ten thousand friends

The fanout is O(recipients) inserts, batched 500 at a time, resumable by cursor. Do the arithmetic out loud:

• 100 friends: 100 rows, one batch, milliseconds. Fanout-on-write is obviously correct.
• 10,000 friends: 10,000 rows per broadcast, 20 batches. An account posting ten times a day generates 100,000 notification rows a day — from one account. Write amplification is now a capacity problem, and the notification table's growth rate is dominated by the largest accounts rather than by total user count.
• 1,000,000 followers: fanout-on-write is simply wrong. One post becomes a million-row write burst that saturates the database, and the broadcast job's runtime exceeds its own retry window.

The standard answer is the hybrid (Day 63): fan out on write below a follower threshold, and above it store the post once and merge it into follower timelines at read time. The threshold is derived from the write-amplification budget, not chosen for elegance — "if a broadcast may cost at most N rows, the threshold is N." And the transition needs a backfill story: when an account crosses the threshold, its followers must stop receiving fanout rows, while its historical posts remain visible through the merge path.

Two smaller consequences worth naming: the cursor is what makes the large case survivable at all (a 10,000-recipient fanout that dies at recipient 7,000 resumes instead of restarting), and coalescing becomes mandatory rather than nice at scale, because a busy account otherwise floods every follower's inbox with one row per action. Both mechanisms you already built are exactly the two you would need — which is the real point of this article.