Event contracts, fanout patterns, and rule engines

Events are APIs: consumers you've never met will couple to them. Contract hygiene:

• Naming: past-tense facts (access.granted), not commands (grantAccess) — facts are re-consumable; commands presume intent.
• Payload shape: full state, not deltas (request_id, tenant_id, user_id, resource_id, action, timestamp) — a delta requires the consumer to have seen prior events (ordering coupling); full state makes every event self-contained (zoth's events follow this — say it).
• Identity: event_id for dedup, aggregate_id for correlation (yours: request_id threads everything).
• Versioning: additive fields only; new topic/event_type for breaking changes.
• Metadata over payload duplication: trace context in headers (propagating your W3C traceparent through messages is what makes cross-service traces stitch in Better Stack).

The fanout taxonomy (which "fanout" means depends who's asking):

1. Queue fanout (work distribution): competing consumers, each message once — zoth's emitter.
2. Topic fanout (delivery to all subscribers): RabbitMQ topic exchange → N queues; Kafka consumer groups; zoth's alerting queue binding.
3. Fanout-on-write (push model): at write time, denormalize to every recipient's feed — precomputed reads, N writes per event. Costs write amplification; celebrity problem (10M followers = 10M writes).
4. Fanout-on-read (pull model): compute the feed at read time — cheap writes, expensive reads; fine for small follower counts.
5. Hybrid: push for normal users, pull for celebrities (Twitter/Instagram's actual answer).

Traverse's broadcast system = fanout-on-write, engineered for its scale (the details are Day 64's): recipients enumerated in batched inserts (500/round via createManyAndReturn), cursor-resumable, coalesced notifications. At 100 friends it's trivially correct; the design notes for scale are the classic ones: batching (done), cursor (done), per-user write cost — 100 friends → 100 rows (fine) vs celebrity case → hybrid switch.

Rule engines (your topic-list item, one honest paragraph): the spectrum from hard-coded conditionals → policy tables → JSONB-conditions evaluated by a service (yours!) → full DSL inference engines. Zoth's policy evaluation (match_conditions JSONB: resource sensitivity, group ids, platform; access_rules JSONB: max duration, sensitivity gate; first-match by priority) is a mid-spectrum rule engine — declarative rules, interpreted at runtime, no rule hot-reload caching. The design trade to articulate: interpretive flexibility (rules change without deploys) vs. evaluation cost (JSONB parse per request — measured as fine at your QPS) vs. explainability (a matched policy_id on the request = auditable why — the rule-engine feature that matters most in IAM).

The four ways an event contract breaks, and how each one shows up

1. The additive field that is not additive. Someone adds resource_type to the payload. Consumers ignore unknown fields, so nothing breaks — until a consumer treats a missing field as "not applicable" and a new field as "the old case." The discipline: a new field must have a defined meaning for events emitted before it existed, which usually means a default. "Absent means unknown" is fine for telemetry; it is a bug for anything a consumer branches on.

2. The semantic change that is invisible to schema checks. access.granted has always meant "the grant is active in the platform." Someone makes it fire at approval time instead, because provisioning became asynchronous. Field names identical, types identical, and every consumer is now wrong. No schema registry catches this — which is precisely why the rule is past-tense facts, not intentions: the event name asserts a fact that must remain true forever. If you need "approval happened," that is a different event with a different name.

3. The ordering assumption. A consumer assumes it sees access.requested before access.granted. Nothing in the contract guarantees that across queues, or across a retry that reorders a batch. The fix is the payload rule: full state, not deltas. If the grant event carries the whole relevant state, the consumer does not need to have seen the request, and the ordering coupling disappears. This is exactly why Zoth's events are self-contained rather than diff-shaped.

4. The dedup omission. At-least-once delivery is not a possibility, it is the contract (see the relay trace in Day 62). A consumer that assumes once will double-apply on every relay restart. So: every consumer is idempotent, and idempotency is enforced in the database — a processed_events-style table with event_id as primary key and an insert-on-conflict, so a duplicate is a no-op rather than something the application code has to remember to check. ON CONFLICT DO NOTHING returning zero rows is your duplicate detector, and you can count it as a metric: a rising duplicate rate means a retry loop is misbehaving somewhere upstream.

The idempotent consumer recipe (four steps, in order)

1. Insert the event_id into the dedup table with ON CONFLICT DO NOTHING. Zero rows inserted means it was already processed — return success and do not re-do the work. Success, not failure: the caller's at-least-once retry did its job, and reporting an error would trigger another retry.
2. Do the work inside the same transaction as the dedup insert. Otherwise a crash between the two loses the event forever, because the dedup row now claims it was handled. This is the outbox atomicity rule applied on the consuming side.
3. Make the effect naturally idempotent where you can — an upsert, or setting a state rather than incrementing a counter. Incrementing is the most common accidental non-idempotency in event-driven systems, and the fix is usually to derive the counter from facts rather than maintain it by increment (Day 70's rollup pattern).
4. Bound the dedup table. Otherwise it grows forever (Zoth's named gap, Day 42) — partition by time, or delete rows older than the maximum possible retry window. An unbounded dedup table is a slow-motion outage.

How to review an event schema in five questions

Ask these of any event you design, and answer them in the same breath as you name it:

• What is the fact, stated in the past tense? If you cannot phrase it as a fact, you are designing a command — say so and name it accordingly.
• Is the payload self-contained? Can a consumer that has never seen any earlier event act correctly?
• What is the identity, and what is the correlation key? One for dedup, one for threading a story across events.
• What happens if it arrives twice, late, or out of order? Three separate answers, because they are three separate bugs.
• Who breaks if I add a field, and who breaks if I change a meaning? The first should be nobody; the second should be nobody because you emitted a new event type instead.