Append-only in your systems — the audit chain, outbox, and ledgers
The three append-only structures in your code, their integrity machinery, and their known weaknesses.
Know your own system
Catalog + deep mechanics of every append-only thing you own.
1. The audit hash chain (zoth)
audit_logs: id, tenant_id, action, actor, target, metadata, prev_hash, hash, created_at. audit.LogAction (internal/services/audit/service.go): read last row's hash (GetLastHash) → compute SHA256(prev_hash || canonicalJSON(tenant, action, actor, target, metadata, ts)) → insert. VerifyChain(startID) recomputes walking id order.
Strengths to claim: tamper-evidence without a blockchain — the chain makes silent history edits detectable; per-row metadata is rich enough for audit reconstruction (who/what/when/why + rag fields on the decisions).
Weaknesses to know cold (your SECURITY_AUDIT says them — say them first):
- The M-8 race: GetLastHash→Insert isn't transactional — concurrent writers fork the chain. Single replica = rare (daily, per the audit); 3 replicas = constant. Fix (Day 50): per-tenant sequence-ordered chain (
seqfrom a Postgres sequence) — no read-modify-write. - VerifyChain caps at 10,000 rows (SQLC LIMIT) — a silent completeness limit on verification.
- No route mounts
/audit-logs/verify— the capability exists but is unreachable without code. - Chain hash is canonical-JSON-order dependent — serialization changes break verification (a schema evolution hazard; versioning the hash recipe would fix it).
2. The outbox (zoth + traverse's notifications)
Zoth outbox_events(aggregate_id, type, payload, published_at) — append in the same transaction as state change (the whole point); the partial index of unpublished rows (Day 27) is the "queue" the relay drains. Traverse's notification pattern is the two-table twin: Notification (the intent/ledger — who should get what) + NotificationDelivery (per-channel outcome ledger with attempts/backoff state). Intent/outcome separation: what should happen vs. what did happen — the accounting pattern, which is why "ledger" is the right word.
3. The attempt ledgers (traverse)
RevisionAttempt (each review observed), Submission (each submission captured — mutable analytics columns but append-first), PasswordResetCode/token rows (ephemeral ledger). Plus BullMQ's own Redis structures (hash+zset per job) — append-then-reap via removeOnComplete.
The integrity map across all three: outbox — atomic with state (transactional integrity); chain — cryptographic (tamper-evidence); ledgers — completeness via unique constraints (every attempt recorded exactly once, (solveId, revisionNumber, type)). Three different integrity mechanisms for three different threat models — a taxonomy interviewers rarely hear articulated.
The narrative
"Zoth's audit story has three tiers: completeness (every state change writes an audit row — enforced by routing every transition through services that log), integrity (the hash chain makes edits detectable — with a known race I'd fix via sequence-ordering before scaling), and accessibility (structured fields + OTel export for SIEM ingestion — the cyber-resume line). Traverse's equivalent is the notification intent/outcome split — what should happen vs. what happened per channel, with attempts and backoff as a replayable ledger. Same architectural instinct in two different products: history is a table, not a log file."
Bulletproof your resume
Say these answers aloud before moving on
These are the follow-ups an interviewer will actually ask about your projects.
How does your tamper-evident logging work, and where is it weak?
Each audit row hashes canonical row content + the previous row's hash — SHA256 chaining — so in-place edits break every subsequent verification. Weaknesses I can enumerate: the read-last-then-insert race (single-replica: rare fork; multi-replica: constant — fix is a per-tenant sequence-ordered chain); VerifyChain's 10k-row cap; the verify capability isn't mounted as a route; and the hash recipe isn't versioned, so a canonicalization change would break verification. Knowing the failure modes is the qualification — it's my own audit document that lists them.
Why separate Notification and NotificationDelivery?
Intent vs. outcome. Notification is the business fact ('user X should learn Y') — created in the request/worker flow, one row. Delivery is the per-channel attempt ledger — pending/sent/failed/skipped with attempts and backoff — a row per (notification, channel). That split gives: independent retry state per channel, coalescing on the intent row while deliveries stay faithful, and a clean answer to 'did the user actually get told?' without contaminating the business record with delivery plumbing. It's the same separation as zoth's outbox: what happened (state) vs. what must propagate (event).
Is your audit log actually SIEM-ready?
Two paths: the structured rows (actor, action, target, tenant, timestamps) are queryable and exportable for SOC teams — that's the defensible part of the resume line. The OTel pipeline is the second path: logs flow to Better Stack, which supports alerting and export — so ingestion is plumbing that exists. The honest gaps: no documented export format (CEF/LEEF or JSON contract), no verification route mounted, and retention policy undefined. 'SIEM-ready' in my resume means the data model and telemetry exist; the export contract would be the enterprise next step.