Retries, timeouts, and idempotency

Every network call can fail in four ways: connect timeout, read timeout, reset, or succeed-but-response-lost. The last one is why idempotency is not optional: a retry after a lost response re-executes the operation.

Timeouts — budget them per call and per stack. A request with a 15 s total budget can't contain a 15 s LLM call plus a 2 s DB call. Traverse's AI ladder: 15 s per provider with multiple providers sequenced — the worst case that produced the original 10 s p95 (the budget was unbounded in practice). The lesson you lived: timeout budgets must be explicit, and the async boundary (queue) is how you move unbudgetable work off the request path.

Retries — only retry when (a) the operation is idempotent or deduplicated, and (b) the error is transient. Add exponential backoff + jitter, or you'll DDoS your own dependencies (thundering herd after a blip). Your zlink retry.Do is the textbook: MaxRetries=3, BaseDelay=500ms, MaxDelay=30s, delay = delay/2 + rand(delay/2) — the jitter spreads concurrent retries. Zoth's consumer retries differently — republish with x-retry-count header rather than Nack(requeue=true), because immediate requeue hot-loops without advancing a counter. Five attempts, then dead-letter. Traverse's BullMQ: attempts: 3, backoff: {type:'exponential', delay: 2000}.

Idempotency — the pattern inventory (yours, mapped):

The fencing idea (Day 51): idempotency keys protect against duplication; fencing tokens protect against ordering (late stale writes overwriting newer state). Your systems mostly avoid ordering hazards by making writes conditional (CAS updates) rather than last-write-wins.

The failure-classification piece

Retrying intelligently means classifying errors — zlink's classifyError maps HTTP status → ErrAuth (don't retry, page a human) / ErrRateLimit (retry after X-RateLimit-Reset) / ErrNotFound (don't retry — or in revoke, succeed) / ErrPlatform (retry). Most retry layers people write retry everything and wonder why a 401 loop fills the DLQ. Yours doesn't.

Practice narrative

"When the emitter's GitHub grant fails mid-flight: the handler has a 5-minute timeout; zlink retries 3× with jittered exponential backoff and respects rate-limit headers; if all fail, the consumer's own retry path republishes with an incremented x-retry-count (up to 5), then dead-letters to emitter.access.dlq; and because grants are side effects of state already committed in Postgres, a human can also replay by reconciling — drift detection flags the discrepancy within 5 minutes anyway. Nothing about that chain is speculative — each stage has a file path."