The Gemini pipeline — prompts, fallbacks, BYOK, and JSON discipline

Two LLM integrations, both yours. The engineering (not the prompt text) is what interviewers probe — and you have unusually concrete answers.

Zoth's advisory pipeline (internal/services/rag/)

Calls (all optional-gated on GEMINI_API_KEY):

1. AnalyzeRequest → risk score + reasoning for the approver (stored on the request row).
2. PredictApproval → likelihood + ETA (calibrated against historical decision rows retrieved from the corpus).
3. SuggestDuration → recommended expiry for the requester.
4. SuggestPolicy → policy JSON authored from an uploaded security document (the most ambitious: document → structured JSONB, validated by ParseAccessRules before storage — LLM output must pass the same validation as human input; that sentence is the entire lesson).
5. ShouldAutoApprove → the gate for the LOW-sensitivity auto-approve track (tenant opt-in + >95% historical approval rate conditions).

Failure discipline: every call has a timeout and a fallback (heuristic defaults: risk = MEDIUM, duration = policy max); LLM failure never blocks a request — the advisory layer is best-effort by construction. The advisory write path: separate SetAdvisory/SetRiskScore repository methods, so the advisory data physically cannot touch status.

Traverse's AI ladder (BYOK + providers)

Order: user-supplied Gemini key (BYOK, stored per user) → server Gemini → Groq (with model selection, groq:top_models cached 35 days) → G4F (free proxy). Details that show production thinking: per-provider 15 s timeout (the original latency culprit — Day 7); the ladder is sequential with fallthrough on error; responses are JSON-cleaned before parsing (LLM JSON arrives wrapped in markdown fences, with trailing commas, or prefixed with prose — the cleaning function strips fences, extracts the outermost {...}/[...], repairs trailing commas — because "the model returns valid JSON" is not a guarantee, it's a hope); parsing failures fall back to deterministic heuristics (the scheduling layer must never depend on AI success — Day 74's theme).

Premium gating as a choke point: aiAnalysis.queue.ts:143 nulls cognitiveTier/recallScore for free users — one code path, not scattered checks. Design principle: gate at the boundary where data enters the system, not at every read.

The patterns to name (this is the interview payload)

1. Advisory-vs-authoritative separation: LLM output lands in dedicated columns/statuses; authority lives in deterministic code. Structurally enforced (separate repository methods, policy-gated reachability).
2. Validated LLM output: any LLM-authored artifact that becomes data (policy JSON) passes the same validator as human input. Never JSON.parse → store.
3. Graceful degradation everywhere: heuristic fallbacks per feature (risk, duration, tier, schedule) — the system's correctness never depends on a provider being up.
4. Provider abstraction + BYOK: key resolution is a policy (user key → server key → fallback), and the AI worker is provider-agnostic. That's the shape of every serious LLM integration.
5. Cost/latency control: model selection, response caching where safe, timeouts per provider, and — in traverse's case — moving the whole thing off the request path (the 50× story).
6. JSON discipline: cleaning, extracting, repairing; schema validation before use; failures counted (not silently defaulted — log the fallback so you can measure the ladder's health).