Information retrieval — inverted indexes, TF-IDF, BM25

Inverted index: for each term, the sorted list of documents containing it (posting list). Query = intersect/union posting lists. Costs: index build (tokenize → terms), storage (postings dominate), and updates (immutable segment files + merges in Lucene-style engines — the LSM rhyme again).

Ranking evolution:

• TF-IDF: term frequency (how often in this doc) × inverse document frequency (how rare across the corpus — log(N/df)). Rare-term matches matter more; common terms are downweighted.
• BM25 (the production default): TF-IDF with saturation — term frequency has diminishing returns (tf/(tf + k1·(1-b+b·|d|/avgdl))) and length normalization (b). Two tunable knobs, decades of tuning, still the baseline every neural ranker must beat.
• Phrase/boolean/tiered: exact-match filters (quotes), boolean ops (AND/OR/NOT), and tiered/early-termination (WAND-style: process promising postings first, stop when the remaining can't beat the top-k).

Evaluation: precision/recall, nDCG (graded relevance, position-discounted), MRR (first relevant hit), and — the one interviewers love — offline proxies vs online behavior (CTR, dwell). Your topic list's "search feedback and relevance signals" = the loop: log clicks → judge → re-rank.

Index synchronization (keeping search consistent with the DB): dual-write (fragile), transactional outbox → indexer (your pattern!), CDC, or periodic reindex. Index sharding: partition by document id or by term (global IDF then needs cross-shard stats — the reason BM25 at scale is trickier than it looks).

Where your systems stand: no inverted index at all — and here's the precise, defensible reason. Zoth's retrieval is semantic (pgvector + cosine/HNSW): query = embed text, find nearest embedding. That's a dense retrieval system, strong at fuzzy intent ("who approved similar requests to this one?") and weak at exact-term guarantees (a policy doc mentioning Azure-Contributor may not surface for the literal string unless the embedding model preserves it). Hybrid retrieval (the modern answer): BM25 for exactness + vector for semantics, fused (reciprocal rank fusion / weighted scores) — and you can build the BM25 half inside Postgres: to_tsvector/ts_rank_cd + GIN index, which is exactly the upgrade to name for zoth: "add a tsvector column + GIN index, fuse with vector scores, and get literal-string recall without a search cluster."

A concrete hybrid retrieval design for Zoth (with the SQL)

The upgrade path, in the order you would actually build it.

Step 1 — add the lexical half inside Postgres. No new cluster, no new operational surface:

ALTER TABLE policy_documents
  ADD COLUMN search_tsv tsvector
  GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || content)) STORED;

CREATE INDEX policy_documents_tsv_idx ON policy_documents USING GIN (search_tsv);

A generated stored column means the database maintains the tsvector on every write, so it cannot drift — no application code touches it. The GIN index makes @@ containment fast. The lexical query:

SELECT id, ts_rank_cd(search_tsv, query) AS lex_rank
FROM policy_documents, websearch_to_tsquery('english', $1) query
WHERE search_tsv @@ query
ORDER BY lex_rank DESC
LIMIT 20;

websearch_to_tsquery is the right entry point because it accepts human input ("azure contributor" -readonly) without throwing on malformed syntax. to_tsquery does throw, and that is a 500 waiting for a curious user.

Step 2 — take the vector half from the index you already have. The pgvector query exists (cosine distance over HNSW). The detail to state: HNSW is approximate, so recall depends on ef_search — raising it improves recall at the cost of latency, and that knob should be tuned against a measured recall number rather than guessed. The other detail is subtler: the embedding and the tsvector must describe the same document version. If the embedding was computed before an edit and the tsvector after, the two halves of your hybrid disagree about what the document says. Versioning the embedding alongside the content (Day 66's lesson) is what prevents that.

Step 3 — fuse the two ranked lists. Reciprocal rank fusion is the default because it is scale-free and needs no tuning:

WITH lex AS (SELECT id, row_number() OVER (ORDER BY lex_rank DESC) AS r FROM ... LIMIT 50),
     vec AS (SELECT id, row_number() OVER (ORDER BY embedding <=> $2) AS r FROM ... LIMIT 50)
SELECT COALESCE(lex.id, vec.id) AS id,
       COALESCE(1.0/(60 + lex.r), 0) + COALESCE(1.0/(60 + vec.r), 0) AS score
FROM lex FULL OUTER JOIN vec USING (id)
ORDER BY score DESC
LIMIT 10;

RRF's advantage over weighted score addition: cosine distances and ts_rank_cd values live on incomparable scales, and normalizing them requires calibration data you do not have. Rank position is comparable by construction. The cost is that RRF discards the magnitude of a match — a document that is overwhelmingly the best lexical hit contributes the same rank-1 term as a marginal one. That is the trade, and it is almost always worth taking first.

Step 4 — evaluate it, or do not bother. Recall@k against a hand-labelled set of ~50 queries drawn from the questions a compliance officer actually asks ("who approved similar requests", "policies covering Azure Contributor", "requests denied for sensitivity reasons"), measured separately for lexical-only, vector-only, and fused. The expected shape of the result is instructive: vector wins on fuzzy-intent queries, lexical wins on exact identifiers, and fusion wins on average while losing to lexical on a handful of exact-match cases — which is itself an argument for letting the query path detect an exact identifier and route it lexically. A retrieval change without a labelled evaluation set is a vibe, not an engineering decision — and that sentence lands in interviews, because it is the same instinct as the calibration harness (Day 72) pointed at a different subsystem.