# Stop Words And Champion Lists
Stop words are terms that appear so often they carry little information. Common examples are `the`, `is`, `of`, and `and`, but stop words can be domain-specific.
In a Harry Potter-only corpus, `Harry` may behave like a weak signal even though it is not an English stop word.
## Detection
Do not use document frequency alone. A term is a stop-word candidate when both are high:
```text
df(t) / N
avg_tf(t)
```
This separates:
- `the`: high document frequency and high average term frequency,
- `Harry`: high document frequency, but informative when it spikes in the right document.
## Handling Options
| Option | Upside | Risk |
|---|---|---|
| Drop from index | Smaller index, faster search | Breaks phrase/title queries |
| Index but low-weight | Better recall | Huge posting lists |
| Tiered fallback | Fast normal path | More query logic |
| Champion list | Bounded top docs for common terms | Can miss rare valid intent |
## Decision Checklist
Before dropping a term, ask:
- Is it required for exact titles, names, SKUs, laws, or commands?
- Does it change phrase meaning, such as `to be or not to be`?
- Does it help disambiguate short queries?
- Can the analyzer treat it differently by field?
- Can the system keep it for phrase matching but discount it for scoring?
Many systems keep common terms indexed but give them low weight. That preserves phrase and highlighting behavior while letting the scorer avoid spending most of its budget on weak signals.
## Champion Lists
A champion list stores only the top `K` documents for a very common term. The top docs can be chosen using quality, PageRank, freshness, popularity, or a static score.
```mermaid
flowchart LR
The[term: the] --> Huge[all docs posting list]
The --> Champ[top K champion list]
Champ --> Fast[bounded fallback query]
```
Champion lists are most useful when a query contains at least one expensive common term and no strongly selective term. For `the office`, the phrase and title fields should dominate. For `the`, the system can either reject the query, show editorial suggestions, or use a champion list to return popular navigational results.
| Champion Choice | Operational Effect |
|---|---|
| Small `K` | Fast but may miss valid niche documents. |
| Large `K` | Safer recall but less latency control. |
| Static quality score | Predictable and cacheable. |
| Freshness-aware score | Better for news, worse for stability. |
Refresh champion lists on a schedule or during segment merge. Recomputing them on every write usually costs more than the latency it saves.
## Product Caveat
Stop words are often important in names:
- `The Who`,
- `To Kill a Mockingbird`,
- legal clauses,
- product SKUs,
- exact titles.
This is why production systems often combine stop-word suppression with exact phrase/title matching instead of blindly deleting terms.
## Operational Notes
Track high-frequency terms by corpus, field, language, and tenant. A stop-word list copied from English web search may harm a medical, legal, code, or entertainment corpus.
Log queries where all tokens are stop-word candidates. These are product decisions, not only ranking decisions: the page may show recent searches, popular entities, or a clear "too broad" message instead of pretending the index can rank the whole corpus.
## Related Pages
- [[wiki/tf-idf-relevance-scoring]]
- [[wiki/boolean-tiered-search]]
- [[wiki/inverted-index-and-posting-lists]]