The calibration harness — measuring your own scheduler's predictions

Most projects claim an algorithm works. Traverse measures it — the Revision.predictedAt/predictedIntervalDays/predictedStability/predictedDifficulty/modelVersion/predictionFeatures columns plus the calibration harness (revision-calibration.service.ts) are an evaluation loop built into the schema. That's the story.

The harness mechanics

• Predictions are persisted at schedule time: when the scheduler creates a revision, it writes what it expected — the predicted review date, interval, and the model version that produced it. Without this, evaluation is impossible (you can't retroactively know what the model thought).
• Outcomes are recorded via RevisionAttempt (when the user actually reviewed, and how it went).
• Calibration query: join predictions against outcomes → compare predicted interval vs. observed review timing and quality → bucket by modelVersion → derive error metrics (mean signed error: are we scheduling too early or too late? absolute error; and retrieval-rate calibration: of reviews predicted at R=0.85, what fraction were actually recalled?).
• modelVersion discipline: every prediction is stamped with the algorithm version, so model changes are A/B-analyzable rather than blending into one history — the single most important schema decision in the whole harness. It's the same versioning instinct as embedding-model versioning (Day 66) applied to scheduling.

The error sign insight worth articulating: signed error tells you direction (systematically too-early schedules = the model underestimates stability), which is actionable tuning feedback; unsigned error only says "wrong". Reporting both is what a real ML-evaluation culture does.

The broader ML/analytics lessons you're demonstrating

1. Instrument the prediction, not just the outcome: without persisted predictions you have no ground truth to compare against. Applies to any scoring system (zoth's RAG risk scores are stored on request rows — the same instinct, and a future calibration study: do HIGH-risk requests actually get denied more?).
2. Version your model in the data: enables offline evaluation across versions, rollback justification, and per-version accuracy dashboards.
3. Make evaluation a query, not a notebook: the harness reads the same tables the product writes — so evaluation is always available, not a one-off analysis (the "derived analytics" shape of Day 70).
4. Free-tier as a control group (say this — it's subtle): premium gating nulls AI signals for free users, so the deterministic scheduler path runs continuously on a real population. Any calibration study compares AI-influenced vs. behavioral-only scheduling with production data — a built-in control group you didn't have to design.

What calibration actually measures (the four numbers to quote)

"Calibration" sounds abstract until you name the metrics. Four, in increasing order of usefulness:

1. Mean signed error (bias). For each prediction, observed_interval − predicted_interval; then average. A negative mean means you schedule too late (the user forgot before you asked); a positive mean means too early (you asked while the memory was still strong, wasting the user's time). This is the most actionable number in the harness, because it has a direction. Reporting only absolute error is a real mistake: MAE tells you the model is imprecise; signed error tells you which way to turn the knob.

2. Mean absolute error / interval accuracy. How far off you are on average, in days. Good as a headline ("we predict review dates within X days at the median"), useless for tuning, and dangerous as a target — a model can shrink MAE by predicting a bland middle interval for everything, which is precisely the behavior you do not want.

3. Retrieval-rate calibration — the important one. Of the reviews the scheduler placed at predicted retrievability R = 0.85, what fraction were actually recalled? If the model is calibrated, observed recall should be ≈ 0.85. This tests the algorithm's core claim rather than its arithmetic: FSRS-style schedulers are defined by a target recall probability, and the entire product promise ("you will remember this, and we will not waste your time") rests on that number being real. Plotting observed recall against predicted retrievability in deciles is a reliability diagram whose ideal is the diagonal, and deviations are diagnostic — points above the diagonal in the low-R region mean the model is too pessimistic, and the correction is a parameter refit, not a fudge factor.

4. Brier score / log loss. A proper scoring rule that penalizes miscalibration and overconfidence: predicting 0.99 and being wrong is punished far more than predicting 0.6 and being wrong. This is the number to quote when you want one figure that cannot be gamed, and it is what lets you compare two modelVersions on equal footing.

What you do when the numbers are bad

Persisting modelVersion is what makes the response to miscalibration a versioned response:

• Systematic bias across all users → the parameters are wrong. Refit on observed outcomes, ship as a new version, and keep both in the data so the improvement is measurable rather than asserted.
• Bias for a subset (new users with no history, or one topic) → a cold-start or per-topic problem, which is a modeling change (per-topic parameters, population priors) rather than a global refit.
• Recall calibration is fine but retention is poor → the model is calibrated and the pedagogy is the issue. That distinction matters, because it tells you not to touch the algorithm.

The most valuable property of this harness is not any single metric. It is that the scheduler becomes falsifiable: there is a query you can run that will tell you the algorithm is wrong, and it runs against the same tables the product writes. Most projects claiming an algorithm works have no such query — they have a demo. Saying "here is the query that would prove my scheduler wrong, and here is what it currently reports" is a genuinely senior answer, because it shows you optimize for being correct rather than for appearing correct.