Report every score with an interval, compare models on the same items, and leave the 2021 rerun untouched

Context

The 2021 report, like most coursework, printed one number per model: the mean accuracy over five folds of one shuffle. It ranked models by those numbers without asking whether the gaps were larger than the noise, and it judged the stack mostly on accuracy, which barely registers the 5% of recipes that are slow. The rerun on /models deliberately keeps that protocol, so it can be compared with the report. Changing it would make the comparison meaningless, and leaving the numbers bare would repeat the 2021 omission.

Decision

• Keep experiments.json and stacking.json, the like-for-like rerun, exactly as they are, and add new artefacts beside them.
• Rerun the five report models under RepeatedStratifiedKFold with 5 splits and 3 repeats (seed 30027), and report each mean with the corrected resampled t interval. Compare pairs of models on per-fold differences with the same correction.
• On the 10,000 hold-out recipes, give accuracy and per-class recall Wilson intervals, give macro F1 a seeded percentile bootstrap, and compare models with McNemar's exact test and paired bootstrap differences.
• Check calibration for every model that gives probabilities, with reliability diagrams, ECE, Brier score and log loss.
• Compute the intervals in TypeScript from the committed predictions when the site is built, and test the helpers against SciPy, statsmodels, scikit-learn and base R.

Options considered

1. Change the rerun to stratified, repeated CV. Better statistics, but the rerun would no longer be comparable with the report.
2. Naive t intervals over the fold scores. Easy, but too narrow, because folds share training data.
3. A separate uncertainty layer with corrected intervals (chosen).
4. Bootstrap the whole training pipeline. The most honest interval for "train on a new sample", but each SVM fit takes about ten minutes, so it is not affordable at a useful number of resamples.

Why

Keeping the two layers apart lets the site say clearly what the 2021 work found and, separately, how sure we can be of it. The corrected resampled t interval is the standard answer to correlated fold scores, and it costs nothing beyond the fold scores themselves. On the hold-out set the recipes are independent test items, so Wilson intervals and McNemar's test are exact enough and easy to explain. Computing the intervals in the site from committed predictions, rather than copying them from Python, means the tests can check the statistics against reference implementations on the real data.

What happened

• Over the 15 folds the Random Forest leads the SVM by +5.3 pts (+5.0 to +5.6 pts) and wins on all 15 folds. The model ranking from the report holds with room to spare.
• The correction matters: the naive interval for the forest would have been 76.3% to 76.7%, against 76.1% to 76.9% with the correction. The corrected interval is about twice as wide.
• The hold-out tests change the reading of one 2021 finding. The stack is no more accurate than the SVM, as the report said, but its macro F1 is clearly higher (DR-003).
• The browser model, a TF-IDF Logistic Regression trained for this site, beats every 2021 model on accuracy and macro F1, and is close to calibrated. Gaussian Naive Bayes is badly over-confident and the forest is under-confident.
• The repeated cross-validation needs 45 SVM fits and took about 1.5 hours on 10 cores. It is checkpointed after every fit.

What I'd change

• Use 10 repeats instead of 3 if the compute is there. The corrected interval is wide partly because 15 folds give a rough standard deviation.
• Add per-class precision and recall to the cross-validation summaries, not only to the hold-out tables.
• Report a stratified bootstrap for the hold-out metrics, so each resample keeps exactly 512 slow recipes.