Evaluation

How sure can we be about each number?

The report printed one accuracy per model. Here every model gets an interval, every comparison between two models is paired on the same recipes, and the models that give probabilities are checked for calibration. The 2021 rerun on Models is left exactly as it was, and this page adds the uncertainty around it.

Repeated cross-validation · 3 × 5 folds · seed 30027

Cross-validation, repeated and stratified

The report used one shuffle of plain five-fold cross-validation. Rerunning the same five models over 15 stratified folds (3 different shuffles, each fold with 32,000 training and 8,000 test recipes) shows how much the score moves with the split. Fold scores share most of their training recipes, so the interval is the corrected resampled t interval (Nadeau and Bengio, 2003), which is wider than the naive one.

Accuracy over 15 folds, with corrected 95% intervals

Estimate with 95% interval
2021 report

ModelAccuracy

Random Forestfolds 75.8% to 76.9%76.5% (76.1% to 76.9%)

SVM (RBF, one-vs-rest)folds 70.6% to 71.8%71.2% (70.8% to 71.7%)

Stacked Naive Bayesfolds 70.4% to 71.5%70.8% (70.4% to 71.3%)

Logistic Regressionfolds 67.0% to 68.2%67.6% (67.2% to 68.1%)

Gaussian Naive Bayesfolds 53.0% to 55.2%54.2% (53.4% to 55.0%)

Mean of 15 stratified folds with its corrected resampled t interval (14 degrees of freedom). Hollow markers: the 2021 report's figure for the same model on the staff data, which a rebuild on other recipes is not expected to match.

Mean accuracy, macro F1 and slow recall over 15 folds, with corrected and naive intervals, next to the rebuild's single plain five-fold split and the 2021 report's own figure
Model	Accuracy, corrected 95%	Naive interval (too narrow)	Macro F1	Slow recall	Rebuild, report's protocol	2021 report
Random Forest	76.5%76.1% to 76.9%	76.3% to 76.7%	0.6340.625 to 0.643	22.0%20.0% to 24.0%	76.6%one plain 5-fold split	78.9%staff data, Table 4
SVM (RBF, one-vs-rest)	71.2%70.8% to 71.7%	71.0% to 71.4%	0.4900.486 to 0.494	0.4%0.0% to 0.8%	71.3%one plain 5-fold split	73.0%staff data, Table 4
Stacked Naive Bayes	70.8%70.4% to 71.3%	70.6% to 71.0%	0.5620.555 to 0.568	15.6%14.1% to 17.1%	71.0%one plain 5-fold split	73.0%staff data, implied by Figure 5
Logistic Regression	67.6%67.2% to 68.1%	67.4% to 67.9%	0.5420.534 to 0.550	16.3%14.4% to 18.2%	67.8%one plain 5-fold split	72.0%staff data, implied by Figure 2
Gaussian Naive Bayes	54.2%53.4% to 55.0%	53.8% to 54.6%	0.4460.435 to 0.456	35.1%32.5% to 37.7%	54.2%one plain 5-fold split	62.6%staff data, Table 4

Paired, fold by fold

Random Forest against the SVM

The forest wins on 15 of 15 folds. Its accuracy is 5.3 pts higher (corrected 95% +5.0 to +5.6 pts, p < 0.0001), and its macro F1 higher by 0.145 (0.135 to 0.154): the gap is mostly the SVM's near-total neglect of slow recipes.

The report's stacking claim

The stack against its best member

The stack minus the SVM: −0.4 pts accuracy (corrected 95% −0.6 to −0.2 pts, p = 0.002), winning on 0 of 15 folds, and +0.072 macro F1 (0.065 to 0.079). The accuracy interval lies below zero. The stack never beats its best member, as the report said, and is in fact slightly less accurate, while its macro F1 is higher because it finds more slow recipes.

Hold-out set · 10,000 unseen recipes

On 10,000 recipes no model saw

Every model was refitted on all 40,000 training recipes and scored once on the hold-out set (4,426 quick, 5,062 medium, 512 slow). Accuracy and recall are proportions, so they get Wilson intervals. Macro F1 gets a percentile bootstrap (2,000 resamples, seed 30027). The two TF-IDF models are this site's browser predictor, trained on the same recipes and labels.

Hold-out accuracy with 95% Wilson intervals

Estimate with 95% interval

ModelAccuracy

TF-IDF + Logistic Regressionthis site's predictor79.0% (78.2% to 79.8%)

Random Forest2021 report, rebuilt76.4% (75.5% to 77.2%)

TF-IDF + Naive Bayesthis site's predictor73.3% (72.4% to 74.2%)

SVM (RBF, one-vs-rest)2021 report, rebuilt71.2% (70.3% to 72.0%)

Stacked Naive Bayes2021 report, rebuilt70.9% (70.0% to 71.8%)

Logistic Regression2021 report, rebuilt68.2% (67.2% to 69.1%)

Gaussian Naive Bayes2021 report, rebuilt53.6% (52.6% to 54.5%)

With 10,000 recipes each interval is under ±1 point wide, so differences of a few points are not noise. Whether they matter is a separate question.

Hold-out accuracy, macro F1 and recall of each class, with intervals
Model	Accuracy	Macro F1	Quick recall	Medium recall	Slow recall
TF-IDF + Logistic Regressionthis site's predictor	79.0%78.2% to 79.8%	0.7060.690 to 0.721	81.1%3,588 of 4,42679.9% to 82.2%	81.1%4,105 of 5,06280.0% to 82.1%	39.8%204 of 51235.7% to 44.1%
Random Forest2021 report, rebuilt	76.4%75.5% to 77.2%	0.6260.609 to 0.643	73.9%3,270 of 4,42672.6% to 75.2%	84.2%4,264 of 5,06283.2% to 85.2%	20.1%103 of 51216.9% to 23.8%
TF-IDF + Naive Bayesthis site's predictor	73.3%72.4% to 74.2%	0.6690.653 to 0.683	71.2%3,153 of 4,42669.9% to 72.6%	78.3%3,966 of 5,06277.2% to 79.5%	41.4%212 of 51237.2% to 45.7%
SVM (RBF, one-vs-rest)2021 report, rebuilt	71.2%70.3% to 72.0%	0.4890.483 to 0.496	75.0%3,318 of 4,42673.7% to 76.2%	75.0%3,797 of 5,06273.8% to 76.2%	0.4%2 of 5120.1% to 1.4%
Stacked Naive Bayes2021 report, rebuilt	70.9%70.0% to 71.8%	0.5670.551 to 0.583	74.7%3,306 of 4,42673.4% to 76.0%	73.2%3,704 of 5,06271.9% to 74.4%	16.4%84 of 51213.5% to 19.9%
Logistic Regression2021 report, rebuilt	68.2%67.2% to 69.1%	0.5520.536 to 0.567	70.3%3,113 of 4,42669.0% to 71.7%	71.4%3,612 of 5,06270.1% to 72.6%	17.6%90 of 51214.5% to 21.1%
Gaussian Naive Bayes2021 report, rebuilt	53.6%52.6% to 54.5%	0.4490.439 to 0.460	68.3%3,023 of 4,42666.9% to 69.7%	42.0%2,124 of 5,06240.6% to 43.3%	41.0%210 of 51236.8% to 45.3%

Accuracy flatters every model here, because 95% of recipes are quick or medium. Macro F1 weighs the three classes equally and tells a different story: the SVM is fourth of 7 on accuracy but sixth on macro F1, because it recalls only 0.4% of slow recipes.

Paired comparisons · McNemar's test

Is one model really better than another?

Two models scored on the same recipes are compared on the recipes where exactly one of them is right. McNemar's exact test asks whether those disagreements favour one side more than chance would. The effect sizes are the accuracy difference, with a paired bootstrap interval, and the odds that a disagreement favours the first model.

McNemar tests and paired differences on the hold-out set
Comparison	Only A right / only B right	Accuracy, A minus B	Macro F1, A minus B	Odds a disagreement favours A	McNemar exact
A: Random ForestB: SVM (RBF, one-vs-rest)The best two of the report's models	1,212 / 692of 10,000 recipes	+5.2 pts+4.3 to +6.1 pts	+0.1370.119 to 0.154	1.751.59 to 1.93	p < 0.0001
A: TF-IDF + Logistic RegressionB: Random ForestThe best model overall against the report's best	823 / 563of 10,000 recipes	+2.6 pts+1.9 to +3.3 pts	+0.0800.064 to 0.096	1.461.31 to 1.63	p < 0.0001
A: Stacked Naive BayesB: SVM (RBF, one-vs-rest)The report's claim: the stack does not beat its best member	218 / 241of 10,000 recipes	−0.2 pts−0.6 to +0.2 pts	+0.0780.063 to 0.093	0.900.75 to 1.09	p = 0.30
A: Stacked Naive BayesB: Stacked NB, out-of-foldThe fix the report proposed: an out-of-fold level-1 learner	0 / 0of 10,000 recipes	0.0 pts0.0 to 0.0 pts	0.0000.000 to 0.000	n/ano disagreements	p = 1.00

Random Forest is more accurate than SVM (RBF, one-vs-rest) by 5.2 pts (+4.3 to +6.1 pts, p < 0.0001).
TF-IDF + Logistic Regression is more accurate than Random Forest by 2.6 pts (+1.9 to +3.3 pts, p < 0.0001).
Stacked Naive Bayes and SVM (RBF, one-vs-rest) cannot be told apart on accuracy: −0.2 pts (−0.6 to +0.2 pts), p = 0.30, across 459 disagreements.
Stacked Naive Bayes and Stacked NB, out-of-fold give the same label to every one of the 10,000 recipes, so the proposed fix changes nothing on this data.

Calibration · 10 bins · bootstrap seed 30027

When a model says 80%, is it right 80% of the time?

Each panel plots, for ten bins of top-label confidence, the share of hold-out predictions that were right against the mean confidence in the bin. On the dashed diagonal a model is calibrated, and below it the model is over-confident. Dot area follows how many recipes fall in the bin. The SVM is left out: as in the report it gives one-vs-rest decision values, not probabilities.

this site's predictor

TF-IDF + Logistic Regression

Confidence vs right: 78.3% vs 79.0% (close)
ECE: 0.009 (0.006 to 0.018)
Brier score: 0.294 (0.286 to 0.302)
Log loss: 0.490 (0.478 to 0.503)

2021 report, rebuilt

Logistic Regression

Confidence vs right: 68.2% vs 68.2% (close)
ECE: 0.021 (0.015 to 0.030)
Brier score: 0.432 (0.423 to 0.439)
Log loss: 0.698 (0.685 to 0.712)

this site's predictor

TF-IDF + Naive Bayes

Confidence vs right: 75.3% vs 73.3% (over-confident)
ECE: 0.026 (0.019 to 0.034)
Brier score: 0.364 (0.355 to 0.373)
Log loss: 0.600 (0.586 to 0.616)

2021 report, rebuilt

Random Forest

Confidence vs right: 68.4% vs 76.4% (under-confident)
ECE: 0.080 (0.074 to 0.088)
Brier score: 0.340 (0.334 to 0.347)
Log loss: 0.572 (0.561 to 0.585)

2021 report, rebuilt

Stacked Naive Bayes

Confidence vs right: 83.4% vs 70.9% (over-confident)
ECE: 0.125 (0.116 to 0.134)
Brier score: 0.447 (0.434 to 0.460)
Log loss: 0.773 (0.751 to 0.793)

2021 report, rebuilt

Gaussian Naive Bayes

Confidence vs right: 95.8% vs 53.6% (over-confident)
ECE: 0.423 (0.412 to 0.432)
Brier score: 0.870 (0.851 to 0.889)
Log loss: 6.487 (6.235 to 6.621)

Lower is better for all three scores. Gaussian Naive Bayes is the textbook case of over-confidence: it multiplies 350 features as if they were independent, so 87% of its predictions claim more than 90% confidence while 54% of all its predictions are right. The Random Forest is the opposite: its probabilities are averaged tree votes and sit below its accuracy. The two Logistic Regressions are close to the diagonal, which is why the browser predictor shows its probabilities.

Reproducibility

Seeds and sample sizes

Sampling, folds, forests and the hold-out split: seed 30027 (the subject code), set in scripts/common.py.
Repeated cross-validation: RepeatedStratifiedKFold, 5 splits × 3 repeats, seed 30027, with 8,000 test recipes in each fold.
Bootstrap intervals on this page: 2,000 resamples with seed 30027 (macro F1 and paired differences), 1,000 resamples with seed 30027 (calibration scores).
Every interval is 95%. Every number is recomputed from the committed artefacts when the site is built, and the test suite checks the statistics against SciPy, statsmodels, scikit-learn and R.