Skip to content
Recipe Timer Lab

Decision record · DR-003

Keep the report's in-sample stack, test the out-of-fold fix it proposed, and judge both on more than accuracy

Status
Accepted
Date
2026-10
Applies to
scripts/report_models.py, scripts/holdout_eval.py, scripts/repeated_cv.py, web/src/lib/categorical-nb.ts, /stacking, /evaluation

Decision in one line

Rebuild the stack exactly as the report described it, a categorical Naive Bayes judge trained on the in-sample labels of Gaussian Naive Bayes, Logistic Regression and the SVM, run the out-of-fold version the report proposed as further work next to it, and compare both with the SVM on accuracy, macro F1 and slow recall with paired tests.

DR-006 corrects a fact stated here. This record is kept as it was written.

Context

The assignment required a stacking ensemble. The report's stack fed the labels of its three Doc2Vec models to a Naive Bayes level-1 learner, which was trained on the level-0 models' predictions for their own training recipes. It matched the SVM's accuracy but did not beat it, and the report gave two reasons. The level-0 errors were correlated, and the level-1 learner had learnt from in-sample predictions, which are more often right than predictions on new recipes, so it learnt to trust the level-0 models too much. The report proposed out-of-fold predictions as the fix but did not run it. It also did not name its Naive Bayes variant.

Decision

  • Rebuild the in-sample stack as the report described it. The level-1 learner is a categorical Naive Bayes (alpha = 1) over the three predicted labels, the natural reading of "Naive Bayes over labels".
  • Run the out-of-fold stack on the hold-out set as well, with the level-1 learner trained on the pooled out-of-fold labels from the five cross-validation folds, which is what scikit-learn's StackingClassifier does with cv=5.
  • Publish the level-1 learner's parameters and all 27 vote patterns, so /stacking can show the judge deciding in the browser.
  • Compare the stack with its best member on the same recipes with McNemar's test and paired bootstrap intervals, on macro F1 and slow recall as well as accuracy (DR-005).

Options considered

  1. Only the in-sample stack, as in 2021. Faithful, but it leaves the report's own proposed fix untested.
  2. Only the out-of-fold stack. Better practice, but it would change what the report measured.
  3. Both, side by side (chosen).
  4. A different level-1 learner, such as Logistic Regression on the level-0 probabilities. Likely stronger, but it answers a question the report did not ask.

Why

Running both keeps the rerun faithful and tests the report's explanation directly. If the in-sample training were the problem, the out-of-fold judge should decide differently on the recipes that matter. Judging on macro F1 and slow recall as well as accuracy matters here because the stack's one real ingredient is Gaussian Naive Bayes, the weakest model overall but the one that finds the most slow recipes, and accuracy barely notices slow recipes at 5% of the data.

What happened

  • On the hold-out set the stack minus the SVM is −0.2 pts (−0.6 to +0.2 pts, McNemar p = 0.30) in accuracy. The report's reading, that the stack matches its best member but does not beat it, holds.
  • The same comparison on macro F1 is +0.078 (0.063 to 0.093). The stack finds 84 of the 512 slow recipes against the SVM's 2, because the judge calls a recipe slow whenever Gaussian Naive Bayes and Logistic Regression both do, even when the SVM says medium. That one vote pattern covers 125 hold-out recipes, 68 of them slow. On the measure the report did not use, the stack is clearly better than its best member.
  • The out-of-fold judge decides differently on only three of the 27 vote patterns, and none of those patterns occurs in the hold-out set. It gives the same label as the in-sample stack on all 10,000 hold-out recipes, so on this data the proposed fix changes nothing. The correlated errors explain the result better: Logistic Regression and the SVM agree on 82% of hold-out recipes (Yule's Q 0.90).
  • Over 15 repeated stratified folds the stack is slightly less accurate than the SVM, by −0.4 pts (−0.6 to −0.2 pts, corrected resampled t, p = 0.002), and it loses on all 15 folds. Its macro F1 is higher by 0.072 (0.065 to 0.079). The stack never beats its best member on accuracy, so the report's word "matches" is, if anything, generous.

What I'd change

  • Feed the level-1 learner probabilities rather than labels, so it can tell a confident vote from a coin toss.
  • Add a level-0 model that reads different features, such as the bag-of-words Random Forest, the report's most accurate model, and measure whether its errors are less correlated with the three Doc2Vec models.
  • Report macro F1 next to accuracy from the start, as the 2026 pages now do.