Stacking

Three cooks and a judge

The team's ensemble asked three level-0 models for a label, then let a Naive Bayes level-1 model learn which combinations of votes to trust. It matched its best member but never beat it. Here is why.

How it works

From recipe to verdict

01
Encode
Each recipe becomes 350 Doc2Vec numbers from its name, ingredients and steps.
02
Level 0
Gaussian NB, Logistic Regression and an RBF SVM each predict quick, medium or slow.
03
Level 1
A categorical Naive Bayes reads only the three labels and scores every class.
04
Verdict
The class with the highest score wins, even if it overrules the majority.

Try it

Cast the votes yourself

Pick what each level-0 model says. The judge's probabilities come from the categorical Naive Bayes fitted on the 40,000 training recipes, recomputed here in your browser.

Level 1 · the Naive Bayes judge

Medium

majority vote would say Medium

Quick4.4% Quick
Medium83.4% Medium
Slow12.2% Slow

How the judge scores each class: log prior plus one log-likelihood per vote. The highest total wins.

The judge's log scores for each class
Term	Quick	Medium	Slow
log P(class)	-0.82	-0.68	-2.97
GNB said slowlog P(GNB = slow \| class)	-2.59	-1.88	-1.00
LR said mediumlog P(LR = medium \| class)	-1.18	-0.31	-0.61
SVM said mediumlog P(SVM = medium \| class)	-1.49	-0.26	-0.46
Total	-6.07	-3.12	-5.04

The judge learnt from 2,761 of 40,000 training recipes that got this exact vote.

It happened for 674 of 10,000 hold-out recipes (6.7%). Their true classes:

Quick 84
Medium 490
Slow 100

Diagnosis

Why stacking could not beat the SVM

A stack only helps when its members make different mistakes. These ones mostly do not.

LR and SVM agree on

82%

Yule's Q 0.90: near 1, they tend to be right and wrong on the same recipes

All three cooks agree on

49%

of hold-out recipes, leaving the judge nothing to decide

Judge overrules the SVM on

4.9%

of hold-out recipes

Ceiling if a perfect judge picked the right cook

85%

best single cook: SVM, 71.2%

Hold-out, pairs of level-0 models

Correlated errors

Agreement and shared errors between each pair of level-0 models
Pair	Agree	Both wrong	Q
GNB + LR	53.8%	18.6%	0.34
GNB + SVM	60.1%	19.8%	0.57
LR + SVM	82.2%	21.9%	0.90

Yule's Q is 0 when two models err independently and approaches 1 when they tend to be right, and wrong, on the same recipes; a negative Q means their mistakes fall on different recipes. It reads the pattern, not the amount: “Both wrong” is how often they actually fail together. The report made the same point from its confusion matrices: LR and the SVM learnt the same thing from the same features.

In-sample against out-of-fold

Training the judge on its own homework

Level-0 accuracy on training rows they had seen, on unseen folds, and on the hold-out set
Model	SeenSeen rows	OOFUnseen folds	Hold-out
GNBGaussian NB	54.4%	54.2%	53.6%
LRLogistic Regression	68.6%	67.8%	68.2%
SVMSVM	73.5%	71.3%	71.2%

The report trained its judge on level-0 predictions for the same rows those models had just learnt, which flatters whichever model memorises most. It suggested out-of-fold predictions instead. In the rebuild both versions score 70.9% on the hold-out set. Retraining the judge on out-of-fold votes changes its verdict for only 3 of the 27 vote patterns, and no hold-out recipe produces any of them.

The judge's rulebook

All 27 possible votes

Every combination of three labels, most common first: what the judge decides, how many of the 40,000 training recipes it learnt the vote from, and what the 10,000 hold-out recipes really were. 6 of the votes never happened in training, yet the judge still rules on them.

The ten most common votes with the level-1 decision and how often each occurred
Judge	Training recipes	Hold-out recipes	Truly quick / medium / slow
Quick	11,917	2,850	2,181 / 592 / 77
Medium	8,039	2,083	364 / 1,653 / 66
Medium	6,914	1,584	368 / 1,144 / 72
Quick	3,154	700	396 / 266 / 38
Medium	2,761	674	84 / 490 / 100
Quick	2,266	668	446 / 207 / 15
Quick	1,023	357	206 / 133 / 18
Medium	1,162	351	130 / 214 / 7
Medium	740	224	73 / 139 / 12
Quick	757	187	77 / 105 / 5

judge saysQuick
11,917 training · 2,850 hold-out
truly2,181quick592medium77slow
judge saysMedium
8,039 training · 2,083 hold-out
truly364quick1,653medium66slow
judge saysMedium
6,914 training · 1,584 hold-out
truly368quick1,144medium72slow
judge saysQuick
3,154 training · 700 hold-out
truly396quick266medium38slow
judge saysMedium
2,761 training · 674 hold-out
truly84quick490medium100slow
judge saysQuick
2,266 training · 668 hold-out
truly446quick207medium15slow
judge saysQuick
1,023 training · 357 hold-out
truly206quick133medium18slow
judge saysMedium
1,162 training · 351 hold-out
truly130quick214medium7slow
judge saysMedium
740 training · 224 hold-out
truly73quick139medium12slow
judge saysQuick
757 training · 187 hold-out
truly77quick105medium5slow

Show the other 17 votes

Less common votes with the level-1 decision and how often each occurred
Judge	Training recipes	Hold-out recipes	Truly quick / medium / slow
Slow	459	125	11 / 46 / 68
Medium	470	120	69 / 45 / 6
Medium	119	34	14 / 14 / 6
Slow	40	13	5 / 4 / 4
Medium	33	10	1 / 5 / 4
Slow	68	7	0 / 1 / 6
Slow	35	6	1 / 1 / 4
Slow	35	5	0 / 3 / 2
Medium	3	2	0 / 0 / 2
Slow	3	0	0 / 0 / 0
Slow	2	0	0 / 0 / 0
Slow	0	0	0 / 0 / 0
Slow	0	0	0 / 0 / 0
Slow	0	0	0 / 0 / 0
Slow	0	0	0 / 0 / 0
Slow	0	0	0 / 0 / 0
Slow	0	0	0 / 0 / 0

judge saysSlow
459 training · 125 hold-out
truly11quick46medium68slow
judge saysMedium
470 training · 120 hold-out
truly69quick45medium6slow
judge saysMedium
119 training · 34 hold-out
truly14quick14medium6slow
judge saysSlow
40 training · 13 hold-out
truly5quick4medium4slow
judge saysMedium
33 training · 10 hold-out
truly1quick5medium4slow
judge saysSlow
68 training · 7 hold-out
truly0quick1medium6slow
judge saysSlow
35 training · 6 hold-out
truly1quick1medium4slow
judge saysSlow
35 training · 5 hold-out
truly0quick3medium2slow
judge saysMedium
3 training · 2 hold-out
truly0quick0medium2slow
judge saysSlow
3 training · 0 hold-out
judge saysSlow
2 training · 0 hold-out
judge saysSlow
0 training · 0 hold-out
judge saysSlow
0 training · 0 hold-out
judge saysSlow
0 training · 0 hold-out
judge saysSlow
0 training · 0 hold-out
judge saysSlow
0 training · 0 hold-out
judge saysSlow
0 training · 0 hold-out

Circles show each model's vote: Q for quick, M for medium, S for slow. Decisions in this table are recomputed in TypeScript and checked against scikit-learn by the test suite.

From the test kitchen

Eighteen hold-out recipes

Amarula Cream Shake
3 min · 5 ingredients · 2 steps
GNB: QuickLR: QuickSVM: Quick
Judge Quickright
Glazed Orange Carrots
30 min · 3 ingredients · 5 steps
GNB: QuickLR: QuickSVM: Medium
Judge Quickright
Very Raspberry Frappes
15 min · 5 ingredients · 10 steps
GNB: MediumLR: QuickSVM: Quick
Judge Quickright
Green Mango Salad
10 min · 11 ingredients · 4 steps
GNB: QuickLR: QuickSVM: Quick
Judge Quickright
Koukia Yiahnt (Fava Bean Stew)
25 min · 9 ingredients · 6 steps
GNB: QuickLR: MediumSVM: Quick
Judge Quickright
Low Carb Pasta With Gorgonzola and Walnuts
25 min · 8 ingredients · 5 steps
GNB: QuickLR: QuickSVM: Quick
Judge Quickright
Parsnip Pudding
1 h 20 min · 10 ingredients · 5 steps
GNB: MediumLR: MediumSVM: Medium
Judge Mediumright
Oatmeal Soda Bread
50 min · 9 ingredients · 5 steps
GNB: MediumLR: MediumSVM: Medium
Judge Mediumright
Upside-Down Pear Gingerbread
55 min · 17 ingredients · 5 steps
GNB: MediumLR: MediumSVM: Medium
Judge Mediumright
Tuckey's Chicken and Dumplings
1 h 10 min · 5 ingredients · 1 step
GNB: MediumLR: MediumSVM: Medium
Judge Mediumright
The Mac Daddy of All Mac N' Cheeses
46 min · 17 ingredients · 7 steps
GNB: MediumLR: MediumSVM: Medium
Judge Mediumright
Roasted Cauliflower With Capers
50 min · 6 ingredients · 7 steps
GNB: QuickLR: MediumSVM: Medium
Judge Mediumright
All Day Beef Stew
8 h 25 min · 12 ingredients · 3 steps
GNB: QuickLR: MediumSVM: Quick
Judge Quickwrong, really slow
Navy Bean Soup
8 h 20 min · 14 ingredients · 3 steps
GNB: QuickLR: MediumSVM: Quick
Judge Quickwrong, really slow
Crock-Pot Italian Sausage and Peppers
6 h 25 min · 15 ingredients · 7 steps
GNB: SlowLR: SlowSVM: Medium
Judge Slowright
Garlic Lover's Chili
5 h 25 min · 16 ingredients · 4 steps
GNB: MediumLR: MediumSVM: Medium
Judge Mediumwrong, really slow
Cholent
20 h 20 min · 7 ingredients · 1 step
GNB: QuickLR: QuickSVM: Quick
Judge Quickwrong, really slow
Grandma Stevens Baked Beans
6 h 10 min · 5 ingredients · 6 steps
GNB: SlowLR: MediumSVM: Medium
Judge Mediumwrong, really slow

Six random recipes from each true class of the hold-out set. Five-fold CV accuracy of the stack: 71.0%.