Methods

How the rebuild works, and what it cannot tell you

Where the data comes from, how every number is tested, how uncertain it is, the decisions behind the design and how the optional AI features are governed.

Data provenance

The 2021 staff data is not public, so the rebuild uses the public-domain (CC0) Food.com crawl published on Kaggle by Alvin (irkaal), checked by SHA-256 before use. Recipes published up to 2018 with a valid total time and text are labelled from their listed total time, then sampled to the 2021 report's exact class counts: 40,000 training recipes and a 10,000-recipe hold-out set, all with seed 30027. The data statement has the funnel, the label rules and the differences from the course data, and DR-002 explains the thresholds.

Method

Six uv scripts in scripts/, each with pinned dependencies, rebuild everything the site shows.

prepare_data.py downloads and checks the crawl, labels the recipes and draws the samples.
features.py trains the six Doc2Vec models (50 and 100 dimensions for the name, the ingredients and the method) on the training recipes and builds the "steps" bag of words.
report_models.py reruns every experiment in the 2021 report with the report's estimators and settings (DR-001).
browser_model.py trains the two TF-IDF models that run on /predict and writes parity fixtures for the TypeScript ports.
repeated_cv.py reruns the five report models under repeated stratified cross-validation.
holdout_eval.py writes every model's hold-out predictions, the calibration analysis and the LLM evaluation set. It checks that each refitted model reproduces the published hold-out labels before it writes anything.

The site turns these artefacts into intervals and tests when it is built. The statistics are in web/src/lib/stats/ and are tested against SciPy, statsmodels, scikit-learn and R.

Evaluation design

The design has two layers, kept apart on purpose (DR-005).

The 2021 rerun (/models) uses the report's own protocol, plain five-fold cross-validation with one shuffle, so its numbers can sit next to the report's. Nothing in this layer was changed by the 2026 upgrade.

The uncertainty layer (/evaluation) asks how far each number can be trusted.

Repeated stratified cross-validation: 5 splits by 3 repeats with seed 30027, so every one of the 15 folds keeps the class mix. Each fold trains on 32,000 recipes and tests on 8,000. Means get the corrected resampled t interval of Nadeau and Bengio (2003), because fold scores share most of their training data and the naive t interval is too narrow. Two models are compared on per-fold differences with the same correction.
The hold-out set: every model refitted on all 40,000 training recipes and scored once on the 10,000 hold-out recipes. Accuracy and per-class recall get Wilson intervals. Macro F1 gets a percentile bootstrap over recipes (2,000 resamples, seed 30027).
Paired comparisons on the hold-out set: McNemar's exact test on the recipes where exactly one of two models is right, paired bootstrap intervals for the difference in accuracy and in macro F1, and the odds that a disagreement favours the first model, with an exact interval.
Calibration, for the models that give probabilities: reliability diagrams over 10 confidence bins, the expected calibration error, the Brier score and log loss, each with a bootstrap interval (1,000 resamples, seed 30027).
The LLM evaluation on /llm-eval follows its own evaluation card.

Accuracy is reported because the report used it, but macro F1 and slow recall are reported next to it, because 95% of recipes are quick or medium and accuracy barely notices how a model handles the slow ones.

Results with uncertainty

Mean accuracy over the 15 stratified folds, with the corrected 95% interval:

Model	Accuracy over 15 folds
Random Forest	76.5% (76.1% to 76.9%)
SVM (RBF, one-vs-rest)	71.2% (70.8% to 71.7%)
Stacked Naive Bayes	70.8% (70.4% to 71.3%)
Logistic Regression	67.6% (67.2% to 68.1%)
Gaussian Naive Bayes	54.2% (53.4% to 55.0%)

The ranking is the report's. No two neighbouring models have overlapping intervals except the SVM and the stack, and their paired difference still favours the SVM by 0.4 points (−0.6 to −0.2 points). The three repeats agree with each other to within about 0.1 points for every model. On the hold-out set the TF-IDF Logistic Regression behind /predict is the most accurate model at 79.0% (78.2% to 79.8%), ahead of the rebuilt Random Forest at 76.4% (75.5% to 77.2%). The SVM finds 2 of the 512 slow hold-out recipes. The stack is no more accurate than the SVM but recalls far more slow recipes (DR-003).

Assumptions

The public crawl is a fair stand-in for the staff sample. Both come from the same website, but the staff sample's selection rules are unknown.
The listed total time is the truth the models are scored against, even though authors enter it themselves.
Recipes are independent of each other. Some Food.com recipes are near-duplicates, which would make every interval on this site a little too narrow.
The hold-out set behaves like the Kaggle test set did in 2021. It was drawn from the same pool with the same class mix.

Limitations

The rebuild cannot reproduce the 2021 numbers exactly, because the recipes, the label thresholds and the Doc2Vec vectors all differ. The site compares the two side by side and never treats one as the other.
Gensim trains Doc2Vec with six threads, so rebuilding the vectors moves scores in the third decimal. The cached vectors reproduce the committed results exactly.
The tuning sweeps from the report are scored on one split, as the cheapest reading of a protocol the report did not name. Near-ties in them, such as C = 10 against C = 1, can flip on a rerun.
Slow recipes are 5% of the data. Results about them rest on 2,049 training and 512 hold-out recipes, and on 10 recipes in the LLM evaluation set.
The Doc2Vec models and the "steps" vocabulary were fitted once on all 40,000 training recipes, and the repeated cross-validation then splits those same recipes, so each fold's test recipes helped shape the features. The cross-validation intervals cover the classifiers but not the feature learning. The clean hold-out set, which played no part in the features, agrees within 0.1 points: the Random Forest scores 76.5% in cross-validation and 76.4% on the hold-out set, and the SVM 71.2% in both. The browser model's TF-IDF vocabulary was also fitted before its tuning cross-validation.
The calibration and paired tests use the hold-out set once. They say how sure we can be about these models on this data, not how they would do on recipes from another site or another decade.

What I'd change

Run the threshold sensitivity analysis proposed in DR-002 and report how much each finding moves.
Deduplicate near-identical recipes before sampling, then rerun the intervals.
Use nested cross-validation for the tuning sweeps instead of a single tuning split.
Recalibrate the forest and Gaussian Naive Bayes (isotonic or Platt scaling on held-out folds) and show the reliability diagrams before and after.
Fit Doc2Vec, the bag of words and the TF-IDF vocabulary inside each cross-validation fold, so the intervals cover the feature learning as well.

Model card

The models this site runs for you

The two TF-IDF classifiers behind the Predict page, with the 2021 Random Forest for reference, scored on the 10,000 hold-out recipes.

Hold-out evaluation of the served models, with 95% intervals
Model	Accuracy (Wilson)	Macro F1 (bootstrap)	Slow recall (Wilson)	ECE
TF-IDF + Logistic Regression	79.0%78.2% to 79.8%	0.7060.690 to 0.721	204 of 51235.7% to 44.1%	0.0090.006 to 0.018
Random Forest	76.4%75.5% to 77.2%	0.6260.609 to 0.643	103 of 51216.9% to 23.8%	0.0800.074 to 0.088
TF-IDF + Naive Bayes	73.3%72.4% to 74.2%	0.6690.653 to 0.683	212 of 51237.2% to 45.7%	0.0260.019 to 0.034

Models

Model	Role	Features	Where it runs
TF-IDF + Logistic Regression (C = 1)	Default predictor on `/predict`	6,000 word and word-pair TF-IDF weights from the name, ingredients and method	In the visitor's browser
TF-IDF + Multinomial Naive Bayes (alpha = 0.01)	Second predictor on `/predict`	The same TF-IDF features	In the visitor's browser
Random Forest, 200 trees	The 2021 report's best model, rebuilt	Bag of words of the method steps	Offline only (scripts)
RBF SVM, Logistic Regression, Gaussian Naive Bayes, stacked Naive Bayes	The rest of the 2021 lineup, rebuilt	350 Doc2Vec features, or the three models' labels for the stack	Offline only (scripts)

The two browser models are companions trained for this site, not models from the 2021 report. Their weights are exported to JSON and re-implemented in TypeScript, and a test checks the TypeScript gives the same probabilities as scikit-learn on preset and hold-out recipes.

Intended use

Sorting a recipe into quick (under 35 minutes), medium (35 minutes to 4 hours) or slow (more than 4 hours) from its text, as a teaching demo of text classification, and as the subject of the evaluation on /evaluation and /llm-eval. The models are not meant for planning meals around, and never for food safety: cooking and chilling times for safety come from the recipe and from food safety guidance, not from a classifier.

Training data

40,000 Food.com recipes from the public-domain crawl, sampled to the 2021 report's class counts (17,705 quick, 20,246 medium and 2,049 slow) with seed 30027. Labels come from each recipe's listed total time. The data statement covers the source, the filters and the differences from the 2021 course data. The browser models' hyperparameters were chosen by three-fold stratified cross-validation on the training recipes only.

Evaluation

Scored once on the 10,000 hold-out recipes (4,426 quick, 5,062 medium, 512 slow) that no model saw in training. Intervals are 95%: Wilson for accuracy and recall, a percentile bootstrap (2,000 resamples, seed 30027) for macro F1, and a percentile bootstrap (1,000 resamples, seed 30027) for the expected calibration error (ECE).

Model	Accuracy	95% interval	Macro F1	Slow recall	ECE
TF-IDF + Logistic Regression	79.0%	78.2% to 79.8%	0.706 (0.690 to 0.721)	204 of 512 (35.7% to 44.1%)	0.009 (0.006 to 0.018)
TF-IDF + Naive Bayes	73.3%	72.4% to 74.2%	0.669 (0.653 to 0.683)	212 of 512 (37.2% to 45.7%)	0.026 (0.019 to 0.034)
Random Forest (2021, rebuilt)	76.4%	75.5% to 77.2%	0.626 (0.609 to 0.643)	103 of 512 (16.9% to 23.8%)	0.080 (0.074 to 0.088)

The TF-IDF Logistic Regression is more accurate than the rebuilt Random Forest on the same recipes by 2.6 points (paired bootstrap 1.9 to 3.3 points, McNemar p < 0.0001), and it recalls about twice as many slow recipes. Its probabilities are also close to calibrated, which is why the predictor shows them. /evaluation has every model's intervals, the paired tests and the reliability diagrams.

Known failure modes

Slow recipes. Even the best model finds fewer than half of them, and most of the misses are called medium. Slow is 5% of the data, so the models have seen few examples.
Unstated waiting time. "Refrigerate until firm" or "let the dough rise" with no time given looks like a short recipe.
Single digits are dropped. The tokeniser keeps tokens of two or more characters, so "bake for 5 minutes" reaches the model as "bake for minutes". Two-digit times such as "40" survive.
Unfamiliar text. Recipes in other languages, or a few words with no method, share few terms with the training data. When no term is known the predictor says so and gives no verdict.
Label noise. The labels are authors' own listed times, so some "errors" are recipes whose listed time does not match their method.

Ethical considerations

The task is low stakes and the data is public domain, with no information about Food.com users. A recipe typed into /predict is classified in the browser and never sent anywhere, unless the visitor asks for an LLM second opinion with their own key, in which case it goes to the provider they chose (AI use statement). The models carry the biases of the crawl, whose recipes, units and ingredients are mostly American and stop in 2018.

Data statement

The recipes and how they were labelled

Source

The recipes come from the public-domain (CC0) "Food.com - Recipes and Reviews" crawl, published on Kaggle by Alvin (irkaal) in 2020 and read from a byte-identical mirror on the Hugging Face Hub. scripts/prepare_data.py checks the file's SHA-256 (7d564e2fabb9…, the full hash is in web/public/data/dataset.json) before using it. The 2021 assignment asked every team to cite the dataset paper, and the site does the same: Majumder, B. P., Li, S., Ni, J. and McAuley, J. (2019). Generating Personalized Recipes from Historical User Preferences. EMNLP-IJCNLP 2019.

From raw rows to the training set

Step	Recipes left
Rows in the crawl	522,517
Published in 2018 or earlier	519,174
A valid total time, more than zero and no more than 30 days	516,821
A name, at least one ingredient and at least one step	514,623

Of the 514,623 recipes, 42.0% are quick, 52.4% medium and 5.7% slow. The training set is a random sample of 40,000 with the 2021 report's exact class counts (17,705 quick, 20,246 medium, 2,049 slow), and the hold-out set is another 10,000 in the same proportions (4,426, 5,062 and 512). Both are drawn with seed 30027.

How the labels were derived

Each recipe's label comes from its listed total time, an ISO 8601 duration such as PT1H20M that the recipe's author entered on Food.com. Under 35 minutes is quick, 35 minutes to 4 hours is medium and more than 4 hours is slow. The staff thresholds from 2021 were lost, so these round numbers were chosen because they give class shares closest to the report's (DR-002). The listed time is the author's own estimate, and authors are not consistent about whether it includes overnight chilling or marinating, so the labels carry noise that no model can remove.

How this differs from the 2021 course data

Different recipes. In 2021 the staff supplied a sample of Majumder et al.'s crawl that is not public. The rebuild uses a later public crawl of the same website. The two overlap in kind but not, as far as anyone can check, recipe for recipe.
Rebuilt labels. The staff labels came with the bundle. The rebuild derives them from minutes with guessed thresholds.
Regenerated features. The staff Doc2Vec vectors and CountVectorizer matrices were part of the bundle. The rebuild trains its own with gensim and scikit-learn, on the training recipes only.
A hold-out set instead of Kaggle. The 2021 test set was scored on a Kaggle leaderboard. The rebuild keeps 10,000 recipes aside as its stand-in.

What is published

Aggregates and model weights in web/public/data/: the sampling funnel, class counts, minute histograms, every experiment's scores, and the parameters of the browser models and the stacking judge.
The true class and every model's predicted class for the 10,000 hold-out recipes, as strings of digits, with no recipe text (holdout-predictions.json).
200 hold-out recipes with their full text, listed time and every model's label, for the LLM evaluation (llm-eval-set.json). The text is public domain.
Nothing about Food.com users: the crawl's author ids, reviews and ratings are not read.

Known issues

Listed times are noisy, as above. A recipe that says "refrigerate overnight" in its method can list 20 minutes.
Slow recipes are only 5% of the data, so every result about them rests on 512 hold-out recipes, and on 10 in the LLM evaluation set.
The crawl is from 2020 and the recipes are mostly American. Units, ingredients and phrasing outside that context are under-represented.

Evaluation card

LLM against the 2021 models

Question

Can a general-purpose LLM, given only a recipe's name, ingredients and method, sort it into quick, medium or slow better than the models the 2021 team trained on 40,000 labelled recipes? And how sure can a visitor be of the answer from one run?

Who is compared

The LLM the visitor chooses: Claude Haiku 4.5 by default (temperature 0), Claude Sonnet 5.5 at low effort, or any OpenAI chat model that supports JSON-schema output (reasoning models run at low reasoning effort, the rest at temperature 0). It is zero-shot. The system prompt defines the three classes and says the recipes are data, not instructions.
The rebuilt 2021 models: the Random Forest on the "steps" bag of words, the RBF SVM, the stacked Naive Bayes, Logistic Regression and Gaussian Naive Bayes on Doc2Vec features.
The browser models: the TF-IDF Logistic Regression and Naive Bayes behind /predict.
The rule baseline: add up the times written in the method, add 10 minutes for preparation, and apply the class thresholds. It needs no training and no AI.

Every comparator's label for each recipe is fixed before the LLM runs. The 2021 models' labels are their hold-out predictions from scripts/holdout_eval.py, and the rule runs in the browser. When the rule baseline stands in for the LLM, it is not compared with itself.

Primary comparison

One comparison was named before any run: the LLM against the 2021 Random Forest, the most accurate model the 2021 team trained, on accuracy, with McNemar's exact test and the paired bootstrap difference. It is the headline on /llm-eval and is marked as primary in every export. Every other comparator is exploratory, and its McNemar p-value is shown both as it is and Holm-adjusted across the exploratory family (DR-006).

Inputs

200 recipes from the 10,000-recipe hold-out set, which no model was trained on. They are a proportional stratified sample drawn with seed 30027 (89 quick, 101 medium and 10 slow), ordered so that the first 30, 60 or 100 keep the same mix. The first 60, the default, hold 27 quick, 30 medium and 3 slow recipes.

On all 200, the Random Forest gets 149 right (74.5%), the TF-IDF Logistic Regression 152 (76.0%) and the rule baseline 136 (68.0%).

Scoring rules

These were fixed before any paid LLM run. Rule 3, the position-keyed schema and the primary comparison above were added by DR-006, still before any paid run.

The class the LLM gives is what is scored. Its minutes estimate and stated confidence are analysed separately.
A reply that fails validation, or a refusal, counts as no answer for every recipe in the call. A no-answer is wrong and counts as a miss for the recipe's true class.
A reply cut off at the site's output-token limit is not the model's answer. The call is retried once with twice the limit, and if the reply is cut off again its recipes are left out for every model and counted separately.
A call that fails (no network, or rate-limited after two retries) is left out for every model, and the page reports how many recipes were left out.
Errors that would repeat on every call (an invalid key, an unknown model) stop the run.

The reply schema has one required answer per recipe in the call, keyed by its position, and no other keys, so the schema itself rules out a skipped or repeated recipe.

Statistics

Accuracy and per-class recall: 95% Wilson intervals.
Macro F1: 95% percentile bootstrap over recipes, stratified by true class (every resample keeps the run's class counts), 5,000 resamples, seed 30027.
LLM against each comparator: McNemar's exact test on the recipes where exactly one of the two is right, and paired bootstrap intervals for the difference in accuracy and in macro F1, from the same stratified resamples.
Multiple comparisons: the primary comparison is tested on its own at the 5% level. The exploratory p-values are Holm-adjusted as one family (seven comparisons for an LLM, six for the rule baseline).
Stated confidence: expected calibration error over 10 bins and the Brier score of confidence against correctness, marked exploratory.
Minutes estimate: the share of answers whose estimate falls in the chosen class, the median factor between the estimate and the listed time, and how often the class read off the estimate alone is right.

How much a run can show

At 60 recipes a 95% interval on accuracy is about plus or minus 11 points wide. Two models that differ by 5 points will usually not separate, and McNemar's test only uses the recipes where they disagree. With 3 slow recipes in the default run, slow recall is close to unmeasured. Running all 200 narrows accuracy intervals to about plus or minus 6 points, and still leaves 10 slow recipes. The results panel says next to the primary result that a p-value of 0.05 or more means the run cannot tell the models apart, not that they are equally accurate, and it flags contamination there too. The JSON export carries the same caveats.

Known risks

Contamination. Food.com recipes are public and may be in the LLM's training data, perhaps with their listed times. A strong score is evidence about this public data, not about recipes the model has never seen.
Label noise. The labels come from times that authors entered, and some leave out overnight steps (DR-002). An LLM that reads "chill overnight" and answers slow can be marked wrong.
Prompt and batch sensitivity. The result depends on the wording of the prompt and on how many recipes share a call. Both are fixed and shown on the page, and the batch size is recorded with every run.
Run-to-run variation. Even at temperature 0, providers do not promise identical outputs, and Sonnet 5.5 runs with the provider's default sampling.

What is not claimed

No LLM result is published in this repository. A run needs a key and spends the visitor's money, so the page reports only runs made in the visitor's own browser, and the audit log holds the evidence for each one. See DR-004 and DR-006 for the design decisions.

Run it

Decision records

Why it is built this way

Each record states the decision first, then the options weighed, why, what happened (weak numbers included) and what I would change. A past record is never edited, and a later one supersedes it.

Transparency

AI use statement

This statement covers the generative AI features on this site. It is informed by the Australian Government's Policy for the responsible use of AI in government (Digital Transformation Agency), the transparency principles of the EU AI Act and the NIST AI Risk Management Framework. It makes no claim of compliance with any of them.

What AI does here

There are two optional features, and both need the visitor's own API key.

LLM evaluation (/llm-eval). An LLM sorts held-out recipes into quick, medium or slow, and the site scores its answers against the rebuilt 2021 models on the same recipes.
Second opinion (/predict). An LLM reads the recipe the visitor typed and gives a class, a minutes estimate, a stated confidence and the phrase it relied on, next to the browser model's prediction.

The browser predictor, the rerun of the report, the evaluation page and the stacking explainer all work without a key and without AI.

What AI never does here

It never runs unless the visitor adds a key and starts a call.
It never changes anything on the site by itself. An LLM answer is shown, labelled and logged, and a person decides what to make of it.
It never sees a recipe's listed time or label in the evaluation.
This site's server never receives the API key, because there is no AI server code. The key goes from the visitor's browser straight to the provider they chose.
It is never used to grade itself. The evaluation scores answers against known labels, not against another model's opinion.

What data is sent to the provider

For the evaluation, the system prompt and the name, ingredients and method of each recipe in the call, all from the public-domain evaluation set. For a second opinion, the system prompt and whatever the visitor typed into the recipe form, cut to 4,000 characters per field. No listed times, labels, cookies or referrer details are sent. The provider's own terms govern what it does with the request.

Labelling

Every AI answer on the site carries an "AI-generated" label with the model's name. The rule baseline that stands in for an LLM when there is no key carries a "Rule baseline, not AI" label instead, so the two cannot be confused. Its runs are logged under their own feature, "Rule baseline dry run (not AI)", and /ai-log counts them apart from AI calls.

Human in the loop and the audit log

Every call is written to an audit log in the visitor's browser (IndexedDB), viewable at /ai-log and exportable as JSON or CSV. Each entry records the time, the feature, the provider and model, the full prompt and the JSON schema, the generation settings, the raw and parsed reply, whether it passed validation, the stop reason, the latency, the token counts, the provider's request id and the human decision. The key is never stored in it.

The visitor records the decision. A whole evaluation run can be accepted or rejected, and a second opinion can be accepted, edited (with the class the visitor thinks is right) or rejected. Evaluation answers cannot be edited, because editing them would change the measurement. A free-text note can go with any decision.

Risks and how they are handled

Key exposure. The key lives in sessionStorage unless the visitor asks for it to be remembered, a Content-Security-Policy limits where the page can send data, and the site loads no third-party scripts. Browser extensions can still read a page, so the site advises a key with a spending limit.
Prompt injection. Recipe text is data. The system prompt says so, the reply is held to a JSON schema and validated, and an answer cannot trigger any action.
Contamination. Food.com recipes are public and may be in an LLM's training data. The results panel on /llm-eval flags this next to the primary result, and the JSON export of a run and the evaluation card say it too.
Cost. The page estimates tokens and cost before a run, shows the tokens used after it, and stops a run on errors that would repeat on every call.

How this site was built

The 2026 revival was written by Rin with help from an AI coding assistant. The 2021 report and its numbers are the team's own work and are kept unchanged in coursework/.

Open the AI audit log