Skip to content
Recipe Timer Lab

Decision record · DR-001

Rebuild the lost 2021 pipeline from the report, on public data, and label every number with its source

Status
Accepted
Date
2026-10
Applies to
scripts/prepare_data.py, scripts/features.py, scripts/report_models.py, web/src/lib/report.ts, web/src/lib/findings.ts, /models

Decision in one line

Rerun every experiment the 2021 report describes, with the same estimators and settings, on a public-domain crawl of the same website, and show the report's numbers and the rebuild's numbers side by side without letting one stand in for the other.

Context

The only thing that survives from COMP30027 Project 2 is the six-page report. The notebooks, the Kaggle prediction files and the staff data bundle are gone, and the staff bundle was never ours to publish. The report is detailed enough to follow. It names the features, the estimators, the settings it tuned, the cross-validation it used and the tables it printed. A revival can either show the report's numbers as they are, or rebuild the work and compare.

Decision

  • Rebuild the pipeline as four reproducible uv scripts that follow the report step by step: data preparation, the six Doc2Vec files and the "steps" bag of words, every experiment in the report, and the two small browser models.
  • Use the public-domain (CC0) Food.com crawl, verified by SHA-256, sampled to the report's exact class counts (DR-002).
  • Keep the report's own choices for the final models even where the rebuilt sweep prefers something else. The final SVM keeps C = 1 and gamma = 0.01 although C = 10 scores slightly higher on the rebuilt data.
  • Transcribe the report's numbers into web/src/lib/report.ts and never recompute them. Every chart and table on /models shows which number is the report's and which is the rebuild's.
  • Score each of the report's qualitative findings against the rebuild in code (web/src/lib/findings.ts), and pin every verdict in a test, so the copy cannot claim more than the data shows.

Options considered

  1. Show the report only. Honest, but nothing a visitor could check or run.
  2. Rebuild and quietly replace the report's numbers. Tidy, and wrong: the rebuild uses different recipes and regenerated features, so its numbers are not the 2021 results.
  3. Rebuild, compare side by side, and label everything (chosen).
  4. Rebuild with modern models instead. Better scores, but it would answer a different question from the one the report asked.

Why

The point of the revival is to show that the 2021 reasoning holds up, or where it does not. That needs the same models on comparable data, with the old and new numbers kept apart. Keeping the report's settings for the final models makes the five-fold results a like-for-like rerun of what we actually submitted, and the scorecard makes the comparison falsifiable instead of a matter of tone.

What happened

  • Ten of the report's eleven qualitative findings carry over, including the model ranking (Random Forest, then the SVM, the stack, Logistic Regression and Gaussian Naive Bayes).
  • The exception is the C curve. In the report accuracy peaks at C = 1. On the rebuilt data it keeps rising, and C = 10 beats C = 1 by 0.0029 on the tuning split.
  • Absolute scores are lower than in 2021: by about 2 points for the SVM, the forest and the stack, 4 for Logistic Regression and 8 for Gaussian Naive Bayes. The site says so next to every comparison.
  • Gensim trains Doc2Vec with six threads, so rebuilding the vectors moves stage 3 scores in the third decimal. Rerunning stage 3 on the cached vectors reproduces the committed JSON exactly.

What I'd change

  • Train Doc2Vec with one worker thread for bit-for-bit repeatable vectors, and accept the longer run.
  • Report the tuning sweeps with cross-validation instead of a single tuning split. The report did not name its protocol, and a single split is the cheapest reading of it.
  • Ask the subject coordinator whether the staff label thresholds can be shared, which would remove the largest source of difference (DR-002).