Name one primary comparison for the LLM test, adjust the others for multiple testing, and stop scoring the site's own limits as the model's errors

Context

Two reviews of the upgrade found gaps in the harness DR-004 describes. Its scoring rules were fixed before any run, but four things were not.

• No main comparison. The page compared the LLM with eight models, each with an unadjusted McNemar test, and a drop-down let the visitor choose which comparison to headline after the results were on screen. With eight tests at the 5% level and no real differences, the chance that at least one looks significant is about one in three if the tests are independent.
• A bootstrap that ignored the design. The evaluation set has fixed class counts, but the intervals resampled recipes without regard to class. With 3 slow recipes in the default run of 60, about 4% of resamples had none, which forces slow F1 to zero. At 30 recipes it was 12%.
• Cut-off replies scored as wrong. A reply that hit the output-token limit failed validation and counted as a wrong answer for every recipe in the call, although the limit is this site's setting and not the model's choice. OpenAI reasoning models also ran at the provider's default reasoning effort, so their hidden reasoning used more of that limit than Claude Sonnet's at low effort.
• A schema that could not say "each recipe once". The reply schema was an array whose items carried an id from a list. JSON Schema cannot require each id exactly once, so a reply could pass the schema, skip a recipe and then fail the zod check, losing the whole call. The list of ids also changed on every call, so the provider had to compile a new schema each time.

The reviews also found two factual slips in earlier records, which stay as written. DR-003 says Logistic Regression and the SVM "agree on 82% of hold-out recipes (Yule's Q 0.90)". The 82% is agreement on labels (82.2%), but Q is computed from the table of right and wrong answers, on which they agree for 83.1% of recipes. DR-005 says the repeated cross-validation took about 1.5 hours. The whole run takes about 45 to 60 minutes on 10 cores.

Decision

• The primary comparison is the LLM against the Random Forest, the most accurate model the 2021 team trained: accuracy with its paired bootstrap difference and McNemar's exact test. It is named in the code, in the evaluation card and in every export, and the headline on /llm-eval shows only this comparison.
• Every other comparator is exploratory. Each stays in the results table, labelled "exploratory", with its McNemar p-value and the Holm-adjusted p-value for that family: seven comparisons for an LLM, and six when the rule baseline stands in, because it is no longer compared with itself. The per-recipe table can still show any comparator's labels, but the headline no longer changes.
• The bootstrap intervals in the harness resample within each true class, so every resample keeps the run's class counts.
• A reply that stops at the output-token limit (stop reason max_tokens or length) is retried once with twice the limit. If it is cut off again, its recipes are left out for every model, like a failed call, and counted separately in the results and the export. OpenAI reasoning models are sent reasoning_effort: "low", the effort Claude Sonnet runs at.
• The reply schema is an object with one required key per recipe in the call, its position from "1" to the batch size, and no other keys. The code maps positions back to recipes. The schema depends only on the batch size, so the provider can reuse it.
• The results panel says, next to the primary result, that a p-value of 0.05 or more means the run cannot tell the models apart, not that they are equally accurate, and that Food.com recipes may be in the LLM's training data. The JSON export carries the same caveats.
• Rule-baseline runs are written to the audit log under their own feature, "Rule baseline dry run (not AI)", and /ai-log counts them apart from AI calls.

Options considered

1. Keep the drop-down and adjust all eight p-values. This controls false positives, but the headline could still be chosen after seeing the results.
2. Make the TF-IDF Logistic Regression the primary comparator. It is the most accurate model on the hold-out set, so it is the tougher bar, but it was built for this site in 2026, and the page asks about the 2021 models.
3. The Random Forest as the primary comparator, the rest exploratory with Holm's adjustment (chosen).
4. For cut-off replies, raise the limit for every call. This would avoid most retries, but it raises the worst-case cost of every run to deal with a rare event.

Why

A single comparison named in advance is what makes "the scoring rules were fixed before any run" mean something: the result that gets reported cannot be picked once the numbers are known. The Random Forest is the 2021 team's best model, the one the page's question is about, and it is fixed before any paid run, because none has been made. Holm's adjustment controls the chance of any false positive across the family, like Bonferroni, and is never less powerful. Resampling within class matches how the evaluation set was drawn, so the intervals describe the models rather than the luck of the resample. Retrying once with more room removes the usual cause of a cut-off reply, and leaving out the rare reply that is cut off twice keeps every model scored on the same recipes. A schema that names every position makes "one answer per recipe" a property of the reply rather than a check that throws away a whole call.

What happened

• On the first 30 recipes, with the rule baseline standing in for the LLM, its comparison with Gaussian Naive Bayes gives p = 0.02 unadjusted and p = 0.13 after Holm's adjustment. On all 200, its comparison with the TF-IDF Logistic Regression gives p = 0.04 unadjusted and p = 0.26 after adjustment. Without the adjustment, either would have read as a finding.
• On the default 60 recipes the rule baseline's primary result is 48 right against the Random Forest's 43, a difference of +8.3 pts (−5.0 to +21.7 pts) with McNemar p = 0.38. The run cannot tell the two apart, and the panel now says so.
• At 30 recipes the rule baseline's macro F1 interval is 0.576 to 0.975 when resampling ignores class and 0.670 to 0.976 when it resamples within class. The lower bound moves by almost a tenth because some plain resamples have no slow recipe at all.
• Every call with five recipes now sends the same schema. The tests check Holm's adjustment against statsmodels and R, that a stratified resample never loses a class, that a cut-off reply is retried and then left out, and that the schema rejects a missing or extra recipe.
• No paid LLM run has been made, so the primary comparison has not yet been run with an LLM.

What I'd change

• Size the evaluation set from a power calculation for the primary comparison, instead of offering 30 to 200 recipes. At 60 recipes the interval on the difference above is almost 27 points wide, so only a large difference can show.
• Add slow recall as a second named comparison once the set holds enough slow recipes to measure it. With 10 in all 200 it cannot carry a test.
• Record each run's retry count and output limit on the results panel as well as in the audit log.