Test an LLM against the 2021 models with the visitor's own key, from the browser, with a local audit log

Context

The obvious question about a 2021 text classifier is whether a general-purpose language model now does the job better, and at what cost. A recipe's method often states its own times ("chill overnight", "bake for 40 minutes"), which an LLM can read and the 2021 models could only count as words. The site is static and has no budget for AI calls. A shared key held by the site would cost money on every visit and would need secret management and abuse protection.

Decision

• The visitor picks Anthropic (default Claude Haiku 4.5 at temperature 0, with Claude Sonnet 5.5 at low effort as the option) or OpenAI (the model id is free text) and pastes their own key. The key is kept in sessionStorage, moves to localStorage only if they turn on "remember on this device", and "Forget keys" clears both.
• Requests go from the browser straight to the provider, with Anthropic's anthropic-dangerous-direct-browser-access header. The site has no AI server code and never receives the key. Every page is served with a Content-Security-Policy whose connect-src lists only this site and the two provider APIs.
• The evaluation set is 200 hold-out recipes, a proportional stratified sample drawn with seed 30027 and ordered so that every prefix keeps the hold-out class mix. The visitor runs the first 30, 60, 100 or 200, in calls of 1, 5 or 10 recipes. The model sees the name, ingredients and method, never the listed time or the label.
• The reply is held to a JSON schema whose ids are exactly that call's recipes, and is checked again with zod. Each answer gives a class, a minutes estimate, a stated confidence and the cue it relied on.
• Scoring rules were fixed before any run. A reply that fails validation, or a refusal, counts as no answer (wrong) for every recipe in the call. A failed call is left out for every model and reported. Errors that would repeat on every call stop the run.
• Every model is scored on the same recipes: accuracy and recall with Wilson intervals, macro F1 and the paired differences with a seeded percentile bootstrap, and McNemar's exact test against each comparator.
• Every call is appended to an audit log in IndexedDB with the prompt, schema and settings, the raw and parsed reply, the validation result, the stop reason, the latency, the tokens, the provider's request id and the human decision. The log is scrubbed for the key before it is stored, and /ai-log shows it with JSON and CSV export. A run can be accepted or rejected as a whole. Labels in an evaluation are judged, not edited, because editing them would change the measurement.
• A second, smaller feature on /predict asks the LLM for a second opinion on the recipe the visitor typed. There the visitor can accept, edit (record the class they think is right) or reject the answer.
• Without a key, the rule baseline (add up the times written in the method, plus 10 minutes) runs through the same pipeline, statistics and audit log. It is labelled "Rule baseline, not AI" wherever it appears, and its log entries say that no AI was called.

Options considered

1. A server route with my key. Every visit would cost me money and need rate limits.
2. A server route with the visitor's key. The key would pass through a server I run.
3. Browser-direct calls with the visitor's key (chosen).
4. Publish results from one paid run. Cheap to show, but visitors could not reproduce or vary it, and one run says nothing about run-to-run variation.
5. No AI feature. Simplest, but it leaves out the LLM evaluation and governance work the upgrade is meant to show.

Why

Option 3 costs nothing to host, keeps the key between the visitor and their provider, and keeps the audit record on the visitor's device, where they can inspect and export it. Scoring against known labels, next to models whose labels are fixed in advance, means there is no judge model and no rubric to argue about. Fixing the scoring rules before any run stops them being tuned to make one model look good. The rule baseline lets anyone see the harness, the statistics and the audit log work without spending anything, and labelling it plainly keeps it from being mistaken for evidence about a model.

What happened

• The adapters, validation, retries, scoring rules, audit log and export are covered by unit tests with mocked fetch responses. No paid run is recorded in this repository: a run needs a key and spends money, so the page reports only runs made in the visitor's own browser.
• On all 200 recipes the Random Forest gets 149 right (74.5%) and the TF-IDF Logistic Regression behind /predict gets 152 (76.0%). The rule baseline gets 136 (68.0%) with no training at all. That is ten more than the 2021 Gaussian Naive Bayes (126) and five fewer than its Logistic Regression (141), so a fair share of the task is reading the times the author wrote down.
• With 60 recipes a 95% interval on accuracy is about plus or minus 11 points, and only 3 of the 60 are slow. The page says so next to every result, so a non-significant McNemar test is not read as "no difference".
• Food.com pages are public, so an LLM may have seen these recipes, perhaps with their listed times, during training. The evaluation card and the page flag this. A high LLM score here is evidence about this public data, not about unseen recipes.
• A static Content-Security-Policy still allows inline scripts, because the pages are prerendered without per-request nonces. It limits where a script could send data, not what it could read, and browser extensions are not bound by it, so the advice to use a key with a spending limit stands.

What I'd change

• Run each configuration several times and report run-to-run variation as well as the mean.
• Add a set of recipes written after the model's training cut-off, to separate reading the recipe from remembering it.
• Try a few-shot prompt and report the difference as a paired comparison on the same recipes.