Paper deep dive
FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth
Josef Chen, Erim Hayretci
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/24/2026, 4:47:48 AM
Summary
The paper introduces FlavourBench, an automated benchmark for evaluating frontier language models on culinary reasoning tasks. It utilizes a versioned culinary system called Epicure to provide dense, executable ground truth by scoring all possible three-ingredient portfolios from a set of eight. The study evaluates 27 frontier models on a common core of 534 tasks, ensuring statistical rigor through simultaneous confidence intervals and multiplicity-controlled paired tests. Grok 4.6 achieved the highest score.
Entities (16)
Relation Signals (10)
Grok-4.6 â achievedhighestscorein â FlavourBench
confidence 98% ¡ Grok 4.6 has the largest point estimate at 65.1
FlavourBench â evaluates â 27 frontier endpoints
confidence 95% ¡ We evaluate 27 frontier endpoints on an identical 534-task core
FlavourBench â uses â Epicure
confidence 95% ¡ FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth... Epicure represents 1,790 ingredients
Meta â developed â llama-4-maverick
confidence 90% ¡ The panel spans... Meta... Llama 4 Maverick
Anthropic â developed â claude-opus-5
confidence 90% ¡ The panel spans... Anthropic... Claude Opus 5
Google â developed â Gemini 3.1 Pro
confidence 90% ¡ The panel spans... Google... Gemini 3.1 Pro
OpenAI â developed â GPT-5.6-Sol-Pro
confidence 90% ¡ The panel spans OpenAI... It includes GPT-5.6 Sol Sol
xAI â developed â Grok-4.6
confidence 90% ¡ The panel spans... xAI... Grok 4.6
FlavourBench â usesstatisticalmethod â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and family (14,418 model-task cells total), eliminating differential missingness from the leaderboard. The FlavourBench Score is the equal-family mean of the frozen task scores. We use 50,000 anchor-cluster bootstrap replicates for simultaneous 95% score bands and 100,000 sign-flip draws for all 351 paired model contrasts, with Holm control. The two independently compiled panels correlate at r = 0.89 (rank rho = 0.80). Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0-69.2); 101 of 351 model pairs are resolved. The release includes the prompts, all portfolio score maps, raw responses, exact routes, content hashes, and an offline verifier that reconstructs every result.
Tags
Links
- Source: https://arxiv.org/abs/2608.20574v1
- Canonical: https://arxiv.org/abs/2608.20574v1
Trouble viewing inline? Open PDF directly â
Full Text
36,028 characters extracted from source content.
Expand or collapse full text
FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth Josef Chen Independent Researcher Erim Hayretci Imperial College London August 2026 Abstract Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and family (14418 modelâtask cells total), eliminating differential missingness from the leaderboard. The FlavourBench Score is the equal-family mean of the frozen task scores. We use 50000 anchor-cluster bootstrap replicates for simultaneous 95% score bands and 100000 sign-flip draws for all 351 paired model contrasts, with Holm control. The two independently compiled panels correlate atr= 0.89 (rankĎ= 0.80). Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0â69.2); 101 of 351 model pairs are resolved. The release includes the prompts, all portfolio score maps, raw responses, exact routes, content hashes, and an offline verifier that reconstructs every result. 1 Introduction Benchmarks are strongest when the evaluator can be executed. Code can be tested, database states can be inspected, and games can be scored. Culinary reasoning is usually evaluated with a model judge, a handful of human preferences, or factual questions whose answers do not capture the quality of a complete decision. That leaves two problems: the judge is entangled with the systems being tested, and exact match throws away useful differences between plausible answers. FlavourBench turns a culinary runtime into a bench- mark environment. Epicure represents 1,790 ingredients in a 300-dimensional space and exposes deterministic op- erations for substitution, pairing, dietary feasibility, and regional composition [12,13]. A task compiler converts these operations into three-of-eight selection problems. It enumerates every candidate portfolio and freezes a contin- uous 0â100 score map before model execution. An answer is therefore scored by table lookup, without a model judge and without post-hoc interpretation. This setup separates the reference system from the evaluated systems. Epicure is not a leaderboard entry and is not â100% accurate.â It defines the task-specific reward surface. A model receives 100 on a task only if it selects that taskâs highest-scoring portfolio; other valid portfolios receive partial credit. The benchmark measures agreement with a published executable culinary environment, not universal human taste. We use that environment to evaluate 27 current end- points. Every model faces the same 534 tasks and 56 frozen answer scores per task. The complete common core supports direct paired comparisons on 534 shared observations per model pair instead of inferring a ranking from sparse votes or comparing endpoints on different task subsets. The paper makes three contributions: 1.an executable culinary benchmark with 29904 model- independent portfolio scores and a single interpretable 0â100 metric; 2.a 27-model, 534-task complete-core evaluation with simultaneous uncertainty and paired, multiplicity- controlled tests; and 3.a content-addressed release in which every public num- ber traces to frozen tasks, exact routes, raw responses, and a dependency-light offline scorer. 2Executable culinary ground truth 2.1 A reference environment, not a model judge The benchmark binds an exact Epicure release, evidence bundle, application build, ingredient count, and embed- ding dimensionality by hash. Changing any of these inputs creates a new benchmark task-set identity. This is analogous to fixing a simulator or a code-test suite: the reference is explicit and reproducible, but its exter- nal validity remains a scientific question rather than an assumption hidden inside a judge prompt. For each taskt, letA t contain all 56 three-item subsets of eight candidates. Epicure assigns a raw utilityu t (a) to every validaâA t . Invalid portfolios under a dietary or category constraint receive zero. Valid utilities are 1 arXiv:2608.20574v1 [cs.AI] 20 Aug 2026 Epicure runtime 1,790 ingredients versioned 300-D space Task compiler 534 shared prompts 3 decision families Exhaustive oracle 8 3 = 56 portfolios dense 0â100 map Frontier panel 27 exact endpoints no fallback Set parser three distinct labels exact common core Statistical release score + simul- taneous CI all 351 paired tests Fixed before score inspection: task el- igibility, all score maps, parser, task- selection hash, inference seeds, and the exact route for each model block Figure 1: FlavourBench compiles a versioned culinary runtime into dense selection tasks. The evaluated models never judge one another and do not access Epicure. All score maps and the per-panel contracts are fixed before collection. The complete-core task set is selected from response validity alone, before inspecting any model selection or score. normalized within the task: s t (a) = 100 u t (a)â min bâA t u t (b) max bâA t u t (b)â min bâA t u t (b) . (1) Every task has a unique 100-point optimum. The ex- act chance baseline is not a generic percentage; it is |A t | â1 P a s t (a) for that frozen task. 2.2Three complete-core decision families The ranked core contains 534 tasks: 89 from each family in each of two independently compiled panels, spanning 534 unique anchor ingredients. Tasks sharing an anchor move together as one cluster in inference. Candidate sets are selected across validation strata, then ordered and labeled by a task-derived hash. No evaluated model writes an item, distractor, or score. Regional-composition tasks are released as a supplemental track but are not in the 27-model leaderboard because they do not support an equally complete common core. The score maps have substantial resolution rather than one correct key plus 55 equivalent mistakes. Table 2 reports the exact random-portfolio baseline, separation of the best and second-best portfolios, and the number of distinct score values. Constraint tasks are intentionally sparse because infeasible portfolios score zero; the other families provide much finer graded rewards. 2.3 Prompt and parser contract Prompts show the decision context and eight labeled candidates. The final line must begin with the FINAL_SELECTIONmarker and contain three distinct AâH labels separated by commas. No prompt supplies a con- crete valid triplet. The parser takes the final exact marker line, uppercases the labels, sorts them as an unordered set, and performs one lookup in the frozen score map. Ex- planatory text does not affect the result. Missing markers, duplicate labels, provider failures, and abnormal finishes remain in the raw response ledger but cannot enter the common core. The model-only condition is deliberate. Epicure sup- plies the ground truth but is not exposed as a tool during ranking. Thus the score measures whether a model can make the culinary decision, not whether it can copy a runtime return value after being told which operation to call. 3 Evaluation design 3.1 One primary score For modelm, panelp, familyf, and taskt, letY mt be the frozen portfolio score and letT pf be the score-blindly selected set of 89 tasks completed by every model. The FlavourBench Score is F m = 1 2¡ 3 2 X p=1 3 X f=1 1 |T pf | X tâT pf Y mt .(2) It ranges from 0 to 100 and equals the ordinary mean over all 534 cells because the six panelâfamily strata have equal size. Every ranked endpoint contributes exactly 534 valid responses. Failures are neither assigned zero nor silently dropped model by model: they determine whether a task can enter the shared core, using response status and parser validity only. The task-selection hash is fixed before loading any selected portfolio or score. 3.2 Frontier endpoint panel The panel spans OpenAI, Anthropic, Google, xAI, Meta, Kimi, Qwen, Z.ai, DeepSeek, ByteDance, Thinking Ma- chines, MiniMax, NVIDIA, Mistral, Tencent, and Cohere. It includes GPT-5.6 Sol, Terra, and Luna; Claude Opus, Sonnet, and Fable 5; Gemini 3.1 Pro and 3.6 Flash; Grok 4.6; Llama 4 Maverick; Muse Spark 1.2 and Muse Glimmer 30B; Kimi K3; Qwen 3.8 Max and Qwen3.8 2.4T A95B; GLM 5.2 and GLM 5.3; DeepSeek V4 Pro 0813 and V4 Flash; Seed 2.1 Turbo; Inkling; MiniMax M3; Nemotron 3.5 Lightning; Mistral Large 3; Tencent HY 3; Cohere Command A; and Cohere Command R+ (08-2024). Z.ai supplied written permission for one finite GLM 5.3 bench- mark run, conditioned on the adapter not becoming a permanent running function. We bind that permission and its hash in the release, require the returned identity to equal glm-5.3, and expose no standing endpoint. 1 1 https://docs.z.ai/guides/llm/glm-5.3 2 FamilyDecisionEpicure utilityHard constraintTasks Substitutionchoose a three-item replacement port- folio for an anchor ingredient 0.8 anchor similarity +0.2 portfolio coherence none178 Pairingchoose three ingredients to accompany an anchor 0.65 anchor affinity +0.35 portfolio coherence none178 Constraints choose a feasible portfolio under diet and processing limits 0.7 anchor similarity +0.3 coherence diet and maximum NOVA level178 Table 1: The three complete-core task families share one response and scoring interface but probe different culinary decisions. Each task contains eight candidates and 56 frozen scores. FamilyExact chance Top gap Distinct scores Substitution45.25.656 Pairing45.05.656 Constraints3.437.34 Table 2: Frozen task-map diagnostics for 178 sched- uled tasks per family, computed before model execution. Chance is the uniformly random portfolio mean; top gap is the median bestâsecond-best margin. Routes were chosen before each scored block and auto- matic fallback is disabled. Most endpoints use an exact OpenRouter route; Qwen 3.8 Max uses Alibaba, Fable 5 uses Google Vertex, GLM 5.3 uses the finite Z.ai route above, and selected DeepSeek blocks use an exact GMI- Cloud or DeepInfra route. When transport validation required a replacement, the complete model block was re- run under one new route; successful cells from superseded blocks are not pooled. The manifest binds the requested model, returned identity contract, endpoint tag, provider, supported parameters, and execution contract for both panels. The exact per-model routes appear in Section A. For endpoints that advertise a reasoning-effort control, hidden reasoning is excluded and effort is fixed to the low- est transport-stable setting recorded in the exact analysis plan. Endpoints without that control use their provider- fixed behavior. This prevents hidden deliberation from silently consuming the answer budget while leaving the scored response format identical across routes. 3.3Score-blind complete-core construc- tion The two task compilers each produce 160 candidates per family. After collection, the release re-scores response syntax with one versioned parser and constructs a validity matrix. Within every panelâfamily stratum it selects the first 89 eligible task IDs under a fixed SHA-256 ordering. A task is eligible only when all 27 models completed it and produced a parseable three-item set. The selection code does not load the chosen ingredients or their scores until the task IDs are fixed. This yields a balanced common core without favoring any model or post-selecting easy tasks by observed score. 3.4 Uncertainty and multiplicity All inferential choices are bound in the content-addressed analysis plan. â˘Scores. 50000 bootstrap samples are drawn over ingre- dient anchors; tasks that share an anchor across panels always move together. Each resample is reduced to six panelâfamily means and then equally weighted. We re- port pointwise percentile intervals and a simultaneous max-t band across all 27 model scores. ⢠Pairs. Every one of the 351 model pairs uses the same 534 tasks. Two-sided anchor-cluster sign-flip tests use 100000 Monte Carlo draws and Holm correction at familywiseÎą=.05. We also report paired mean differences, bootstrap intervals, and Cohenâs d z . â˘Chance. Each model is compared with the exact taskwise mean over all 56 portfolios. The 27 paired sign-flip tests form a separate Holm family. ⢠Ranks. Point ranks are accompanied by bootstrap rank intervals and contiguous statistical groups. Mod- els that are not significantly separated are not pre- sented as a resolved total order. The design has 534 independent anchor clusters and 534 paired cells per contrast. Statistical claims are read directly from simultaneous intervals and Holm-adjusted tests; point-rank differences without corrected support are shown as unresolved. 4 Results 4.1 A statistically resolved FlavourBench leaderboard All 27 endpoints receive a score on the same 534 tasks. Grok 4.6 has the largest point estimate, 65.1, with simulta- neous 95% interval [61.0, 69.2]. Across all 351 prespecified paired contrasts, 101 remain significant after Holm cor- rection. A point rank is therefore a compact summary, not a claim that every adjacent model is distinguishable. Figure 2 shows the score scale, simultaneous intervals, and rank groups; Table 3 gives exact values. The ranked matrix is rectangular: every row contains 534 valid, parser-confirmed responses, so no endpoint is advantaged by answering a different subset. Provider failures and filtered calls remain available in the full exe- cution ledger, but none appears as a zero-valued culinary 3 decision and no incomplete row enters this leaderboard. 4.2Does the ordering replicate on a sec- ond task panel? The two panel-specific scores use the same frozen metric and 89 tasks per family, but disjoint task IDs. Their model-level Pearson correlation is 0.89 and their rank correlation is 0.80. These are descriptive stability diagnos- tics, not a model-selection rule; the primary leaderboard and its uncertainty use both panels and cluster shared ingredient anchors. Figure 3 exposes every model rather than reducing replication to a single coefficient. 4.3 Which pairwise conclusions survive correction? The score is continuous, and each model pair is compared on the same 534 tasks rather than on a small number of arena votes. Figure 4 visualizes the multiplicity-controlled results. Grey cells are scientifically useful: they mark pairs for which the shared task evidence does not support a directional claim at familywiseÎą=.05. This is different from declaring the two models equal. 4.4Aggregate scores conceal different culinary profiles Family-level results in Figure 5 show that similar aggre- gate scores can arise from different strengths. Constraint tasks penalize infeasible portfolios directly, while pairing and substitution reward graded semantic structure. The three columns are related but not interchangeable. 4.5Rank uncertainty is visible, not hid- den Bootstrap rank intervals expose how often small score differences change order under plausible resampling of ingredient anchors. Figure 6 separates stable regions of the table from visually tempting but statistically unsupported micro-ranks. This is stricter than sorting point estimates and printing an ordinal list without uncertainty. 4.6 Real prompts and model decisions Table 4 reports one deterministically selected common- core task from each family with responses from the highest- and lowest-point-estimate endpoints. Selection follows a fixed hash rule rather than task score. The public case-study JSON contains the complete prompt, all eight choices, the optimal portfolio, both raw model responses, selected ingredient identities, and exact scores. The ex- amples illustrate why continuous scoring is preferable to exact match: two non-optimal portfolios can differ substantially in culinary utility. 5From benchmark to training sig- nal The exhaustive task map makes FlavourBench useful beyond a static leaderboard. For a task statex t and selected portfolioa, Epicure supplies a deterministic rewardr(x t ,a) =s t (a). This creates three immediate training interfaces: 1.supervised learning from the optimal portfolio and the full ordering of 56 candidates; 2. preference or contrastive learning from any pair (a i ,a j ) with known reward difference; 3.offline or online policy optimization using dense bounded rewards and held-out anchors. The present experiment evaluates models rather than training them. The release supplies the environment, re- ward function, and anchor identifiers required for a clean next experiment: train on development reward maps and measure transfer on disjoint held-out anchors. Direct optimization on the public test map would measure mem- orization rather than culinary learning. 6 Reproducibility The release is organized around content-addressed inputs and append-only outputs. It includes: â˘both exact content-addressed task sets (534 scheduled tasks and 534 unique anchors), with prompts, candi- dates, all 56 portfolio scores, oracle provenance, task strata, and task-set digests; â˘the 27-model manifest, exact provider route per model, fallback policy, supported parameters, endpoint con- tract, and route digest; â˘all response records used to construct the 14418-cell ranked matrix, including raw text, completion state, usage, latency, parser output, task score, and digest; ⢠all content-addressed provider-attempt events refer- enced by those responses, including normalized and native finish reasons used to distinguish refusals from ordinary completions; ⢠the frozen joint analysis plan, anchor-cluster contract, seeds, 50,000 bootstrap and 100,000 sign-flip counts, all 351 pairwise rows, cross-panel diagnostics, and the final statistical release; and ⢠a verifier and asset builder that reconstruct the leader- board CSV, pairwise CSV, LaTeX tables, and vector figures from local files without provider access. Responses are written once under a model slot and cell identifier. Existing valid cells are skipped on resume; con- flicting artifacts fail closed. Provider fallback is disabled, so the route shown in the manifest is the route measured. The task maps and analysis plan were frozen before the clean primary lineage; transport-calibration responses are retained separately and never enter the leaderboard. 4 303540455055606570 FlavourBench Score (0â100) Command R+ Llama 4 Maverick Mistral Large 3 DeepSeek V4 Flash Command A Nemotron 3.5 Lightning GLM 5.2 Claude Sonnet 5 Inkling Seed 2.1 Turbo Muse Glimmer 30B GLM 5.3 MiniMax M3 Tencent HY 3 Qwen 3.8 Max DeepSeek V4 Pro 0813 Gemini 3.6 Flash Kimi K3 Qwen3.8 A95B Claude Opus 5 GPT-5.6 Luna Pro Claude Fable 5 GPT-5.6 Terra Pro Muse Spark 1.2 GPT-5.6 Sol Pro Gemini 3.1 Pro Grok 4.6 47.9 G3 53.7 G3 55.4 G2 55.4 G2 56.7 G2 57.4 G2 58.5 G2 59.5 G2 59.6 G2 59.7 G2 59.9 G2 60.6 G1 60.9 G1 61.5 G1 61.5 G1 62.0 G1 62.0 G1 62.1 G1 62.1 G1 62.5 G1 62.6 G1 63.4 G1 63.7 G1 63.8 G1 64.2 G1 65.0 G1 65.1 G1 27 frontier models on the identical 534-task core Mean exact chance Figure 2: FlavourBench Score with simultaneous 95% max-tintervals. The letter-free group numbers are inferential tiers: a point rank is shown for navigation, while models inside an unresolved group should not be read as a statistically established ordering. 7 Related work General-purpose evaluation spans broad suites such as HELM [10], difficult knowledge tests such as MMLU-Pro and GPQA [16,14], live contamination- resistant sets [17,7], and human preference systems such as Chatbot Arena [2]. Model judges increase scale but introduce their own bias and calibration problems [20]. FlavourBench instead takes the executable- environment route used by software and agent benchmarks: SWE-bench runs repository tests [8], BFCL evaluates function calls [11], andĎ-bench checks interaction outcomes against database state [19]. Culinary datasets and benchmarks cover ingredient networks, recipe understanding, planning, nutrition, and food knowledge [1,5,15,3,18,6,4,9]. Our focus is nar- rower and complementary: common-task frontier model measurement against a versioned culinary decision envi- ronment, with exhaustive partial-credit maps and paired inference. 8 Limitations FlavourBench Score measures agreement with one pub- lished Epicure release, not universal human taste. Within-task minâmax normalization makes the 0â100 scale comparable while discarding absolute raw-utility magnitudes. The tasks evaluate constrained ingredient selection rather than full recipe generation, sensory ex- ecution, safety advice, or long-horizon kitchen planning. The common-core requirement removes differential miss- 5 Rank ModelFB Score simultaneous 95% CI rank 95% CI group 1 Grok 4.665.1[61.0, 69.2][1, 5]1 2 Gemini 3.1 Pro65.0[60.8, 69.1][1, 6]1 3 GPT-5.6 Sol Pro64.2[60.1, 68.4][1, 8]1 4 Muse Spark 1.263.8[59.6, 67.9][1, 10]1 5 GPT-5.6 Terra Pro63.7[59.5, 67.8][1, 11]1 6 Claude Fable 563.4[59.2, 67.5][2, 13]1 7 GPT-5.6 Luna Pro62.6[58.5, 66.8][4, 14]1 8 Claude Opus 562.5[58.4, 66.6][3, 15]1 9 Qwen3.8 A95B62.1[57.9, 66.2][4, 17]1 10 Kimi K362.1[57.9, 66.2][5, 16]1 11 Gemini 3.6 Flash62.0[57.7, 66.2][4, 17]1 12 DeepSeek V4 Pro 081362.0[57.8, 66.1][5, 17]1 13 Qwen 3.8 Max61.5[57.4, 65.6][7, 18]1 14 Tencent HY 361.5[57.3, 65.7][6, 19]1 15 MiniMax M360.9[56.8, 65.1][8, 20]1 16 GLM 5.360.6[56.4, 64.8][8, 20]1 17 Muse Glimmer 30B59.9[55.9, 63.9][12, 21]2 18 Seed 2.1 Turbo59.7[55.4, 64.0][12, 22]2 19 Inkling59.6[55.4, 63.8][12, 22]2 20 Claude Sonnet 559.5[55.4, 63.7][13, 22]2 21 GLM 5.258.5[54.2, 62.7][16, 23]2 22 Nemotron 3.5 Lightning57.4[53.2, 61.6][19, 25]2 23 Command A56.7[52.5, 60.9][20, 25]2 24 DeepSeek V4 Flash55.4[51.1, 59.7][22, 26]2 25 Mistral Large 355.4[51.4, 59.5][22, 26]2 26 Llama 4 Maverick53.7[49.6, 57.7][24, 26]3 27 Command R+47.9[43.7, 52.0][27, 27]3 Table 3: The complete automated leaderboard. Every score and every pairwise comparison uses the identical 534-task core. Rank intervals come from the anchor-cluster bootstrap; groups summarize the Holm-controlled pairwise graph. FamilyPromptHigher-ranked selectionLower-ranked selection SubstitutionSelect three alternatives to âboursin cheeseâ that best preserve its dairy role, regional context, and mutual portfolio coherence. caciocavallo, fromage blanc, grana padano (100) fromage blanc, quail egg, goose egg (15) PairingSelect the three-ingredient bundle with the strongest learned pairing to âsweetbreadâ and the best internal coherence. parsnip, cipollini onion, porcini mushroom (92) parsnip, pickled onion, peppadew pepper (0) Constraints For a dish centered on âpotatoâ, select three vegetarian complements with NOVA pro- cessing level at most 2; among valid port- folios, maximize learned pairing and coher- ence. parsley, black pepper, thyme (100)cheese, bay leaf, parsley (0) Table 4: Three common-core response examples. Parentheses give the fixed task score. The complete prompts and unabridged responses are released in machine-readable form. ingness from ranking, but it also restricts the estimand to tasks completed by every endpoint; the released sup- plemental tracks retain the rest. Results bind particular model routes and collection dates. The dense reward sur- face supports learning experiments, but this paper does not test whether optimizing Epicure reward transfers to human cooking outcomes. 9 Conclusion FlavourBench makes culinary reasoning measurable without asking one language model to grade another. A versioned runtime enumerates every candidate decision, a 27-model complete panel produces one continuous score, and paired uncertainty distinguishes resolved differences from apparent rank order. The result is an automated leaderboard that can be rerun, audited, and used as a dense training environment. Under this executable culi- nary ground truth and the released 534-task, 534-cluster distribution, every score, interval, contrast, and figure is reconstructable from public artifacts. References [1] Yong-Yeol Ahn, Sebastian E. Ahnert, James P. Bagrow, and Albert-LĂĄszlĂł BarabĂĄsi. Flavor Net- work and the Principles of Food Pairing. Scientific Reports, 1:196, 2011. [2]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anas- tasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pages 8359â 8388. PMLR, 2024. [3]Donghee Choi, Mogan Gim, Donghyeon Park, Mu- jeen Sung, Hyunjae Kim, Jaewoo Kang, and Jihun 6 50556065 Panel 1 FlavourBench Score 50 55 60 65 Panel 2 FlavourBench Score GPT-5.6 Sol Pro GPT-5.6 Terra Pro GPT-5.6 Luna Pro Llama 4 Maverick Claude Opus 5 Claude Sonnet 5 Gemini 3.1 Pro Gemini 3.6 Flash Grok 4.6 Kimi K3 Qwen 3.8 Max GLM 5.2 DeepSeek V4 Pro 0813 DeepSeek V4 Flash MiniMax M3 Nemotron 3.5 Lightning Mistral Large 3 Tencent HY 3 Command A Command R+ Muse Spark 1.2 Muse Glimmer 30B Claude Fable 5 Qwen3.8 A95B Seed 2.1 Turbo Inkling GLM 5.3 Pearson r=0.89 Spearman Ď=0.80 Independent task panels test ranking stability Figure 3: Panel-1 versus panel-2 FlavourBench Score for every endpoint. The diagonal marks exact agreement. Each point aggregates 89 tasks per family; the joint anal- ysis uses 534 unique anchor clusters. Choi. CookingSense: A Culinary Knowledgebase with Multidisciplinary Assertions. In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Eval- uation (LREC-COLING 2024), pages 3983â3996, Torino, Italia, 2024. ELRA and ICCL. [4]Tome Eftimov, Ana Gjorgjevikj, Matej Martinc, Gjorgjina Cenikj, Saso Dzeroski, and Barbara Ko- rouĹĄiÄ Seljak. FoodBench-QA: Overview of the Shared Task on Grounded Food and Nutrition Question Answering. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) at LREC 2026, 2026. [5] Neelansh Garg, Apuroop Sethupathy, Rudraksh Tuwani, et al. FlavorDB: A Database of Flavor Molecules. Nucleic Acids Research, 46(D1):D1210â D1216, 2018. [6] Andong Hua, Mehak Preet Dhaliwal, Laya Pullela, Ryan Burke, and Yao Qin. NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Es- timation from Meal Descriptions. In The Thirteenth International Conference on Learning Representa- tions, 2025. [7]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Live- CodeBench: Holistic and Contamination Free Eval- Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5 Qwen3.8 A95B Kimi K3 Gemini 3.6 Flash DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash Mistral Large 3 Llama 4 Maverick Command R+ Column model Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5 Qwen3.8 A95B Kimi K3 Gemini 3.6 Flash DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash Mistral Large 3 Llama 4 Maverick Command R+ Row model Teal: row higher Red: row lower Grey: unresolved at familywise Îą=.05 Holm-controlled paired comparisons Figure 4: All paired model contrasts after Holm correction. Teal means the row model scores higher; red means lower; grey is unresolved. uation of Large Language Models for Code. arXiv preprint arXiv:2403.07974, 2024. [8]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In The Twelfth International Conference on Learning Representa- tions, 2024. [9] Song Jin, Juntian Zhang, Xun Zhang, Zeying Tian, Fei Jiang, Guojun Yin, Wei Lin, Yong Liu, and Rui Yan. DiningBench: A Hierarchical Multi-view Bench- mark for Perception and Reasoning in the Dietary Domain. In Proceedings of the 64th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35289â35310. Asso- ciation for Computational Linguistics, 2026. [10] Percy Liang, Rishi Bommasani, Tony Lee, et al. Holis- tic Evaluation of Language Models. Transactions on Machine Learning Research, 2023. [11]Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Char- lie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Eval- uation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learn- ing, volume 267 of Proceedings of Machine Learning Research, pages 48371â48392. PMLR, 2025. 7 SubstitutionPairingConstraints Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5 Qwen3.8 A95B Kimi K3 Gemini 3.6 Flash DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash Mistral Large 3 Llama 4 Maverick Command R+ 667158 697254 667254 667055 667253 687151 657054 677050 666852 657447 686652 667050 676849 666851 657048 647048 656748 656648 646650 656944 646744 656443 656442 636340 606640 596140 575532 Performance decomposes by culinary decision family 0 20 40 60 80 100 Mean Epicure score Figure 5: Mean score by family on the common core. Each family contributes 178 tasks per model. Values share the 0â100 scale but retain family-specific decision semantics. [12]Jakub Radzikowski and Josef Chen. Epicure: Mul- tidimensional Flavor Structure in Food Ingredient Embeddings. arXiv preprint arXiv:2604.22776, 2026. [13]Jakub Radzikowski and Josef Chen. Epicure: Navi- gating the Emergent Geometry of Food Ingredient Embeddings. arXiv preprint arXiv:2605.22391, 2026. [14]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling, 2024. [15]Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Tor- ralba. Learning Cross-Modal Embeddings for Cook- ing Recipes and Food Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3020â3028, 2017. [16]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understand- ing Benchmark. In Advances in Neural Information Processing Systems, volume 37, 2024. [17] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti 13579111315171921232527 Rank (lower is better) Command R+ Llama 4 Maverick Mistral Large 3 DeepSeek V4 Flash Command A Nemotron 3.5 Lightning GLM 5.2 Claude Sonnet 5 Inkling Seed 2.1 Turbo Muse Glimmer 30B GLM 5.3 MiniMax M3 Tencent HY 3 Qwen 3.8 Max DeepSeek V4 Pro 0813 Gemini 3.6 Flash Kimi K3 Qwen3.8 A95B Claude Opus 5 GPT-5.6 Luna Pro Claude Fable 5 GPT-5.6 Terra Pro Muse Spark 1.2 GPT-5.6 Sol Pro Gemini 3.1 Pro Grok 4.6 Rank uncertainty remains visible behind point estimates Figure 6: Point rank and 95% bootstrap rank interval for every endpoint. Narrow intervals imply a stable location; overlapping intervals mark unresolved local order. Dey, Shubh Agrawal, Sandeep Singh Sandha, Sid- dartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Gold- blum. LiveBench: A Challenging, Contamination- Limited LLM Benchmark. In The Thirteenth In- ternational Conference on Learning Representations, 2025. [18]Zirui Wu, Xiao Liu, Jiayi Li, Lingpeng Kong, and Yansong Feng. Recipe2Plan: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multi- tasking with Time Constraints Between Actions. In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 4279â4301, Suzhou, China, 2025. Association for Computational Linguis- tics. [19]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.Ď-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In The Thirteenth International Conference on Learn- ing Representations, 2025. [20]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. Ad- vances in Neural Information Processing Systems, 36, 2023. 8 A Exact execution routes ModelBackendPanel 1 routePanel 2 route Grok 4.6openrouterxai/zdrxai/zdr Gemini 3.1 Proopenroutergoogle-ai-studiogoogle-ai-studio GPT-5.6 Sol Proopenrouteropenai/flexopenai/flex Muse Spark 1.2openroutermetameta GPT-5.6 Terra Proopenrouteropenai/flexopenai/flex Claude Fable 5openroutergoogle-vertex/globalgoogle-vertex/global GPT-5.6 Luna Proopenrouteropenai/flexopenai Claude Opus 5openrouterazure/usazure/us Qwen3.8 A95Bopenrouteralibabaalibaba Kimi K3openroutermorphmorph Gemini 3.6 Flashopenroutergoogle-vertex/usgoogle-vertex/us DeepSeek V4 Pro 0813 openroutergmicloud/fp8gmicloud/fp8 Qwen 3.8 Maxopenrouteralibabaalibaba Tencent HY 3openroutergmicloud/bf16gmicloud/bf16 MiniMax M3openroutervenice/fp8venice/fp8 GLM 5.3zai_coding_direct zai-coding-plan-directzai-coding-plan-direct Muse Glimmer 30Bopenrouterfireworksfireworks Seed 2.1 Turboopenrouterseed/fp8seed/fp8 Inklingopenrouterdeepinfra/fp8deepinfra/fp8 Claude Sonnet 5openrouteramazon-bedrock/claude-on-aws amazon-bedrock/claude-on-aws GLM 5.2openroutercoreweave/fp4coreweave/fp4 Nemotron 3.5 Lightning openroutervenice/fp4venice/fp4 Command Aopenroutercoherecohere DeepSeek V4 Flashopenrouterdeepinfra/fp4deepinfra/fp8 Mistral Large 3openroutermistralmistral Llama 4 Maverickopenrouterdigitaloceandigitalocean Command R+openroutercoherecohere Table 5: Frozen execution routes by panel. Automatic fallback is disabled; route differences indicate complete, score-blind model-block replacements rather than cell pooling. B Per-family scores ModelSubstitution Pairing Constraints Grok 4.666.170.958.2 Gemini 3.1 Pro69.071.654.2 GPT-5.6 Sol Pro66.272.554.1 Muse Spark 1.265.970.355.0 GPT-5.6 Terra Pro66.172.152.8 Claude Fable 568.270.651.3 GPT-5.6 Luna Pro64.569.753.7 Claude Opus 567.269.950.4 Qwen3.8 A95B66.067.852.5 Kimi K365.573.747.0 Gemini 3.6 Flash68.166.251.6 DeepSeek V4 Pro 081366.469.549.9 Qwen 3.8 Max66.668.549.5 Tencent HY 366.067.950.5 MiniMax M365.469.947.6 GLM 5.363.769.648.4 Muse Glimmer 30B65.166.847.8 Seed 2.1 Turbo65.366.247.6 Inkling63.665.849.5 Claude Sonnet 565.369.244.1 GLM 5.263.667.244.5 Nemotron 3.5 Lightning65.364.342.5 Command A64.763.941.6 DeepSeek V4 Flash63.163.240.1 Mistral Large 360.465.840.0 Llama 4 Maverick59.461.440.2 Command R+56.754.932.0 Table 6: Mean score in each 178-task family. Every entry uses the same tasks for every model; the equal-family mean is the FlavourBench Score. 9 C Protocol constants QuantityFrozen value Models27 Collection panels2 Primary tasks per model534 Unique anchor clusters534 Tasks per family178 Candidate ingredients8 Selected ingredients3 Scored portfolios per task56 Complete response cells14418 Bootstrap replicates50000 Sign-flip replicates100000 Pairwise hypotheses351 Shared cells per pair534 Common-core ruleStatus/parser only; score-blind Table 7: Core design and inference constants fixed before clean primary execution. 10