Paper deep dive
RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology
Lingli Ge, Yubin Wang, Junyuan Gao, Jiahe Song, Jiaxing Sun, Boyu Zhu, Haote Yang, Jingchao Wang, Lixin Ma, Jiang Wu, Yuqiang Li, Conghui He
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/5/2026, 5:08:19 AM
Summary
The paper introduces RxnOptBench, a benchmark designed to evaluate Large Language Models (LLMs) on reaction-condition optimization in organic methodology. Unlike existing benchmarks that focus on retrosynthesis or property prediction, RxnOptBench requires models to select the best experimental conditions from real condition-screening tables published in 2025. The benchmark uses a continuous scoring system based on yield and stereoselectivity metrics (ee, dr, rr) and includes a paired design to test the model's ability to use in-context literature precedents versus relying on parametric memorization. Evaluation of nine frontier general-purpose LLMs and three chemistry-specialized LLMs reveals that even the best models perform significantly below human levels, with chemistry-specialized models often underperforming general-purpose ones on complex multi-axis selection tasks.
Entities (13)
Relation Signals (12)
RxnOptBench → evaluates → LLM
confidence 95% · RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology
RxnOptBench → usesdatafrom → condition-screening table
confidence 95% · RxnOptBench, a benchmark whose every option and precedent is a real wet-lab entry mined from the optimization tables of organic-methodology papers
ChemLLM → istype → Chemistry LLM
confidence 90% · three Chemistry LLMs (ChemLLM [19], ChemVLM [14], ChemDFM [20])
ChemDFM → istype → Chemistry LLM
confidence 90% · three Chemistry LLMs (ChemLLM [19], ChemVLM [14], ChemDFM [20])
ChemVLM → istype → Chemistry LLM
confidence 90% · three Chemistry LLMs (ChemLLM [19], ChemVLM [14], ChemDFM [20])
GPT-5.5 → istype → Proprietary LLM
confidence 90% · three Proprietary LLMs (Claude-Opus-4.7 [49], GPT-5.5 [50], Gemini-3.1 Pro thinking variant [51])
RxnOptBench → scoresusing → Yield
confidence 90% · graded by a continuous relative score derived from a declared headline utility that combines reported yield with enantiomeric excess (ee), diastereomeric ratio (dr), and regioisomeric ratio (rr)
RxnOptBench → scoresusing → enantiomeric excess
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Chemical reaction-condition optimization -- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize yield and stereoselectivity -- is a central, judgement-laden subtask of organic methodology research that large language models are increasingly expected to support. Yet existing chemistry benchmarks evaluate reaction-class labelling, retrosynthesis, or SMILES manipulation, and do not ask models to read a real condition-screening table and pick the best set. We introduce RxnOptBench, a benchmark whose every option and precedent is a real wet-lab entry mined from the optimization tables of organic-methodology papers published in 2025, graded by a continuous relative score derived from a declared headline utility that combines reported yield with enantiomeric excess (ee), diastereomeric ratio (dr), and regioisomeric ratio (rr), and equipped with a paired precedents-vs-no-precedents design that isolates in-context use of literature evidence from parametric memorization. Across nine frontier LLMs and three Chemistry LLMs, even the best models leave substantial headroom: chemistry-specialized models fall to the random-baseline floor on multi-axis selection, while open-weight models have closed most of the gap to proprietary frontier models. We release the final human-reviewed benchmark test set and evaluation code.
Tags
Links
- Source: https://arxiv.org/abs/2610.02242v1
- Canonical: https://arxiv.org/abs/2610.02242v1
Trouble viewing inline? Open PDF directly →
Full Text
84,188 characters extracted from source content.
Expand or collapse full text
RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology Lingli Ge1,2,3 Yubin Wang3 Junyuan Gao3 Jiahe Song1,3 Jiaxing Sun3 Boyu Zhu3 Haote Yang3 Jingchao Wang3 Lixin Ma3 Jiang Wu3* Yuqiang Li2* Conghui He3* 1Shanghai Jiao Tong University, Shanghai, China 2Shanghai Innovation Institute, Shanghai, China 3Shanghai Artificial Intelligence Laboratory, Shanghai, China Abstract Chemical reaction-condition optimization — choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize yield and stereoselectivity — is a central, judgement-laden subtask of organic methodology research that large language models are increasingly expected to support. Yet existing chemistry benchmarks evaluate reaction-class labelling, retrosynthesis, or SMILES manipulation, and do not ask models to read a real condition-screening table and pick the best set. We introduce RxnOptBench, a benchmark whose every option and precedent is a real wet-lab entry mined from the optimization tables of organic-methodology papers published in 2025, graded by a continuous relative score derived from a declared headline utility that combines reported yield with enantiomeric excess (e), diastereomeric ratio (dr), and regioisomeric ratio (r), and equipped with a paired precedents-vs-no-precedents design that isolates in-context use of literature evidence from parametric memorization. Across nine frontier LLMs and three Chemistry LLMs, even the best models leave substantial headroom: chemistry-specialized models fall to the random-baseline floor on multi-axis selection, while open-weight models have closed most of the gap to proprietary frontier models. We release the final human-reviewed benchmark test set and evaluation code. **footnotetext: Corresponding authors. Figure 1: RxnOptBench overview. (A) Question Construction: each condition-screening table yields MC/FIB items in matched precedents (prior rows shown, top withheld) and no-precedents (no prior rows) modes. (B) Scoring Rules: every observed row carries a precomputed relative score fusing yield with e/dr/r. MC reports Accuracy and Score; FIB separately reports observed-row Hit, Strict Accuracy, and Score conditioned on Hit. 1 Introduction Over the past two decades, organic synthesis methodology has remained a central driver of progress in chemistry, reflected by multiple Nobel Prizes recognizing advances such as asymmetric catalysis, olefin metathesis, cross-coupling, and click chemistry. These breakthroughs were not defined solely by the discovery of new bond-forming transformations, but by the development of reaction conditions that rendered them broadly practical and reproducible across diverse substrates. Catalysts, ligands, solvents, additives, and reaction environments collectively determine whether a transformation can achieve the efficiency and selectivity required for applications in pharmaceutical, agrochemical, and materials synthesis. Contemporary methodology research continues to expand this landscape, with new catalytic systems and activation modes reported each year across thermal, photochemical, electrochemical, and multi-catalytic regimes. A central challenge underlying this process is reaction condition optimization: identifying the combination of catalyst, ligand, solvent, reagent, temperature, and other controllable variables that maximizes yield and stereoselectivity for a target transformation. Existing computational approaches address this problem only partially. Bayesian optimization and high-throughput experimentation frameworks [1, 2, 3] can efficiently search within predefined experimental spaces, but they typically assume a fixed reaction class and a closed candidate set. Such approaches therefore offer limited support when reasoning about newly reported transformations or unfamiliar catalytic systems. In parallel, large language models (LLMs) have recently emerged as general-purpose reasoning systems for chemistry and now constitute the core component of many autonomous or semi-autonomous chemistry agents [4, 5]. These systems derive their capabilities from large-scale exposure to the chemical literature, suggesting the possibility of leveraging accumulated methodological knowledge for reaction-condition recommendation. However, whether current LLMs can reliably perform experimentally meaningful condition selection remains largely unexamined. Existing chemistry benchmarks provide little insight into this capability. Widely used reaction datasets, including USPTO-50k and related variants [6, 7, 8], primarily support retrosynthesis and reaction prediction tasks. Larger reaction corpora such as Pistachio [9] and the Open Reaction Database [10] record successful experimental outcomes, but generally preserve reactions as isolated entries rather than as matched sets of tested alternatives with directly comparable outcomes. More recent chemistry benchmarks — including ChemLLMBench [11], ChemBench [12], SciBench [13], and ChemVLM [14] — evaluate molecular property prediction, reaction understanding, SMILES manipulation, or general scientific reasoning. Collectively, these benchmarks measure whether models can retrieve or manipulate chemical knowledge, but not whether they can make the comparative decisions required during experimental optimization. Organic-methodology publications commonly include condition-screening tables, which disclose a curated subset of experimentally tested conditions supporting a reported transformation. This curation defines which local decision spaces are visible; among the disclosed rows, the benchmark uses the measured outcomes and their ordering directly as labels. Although these tables are not complete optimization histories, they preserve matched comparisons for the same reaction while varying catalyst, ligand, solvent, base, additive, temperature, or related factors. They therefore support an experimentally grounded conditional task: selecting the best reported condition among the candidates disclosed in that table (Figure 1A). Without additional wet-lab validation, unreported combinations have no defensible outcome label and are treated as outside measured support, not as failures or negative examples; the best disclosed row is not interpreted as a global optimum over chemical space. Motivated by this observation, we introduce RxnOptBench, a benchmark designed to evaluate LLMs on reaction-condition recommendation. We transform condition-screening tables into multiple-choice and fill-in-the-blank tasks that require models to identify optimal or near-optimal reaction conditions. Following a four-stage extraction pipeline with full human verification (Section 3.2), RxnOptBench comprises 4,773 questions across eight task sub-types (3,911 multiple-choice and 862 fill-in-the-blank questions), derived from 552 screening tables spanning 232 organic-methodology papers published in 2025 across 16 journals. This recent sampling frame deliberately targets contemporary, potentially unfamiliar condition-selection problems rather than canonical-recipe recall; recency reduces but does not eliminate exact source exposure. Answers receive partial credit through the benchmark’s declared headline aggregate utility, which combines reported yield and stereoselectivity but is not intended to encode every chemical objective or chemist preference. RxnOptBench evaluates both cold-start and precedent-assisted reasoning and separately probes single-factor versus full-condition selection. We evaluate nine frontier general-purpose LLMs together with three chemistry-specialized LLMs on RxnOptBench. The results reveal four main findings. First, substantial performance headroom remains even for the strongest proprietary models, indicating that reaction-condition recommendation is far from a solved problem. Second, tasks requiring global reasoning over complete condition sets are consistently more challenging than single-factor selection. Third, chemistry-specialized models do not consistently outperform frontier general-purpose LLMs, particularly on the most difficult task settings. Finally, providing experimentally related precedents yields measurable improvements, suggesting that LLMs can partially leverage literature context for condition selection rather than relying solely on memorized associations. Figure 2: Headline accuracy on the no-precedents subsets. Per-model Accuracy on MC_single (n=1,039n=1,039), MC_all (n=552n=552), and FIB (n=552n=552); sectors group models by tier. FIB uses Strict Accuracy over submitted predictions. Numbers in Table 2; precedents comparisons use the remaining matched subsets. 2 Related Work Reaction datasets and condition-prediction baselines. USPTO-50k and its variants [6, 7, 8] and Pistachio [9] have driven a decade of work on reaction-class labelling and retrosynthesis, but record only individual reactions, stripped from the screening context in which they were chosen. The Open Reaction Database (ORD) [10] provides a structured schema for sharing reaction records, and ORDerly [15] cleans ORD into ML-ready splits, including the first ORD-derived condition-prediction benchmark. Coley-style predictors [3], multilabel cross-coupling models [16], sequence-to-sequence models that infer full experimental procedures from reaction SMILES [17], and Bayesian-optimization workflows [1, 2] all output condition vectors directly, but assume a fixed reaction template or rely on dense HTE coverage that real methodology tables rarely provide. None, however, frames the task as a chemist does at the bench: choosing the empirically best entry from a small table of pre-screened candidates. Chemistry LLMs and benchmarks. Existing benchmarks—ChemLLMBench [11], ChemBench [12], Mol-Instructions [18]—evaluate LLMs on property prediction, name-reaction trivia, and SMILES manipulation. Domain models like ChemLLM [19] and ChemDFM [20] are pretrained on chemistry corpora, while ChemVLM [14] extends to the multimodal regime. Broader suites [13, 21, 22] probe curriculum-level reasoning, and ChemTable [23] and MMSci-Eval [24] target scientific tables. A parallel line targets condition or yield prediction: M-RCR [25], Chemma [26], and SynAsk [27] extend the Coley-style line [3] to LLMs, but are trained on USPTO corpora or single-template HTE screens [28]. None separate single-axis from whole-condition reasoning, ground options in real screening-table entries, or provide continuous partial credit over reported yield and selectivity. Agentic systems such as ChemCrow [5] and Coscientist [4], surveyed by Ramos and others [29], sit one level higher and presuppose the condition-decision capability we benchmark. Condition recommendation methods and benchmarks. We group prior automatic condition-recommendation methods into two regimes. Global models train on patent- or literature-scale corpora, including the USPTO-Condition and Reaxys-TotalSyn-Condition benchmarks released with Parrot [30], the USPTO reagent corpus used by Andronov et al. [31], and the USPTO-500MT condition split [32] adopted by TextReact [33] and M-RCR [25], with ChemMAS [34] further scaling to 544k proprietary reactions. Local models target a single reaction family with HTE data: Bayesian optimization [1, 2] has been extended to multi-objective yield-e tuning 35, bandit-style identification of general conditions [36], and highly parallel HTE-ML campaigns [37]. All of these produce a condition vector rather than selecting the best entry of candidates, sitting orthogonal to RxnOptBench. 3 RxnOptBench 3.1 Source We mine condition-screening tables from organic-methodology papers first published in 2025 across 16 prespecified venues: ACS Catal., ACS Cent. Sci., Angew. Chem. Int. Ed., Chem, Chem Catalysis, Chem. Sci., Green Chem., JACS, JACS Au, Nature, Nat. Catal., Nat. Chem., Nat. Commun., Nat. Synth., Org. Lett., and Sci. Adv. Following the revised venue categorization, JACS and Angew. Chem. Int. Ed. are grouped as chemistry-discipline flagships, whereas ACS Cent. Sci. and Nat. Commun. are grouped as broad-scope flagships. This presentation-level regrouping does not change venue inclusion or sampling. Both the main text and the full Supporting Information are fed to the pipeline. This one-year frame is an intended evaluation distribution rather than an estimate of historical publication prevalence. It emphasizes contemporary, recently disclosed transformations, catalytic systems, and condition regimes, thereby probing transfer to recent chemical decision problems. We do not assume that every transformation constitutes a new reaction class. After passing the extraction pipeline (Section 3.2) and full manual expert verification, the resulting corpus statistics are deferred to Section 3.2. The exclusion of patent-mined and aggregator sources, whose row-level outcomes are routinely truncated, re-normalized, or re-typeset, keeps every source artifact directly auditable against the native PDF. To avoid inflating chemical-space counts by the multiple questions derived from one source table, we analyze composition over the 552 parent screening tables. They span 20 primary transformation classes, 26 condition fields, and 341 unique canonical products containing 232 non-empty Bemis–Murcko scaffolds; the reaction, condition, and product distributions are reported in Appendix C. 3.2 Extraction Pipeline A four-stage semi-automated pipeline (Figure 3) converts the 232 source PDFs into 552 retained tables (of 911 candidates) and 5,451 row-level reaction records, each carrying resolved substrate/product/catalyst/ligand SMILES, a canonical condition string, measured outcomes (yield, e, d.r.), and a back-pointer to paper/page/row. Visual stages run on 600 DPI page PNGs; each stage pairs a strict-format VLM call with a deterministic check. Figure 3: Four-stage semi-automated extraction pipeline. PDF pages are screened, parsed, converted into row-level reaction records, and manually audited before question construction. The GTR-VL label in Stage 3 denotes the MinerU.Chem recognizer; details in §3.2. Page Selection & Table Understanding. A regex-then-VLM cascade shortlists pages whose captions or body text mention optimization/screening and separates true screening tables from substrate-scope, mechanistic, and characterization tables; a second VLM emits structured caption, footnotes, columns, rows per table, with cross-page tables reconstructed from neighbours. All VLM calls in the extraction pipeline other than molecular-structure recognition use Kimi-K2.6. These stages are described for methodological transparency, but their prompts, parsing code, and runtime implementation are not part of the public release. Molecule Parsing. DECIMER-Segmentation [38] (Mask R-CNN [39]) localizes molecule drawings. We use the GTR-VL-based molecular-structure recognition module of MinerU.Chem [40], accessed through an API, to recognize each crop; model-specific post-processing converts its output into structures validated by RDKit [41]. A VLM then binds abstract labels (1a, L1, Pd-1) to these structures via SMILES lookup — it never writes structures. The MinerU.Chem recognition service, our model-specific post-processing, and the broader extraction and benchmark-construction code are not redistributed with RxnOptBench. The public release is limited to the final human-reviewed benchmark test set and evaluation code; it supports consistent scoring of model predictions but not reconstruction from raw PDFs. Reaction Construction & Final Expert Review. Per-row constrained VLM calls type each cell (yield, e, dr, solvent, temperature, …), populate substrate/product/catalyst/ligand slots, and expand footnote/arrow annotations into a canonical condition string; text-only reagents are resolved via curated dictionary → local cache → PubChem [42]. The checks within Stages 1–4 are deterministic rather than human reviews. After Stage 4 has assembled complete table- and row-level records, every extracted table is compared with the source PDF by four master’s-level and two doctoral-level paid chemistry interns based in mainland China, who edit, accept, or reject it; only records passing this final gate enter question construction. The automatic stages support batched inputs, concurrent calls, and resumable intermediate outputs; expert verification is the deliberate final quality gate and the main curation bottleneck, so future growth proceeds through versioned reviewed releases. 3.3 Question Construction A retained table reaches question construction only if it contains ≥2≥\!2 rows whose normalized condition strings differ on at least one axis (normalization: Unicode NFKC, sub-/super-script folding, whitespace and case folding); single-row tables and tables collapsing to a single condition string are excluded. A condition category is varying if its normalized value differs across ≥2≥\!2 entries; table-level constants enter the prompt as context, while varying axes define the candidate space. RxnOptBench then defines two response types and eight evaluated subsets (Table 1). Table 1: The eight sub-types of RxnOptBench. Type Sub-type #Q Description MC single_var 1,039 One target axis; all other varying axes locked. MC all_var 552 Choose one observed joint-condition row. MC single_var_precs 850 single_var with precedents. MC all_var_precs 310 all_var with precedents. MC single_var_precs_paired 850 Twin of single_var_precs without precedents. MC all_var_precs_paired 310 Twin of all_var_precs without precedents. FIB all_blank 552 One index per axis; observed joint tuple required. FIB all_blank_precs 310 all_blank with precedents. Total 4,773 3,911 MC + 862 FIB. Both MC families are single-answer: MC_single locks all but one varying axis to a real fixed tuple and scores within that local group, whereas each MC_all option is a complete observed row. FIB returns one index per varying key; independently observed values can recombine into an untested tuple, so Hit requires the complete tuple to match an observed row. Normalized duplicates retain the highest-scoring row and all maximum ties are accepted as gold (formal construction in Appendix A). Two construction details not visible in the table: precedents are condition-disjoint from every option (their yield-band sampling and matched no-precedents pairing rule are deferred to §3.5), and all formal evaluations use shuffled candidate order. MC option bundles are shuffled by a question-ID-derived seed; each FIB candidate list is shuffled independently by question and condition key using a stable hash and seed 20260504. The same resulting question version and index space are used for all 12 models, and the public test-set records store each exact permutation. 3.4 Metrics Each entry carries the benchmark’s headline aggregate utility: a monotone, bounded summary combining reported yield with available selectivity. It provides one reproducible scalar ranking but is not intended to encode every chemical objective or chemist preference, effective(s)=yield⋅fenantio⋅fdr⋅frr,effective(s)=yield· f_enantio· f_dr· f_r, (1) where fenantio=(100+|ee|)/200,eereported,ρer/(ρer+1),erreported but eeabsent,1,otherwise,fdr=ρdrρdr+1,frr=ρrrρrr+1.f_enantio= cases(100+|e|)/200,&e\ reported,\\ _er/( _er+1),&er\ reported but e\ absent,\\ 1,&otherwise, cases f_dr= _dr _dr+1, f_r= _r _r+1. (2) Here ee denotes enantiomeric excess, while erer, drdr, and rr denote enantiomeric, diastereomeric, and regioisomeric ratios, respectively. Each factor is the major-product fraction on its selectivity axis: major enantiomer for ee/eree/er, major diastereomer for drdr, and major regioisomer for rr. For each ratio metric, ρ is the normalized major:minor ratio (thus either reported order, e.g. 20:120:1 or 1:201:20, maps to ρ=20ρ=20). An unreported selectivity axis contributes the neutral factor 11; this omits that axis from the utility rather than assuming perfect selectivity. For question q with scorable set qO_q, define rq(o)=clamp(effective(o)/Uq⋆,0,1),Uq⋆=maxo′∈qeffective(o′),r_q(o)=clamp\! (effective(o)/U_q ,0,1 ), U_q = _o _qeffective(o ), (3) with equal zero numerator/denominator defined as 11. Here qO_q is global for MC_all/FIB and local for MC_single. For Aq=i:rq(oq,i)=1A_q=\i:r_q(o_q,i)=1\ and MC prediction i^q i_q, over N submissions AccMC=1N∑q[i^q∈Aq],ScoreMC=1N∑qrq(oq,i^q),Acc_MC= 1N _q1[ i_q∈ A_q], _MC= 1N _qr_q(o_q, i_q), (4) where an invalid index contributes zero to both. For FIB, let VqV_q mark a valid in-range mapping ^q x_q, and define Hq=[Vq=1∧^q∈ℒq]H_q=1[V_q=1 x_q _q] and Bq=Hq 1[rq(^q)=1]B_q=H_q\,1[r_q( x_q)=1]. The reported metrics are Hit=∑qHqN,StrictAcc=∑qBqN,Score∣Hit=∑q:Hq=1rq(^q)∑qHq.Hit= _qH_qN, = _qB_qN, = _q:H_q=1r_q( x_q) _qH_q. (5) Thus StrictAcc=HitPr(Bq=1∣Hq=1)StrictAcc=Hit (B_q=1 H_q=1). Invalid and non-Hit outputs fail Hit/Strict but lack measured outcomes and are excluded from Score∣\, \,Hit; non-Hit does not mean chemically invalid. Appendix D gives score-sensitivity analyses. Table 2 reports a matched uniform-random floor (∼0.54 \!0.54–0.670.67), against which Scores should be read. 3.5 Precedents vs. No-precedents: Paired Design Each *_precs question is paired with a *_precs_paired twin sharing identical option space, answer key, and scoring; they differ only in whether 2–5 precedent experiments are appended. Precedents are drawn from the lower-to-mid yield band, the optimum is withheld, and (FIB) precedent combinations are removed from legal answers — so Δr=rprecs−rnoprec r=r_precs\!-\!r_noprec isolates trend extrapolation, free of difficulty/option/scoring confounds. 3.6 Models and Inference Protocol We evaluate nine frontier LLMs — six Open Source LLMs (DeepSeek-V4-Pro [43], Kimi-K2.6 [44], GLM-5.1 [45], Qwen3.5-397B-A17B [46], Intern-S1-Pro [47], MiniMax-M2.7 [48]) and three Proprietary LLMs (Claude-Opus-4.7 [49], GPT-5.5 [50], Gemini-3.1 Pro thinking variant [51]) — plus three Chemistry LLMs (ChemLLM [19], ChemVLM [14], ChemDFM [20]). All models share one prompt template per question type. Sampling is disabled (temperature=0 where available, otherwise provider greedy/non-sampling mode). The public evaluator deterministically scores supplied predictions against the fixed, human-reviewed test set; hosted backends need not reproduce generations bit-for-bit (Appendix D). 4 Empirical Results Table 2: No-precedents results (MC_single: 1,039; MC_all/FIB: 552 each). FIB Strict uses all submissions; Score is conditional on Hit. Cells are heat-mapped per column; models are ordered weakest→ by mean Score within tier. Model MC_single MC_all FIB Acc. Score Acc. Score Hit Strict Score||Hit Baseline Uniform random 36.5% 0.669 17.8% 0.540 28.1% 5.9% 0.540 Chemistry LLMs ChemLLM 36.1% 0.666 20.7% 0.545 25.2% 6.6% 0.554 ChemDFM 36.7% 0.670 20.5% 0.547 33.5% 7.6% 0.557 ChemVLM 41.4% 0.712 23.6% 0.602 27.7% 6.7% 0.610 Open Source LLMs Intern-S1-Pro 43.4% 0.734 27.9% 0.644 41.7% 9.6% 0.576 Qwen3.5-397B-A17B 42.2% 0.711 32.2% 0.681 42.2% 10.7% 0.615 Kimi-K2.6 41.3% 0.707 31.9% 0.674 40.9% 13.0% 0.637 GLM-5.1 44.6% 0.729 32.2% 0.675 44.9% 13.6% 0.646 MiniMax-M2.7 43.1% 0.729 26.8% 0.656 42.8% 12.1% 0.666 DeepSeek-V4-Pro 44.6% 0.745 32.4% 0.676 47.5% 13.6% 0.672 Proprietary LLMs GPT-5.5 45.4% 0.754 32.4% 0.674 49.1% 16.1% 0.699 Gemini-3.1-Pro 46.8% 0.760 35.0% 0.709 49.6% 15.0% 0.675 Claude-Opus-4.7 48.6% 0.764 36.8% 0.725 44.4% 14.3% 0.683 4.1 Headline Results All 12 models are evaluated on the same 4,773-question benchmark. Figure 2 and Table 2 report the three no-precedents subsets; Figures 4–5 report the matched precedents comparisons. We highlight three top-line observations below; §4.2 and Appendix G ground them in per-key and per-reaction analyses. The task is far from saturated. No model approaches ceiling on the table-anchored selection task. The best MC_all Accuracy is Claude-Opus-4.7 at 36.8%36.8\% (Score 0.7250.725); the best FIB Strict Accuracy is GPT-5.5 at 16.1%16.1\% (Score∣\, \,Hit 0.6990.699); even the loosest FIB criterion—Hit, requiring the structurally valid index tuple to match an observed source-table row—peaks at 49.6%49.6\%. For that model, the 552 outputs comprise 0 format/index-invalid responses, 278 valid but unobserved tuples, 191 observed suboptimal rows, and 83 best observed rows. Hit therefore diagnoses content-level joint-condition composition rather than output-format compliance; it does not label those non-Hits chemically infeasible. Every model also loses Score from MC_single to MC_all (Δaxis∈[0.030,0.090] _axis\!∈\![0.030,0.090] for frontier LLMs), confirming that joint-condition reasoning, not single-axis ranking, is the binding constraint. Chemistry-specialized models sit at the random-baseline floor. Score must be read against the uniform-random floor of §3.4 (∼0.54 \!0.54–0.670.67, top row of Table 2), not against zero. On MC_all, ChemLLM scores 20.7%20.7\% Accuracy / 0.5450.545 Score against Claude’s 36.8%36.8\% / 0.7250.725—a 16.116.1-p Accuracy gap, with the Score within 0.0050.005 of the random floor (0.5400.540). On FIB, ChemLLM reaches 6.6%6.6\% Strict Accuracy / 0.5540.554 Score∣\, \,Hit against GPT-5.5’s 16.1%16.1\% / 0.6990.699, only 0.0140.014 above the random floor (0.5400.540). Thus chemistry-domain tuning yields no measurable aggregate advantage in this table-anchored decision setting. Appendix G (M2) reports the observed position-zero selection pattern as a diagnostic rather than a causal explanation. Open-source LLMs close most of the gap. The strongest open model, DeepSeek-V4-Pro, trails the strongest proprietary model by 4.04.0 p Accuracy / 0.0190.019 Score on MC_single (44.6%44.6\% / 0.7450.745 vs. Claude 48.6%48.6\% / 0.7640.764), ties GPT-5.5 on MC_all Accuracy (32.4%32.4\%), and reaches FIB Score∣\, \,Hit 0.6720.672 against the top proprietary 0.6830.683 (Δ=0.011 \!=\!0.011). GLM-5.1, Qwen3.5-397B-A17B, and Kimi-K2.6 sit in the same band. The proprietary tier therefore retains a measurable but no longer decisive lead on RxnOptBench. Figure 4: Precedents vs. no-precedents on MC. Solid = with precedents, hatched = no-precedents; bars annotated with Δprecs _precs. Top: MC_single; bottom: MC_all. Figure 5: Precedents vs. no-precedents on FIB. Solid = with precedents, hatched = no-precedents; bars annotated with Δprecs _precs. Observed-row Hit, Strict Accuracy (best observed row over all submitted predictions), and Score∣\, \,Hit are reported separately. 4.2 Per-Condition-Key Accuracy We break Accuracy down by the condition key each question targets, on the no-precedents subsets (MC single_var, FIB all_blank). The eight most frequent keys cover 933/1,039933/1,039 MC questions and 1,561/1,9121,561/1,912 FIB key occurrences; rarer keys are pooled into the headline only. Table 3: Per-condition-key Accuracy (%) on no-precedents subsets, with 12 models grouped by tier (Chemistry LLMs / Open Source LLMs / Proprietary LLMs) and sorted ascending by overall within each tier (best at bottom). Column abbreviations: rea=reagents, sol=solvents, cat=catalysts, T=temperature, t=time, hv=light, atm=atmosphere, lig=ligand, ov=overall. Bold = best in column across all 12 models. (a) MC single_var Model rea sol cat T t hv atm lig ov n 227 201 145 104 95 73 48 40 933 ChemLLM 33.5 30.8 29.7 41.3 40.0 35.6 47.9 35.0 34.8 ChemDFM 34.8 32.8 29.0 40.4 42.1 37.0 50.0 32.5 35.7 ChemVLM 43.6 34.8 37.2 41.3 45.3 50.7 56.2 37.5 41.6 Kimi-K2.6 40.1 42.3 34.5 44.2 42.1 43.8 43.8 27.5 40.3 Qwen3.5-397B-A17B 45.8 35.8 33.8 41.3 44.2 57.5 45.8 45.0 42.0 Intern-S1-Pro 44.9 39.8 30.3 44.2 35.8 61.6 56.2 47.5 42.6 MiniMax-M2.7 39.2 39.8 38.6 49.0 46.3 58.9 47.9 32.5 42.8 DeepSeek-V4-Pro 42.3 38.8 35.9 52.9 47.4 61.6 50.0 42.5 44.2 GLM-5.1 42.3 42.8 41.4 49.0 42.1 56.2 50.0 42.5 44.5 GPT-5.5 43.2 42.3 40.7 50.0 41.1 61.6 41.7 55.0 45.0 Gemini-3.1-Pro 42.7 40.3 42.8 46.2 45.3 64.4 56.2 50.0 45.6 Claude-Opus-4.7 44.9 42.3 42.8 45.2 52.6 64.4 64.6 57.5 47.9 (b) FIB all_blank Model rea sol cat T t hv atm lig ov n 334 263 204 194 202 139 158 67 1561 ChemLLM 27.5 25.5 24.0 35.1 32.2 25.9 32.3 16.4 28.1 ChemVLM 34.7 38.0 32.4 47.9 39.1 40.3 40.5 44.8 38.7 ChemDFM 39.2 36.9 35.3 44.3 41.6 42.4 53.2 34.3 40.7 Qwen3.5-397B-A17B 43.4 45.2 34.8 49.0 51.5 46.8 45.6 43.3 44.8 Intern-S1-Pro 40.7 44.1 44.1 48.5 42.6 51.8 52.5 52.2 45.6 GLM-5.1 47.3 53.2 41.2 50.5 49.0 54.7 47.5 41.8 48.6 MiniMax-M2.7 46.7 46.0 45.1 53.6 46.5 56.8 51.9 50.7 48.8 DeepSeek-V4-Pro 47.0 46.4 44.6 52.6 40.1 60.4 57.0 50.7 48.8 Kimi-K2.6 46.4 44.5 48.5 55.7 46.5 51.1 56.3 44.8 48.9 Claude-Opus-4.7 45.8 43.3 47.5 50.0 49.5 52.5 55.7 49.3 48.4 GPT-5.5 47.0 51.7 54.4 53.6 43.1 51.1 47.5 52.2 49.7 Gemini-3.1-Pro 47.3 51.7 49.0 52.1 43.6 51.1 57.6 53.7 50.1 Observations. (i) The proprietary-tier advantage is concentrated on small-arity keys. Leads exceed 2020 pt for irradiation/light source and reaction atmosphere (typically 2–3 candidates) but compress to ≤12≤\!12 pt for reagents and reaction time, with Qwen3.5 even outperforming Claude on reagents — the tier gap in Table 2 is largely a discrimination advantage on easy keys, not a deep-chemistry one. (i) Catalysts are a persistent cross-format bottleneck. This category sits near the bottom in both MC and FIB and across all three tiers. The result establishes category-specific difficulty, but does not distinguish catalyst-class knowledge limitations from candidate ambiguity or other causes. (i) FIB difficulty lives in composition, not in fields. Frontier-model overall per-key FIB Accuracy is 44.844.8–50.1%50.1\%, while FIB Strict remains below 17%17\%: individual axes carry usable signal, but models fail to compose them into the joint optimum, echoing the saturation gap reported in §4.1. 5 Conclusion and Future Work RxnOptBench turns reaction-condition screening into a benchmark anchored in real wet-lab tables, paired across precedents/no-precedents, and scored with a declared headline aggregate utility over reported yield and stereoselectivity. Across 12 frontier and chemistry-specialised models the task is far from saturated: chemistry-specialised models sit at the random-baseline floor, open-source LLMs close most of the proprietary lead on MC_single but not on MC_all or FIB, and the illustrative cases in Appendix G expose item-level patterns that complement aggregate accuracy. The results point to substantial difficulty in ordering disclosed candidates and composing multiple condition axes, rather than identifying a single universal cause of model error. Future work. RxnOptBench can grow through versioned, expert-verified releases rather than unbounded automatic ingestion. The automatic stages support batching, concurrency, deterministic checks, and resumable intermediates, while manual verification remains the deliberate quality-control bottleneck. Periodic, versioned refreshes from newly published methodology papers can test transfer to newly disclosed chemistry; alongside each refresh we will broaden undersampled condition families and non-English venues and tighten FIB grading with an order-invariant chemical comparator. References [1] B. J. Shields et al. (2021) Bayesian reaction optimization as a tool for chemical synthesis. Nature 590, p. 89–96. Cited by: §1, §2, §2. [2] D. T. Ahneman, J. G. Estrada, S. Lin, S. D. Dreher, and A. G. Doyle (2018) Predicting reaction performance in c–n cross-coupling using machine learning. Science 360 (6385), p. 186–190. Cited by: §1, §2, §2. [3] H. Gao, T. J. Struble, C. W. Coley, Y. Wang, W. H. Green, and K. F. Jensen (2018) Using machine learning to predict suitable conditions for organic reactions. ACS Central Science 4 (11), p. 1465–1476. Cited by: §1, §2, §2. [4] D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes (2023) Autonomous chemical research with large language models. Nature 624, p. 570–578. Cited by: §1, §2. [5] A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller (2024) Augmenting large language models with chemistry tools. Nature Machine Intelligence 6 (5), p. 525–535. Cited by: §1, §2. [6] D. M. Lowe (2012) Extraction of chemical structures and reactions from the literature. Ph.D. Thesis, University of Cambridge. Cited by: §1, §2. [7] N. Schneider, N. Stiefl, and G. A. Landrum (2016) What’s what: the (nearly) definitive guide to reaction role assignment. Journal of Chemical Information and Modeling 56 (12), p. 2336–2346. Cited by: §1, §2. [8] B. Liu, B. Ramsundar, P. Kawthekar, J. Shi, J. Gomes, Q. Luu Nguyen, S. Ho, J. Sloane, P. Wender, and V. Pande (2017) Retrosynthetic reaction prediction using neural sequence-to-sequence models. ACS Central Science 3 (10), p. 1103–1113. Cited by: §1, §2. [9] NextMove Software (2017) Pistachio. Note: https://w.nextmovesoftware.com/pistachio.html Cited by: §1, §2. [10] S. M. Kearnes, M. R. Maser, M. Wleklinski, A. Kast, A. G. Doyle, S. D. Dreher, J. M. Hawkins, K. F. Jensen, and C. W. Coley (2021) The open reaction database. Journal of the American Chemical Society 143 (45), p. 18820–18826. Cited by: §1, §2. [11] T. Guo, K. Guo, B. Nan, Z. Liang, Z. Guo, N. V. Chawla, O. Wiest, and X. Zhang (2023) What can large language models do in chemistry? A comprehensive benchmark on eight tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Cited by: §1, §2. [12] A. Mirza, N. Alampara, S. Kunchapu, M. Ríos-García, B. Emoekabu, A. Krishnan, T. Gupta, M. Schilling-Wilhelmi, M. Okereke, A. Aneesh, et al. (2025) A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry 17 (7), p. 1027–1034. Cited by: §1, §2. [13] X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang (2024) SCIBENCH: evaluating college-level scientific problem-solving abilities of large language models. In Proceedings of the 41st International Conference on Machine Learning, p. 50622–50649. Cited by: §1, §2. [14] J. Li, D. Zhang, X. Wang, Z. Hao, J. Lei, Q. Tan, C. Zhou, W. Liu, Y. Yang, X. Xiong, et al. (2025) Chemvlm: exploring the power of multimodal large language models in chemistry area. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 415–423. Cited by: §1, §2, §3.6. [15] D. S. Wigh, J. Arrowsmith, A. Pomberger, K. C. Felton, and A. A. Lapkin (2024) Orderly: data sets and benchmarks for chemical reaction data. Journal of Chemical Information and Modeling 64 (9), p. 3790–3798. Cited by: §2. [16] M. R. Maser, A. Y. Cui, S. Ryou, T. J. DeLano, Y. Yue, and S. E. Reisman (2021) Multilabel classification models for the prediction of cross-coupling reaction conditions. Journal of Chemical Information and Modeling 61 (1), p. 156–166. Cited by: §2. [17] A. C. Vaucher, P. Schwaller, J. Geluykens, V. H. Nair, A. Iuliano, and T. Laino (2021) Inferring experimental procedures from text-based representations of chemical reactions. Nature Communications 12, p. 2573. Cited by: §2. [18] Y. Fang, X. Liang, N. Zhang, K. Liu, R. Huang, Z. Chen, X. Fan, and H. Chen (2024) Mol-instructions: a large-scale biomolecular instruction dataset for large language models. International Conference on Learning Representations (ICLR). Cited by: §2. [19] D. Zhang, W. Liu, Q. Tan, J. Chen, H. Yan, Y. Yan, J. Li, W. Huang, X. Yue, D. Zhou, S. Zhang, M. Su, H. Zhong, Y. Li, and W. Ouyang (2024) ChemLLM: a chemical large language model. arXiv preprint arXiv:2402.06852. Cited by: §2, §3.6. [20] Z. Zhao, D. Ma, L. Chen, L. Sun, Z. Li, Y. Xia, H. Xu, Z. Zhu, S. Zhu, S. Fan, et al. (2024) Chemdfm: a large language foundation model for chemistry. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, Cited by: §2, §3.6. [21] L. Sun, Y. Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu (2024) SciEval: a multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §2. [22] P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [23] Y. Zhou, M. Chen, Q. Mao, Y. Luo, Q. Liu, Y. Li, X. Zhang, D. Liu, X. Li, and E. Chen (2025) Benchmarking multimodal llms on recognition and understanding over chemical tables. arXiv preprint arXiv:2506.11375. Cited by: §2. [24] B. Yang, Y. Zhang, D. Liu, A. Freitas, and C. Lin (2025) Does table source matter? benchmarking and improving multimodal scientific table understanding and reasoning. arXiv preprint arXiv:2501.13042. Cited by: §2. [25] Y. Zhang, R. Yu, K. Zeng, D. Li, F. Zhu, X. Yang, Y. Jin, and Y. Xu (2024) Text-augmented multimodal llms for chemical reaction condition recommendation. arXiv preprint arXiv:2407.15141. Cited by: §2, §2. [26] Y. Zhang, Y. Han, S. Chen, R. Yu, X. Zhao, X. Liu, K. Zeng, M. Yu, J. Tian, F. Zhu, et al. (2025) Large language models to accelerate organic chemistry synthesis. Nature Machine Intelligence 7 (7), p. 1010–1022. Cited by: §2. [27] C. Zhang, Q. Lin, B. Zhu, H. Yang, X. Lian, H. Deng, J. Zheng, and K. Liao (2025) SynAsk: unleashing the power of large language models in organic synthesis. Chemical science 16 (1), p. 43–56. Cited by: §2. [28] A. L. Haywood et al. (2024) When yield prediction does not yield prediction: an overview of the current challenges. Journal of Chemical Information and Modeling. Cited by: §2. [29] M. C. Ramos et al. (2025) A review of large language models and autonomous agents in chemistry. Chemical Science. Cited by: §2. [30] X. Wang, C. Hsieh, X. Yin, J. Wang, Y. Li, Y. Deng, D. Jiang, Z. Wu, H. Du, H. Chen, et al. (2023) Generic interpretable reaction condition predictions with open reaction condition datasets and unsupervised learning of reaction center. Research 6, p. 0231. External Links: Document Cited by: §2. [31] M. Andronov, V. Voinarovska, N. Andronova, M. Wand, D. Clevert, and J. Schmidhuber (2023) Reagent prediction with a molecular transformer improves reaction data quality. Chemical Science 14, p. 3235–3246. External Links: Document Cited by: §2. [32] J. Lu and Y. Zhang (2022) Unified deep learning model for multitask reaction predictions with explanation. Journal of Chemical Information and Modeling 62 (6), p. 1376–1387. External Links: Document Cited by: §2. [33] Y. Qian, Z. Li, Z. Tu, C. W. Coley, and R. Barzilay (2023) Predictive chemistry augmented with text retrieval. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 12731–12745. Cited by: §2. [34] C. Yang, J. Lu, H. Wan, J. Yu, and F. Qin (2025) From what to why: a multi-agent system for evidence-based chemical reaction condition reasoning. arXiv preprint arXiv:2509.23768. Cited by: §2. [35] J. A. Garrido Torres, S. H. Lau, P. Anchuri, J. M. Stevens, J. E. Tabora, J. Li, A. Borovika, R. P. Adams, and A. G. Doyle (2022) A multi-objective active learning platform and web app for reaction optimization. Journal of the American Chemical Society 144 (43), p. 19999–20007. External Links: Document Cited by: §2. [36] J. Y. Wang, J. M. Stevens, S. K. Kariofillis, M. Tom, D. L. Golden, others, and A. G. Doyle (2024) Identifying general reaction conditions by bandit optimization. Nature 626 (8001), p. 1025–1033. External Links: Document Cited by: §2. [37] J. W. Sin, S. L. Chau, R. P. Burwood, K. Püntener, R. Bigler, and P. Schwaller (2025) Highly parallel optimisation of chemical reactions through automation and machine intelligence. Nature Communications 16, p. 6464. External Links: Document Cited by: §2. [38] K. Rajan, H. O. Brinkhaus, M. Sorokina, A. Zielesny, and C. Steinbeck (2021) DECIMER-segmentation: automated extraction of chemical structure depictions from scientific literature. Journal of Cheminformatics 13, p. 20. Cited by: NeurIPS Paper Checklist, §3.2. [39] K. He, G. Gkioxari, P. Dollár, and R. Girshick (2017) Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 2961–2969. External Links: Document Cited by: §3.2. [40] H. Yang, J. Wu, J. Wang, X. Wei, L. Ma, L. Li, C. Zhu, X. Wu, Y. Lu, Z. Zhu, et al. (2026) MinerU.Chem: a high-precision system for optical chemical structure and reaction recognition. arXiv preprint arXiv:2608.03525. External Links: Document, Link Cited by: §3.2. [41] G. Landrum (2024) RDKit: open-source cheminformatics. Note: https://w.rdkit.org Cited by: NeurIPS Paper Checklist, §3.2. [42] S. Kim, J. Chen, T. Cheng, A. Gindulyte, J. He, S. He, Q. Li, B. A. Shoemaker, P. A. Thiessen, B. Yu, L. Zaslavsky, J. Zhang, and E. E. Bolton (2025) PubChem 2025 update. Nucleic Acids Research 53 (D1), p. D1516–D1525. External Links: Document Cited by: NeurIPS Paper Checklist, §3.2. [43] DeepSeek-AI (2026) DeepSeek-V4. Note: https://w.deepseek.com Cited by: §3.6. [44] Moonshot AI (2026) Kimi K2.6. Note: https://kimi.moonshot.cn Cited by: §3.6. [45] Zhipu AI (2026) GLM 5.1. Note: https://w.zhipuai.cn Cited by: §3.6. [46] Qwen Team (2025) Qwen3 technical report. Note: https://qwenlm.github.io Cited by: §3.6. [47] Y. Zou, D. Zhu, L. Zhu, T. Zhu, Y. Zhou, P. Zhou, X. Zhou, D. Zhou, Z. Zhou, Y. Zhou, et al. (2026) Intern-s1-pro: scientific multimodal foundation model at trillion scale. arXiv preprint arXiv:2603.25040. Cited by: §3.6. [48] MiniMax Team (2026) MiniMax m2.7: early echoes of self-evolution. Note: https://w.minimax.io/news/minimax-m27-enMiniMax Official Blog Cited by: §3.6. [49] Anthropic (2026) Claude Opus 4.7. Note: https://w.anthropic.com Cited by: §3.6. [50] OpenAI (2026) GPT-5.5. Note: https://openai.com Cited by: §3.6. [51] Google DeepMind (2026) Gemini 3.1 Pro. Note: https://deepmind.google Cited by: §3.6. [52] K. Rajan, H. O. Brinkhaus, M. I. Agea, A. Zielesny, and C. Steinbeck (2023) DECIMER.ai: an open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications. Nature Communications 14, p. 5045. Cited by: NeurIPS Paper Checklist. [53] J. Wang, Y. He, H. Yang, J. Wu, L. Ge, X. Wei, Y. Wang, L. Li, H. Ao, C. Liu, et al. (2025) GTR-cot: graph traversal as visual chain of thought for molecular structure recognition. arXiv preprint arXiv:2506.07553. Cited by: NeurIPS Paper Checklist. Appendix A Question Sub-Types and Worked Examples See Table 1 of the main paper for the formal sub-type definitions and per-sub-type counts. We work through one MC item (MC_single; dataset type single_varying) and one FIB item (all_blank) below; the MC_all variants share the same payload structure with the candidate set widened to all varying axes, and the with-precedents and matched-control variants of each sub-type share the same item structure, differing only by the presence of a Reference Conditions And Outcomes block (cf. Appendix F). Formal candidate sets. Let ℛtR_t be the distinct observed rows of table t and tK_t its varying keys. For MC_single, target k varies within r∈ℛt:πt∖k(r)=c\r _t: _K_t \k\(r)=c\, where the observed tuple c maximizes distinct target-value coverage. MC_all uses the distinct projections πt(r):r∈ℛt\ _K_t(r):r _t\. FIB exposes marginal lists q,k=πk(r):r∈ℛtC_q,k=\ _k(r):r _t\, but its observed-row support is only ℒq=πt(r):r∈ℛt⊆∏kq,kL_q=\ _K_t(r):r _t\ _kC_q,k. A.1 Worked example: MC_single We illustrate MC_single with a high-discrimination item drawn from Org. Lett. 10.1021/acs.orglett.4c04736 (Wang et al., 2025), question id 10_1021_acs_orglett_4c04736_1_table_0_q2. The full question payload (input, options, and gold label) is included in the public human-reviewed test set under this question id. Reaction (constant background). Pd-catalysed 3,4-addition of pyrazole to a phenyl-carbamate-protected 1,2-dihydropyridine, with all conditions except the ligand held fixed: catalyst [Pd(η3-C3H5)Cl]2[Pd(η^3-C_3H_5)Cl]_2, solvent PhCF3, temperature 50 ∘C. Target axis and options. The model is asked to choose the optimal value of the single varying axis ligand from four candidates, all four of which are real entries in Table 1 of the cited paper: [0] BINAP, [1] DPPF, [2] Dpephos, [3] PPh3. Their measured target-product yields are 35%, 21%, 96%, and 0%, respectively. Gold: [2] Dpephos; the next-best option, BINAP, has relative score r≈0.36r\!≈\!0.36. Model behaviour. The three Proprietary LLMs, ChemVLM, and DeepSeek-V4-Pro recover the gold ligand. ChemLLM, ChemDFM, GLM-5.1, and Kimi-K2.6 choose BINAP; MiniMax-M2.7 chooses DPPF; and Intern-S1-Pro and Qwen3.5-397B-A17B choose PPh3. The item therefore discriminates among closely related ligand choices whose target-product yields differ substantially. Aggregated per-model behaviour across all sub-types is reported in Section 4.1. A.2 Worked example: FIB all_blank We illustrate FIB all_blank with a 3-key item drawn from Org. Lett. 10.1021/acs.orglett.5c02358 (SI Table S2, p. S14), question id 10_1021_acs_orglett_5c02358_si_13_table_0_q1. The full question payload (input, candidate values, and gold label) is included in the public human-reviewed test set under this question id. Reaction (constant background). Photoredox spirocyclopropanation of N-phenyl oxamic acid (1a) with methyl 2-(chloromethyl)acrylate (2a), with photocatalyst 4CzIPN (8 mol%), base KOH, light source 455 nm LED, atmosphere N2. Query keys and candidate values. The model is asked to fill three blanks simultaneously, each constrained to a small enumerated candidate list: temperature 35,40∘\35,40\\, C, time 24,12\24,12\ h, and solvent DMF,DMSO\DMF,DMSO\. The Cartesian product contains eight tuples, while retained measured-row support contains four: (35 ∘C, 24 h, DMF) at 26%, (35 ∘C, 24 h, DMSO) at 80%, (35 ∘C, 12 h, DMSO) at 36%, and (40 ∘C, 24 h, DMSO) at 57%. Gold: (35 ∘C, 24 h, DMSO) with 80% yield. The other four Cartesian combinations are well-formed but do not correspond to a retained measured row, so they are non-Hits and receive no Score∣\, \,Hit. This example directly illustrates why marginally valid field choices need not compose into an observed tuple. Model behaviour. All three Proprietary LLMs plus kimi-k2.6 recover the gold combination. GLM-5.1, MiniMax-M2.7, and Qwen3.5-397B-A17B select the observed but suboptimal (35 ∘C, 24 h, DMF) tuple. ChemDFM, ChemVLM, and DeepSeek-V4-Pro select the unobserved (40 ∘C, 24 h, DMF) tuple, while ChemLLM and Intern-S1-Pro select the unobserved (40 ∘C, 12 h, DMSO) tuple. Thus the errors separate suboptimal observed-row selection from recombinations outside observed support. Appendix B Human Verification and Anti-Leakage Safeguards B.1 Annotation interface The per-table review interface operationalises the expert-review gate of the extraction pipeline (Section 3.2). During local review, a lawfully accessed source crop is displayed in a non-released pane beside editable structured fields for reactants and products (content names, SMILES, idt labels, amounts, yields, conversion rate, diastereomeric/enantiomeric ratio, and free-text other_info). The toolbar (Accept / Save edits / Skip / Reject / Reset / Prev / Next / Next pending / List) lets reviewers triage tables in order, patch any field inline before saving, and persist the verdict together with a timestamp. Only entries with status accepted flow downstream to MC and FIB question construction; rejected and skipped tables are excluded from the released benchmark. Verification was performed by six paid interns based in mainland China: four master’s-level interns received RMB 300 per working day and two doctoral-level interns received RMB 400 per working day. B.2 Anti-leakage safeguards Sources are restricted to papers first published in 2025, and the released metadata retain DOI and first-publication provenance. This recency lowers but cannot eliminate exact-source exposure: hosted and open models have heterogeneous and partly undisclosed training cutoffs, so we do not claim a contamination-free evaluation. All models receive the same frozen question files and candidate permutations, and the release manifest hashes the distributed files. Future versioned refreshes can support temporal contamination audits. Appendix C Benchmark Composition Across Reaction, Condition, and Product Dimensions Counting unit and audit protocol. The 4,773 benchmark questions are derived from 552 source screening tables, so question-level counts would repeatedly count the same chemistry. We therefore use one parent table as the unit throughout this section (N=552N=552). Each table was assigned one of 20 mutually exclusive primary transformations and independent multi-label energy-input and catalyst-class annotations. Two blind Gemini-3.1-Pro-Low passes were followed by field-level blinded adjudication; all 10 flagged records were manually checked against the reviewed table record and, when needed, the publisher source. The released audit contains all per-table decisions and leaves no unresolved record. Reaction and product composition. Figure 6 summarizes complementary aspects of the benchmark chemistry. All 20 primary transformation classes are represented, and the largest accounts for 16.7% of parent tables. The corpus also spans thermal, photochemical, and electrochemical activation together with multiple catalyst regimes. All 552 tables contain a valid target-product structure, yielding 341 unique canonical products and 232 non-empty Bemis–Murcko scaffolds after deduplication. Radius-2 Morgan-fingerprint internal diversity is 0.875. Figure 6: Benchmark composition across reaction and product dimensions. A, mutually exclusive primary transformation classes over 552 parent screening tables. B–C, multi-label energy inputs and catalyst classes; labels report table count and share of all parent tables. D, family sizes over 232 non-empty Bemis–Murcko scaffolds from 341 unique canonical target products. Condition-space structure. The structured schema contains 26 condition fields (Figure 7). Solvent, temperature, time, reagent, and catalyst are the most frequently represented families. A parent table varies a median of three condition axes and reports a median of seven experimentally observed condition combinations, while candidate-set size varies by field. The four panels distinguish field coverage, per-field branching, table dimensionality, and observed screen size. Figure 7: Condition-space structure (N=552N=552). A, fraction of parent tables in which each major condition family is varied/queried or held constant. B, candidate-set arity for varied fields (boxes show quartiles; whiskers show the non-outlier range). C, number of varied condition axes per parent table. D, number of experimentally observed condition combinations per source table. Scope. The 2025 sampling frame is a deliberate target distribution rather than an estimate of historical publication prevalence. Recent methodology papers feature contemporary transformations, catalytic systems, and condition regimes, and their screening tables provide expert-designed local decision spaces grounded in measured experiments. RxnOptBench therefore probes whether models transfer chemical knowledge to recent, potentially unfamiliar condition-selection problems rather than only recall canonical recipes. Recency lowers exact source-exposure risk relative to older corpora but cannot guarantee contamination-free evaluation, and we do not assume that every transformation constitutes a new reaction class. Together, the analyses above characterize substantial reaction, condition, and product heterogeneity within this intended distribution across 552 parent screening tables. Appendix D Benchmark Validity and Robustness Audits Disclosed outcome variation and narrative selection. RxnOptBench evaluates only the candidates disclosed in each source table; it does not interpret the table as a complete optimization history or its best row as a global chemical optimum. Publication curation defines the sampling frame, but it does not alter the measured outcomes or within-table ordering used as labels. For a static offline benchmark without additional wet-lab validation, only observed joint rows have defensible outcome scores; unreported combinations remain outside measured support rather than becoming failures or negative examples. On the fixed FIB task set, after format/index invalidity is separated, lower Hit therefore diagnoses difficulty coordinating multiple field choices into an experimentally observed joint row; it does not adjudicate the wet-lab feasibility of non-Hits. Within this conditional task, the disclosed outcomes are not narrowly concentrated: across the 552 parent tables, the best-to-worst yield difference has median 69 percentage points (IQR 40.75–83; range 1–100). In total, 527/552 tables (95.5%) span at least 10 points, 493 (89.3%) span at least 20, and 383 (69.4%) span at least 50. When tables are stratified into quartiles by yield spread, the ordering of the 12 models remains positively associated with the full-set ordering (Spearman ρ=0.755,0.895,0.902,0.965ρ=0.755,0.895,0.902,0.965 from the lowest to highest quartile), and the Proprietary, Open Source, and Chemistry model tiers retain the same order in every quartile. Thus, publication curation bounds claims about exhaustive coverage, while the reported model-tier conclusion is not confined to tables with negligible disclosed outcome differences. Scoring sensitivity and multi-objective view. The multiplicative score is the benchmark’s headline aggregate utility: a monotone, bounded summary that provides one continuous ranking while preserving partial credit. It is not asserted to encode every chemical objective or to be the unique preference function for experimental optimization. We therefore re-score MC_all in three ways. First, replacing the product by yield-only scoring gives a model-rank Spearman correlation of 0.9650.965 across all 12 models. Second, 43 tables report at least one selectivity axis for every option, permitting complete objective vectors consisting of normalized yield and all complete major-fraction selectivity axes. On these tables, equal-weight additive and maximin utilities give rank correlations of 0.9650.965 and 0.9230.923, respectively, with the product score. Third, for those 43 tables, we compute a per-prediction Pareto flag across all 12 models without collapsing the objective vector. We retain the product score for a single headline ranking and provide these objective vectors and alternatives as complementary diagnostics. These are post hoc sensitivity and multi-objective diagnostics for the present single-selection task, not a claim that RxnOptBench implements full multi-objective optimization. A full formulation would require substantially more tables with complete candidate-level yield and selectivity reporting and a different task definition, such as Pareto-front recovery or recommendation under explicit preferences and constraints. The archived evaluation already used the signed-e and ratio-direction normalization defined in Section 3.4, so Table 2 retains the archived snapshot values. Separately, the evaluation parser was hardened to reject numeric compound labels such as 3a:4a as ratios. A disclosed maintenance recomputation changes only ChemLLM FIB Strict Accuracy from the frozen 6.6% to 5.3%; we report this audit rather than silently replacing the archived result. Candidate-position audit. All models receive the same shuffled candidate order. MC bundles are deterministically shuffled per question; FIB candidate values are independently shuffled per question and condition key with a stable hash and seed 20260504. The exact permutations are stored in the public test-set records. The observed frequency with which a gold answer includes index 0 is 375/1,039 (36.09%) for MC_single, 114/552 (20.65%) for MC_all, and 708/1,912 (37.03%) across FIB key occurrences. Because option arity and ties vary by item, their corresponding arity-conditional uniform expectations are 36.50%, 17.75%, and 36.41%; two-sided Monte Carlo tests give p=0.786p=0.786, 0.0630.063, and 0.5690.569, respectively. Thus none differs significantly from the conditional shuffled null at 0.050.05. The public evaluation code reproduces the complete index counts and percentages stratified by task and candidate arity. Inference reproducibility boundary. Formal runs freeze datasets, prompts, candidate permutations, model IDs, and generation settings and disable sampling (temperature=0 or provider greedy/non-sampling mode). The public evaluator deterministically scores any supplied predictions against the fixed test set. Extraction code, benchmark-construction code, and the archived model outputs used for the paper are not part of the public release. Hosted backends may not reproduce generations bit-for-bit; repeated complete API calls would measure endpoint-level run-to-run variation, and we do not claim that hosted-model responses are invariant across such calls. Appendix E License, Hosting, Intended Use, and Broader Impact The 232 papers and their supporting information were manually downloaded by the authors through open-access channels or university institutional subscriptions; no automated crawling or bulk downloading from publisher websites was used. Processing was performed locally. Before final public release, we will document the acquisition route and verify the applicable publisher TDM term, permission, or legal exception for each source category; records without a confirmed release basis will be excluded. The release is limited to the evaluation code and final human-reviewed benchmark test-set records, which retain DOI/table/page provenance. It does not redistribute the extraction pipeline, benchmark-construction code, review tooling, archived model outputs, raw PDFs, supporting-information files, rendered page/table images, publisher-formatted text, or verbatim source text. The test-set records will be available through a lightweight research-use agreement requiring source attribution, compliance with applicable laws and source terms, and acknowledgement of the intended evaluation/research use. Final license choices, persistent identifiers, and the integrity-manifest hash will be fixed at camera-ready time. Broader impact and misuse mitigation. Machine-readable aggregation lowers the effort required to locate and reuse published reaction-condition information. The records could therefore be misused as unvalidated condition recommendations outside their source context. Users will be instructed not to treat benchmark or model outputs as direct wet-lab guidance without qualified expert review and independent experimental validation. The research-use agreement, retained source provenance, explicit intended-use statement, and a channel for reporting misuse or rights concerns provide accountability and friction, although they cannot eliminate misuse. Using benchmark items as training data would also invalidate future evaluation on RxnOptBench. Appendix F Prompt Templates Although RxnOptBench has eight evaluated subsets (Table 1), they share three prompt templates: mc_single, mc_all, and fib_all, the renderer aliases for MC_single, MC_all, and FIB all_blank, respectively. The reference vs. control distinction is purely a toggle on a single Reference Conditions And Outcomes section — when present, 2–5 prior wet-lab rows are appended; when absent, every other section is identical. This section therefore reproduces the three full prompts in their reference variant; removing the Reference Conditions And Outcomes block recovers the matched control prompt verbatim. Every submitted prompt contains a one-line instruction, Reaction (reactants and target product as name + SMILES), and Constant Conditions, i.e. table-level invariants shared by all source-table rows. The mc_single templates additionally include Fixed Conditions: question-level locked conditions that may vary elsewhere in the table but are held at one observed combination for the selected subgroup. Reference variants add Reference Conditions And Outcomes; each template then gives its candidate set and output format. An audit of the actual task templates and all eight evaluated subsets confirms that all 34,025 stored constant-condition keys and values are rendered across the 4,773 prompts. From the reviewer-archive root, the audit is reproduced bypython scripts/validate_prompt_context.py F.1 mc_single Sample id: 10_1002_anie_202416132_si_5_table_0_q1_ref; expected output: 1. ⬇ You are an expert organic chemist. Given the following reaction, one real fixed condition group from the source table, and several reference condition sets with observed outcomes, choose the best value for **Constant_Current** from the remaining candidate options. ### Reaction **Reactants:** - 2,2-difluoro-3-phenyl-2H-indole (SMILES: FC1(F)C(c2c2)=Nc2c21) **Target Product:** - 2-fluoro-3-phenyl-1H-indole (SMILES: Fc1c(-c2c2)[nH]c2c12) ### Constant Conditions (do not change) - Anode(+): Zn - Cathode(-): C - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - reaction_temperature: rt - speed: 800 rpm ### Fixed Conditions - reaction_time: 4 h - reagents: PS-750-M - solvents: water (SMILES: O) ### Reference Conditions And Outcomes *Reference 1 (yield=59.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 10 mA - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - reaction_temperature: rt - reaction_time: 18 h - reagents: (none) - solvents: 3 wt % aq. PS-750-M - speed: 800 rpm *Reference 2 (yield=92.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 20 mA - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - reaction_temperature: rt - reaction_time: 4 h - reagents: PS-750-M - solvents: (none) - speed: 800 rpm ### Candidate Values For Constant_Current **Option 0:** - Constant_Current: 10 mA **Option 1:** - Constant_Current: 30 mA ### Output Format Respond with ONLY the option index (0-based integer). Do not include any reasoning, explanation, or additional text — just the integer. F.2 mc_all Sample id: 10_1002_anie_202416132_si_4_table_1_q6_ref; expected output: 3. ⬇ You are an expert organic chemist. Given the following reaction and several reference condition sets with observed outcomes, choose the best remaining reaction-condition option. ### Reaction **Reactants:** - 2,2-difluoro-3-phenyl-2H-indole (SMILES: FC1(F)C(c2c2)=Nc2c21) **Target Product:** - 2-fluoro-3-phenyl-1H-indole (SMILES: Fc1c(-c2c2)[nH]c2c12) ### Reference Conditions And Outcomes *Reference 1 (yield=27.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Electrolyte: n-Bu4NClO4 (0.1 M) - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - reagents: n-Bu4NClO4 (SMILES: C[N+](C)(C)C.[O-][Cl+3]([O-])([O-])[O-]) - scale: (none) - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions"] - solvents: 3 wt % aq. PS-750-M - speed: 800 rpm *Reference 2 (yield=32.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Electrolyte: NaI (0.1 M) - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - reagents: NaI (SMILES: [I-].[Na+]) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] - solvents: PS-750-M - speed: 800 rpm *Reference 3 (yield=42.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Electrolyte: n-Bu4NBr (0.1 M) - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - reagents: n-Bu4NBr (SMILES: C[N+](C)(C)C.[Br-]) - scale: (none) - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "indeterminate_descriptions"] - solvents: 3 wt % aq. PS-750-M - speed: 800 rpm ### Candidate Conditions **Option 0:** - Electrolyte: NaBr (0.1 M) - reagents: NaBr (SMILES: [Br-].[Na+]) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "current_conditions", "reaction_time", "reaction_temperature", "speed", "scale"] - solvents: aq. PS-750-M **Option 1:** - Electrolyte: n-Bu4NPF6 (0.1 M) - reagents: n-Bu4NPF6 (SMILES: C[N+](C)(C)C.F[P-](F)(F)(F)(F)F) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] - solvents: PS-750-M **Option 2:** - Electrolyte: NaBF4 (0.1 M) - reagents: NaBF4 (SMILES: F[B-](F)(F)F.[Na+]) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] - solvents: PS-750-M **Option 3:** - Electrolyte: n-Bu4NI (0.1 M) - reagents: n-Bu4NI (SMILES: C[N+](C)(C)C.[I-]) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] - solvents: 3 wt % aq. PS-750-M **Option 4:** - Electrolyte: NaClO4 (0.1 M) - reagents: NaClO4 (SMILES: [Na+].[O-][Cl+3]([O-])([O-])[O-]) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] - solvents: PS-750-M ### Output Format Respond with ONLY the option index (0-based integer). F.3 fib_all Sample id: 10_1002_anie_202416132_si_4_table_1_q1_ref; expected output: a JSON object with one integer per condition key (e.g. "Electrolyte": 0, "reagents": 0, "scale": 0, "selected_conditions": 0, "solvents": 0). ⬇ You are an expert organic chemist. Given the following reaction, several reference condition sets with observed outcomes, and a candidate index table for each condition category, choose one candidate index for each required condition category. ### Reaction **Reactants:** - 2,2-difluoro-3-phenyl-2H-indole (SMILES: FC1(F)C(c2c2)=Nc2c21) **Target Product:** - 2-fluoro-3-phenyl-1H-indole (SMILES: Fc1c(-c2c2)[nH]c2c12) ### Constant Conditions - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - speed: 800 rpm ### Reference Conditions And Outcomes *Reference 1 (yield=27.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Electrolyte: n-Bu4NClO4 (0.1 M) - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - reagents: n-Bu4NClO4 (SMILES: C[N+](C)(C)C.[O-][Cl+3]([O-])([O-])[O-]) - scale: (none) - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions"] - solvents: 3 wt % aq. PS-750-M - speed: 800 rpm *Reference 2 (yield=32.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Electrolyte: NaI (0.1 M) - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - reagents: NaI (SMILES: [I-].[Na+]) - scale: 0.25 mmol - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] - solvents: PS-750-M - speed: 800 rpm *Reference 3 (yield=42.0%):* - Anode(+): Zn - Cathode(-): C - Constant_Current: 30 mA - Electrolyte: n-Bu4NBr (0.1 M) - Main_Mode: Constant - Reaction_Type: Integrated Cell Reaction - other_info: undivided cell, Electrolysis conducted with DC power supply using Electrasyn 2.0 - reaction_temperature: rt - reaction_time: 4 h - reagents: n-Bu4NBr (SMILES: C[N+](C)(C)C.[Br-]) - scale: (none) - selected_conditions: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "indeterminate_descriptions"] - solvents: 3 wt % aq. PS-750-M - speed: 800 rpm Important: the reference condition sets shown above are NOT valid answers for this question. Do NOT copy any reference combination directly. You must propose a NEW combination by selecting candidate indices from the table below. Only combinations formed from the candidate index table are considered valid outputs. ### Candidate Values **Electrolyte:** 0: n-Bu4NI (0.1 M) 1: n-Bu4NPF6 (0.1 M) 2: NaClO4 (0.1 M) 3: NaBr (0.1 M) 4: NaBF4 (0.1 M) **reagents:** 0: n-Bu4NI (SMILES: C[N+](C)(C)C.[I-]) 1: n-Bu4NPF6 (SMILES: C[N+](C)(C)C.F[P-](F)(F)(F)(F)F) 2: NaClO4 (SMILES: [Na+].[O-][Cl+3]([O-])([O-])[O-]) 3: NaBr (SMILES: [Br-].[Na+]) 4: NaBF4 (SMILES: F[B-](F)(F)F.[Na+]) **scale:** 0: 0.25 mmol **selected_conditions:** 0: ["reagents", "solvents", "reaction_time", "reaction_temperature", "speed", "current_conditions", "scale"] 1: ["reagents", "solvents", "current_conditions", "reaction_time", "reaction_temperature", "speed", "scale"] **solvents:** 0: 3 wt % aq. PS-750-M 1: PS-750-M 2: aq. PS-750-M ### Output Format Respond strictly in the following JSON format (no extra text): “‘json "Electrolyte": <candidate_index>, "reagents": <candidate_index>, "scale": <candidate_index>, "selected_conditions": <candidate_index>, "solvents": <candidate_index> “‘ Each value must be a single integer candidate index, not a condition name. Appendix G Illustrative Failure Cases Aggregate accuracy hides item-level behavior. We therefore selected four single-reaction probes (Figure 8) to illustrate item-level behaviors that are not visible in Table 2 or Table 3. These cases are qualitative examples, not an exhaustive taxonomy, prevalence estimate, or causal validation of the underlying mechanism. Figure 8: Per-model predictions on four illustrative cases. Each card shows the target reaction, options (gold ★ ), and the 12 models’ picks (green == correct, red == wrong). M1/M2: MC_single items; M3: paired MC_single/MC_all on the same parent reaction; M4: FIB without vs. with a sibling precedent. (M1) Common-condition selection. On a photoredox optimisation (gold == 4CzIPN at idx =2=2), 5/125/12 models — the Chemistry trio plus Qwen3.5 and Kimi — select the widely used Ir[dF(CF3)ppy]2(dtbpy)PF6 at idx =0=0. This pattern is consistent with a strong default prior, but the example alone cannot distinguish that explanation from other causes. (M2) Index-zero selection. A BF3-catalysed azulene rearrangement places non-default solvents at both idx =0=0 and idx =3=3; the Chemistry trio (plus MiniMax-M2.7) select idx =0=0 while several frontier models converge on the gold. Together with the corpus-level index-frequency diagnostic, this is consistent with position sensitivity; it does not establish chemistry-domain finetuning as its cause. (M3) Combinatorial brittleness. The same parent reaction is asked as a 2-option catalyst question (β-ICD vs. (DHQD)2AQN) and as a 10-option catalyst×solvent question (gold == β-ICD++THF). Three strong models recover the catalyst alone, but every scoring model fails the joint version — single-variable recall does not compose into joint-condition accuracy. This brittleness echoes the aggregate pattern in Figure 4: precedents lift MC_single Accuracy by +13+13–1818 p across frontier models, yet on MC_all the same precedents go neutral or negative — a signal that disambiguates one axis becomes distractor noise once the model must couple two. (M4) Precedent-anchored rescue. A Pd-catalysed cross-coupling (gold catalyst == PdCl2(dppf)⋅·DCM) jumps from 1/121/12 to 10/1210/12 strict when a sibling precedent is supplied, illustrating that a relevant observation can substantially alter the decision. M3–M4 also leave aggregate traces in Table 2 and Figures 4–5; M1–M2 are illustrated here only at the item level. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction claim (i) a real-data benchmark for reaction-condition optimization with paired precedents/no-precedents items, (i) a continuous yield-and-selectivity scoring scheme, (i) public release of the final human-reviewed benchmark test set and evaluation code, and (iv) a systematic evaluation across nine frontier and three Chemistry LLMs showing substantial headroom. Each claim is supported in the paper: (i) by Section 3.3 and Section 3.5; (i) by Section 3.4 (Eq. 1); (i) by Section 3.2 and Appendix E; and (iv) by the results in Section 4.1. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 5 (Conclusion and Future Work) acknowledges current scope and operational caveats through the refinements it commits to in subsequent releases: (a) coverage gaps in undersampled condition families (electrochemistry, photochemistry, flow chemistry, biocatalysis) and in non-English methodology journals; (b) the order-dependence of the multi-component condition comparator (e.g., reagent or solvent mixtures), which a planned order-invariant FIB comparator will address; (c) the single-number nature of the effective score, which the planned multi-objective scoring layer will extend; and (d) the gap exposed by case study M4 between possessing and deploying a relevant prior, which the planned retrieval-augmented evaluation setting targets. Periodic RxnOptBench refreshes drawn from newly published methodology papers further mitigate training-corpus contamination as the field evolves. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [N/A] Justification: The paper makes no formal theoretical claims; it is an empirical benchmark paper. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper? Answer: [Yes] Justification: The four-stage extraction methodology is described in Section 3.2; question construction in Section 3.3; metrics with closed-form definitions in Section 3.4 (Eq. 1); and the inference protocol in Section 3.6. The public release contains the final human-reviewed benchmark test set and deterministic evaluation code, allowing supplied predictions to be scored consistently. Extraction and benchmark-construction implementations and the archived model outputs used in the paper are not part of the public release; exact regeneration from raw PDFs or exact replay of hosted-model responses is not claimed. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The public assets are limited to the final human-reviewed benchmark test-set records and evaluation code, with instructions for formatting predictions and reproducing benchmark scores. The extraction pipeline, benchmark-construction code, review tooling, and archived model outputs used in the paper are not public. The test-set records will use a lightweight research-use agreement in response to the identified aggregation risk. Access terms, hosting, intended use, and worked examples are documented in Appendices E and A. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: No model is trained in this work — all 12 evaluated models are queried as-is. Inference details are reported in Section 3.6 (single shared prompt template per question type, greedy or provider non-sampling mode, and a single sample per question). The 4,773 questions form a single test set; train/validation splits are not applicable. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Models are queried once per question with sampling disabled, which does not yield seed-level variance estimates. All models are evaluated on the same 4,773-item benchmark. The headline no-precedents subsets contain 1,039 MC_single, 552 MC_all, and 552 FIB questions; the remaining items form the matched precedents comparisons. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Open-weight models were served on local GPU nodes and proprietary models were queried through their public APIs. The evaluation consists of one non-sampling pass over the fixed 4,773-question test set; the model serving regime and pipeline components are described in Sections 3.6 and 3.2. 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics? Answer: [Yes] Justification: Source documents were manually obtained through open-access channels or university institutional subscriptions for noncommercial research; no publisher-site crawling was used and no source PDFs, images, or verbatim text are redistributed. Applicable publisher TDM and release terms will be verified before final release. No personal or sensitive data are included. Verification was performed by six paid interns in mainland China at the disclosed rates (Appendix B.1); no human-subjects research is involved. 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The benchmark supports evidence-based evaluation of LLM condition recommendation and reduces overclaiming from shallow proxies. Appendix E now also explains that machine-readable aggregation lowers the barrier to reuse and may enable unvalidated wet-lab recommendations. Mitigations include a research-use agreement, retained provenance, explicit expert-review and experimental-validation warnings, and a misuse/rights reporting channel. 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models with a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Although no pretrained model or source publication is redistributed, structuring published reaction-condition records lowers access friction. Final records will therefore use a lightweight research-use agreement requiring source attribution, compliance with applicable law/source terms, and acknowledgement that outputs require qualified expert review and independent experimental validation (Appendix E). 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and respected? Answer: [Yes] Justification: All 232 source papers are cited by DOI in the released metadata. External tooling — DECIMER-Segmentation [38], DECIMER [52], GTR-VL [53], RDKit [41], PubChem [42] — is cited in Section 3.2; their open-source licenses are respected and listed in Appendix E. Evaluated models are credited in the references and used under their published API terms or open-weight licenses (Section 3.6). 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: A datasheet covering motivation, composition, collection, preprocessing, uses, distribution, and maintenance will accompany the camera-ready release; license, hosting, and intended use are detailed in Appendix E. Worked question examples (Appendix A) and the human-verification protocol (Appendix B.1) document the test set and its curation methodology. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: The verification stage employed four master’s-level and two doctoral-level paid interns based in mainland China, compensated at RMB 300 and RMB 400 per working day, respectively. They were project collaborators rather than anonymous crowdworkers. The interface, decisions (accept / reject / edit), workflow, region, and rates are documented in Appendix B.1. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: Annotators review chemistry tables from public peer-reviewed papers; no human-subjects data is collected, no behavioral or biomedical experimentation is performed on the annotators, and no personal information is processed. IRB review is therefore not applicable under our institution’s policy. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Answer: [Yes] Justification: VLMs are a core component of the extraction pipeline (page selection, table understanding, label-to-structure binding, per-row verification) and their usage, prompting constraints, and human-review gating are described in detail in Section 3.2. LLM/VLM models evaluated as benchmark subjects are listed in Section 3.6. LLMs were not used to write the scientific content of the paper beyond standard editorial assistance.