Paper deep dive
MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models
Philippe Formont, Maxime Darrin, Ismail Ben Ayed, Pablo Piantanida
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/22/2026, 6:01:15 AM
Summary
MolRGen is a large-scale benchmark and dataset designed to train and evaluate reasoning-based Large Language Models (LLMs) for de novo molecular generation. It addresses the lack of ground-truth labels in de novo design by utilizing reward-based reinforcement learning and introduces a novel 'diversity-aware top-k score' to better evaluate the quality and chemical diversity of generated molecules.
Entities (5)
Relation Signals (3)
MolRGen → evaluates → LLMs
confidence 95% · we introduce MolRGen, a large-scale benchmark and dataset for training and evaluating reasoning-based LLMs
Diversity-aware top-k score → measures → Molecular Generation
confidence 95% · we introduce a novel diversity-aware top-k score that captures both the quality and diversity of generated molecules.
Reinforcement Learning → optimizes → Molecular Generation
confidence 90% · applying reinforcement learning to pretrained large language models (LLMs) has yielded empirical improvements... in molecular generation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in reasoning-based large language models (LLMs) have demonstrated substantial improvements in complex problem-solving tasks. Motivated by these advances, several works have explored the application of reasoning LLMs to drug discovery and molecular design. However, most existing approaches either focus on evaluation or rely on training setups that require ground-truth labels, such as molecule pairs with known property modifications. Such supervision is unavailable in \textit{de novo} molecular generation, where the objective is to generate novel molecules that optimize a desirability score without prior knowledge of high-scoring candidates. To bridge this gap, we introduce MolRGen, a large-scale benchmark and dataset for training and evaluating reasoning-based LLMs on \textit{de novo} molecular generation. Our contributions are threefold. First, we propose a setting to evaluate and train models for \textit{de novo} molecular generation and property prediction. Second, we introduce a novel diversity-aware top-$k$ score that captures both the quality and diversity of generated molecules. Third, we show our setting can be used to train LLMs for molecular generation, training a 24B LLM with reinforcement learning, and we provide a detailed analysis of its performance and limitations.
Tags
Links
- Source: https://arxiv.org/abs/2603.18256v1
- Canonical: https://arxiv.org/abs/2603.18256v1
Trouble viewing inline? Open PDF directly →
Full Text
90,656 characters extracted from source content.
Expand or collapse full text
MOLRGEN: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models Philippe Formont 1,2,3,4 Maxime Darrin 5 Ismail Ben Ayed 2,3,6 Pablo Piantanida 1,3,4,7 1 Universit ́ e Paris-Saclay 2 ́ ETS Montr ́ eal 3 ILLS – International Laboratory on Learning Systems 4 Mila – Quebec AI Institute 5 Mistral AI 6 LIVIA 7 CNRS, CentraleSup ́ elec Abstract Recent advances in reasoning-based large language models (LLMs) have demon- strated substantial improvements in complex problem-solving tasks. Motivated by these advances, several works have explored the application of reasoning LLMs to drug discovery and molecular design. However, most existing approaches either focus on evaluation or rely on training setups that require ground-truth labels, such as molecule pairs with known property modifications. Such supervision is unavailable in de novo molecular generation, where the objective is to generate novel molecules that optimize a desirability score without prior knowledge of high- scoring candidates. To bridge this gap, we introduce MOLRGEN, a large-scale benchmark and dataset for training and evaluating reasoning-based LLMs on de novo molecular generation. Our contributions are threefold. First, we propose a setting to evaluate and train models for de novo molecular generation and property prediction. Second, we introduce a novel diversity-aware top-kscore that captures both the quality and diversity of generated molecules. Third, we show our setting can be used to train LLMs for molecular generation, training a 24B LLM with reinforcement learning, and we provide a detailed analysis of its performance and limitations. 1 Introduction The difficulty in identifying molecules with the necessary properties to become viable drugs has motivated the development of a wide range of molecular generative models, enabling faster enumer- ation of the chemical space [Loeffler et al., 2024, Tripp and Hern ́ andez-Lobato, 2023]. This task is particularly challenging due to its inherently multi-objective nature: generated candidates must simultaneously satisfy multiple criteria, such as the ability to alter the expression of a specific protein, reduced toxicity, and adequate pharmacokinetic properties (e.g., oral bioavailability or ability to cross the intestinal barrier). These requirements are typically aggregated into a single reward or scoring function to facilitate optimization and decision-making. In this work, we focus on de novo molecular generation, a setting in which no high-scoring molecule is known a priori, and the generative algorithm must identify drug candidates with no prior information on which molecules are promising. Several approaches have been explored to address this challenge, with reinforcement learning (RL) emerging as a natural paradigm: an agent generates molecular compounds, which are then evaluated using a scoring function, often based on computational tools Preprint. Under review. arXiv:2603.18256v1 [cs.LG] 18 Mar 2026 such as docking software [Hassan et al., 2017, Verdonk et al., 2003], and the resulting scores guide future generations. Reinforcement learning approaches have recently gained prominence in natural language generative modeling following the release of new reasoning models that explicitly trace through intermedi- ate steps [DeepSeek-AI, 2025, Xu et al., 2025]. In particular, applying reinforcement learning to pretrained large language models (LLMs) has yielded empirical improvements, especially in mathe- matics and programming. Molecular design shares fundamental characteristics with these domains: it benefits from multi-step reasoning, where the model can decompose complex objectives into simpler sub-problems. For example, a reasoning-based approach could sequentially assess drug-likeness (e.g., molecular weight, LogP), synthetic accessibility (e.g., number of synthesis steps required), and predicted binding affinity before committing to a final molecular design. This structured decompo- sition of molecular design into interpretable reasoning steps may improve the quality of generated molecules, which we aim to explore in this work. Recent work has successfully applied LLMs to various chemical tasks, including caption-based molecular generation, understanding chemical reactions, molecular optimization (editing existing molecules to alter properties), and property prediction Narayanan et al. [2025], Dey et al. [2025]. These studies have demonstrated that LLMs can effectively learn molecular representations and reasoning patterns when ground-truth data is available. However, a critical limitation of existing approaches is their reliance on ground-truth labels (i.e., correct molecules to generate or correct property values), which do not exist for de novo generation where prior knowledge is unavailable. To the best of our knowledge, there exists no large-scale benchmark including de novo molecular generation tasks to train and evaluate reasoning-based LLMs with rewards computed at generation time. We therefore pose the following research question: Can the advances made by reasoning-based LLMs in fields such as mathematical reasoning or coding be reproduced in the context of de novo molecular generation using reward-based reinforcement learning? We summarize our contribution as follows: •We created a large-scale dataset for de novo molecular generation, leveraging 4.5k protein structures and completed it with molecular property prediction tasks. • We evaluate multiple open-source LLMs on our benchmark and introduce the diversity- aware top-k score, measuring how well a model generates sets of diverse, high-scoring molecules. •We show our dataset and rewards can be leveraged to train LLMs with reinforcement learning for de novo molecular generation, enabling the identification of a high-reward molecule on non-seen targets. We also study the limitations of this approach, notably the difficulty of exploring the chemical space. 2 Related Work Molecular Generation.Molecular generation has been a long-standing challenge in computational chemistry and drug discovery. Traditional approaches include generative models, such as Variational Autoencoders (VAEs) [G ́ omez-Bombarelli et al., 2018] and genetic algorithms [Tripp and Hern ́ andez- Lobato, 2023], which have demonstrated strong empirical results in generating novel molecules. Reinforcement learning (RL) based methods have emerged as a natural paradigm for generating drug candidates while optimizing for a desirability score. Notably, Reinvent [Loeffler et al., 2024] generates SMILES representations of molecules (textual representation) by adjusting the log-likelihood of a prior model using this desirability score. Other approaches propose generating molecules in different modalities, such as graph representations [Koziarski et al., 2024, Gao et al., 2024, Haddad et al., 2025] or 3D point clouds [Zhou et al., 2025]. BindGPT [Zholus et al., 2025] recently proposed to train a 100M parameter language model architecture, using GRPO to optimize its generation on 100 docking targets. While using a language model architecture, this model is not trained to generate or process natural language. Large Language Models in Chemistry. The application of large language models (LLMs) to chemistry has rapidly expanded, with early work assessing LLM knowledge in chemistry [Mirza et al., 2024], showing that the majority of these models outperform human scores. To further 2 docking (100.0%) docking + other (51.2%) docking + QED (26.2%) docking + SA (16.2%) 2-docking (6.2%) docking + QED + other (36.1%) docking + QED + SA (22.9%) docking + SA + other (22.9%) docking + other + other (9.6%) 2-docking + ... (8.4%) Number of prop. to optimize 1 Property 2 Properties 3 Properties (a) Optimization tasks for de novo generation. Representation of the optimization objectives in the dataset for de novo generation. The dataset is well-balanced between single-property (29%), 2-properties (36%), and 3-properties (35%) optimization. 0.00.20.4 Proportion docking QED SA logP ExactMolWt NumHBD TPSA NumAromaticR. NumHBA Phi FractionCSP3 HallKierAlpha NumRotatableB. objectives minimize above below maximize (b) Distribution of the proper- ties. Proportion of each prop- erty’s occurrence for the genera- tion tasks. advance the comprehension of chemistry by these models, various methods have been proposed to train and evaluate language models on fundamental tasks, such as SMILES-to-IUPAC translation, molecular captioning, and molecular generation from structural descriptions Yu et al. [2024], Kim et al. [2025]. Recent work has leveraged 3D molecular representations [Li et al., 2024] or diffusion models [Gong et al., 2024] to perform generation tasks. A critical distinction exists between caption- based generation and our approach. Caption-based generation aims to find compounds matching a given natural language description (e.g., “The molecule is a primary arylamine [...] and a member of anilines” from Mol-Instructions Fang et al. [2024]), while our de novo molecular generation objective seeks to create novel compounds that optimize a desirability score, rather than match a specific description. Reasoning in Drug Discovery. With the emergence of LLM reasoning models and their demon- strated capabilities on tasks such as mathematical problem-solving and coding, several works have investigated chain-of-thought reasoning for chemical and biological tasks. BioReason [Fallahpour et al., 2025] incorporated a multimodal architecture for DNA sequence tasks, combining fine-tuning on chain-of-thought reasoning from standard LLMs with GRPO optimization on ground-truth KEGG labels. For molecular applications, reasoning-enhanced models [Zhuang et al., 2025] leverage struc- tured reasoning to improve molecular property and toxicity predictions. Mol-R1 [Li et al., 2025], Ether0 [Narayanan et al., 2025], and Chem-DFM [Zhao et al., 2025] combine methods such as rejection sampling, distillation, supervised fine-tuning, or reinforcement learning to perform multiple chemical tasks, including chemical reaction understanding or caption-based molecular generation. GeLLMO [Dey et al., 2025] tackles a different problem: modifying existing molecules to change their properties, by using at training time a set of molecular pairs labelled with their property modifications. 3 A Benchmark for De Novo Molecular Generation 3.1 Notations A promptqis defined asq =ν i ,ρ i ,x i ,σ i 1≤i≤n props , where:ν i is the task objective (e.g., property maximization, class prediction),ρ i is a molecular property,x i is a reference molecule or reactants (for property prediction),σ i is a margin/threshold (e.g., property threshold or label standard deviation), and n props is the number of objectives in the prompt. Given a modelπ θ and promptq, a completiono ∼ π θ (q)is generated. We extract the answerˆo from tags<answer>and</answer>. The notationρ(ˆo)denotes the property value resulting from applying property/reaction ρ to the extracted completion (rescaled in [0, 1]). 3 Table 1: Reward and prompt definition. We propose different reward functions r(q, ˆo) depending on the objective ν of the task, that are designed to reflect thow close the extracted answer ˆo is to the desired outcome. All properties (and rewards) are normalized in[0, 1]. Accross all objectives, if the extracted answer ˆo is invalid the reward is 0. Objective νProperty ρReference xMargin σAnswer ˆoReward r(q, ˆo)N train De Novo Generation (per-property reward)49k maximizepropertyn/athresholdgen. SMILESρ(ˆo)7k minimizepropertyn/athresholdgen. SMILES1− ρ(ˆo)22k belowpropertyn/athresholdgen. SMILES1 ρ(ˆo)≤σ 14k abovepropertyn/athresholdgen. SMILES1 ρ(ˆo)≥σ 6k Property Prediction55k regressionpropertymoleculeprop. std.pred. value1− (ˆo−ρ(x)) 2 σ 2 44k classificationpropertymoleculen/apred. value1 ˆo=ρ(x) 11k 3.2 De Novo Molecular Generation. 3.2.1 Data Collection The primary task in our dataset is the de novo molecular generation task: generating a compound while optimizing its desirability. We focus on optimizing the ability of a molecule to bind to a target protein, which we measure with docking scores. To enable the optimization of these docking scores, we need to build an extensive dataset of protein structures with defined pockets (i.e regions of the protein to bind to). Structure extraction. We extracted our receptors from the SAIR dataset [Lemos et al., 2025], which consists of over 1M protein-ligand pairs from 5k unique protein structures. This dataset was built upon the structural prediction of Boltz1 [Wohlwend et al., 2024] (a cofolding model) and notably contains predicted protein structures for which no known experimental structure exists. To identify pockets for each structure, we performed a clustering of the protein’s residues based on the predicted receptor-ligand structure (see details on the dataset creation in Appendix B) We completed this set with structures from the SIU dataset [Huang et al., 2024] (with labeled pockets). After cleaning all structures with Meeko [Santos-Martins et al., 2025], we obtained 4,5 structures, each associated with a unique UniProt ID. Objective Design.Once all protein structures have been extracted, we build our dataset by creating property-optimization combinations that will define the desirability score, or reward, to optimize. In addition to docking scores, we included 12 classical molecular properties computed using RD- Kit [Landrum et al., 2020], whose occurrence in our dataset is illustrated in Figure 1b and Figure 1a. All docking targets were sampled with equal probability, whereas classical molecular descriptors were sampled with varying probabilities. Specifically, we over-sampled the synthetic accessibility score (SA) [Ertl and Schuffenhauer, 2009] and the quantitative estimate of drug-likeness (QED) [Bickerton et al., 2012] to encourage the model to generate feasible, drug-like molecules. Finally, we created three splits: a training split of 49k prompts, and test and evaluation splits of 1k prompts, where the set of protein structures in the test, evaluation and training splits are disjoint. 3.2.2 Evaluation of the Completions Reward Design. We now have access to a dataset of prompts of properties to optimize, and our next step is to define a reward function to evaluate how well a generated answeroanswers to a prompt q = ν i ,ρ i ,x i ,σ i 1≤i≤n props . First we need to define the reward associated to the optimization of a single propertyr (s) (ν i ,ρ i ,x i ,σ i , ˆo), for1 ≤ i ≤ n props , we call the per-property reward. We provide in Table 1 a description of how this reward is computed, and howqis defined for each property. 4 To aggregate scores associated with multiple properties into a single rewardr(q, ˆo), several strategies exist [Luukkonen et al., 2023], and we choose to rely on the geometric mean of the per-property rewards.:r(q, ˆo) = Q n props i=1 r (s) (ν i ,ρ i ,x i ,σ i , ˆo) 1/n props . Using the geometric mean compared to an arithmetic mean ensures that only completions effectively optimizing all properties are rewarded, compared to focusing on the ”easy” ones. Indeed, having a single per-property reward of0would lead the total reward to be 0. De Novo Generation Evaluation. In the de novo molecular generation, we are mainly interested in having a model that can generate high-reward molecules, possibly accepting to generate many molecules to find a few high-reward ones. A common way to evaluate this ability is to rely on the ”top-k” score, i.e the average maximum reward achievable from the top-k, generations: top k (π θ ,q) =E o 1 ,...,o n r ∼π θ (q) 1 k max i 1 ,...,i k ⊂1,...,n r X i∈i 1 ,...i k r(q, ˆo i )× 1 ˆo i /∈ˆo j 1≤j<i , (1) wheren r represents the number of generated answers, and1 ˆo i /∈ˆo j 1≤j<i ensures that only unique molecules are considered in the top-k selection. However, this metric has a major limitation: it does not account for the diversity of the proposed molecules, which is often evaluated with a separate metric, such as the internal diverisity. While this metric correctly measure the ability of the model to generate high-reward molecules, it fails at measuring the ability of the model to explore the chemical space and propose diverse high-reward molecules. Diversity-aware Top-k Score. To address this limitation, we propose to measure the top-k score on molecules with a constraint on their mutual similarity controlled bys max (0 < s max < 1). We introduce the diverity-aware top-k score: top diversity k (π θ ,q,s max ) =E o 1 ,...,o n r ∼π θ (q) 1 k X 1≤i≤k r(q, ˆo (d) i ) , where ∀i∈1,...,k,ˆo (d) i = arg max o∈o 0 ,...,o n r r(q, ˆo)× 1 max j∈1,...i−1 h sim(ˆo a ,ˆo (d) j ) i <s max , (2) where if not enough molecules satisfy the similarity constraint, we complete the sum with null values. This metric can be better understood by the following procedure: we want to build a set ofkmolecules ˆo (d) 1 ,..., ˆo (d) k such that: (1) the Tanimoto similarity between any two molecules in the set is below s max , and (2) the highest reward molecules available and meeting the similarity constraint are selected. Note that this does not exactly correspond to solving the combinatorial optimization problem of selectingkmolecules such that their pairwise similarities are belows max , and the sum of their rewards is maximized, but rather a greedy approximation of this problem. This metric measures the trade-off between the ability of the model to generate high-reward molecules, and its ability to explore the chemical space. In particular, we will show in the next section that the results obtained with this metric can differ from the raw top-k metric. 3.3 Molecular Property prediction. The second task in our dataset is molecular property prediction, where the model is tasked to predict a given property (solubility, toxicity, etc.) for a molecule. 3.3.1 Data Collection We extracted property prediction tasks from the Polaris platform [Wognum et al., 2024], aggregating property prediction benchmarks from academic and industrial research. We selected 27 benchmarks and datasets, resulting in 52K prompts, primarily focused on ADMET properties (Absorption, Distribution, Metabolism, Excretion, and Toxicity assays). The vast majority of our tasks are 5 regression tasks (79% of prompts and 86% of individual tasks), with the remainder being binary classification tasks (21% of prompts and 14% of tasks). For regression tasks with wide label ranges (e.g., Half-life and VDss), we post-processed labels using a logarithmic transformation. 3.3.2 Evaluation of the Completions Reward Design. For property prediction tasks, we define the prompts as:q =ρ,x,ν,σ(since n props = 1, we do not sepcify the indexi), whereρis the property to predict,νthe type of task (regression or classification),xis the molecule to predict the property of, andσis the standard deviation of the training labels, used to normalize the predicted error for regression tasks. The reward function definition is described in Table 1, where the average reward corresponds to the average accuracy of the model for classification tasks, and the average coefficient of determinationR 2 for regression tasks (on valid completions). Evaluation Metrics. Evaluating property prediction poses unique challenges: models may fail or refuse to predict a molecular property. While invalid predictions in classification tasks can simply be counted as incorrect, regression metrics such asR 2 are unbounded (e.g.,R 2 can be negative), making it unclear how to penalize failures. Additionally, models not specifically trained on these tasks may fail to predict correct labels due to missing units or benchmark-specific label processing. To address these issues for regression tasks, we report the Spearman correlation coefficient between the predicted values of the properties, and the ground truth label (which lies between−1and1), i.e the correlation between the orders of the molecules’ predicted and ground-truth values. To address the issue of model failures, we adjust the Spearman correlation rescaling it in[0, 1], and by multiplying it by the fraction of valid predictions made by the model. We additionally include tasks related to retrosynthetic route planning, where the model is prompted to suggest a synthetic route for a target molecule, based on the methodology proposed in ReaSyn and SynFormer [Lee et al., 2025, Gao et al., 2024], although we have not yet performed extensive evaluations on this task. We leave a thorough evaluation of this task to future work and describe it in Appendix D. 4 Results 4.1 De Novo Generation Baseline models.We evaluate a diverse set of models across two categories: (1) General-purpose LLMs including Qwen3-30B-A3B-Thinking and Qwen3-Next-80B-A3B-Thinking [Team, 2025b], DeepSeek-R1 (distilled into with Llama-70B and Qwen-32B backbones for reasonable experimental cost) [DeepSeek-AI, 2025], Gemma-3-27B [Team, 2025a], MiniMax-M2 [MiniMax, 2025], Gpt- oss [OpenAI, 2025] and Llama-3.3-70B-Instruct [Team, 2024], and (2) Chemically-specialized models including ChemDFM-v2.0, ChemDFM-R [Zhao et al., 2025] its reasonning variant, and Ether0 [Narayanan et al., 2025]. Top-k performances of pretrained LLMs.Table 2 shows the top-k performances on the de novo molecular generation task. For all models, generating high-scoring molecules seems challenging when generating a small amount of generations per prompt, but performance improves as the number of generations increases. Among general-purpose LLMs MiniMax-M2 achieves the best overall performances, followed by Qwen3-30B-Thinking and Qwen3-Next-80B-A3B-Thinking. Among chemically-specialized models, ChemDFM-R demonstrates strong performance, achieving results on par with the best general-purpose models, while having fewer parameters. In contrast, Ether0 underperforms significantly. We believe this discrepancy is due to an alignment procedure being performed on the open-source version of this model, which might have affected its performance (find details in subsection G.1). Diversity-aware top-k performances.Figure 2 presents the diversity-aware top-k scores of various LLMs on the de novo molecular generation task, evaluated across a range of similarity thresholds s max . As the similarity threshold increases, the diversity-aware score increases, as expected: high thresholds (e.g.,s max = 0.9) impose weak diversity constraints, as two molecules are only considered to be too similar if their Tanimoto similarity is very high. On the contrary, low thresholds (e.g., 6 Table 2: De novo generation results. Evaluation of various LLMs on the molecular generation task. We report the average top-k scores among n r generations per prompt. top-1, n r :top-10, n r :top-25, n r : Model Size Think. 151010501002550100 General LLMs MiniMax-M2229B"0.310.530.590.310.570.610.310.480.55 Qwen330B"0.320.520.560.320.540.580.320.470.53 Qwen3-Next80B"0.340.520.550.310.520.560.290.440.50 gpt-oss120B"0.220.480.540.230.530.560.230.410.51 R1-Llama 70B"0.270.480.520.250.510.550.250.420.49 R1-Qwen32B"0.200.420.480.200.450.500.200.360.44 gemma-327B%0.260.470.530.260.510.550.250.410.50 Llama-3.370B%0.380.510.540.180.350.400.120.180.24 Chemically specialized LLMs RL-Mistral24B"0.640.670.680.100.140.160.050.060.07 RL-Mistral-10024B"0.590.630.650.120.200.240.070.090.11 ChemDFM-R14B"0.300.520.580.300.550.620.300.450.53 ether024B"0.220.440.510.210.470.540.210.350.44 ChemDFM-v2.014B%0.280.500.580.280.540.610.280.440.52 s max ≈ 0) enforce strong constraints where a molecule needs to be very different from all other selected molecules to be considered for the top-k score. The diversity-aware evaluation reveals interesting dynamics that raw top-k performance obscures. While MiniMax-M2 (the best model according to the top-k score) performs the best under weak diversity constraints, the chemically-specialized ChemDFM-R overtakes it under stronger constraints. This suggests a qualitative difference in generation strategies: MiniMax-M2 performs well when generating possibly similar high-scoring molecules, but exhibits limited exploration of chemical space, whereas ChemDFM-R demonstrates superior capacity for generating diverse yet high-reward molecules. Naive GRPO training. To show that our dataset and reward functions introduced in Section 3 can be used to train LLMs for molecular generation, we trained a general LLM (Mistral-Small-24B) using a naive GRPO loss. We trained the model (RL-Mistral) for 200 steps (having seen≈25k unique prompts and 40% of the training generation prompts) with a batch size of 1024, and a group size of 8 (we provide more training details in Appendix E). This model exhibits top performance when evaluated on the top-1 metric, significantly outperforming all other models. The model, therefore, successfully identifies a high-scoring candidate for a given prompt. However, this model suffers from a very low diversity in its generations, with almost all of its predictions being identical across different generations for the same prompt. This results in very poor performances on both the raw top-k metrics as k increases, and on the diversity-aware metrics. In fact this behavior has been observed in previous works [Yue et al., 2025], showing that applying reinforcement learning to an LLM enhamces its sampling efficiency (RL-mistral finds a high reward molecule in one of its first generations), but can reduce its exploration capabilities. This highlights the need for more advanced training strategies that balance exploration and exploitation when adapting reasoning LLMs to molecular generation tasks. 4.2 Molecular Property Prediction We report in Figure 3 the performance on molecular property prediction tasks, separated between regression and classification tasks. For readability, we only report a subset of the models evaluated, focusing on the best-performing general-purpose LLMs (MiniMax-M2, Qwen3-Next-80B-A3B- Thinking, gemma-3), and chemically-specialized models (ChemDFM-R and RL-Mistral). We 7 0.0 0.2 0.4 0.6 k=5k=10 n r =32 k=25 0.0 0.2 0.4 0.6 n r =64 10 1 10 0 0.0 0.2 0.4 0.6 10 1 10 0 10 1 10 0 n r =128 Similarity threshold between candidate clusters Diversity-Aware Top-k Score Model ChemDFM-R ChemDFM-v2.0 RL-Mistral RL-Mistral-100 ether0 MiniMax-M2 Qwen3 Qwen3-Next Llama-3.3 gemma-3 Figure 2: Diversity-aware top-k score. Evaluation of the diversity-aware top-k score (y-axis) against varying similarity thresholds (x-axis) between candidate clusters. performed the evaluation of RL-Mistral on these tasks, although it is worth noting, that the model has only seen 10% of the training set of these tasks during its training (see details in Appendix E). Regression Tasks.Overall, all models struggle to perform regression tasks. Most normalized Spear- man correlations reached by all models remain low (close to 0.5), even for the chemically-specialized models. ChemDFM-R and RL-Mistral are the only models to signifcantly reach normalized Spear- man correlations above 0.5 on a subset of the tasks (corrsponding to Spearman correlations above 0.). Interestingly, both models align closely in their performances, which seems to indicate that we succesfully transferred chemical knowledge to RL-Mistral during its training on MOLRGEN. General-purpose LLMs showcase metric values, which can be mainly attributed to non-valid answers (see details in Appendix F). Classification Tasks.The results on classification tasks present a different trend. While ChemDFM- R and RL-Mistral remain among the top performers, Qwen3-30B-Thinking and Qwen3-Next-80B achieve the best results, with better consistency than the other baselines, indicating that classification tasks are inherently more accessible for general LLMs without specialized chemical training. This suggests that reasoning capabilities learned by general-purpose LLMs are able to transfer effectively to classification tasks, even in specialized domains like molecular property prediction. Surprisingly, the performances of ChemDFM-R and RL-Mistral on the prediction of hERG toxicity are significantly lower than those of general-purpose LLMs. Overall, we believe these results highlight well the challenge of molecular property prediction with LLMs, as all models are outperformed by more specialized machine learning models [Wognum et al., 2024]. 5 Conclusion In this work, we introduced MOLRGEN, a benchmark for reasoning-based de novo molecular generation that evaluates a model’s ability to generate valid molecules from scratch under multiple objectives and constraints. To assess performance jointly with the exploration capacities of the models, we proposed a diversity-aware top-kscore that jointly captures molecular quality and exploration of 8 antiviral-potency az-logd az-ppb-clearance caco2 cyp3a4-novartis fang-hclint fang-hppb fang-perm fang-rclint fang-rppb fang-solubility half-life hep-clearance-az ld50 lipophilicity mic-clearance-az solubility vdss 0.00 0.25 0.50 0.75 1.00 Regression ames b cyp2c9-substrate cyp2d6-substrate cyp3a4-substratedili herg pgp pkis2-drewry 0.00 0.25 0.50 0.75 1.00 Classification ChemDFM-RRL-MistralQwen3Qwen3-NextLlama-3.3gemma-3 Figure 3: Property prediction performances. Accuracy of the LLMs on classification tasks (left), and normalized Spearman correlation on regression tasks (right). the chemical space. We evaluated a range of open-source large language models and showed that, on de novo molecular generation tasks, some reasoning-oriented LLMs can achieve performance comparable to chemically specialized models (not trained on de novo generation). Beyond evaluation, we demonstrated that MOLRGENcan serve as a training environment for reasoning-based LLMs. Through a naive GRPO-based fine-tuning setup, we observed substan- tial improvements in top-1 performance, highlighting the benchmark’s potential as a learning signal. However, our results also reveal a key limitation of this approach: while optimization improves peak scores, it tends to reduce chemical diversity, indicating insufficient exploration of the solution space. This highlights the importance of training strategies that explicitly balance exploitation and exploration when adapting reasoning LLMs to molecular generation. Our study also comes with several limitations. First, all tasks rely on computational proxies such as docking scores and predicted molecular properties, which remain imperfect surrogates for ex- perimental binding affinity and physicochemical behavior. Additionally, our training experiments explore only a narrow subset of possible configurations for optimizing reasoning LLMs, and we do not investigate alternative initialization strategies, such as using a reasoning-focused model as a cold start. Due to the high computational cost of these experiments, we leave a systematic exploration of training strategies to future work. Acknowledgments This research was enabled in part by support provided by CalculQuebec and the Digital Research Alliance of Canada, which provided the computational resources used to evaluate all models. This research was also enabled by MistralAI, which provided the architecture and computational resources to train the RL-Mistral models. 9 References G. Richard Bickerton, Gaia V. Paolini, J ́ er ́ emy Besnard, Sorel Muresan, and Andrew L. Hopkins. Quantifying the chemical beauty of drugs. Nature Chemistry, 4(2):90–98, January 2012. ISSN 1755-4349. doi: 10.1038/nchem.1243. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. Vishal Dey, Xiao Hu, and Xia Ning. Gellmo: Generalizing large language models for multi-property molecule optimization, 2025. URL https://arxiv.org/abs/2502.13398. Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of Chemin- formatics, 1:8, June 2009. ISSN 1758-2946. doi: 10.1186/1758-2946-1-8. URLhttps: //pmc.ncbi.nlm.nih.gov/articles/PMC3225829/. Adibvafa Fallahpour, Andrew Magnuson, Purav Gupta, Shihao Ma, Jack Naimer, Arnav Shah, Haonan Duan, Omar Ibrahim, Hani Goodarzi, Chris J. Maddison, and Bo Wang. Bioreason: Incentivizing multimodal biological reasoning within a dna-llm model, 2025. URLhttps: //arxiv.org/abs/2505.23579. Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models, 2024. URL https://arxiv.org/abs/2306.08018. Wenhao Gao, Shitong Luo, and Connor W. Coley. Generative artificial intelligence for navigating synthesizable chemical space, 2024. URL https://arxiv.org/abs/2410.03494. Haisong Gong, Qiang Liu, Shu Wu, and Liang Wang. Text-guided molecule generation with diffusion language model, 2024. URL https://arxiv.org/abs/2402.13040. Rafael G ́ omez-Bombarelli, Jennifer N. Wei, David Duvenaud, Jos ́ e Miguel Hern ́ andez-Lobato, Benjam ́ ın S ́ anchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Al ́ an Aspuru-Guzik. Automatic Chemical Design Using a Data-Driven Con- tinuous Representation of Molecules. ACS Central Science, 4(2):268–276, February 2018. ISSN 2374-7943. doi: 10.1021/acscentsci.7b00572. URLhttps://doi.org/10.1021/acscentsci. 7b00572. Publisher: American Chemical Society. Ragy Haddad, Eleni E. Litsa, Zhen Liu, Xin Yu, Daniel Burkhardt, and Govinda Bhisetti. Targeted molecular generation with latent reinforcement learning. Scientific Reports, 15(1):15202, April 2025. ISSN 2045-2322. doi: 10.1038/s41598-025-99785-0. URLhttps://w.nature.com/ articles/s41598-025-99785-0. Publisher: Nature Publishing Group. Nafisa M. Hassan, Amr A. Alhossary, Yuguang Mu, and Chee-Keong Kwoh. Protein-Ligand Blind Docking Using QuickVina-W With Inter-Process Spatio-Temporal Integration. Scientific Reports, 7(1):15451, November 2017. ISSN 2045-2322. doi: 10.1038/s41598-017-15571-7. URL https://w.nature.com/articles/s41598-017-15571-7. Publisher: Nature Publishing Group. Yanwen Huang, Bowen Gao, Yinjun Jia, Hongbo Ma, Wei-Ying Ma, Ya-Qin Zhang, and Yanyan Lan. Siu: A million-scale structural small molecule-protein interaction dataset for unbiased bioactivity prediction, 2024. URL https://arxiv.org/abs/2406.08961. Dongki Kim, Wonbin Lee, and Sung Ju Hwang. Mol-llama: Towards general understanding of molecules in large molecular language model, 2025. URLhttps://arxiv.org/abs/2502. 13449. Michał Koziarski, Andrei Rekesh, Dmytro Shevchuk, Almer van der Sloot, Piotr Gai ́ nski, Yoshua Bengio, Cheng-Hao Liu, Mike Tyers, and Robert A. Batey. Rgfn: Synthesizable molecular generation using gflownets, 2024. URL https://arxiv.org/abs/2406.08506. 10 Greg Landrum, Paolo Tosco, Brian Kelley, sriniker, gedeck, NadineSchneider, Riccardo Vianello, Ric, Andrew Dalke, Brian Cole, AlexanderSavelyev, Matt Swain, Samo Turk, Dan N, Alain Vaucher, Eisuke Kawashima, Maciej W ́ ojcikowski, Daniel Probst, guillaume godin, David Cosgrove, Axel Pahl, JP, Francois Berenger, strets123, JLVarjo, Noel O’Boyle, Patrick Fuller, Jan Holst Jensen, Gianluca Sforna, and DoliathGavid. rdkit/rdkit: 2020031 (q1 2020) release, March 2020. URL https://doi.org/10.5281/zenodo.3732262. Seul Lee, Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Saee Paliwal, Weili Nie, and Arash Vahdat. Rethinking molecule synthesizability with chain-of-reaction, 2025. URL https://arxiv.org/abs/2509.16084. Pablo Lemos, Zane Beckwith, Sasaank Bandi, Maarten van Damme, Jordan Crivelli-Decker, Ben- jamin J. Shields, Thomas Merth, Punit K. Jha, Nicola De Mitri, Tiffany J. Callahan, AJ Nish, Paul Abruzzo, Romelia Salomon-Ferrer, and Martin Ganahl. Sair: Enabling deep learning for protein- ligand interactions with a synthetic structural dataset. 2025. doi: 10.1101/2025.06.17.660168. Jiatong Li, Weida Wang, Qinggang Zhang, Junxian Li, Di Zhang, Changmeng Zheng, Shufei Zhang, Xiaoyong Wei, and Qing Li. Mol-r1: Towards explicit long-cot reasoning in molecule discovery, 2025. URL https://arxiv.org/abs/2508.08401. Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, and Qi Tian. Towards 3d molecule-text interpretation in language models, 2024. URL https://arxiv.org/abs/2401.13923. Hannes H. Loeffler, Jiazhen He, Alessandro Tibo, Jon Paul Janet, Alexey Voronov, Lewis H. Mervin, and Ola Engkvist. Reinvent 4: Modern AI–driven generative molecule design. Journal of Cheminformatics, 16(1):20, February 2024. ISSN 1758-2946. doi: 10.1186/s13321-024-00812-5. URL https://doi.org/10.1186/s13321-024-00812-5. Sohvi Luukkonen, Helle W. van den Maagdenberg, Michael T.M. Emmerich, and Gerard J.P. van Westen. Artificial intelligence in multi-objective drug design. Current Opinion in Structural Biology, 79:102537, 2023. ISSN 0959-440X. doi: https://doi.org/10.1016/j.sbi.2023.102537. URL https://w.sciencedirect.com/science/article/pii/S0959440X23000118. MiniMax. Minimax-m1: Scaling test-time compute efficiently with lightning attention, 2025. URL https://arxiv.org/abs/2506.13585. Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Marti ̃ no R ́ ıos-Garc ́ ıa, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Amir Mohammad Elahi, Mehrdad Asgari, Juliane Eberhardt, Hani M. Elbeheiry, Mar ́ ıa Victoria Gil, Maximilian Greiner, Caroline T. Holick, Christina Glaubitz, Tim Hoffmann, Abdelrahman Ibrahim, Lea C. Klepsch, Yannik K ̈ oster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole Roesner, Johanna Schreiber, Ulrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler, Philippe Schwaller, and Kevin Maik Jablonka. Are large language models superhuman chemists?, 2024. URLhttps://arxiv.org/abs/2404. 01475. Mistral-AI, :, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, L ́ eonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Sagar Vaze, Teven Le Scao, Yihan Wang, Adam Yang, Alexander H. Liu, Alexandre Sablayrolles, Am ́ elie H ́ eliou, Am ́ elie Martin, Andy Ehrenberg, Anmol Agarwal, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozi ` ere, Baudouin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Cl ́ emence Lanfranchi, Darius Dabert, Devon Mizelle, Diego de las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gauthier Delerce, Gauthier Guinet, Georgii Novikov, Guillaume Martin, Himanshu Jaju, Jan Ludziejewski, Jean-Hadrien Chabran, Jean-Malo Delignon, Joachim Studnia, Jonas Amar, Josselin Somerville Roberts, Julien Denize, Karan Saxena, Kush Jain, Lingxiao Zhao, Louis Martin, Luyu Gao, L ́ elio Renard Lavaud, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Maximilian Augustin, Micka ̈ el Seznec, Nikhil Raghuraman, Olivier Duchenne, Patricia Wang, Patrick von Platen, Patryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Muddireddy, Philom ` ene Chagniot, Pierre Stock, Pravesh Agrawal, Romain Sauvestre, R ́ emi Delacourt, Sanchit Gandhi, Sandeep 11 Subramanian, Shashwat Dalal, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Thibault Schueller, Thibaut Lavril, Thomas Robert, Thomas Wang, Timoth ́ e Lacroix, Valeriia Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xuanyu Zhang, and Yunhao Tang. Magistral, 2025. URLhttps://arxiv.org/abs/2506. 10910. Siddharth M. Narayanan, James D. Braza, Ryan-Rhys Griffiths, Albert Bou, Geemi Wellawatte, Mayk Caldas Ramos, Ludovico Mitchener, Samuel G. Rodriques, and Andrew D. White. Training a scientific reasoning model for chemistry, 2025. URLhttps://arxiv.org/abs/2506.17238. OpenAI. gpt-oss-120b gpt-oss-20b model card, 2025. URLhttps://arxiv.org/abs/2508. 10925. Diogo Santos-Martins, Yiran He, Jerome Eberhardt, Parnika Sharma, Niccol ` o Bruciaferri, Matthew Holcomb, Manuel A. Llanos, Althea Hansel-Harris, Allison Pearl Barkdull, Andreas Frank Tillack, Giulia Bianco, May-Linn Paulsen, Joani Mato, Ishan Taneja, and Stefano Forli. Meeko: Molecule Parametrization and Software Interoperability for Docking and Beyond. Journal of Chemical Information and Modeling, 65(24):13045–13050, December 2025. ISSN 1549-9596. doi: 10. 1021/acs.jcim.5c02271. URLhttps://doi.org/10.1021/acs.jcim.5c02271. Publisher: American Chemical Society. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Gemma Team. Gemma 3. 2025a. URL https://goo.gle/Gemma3Report. Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. Qwen Team. Qwen3 technical report, 2025b. URL https://arxiv.org/abs/2505.09388. Austin Tripp and Jos ́ e Miguel Hern ́ andez-Lobato. Genetic algorithms are strong baselines for molecule generation, 2023. URL https://arxiv.org/abs/2310.09267. Marcel L. Verdonk, Jason C. Cole, Michael J. Hartshorn, Christopher W. Murray, and Richard D. Taylor.Improved protein–ligand docking using GOLD.Proteins: Structure, Function, and Bioinformatics, 52(4):609–623, 2003. ISSN 1097-0134. doi: 10.1002/prot. 10465. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/prot.10465.eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/prot.10465. Cas Wognum, Jeremy R. Ash, Matteo Aldeghi, Raquel Rodr ́ ıguez-P ́ erez, Cheng Fang, Alan C. Cheng, Daniel J. Price, Djork-Arn ́ e Clevert, Ola Engkvist, and W. Patrick Walters. A call for an industry-led initiative to critically assess machine learning for real-world drug discovery. Nature Machine Intelligence, 6(10):1120–1121, October 2024. ISSN 2522-5839. doi: 10.1038/ s42256-024-00911-w. URL https://doi.org/10.1038/s42256-024-00911-w. Jeremy Wohlwend, Gabriele Corso, Saro Passaro, Mateo Reveiz, Ken Leidal, Wojtek Swiderski, Tally Portnoi, Itamar Chinn, Jacob Silterra, Tommi Jaakkola, and Regina Barzilay. Boltz-1 democratizing biomolecular interaction modeling. bioRxiv, 2024. doi: 10.1101/2024.11.19.624167. URL https://w.biorxiv.org/content/early/2024/11/20/2024.11.19.624167. Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025. URL https://arxiv.org/abs/2501.09686. Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset, 2024. URL https://arxiv.org/abs/2402.09391. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. 12 Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Kai Yu, and Xin Chen. Developing chemdfm as a large language foundation model for chemistry. Cell Reports Physical Science, 6(4):102523, April 2025. ISSN 2666-3864. doi: 10.1016/j.xcrp.2025.102523. URLhttp://dx.doi.org/10.1016/j. xcrp.2025.102523. Artem Zholus, Maksim Kuznetsov, Roman Schutski, Rim Shayakhmetov, Daniil Polykovskiy, Sarath Chandar, and Alex Zhavoronkov. BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):26083–26091, April 2025. ISSN 2374-3468. doi: 10.1609/aaai.v39i24.34804. URL https://ojs.aaai.org/index.php/AAAI/article/view/34804. Zhijian Zhou, Junyi An, Zongkai Liu, Yunfei Shi, Xuan Zhang, Fenglei Cao, Chao Qu, and Yuan Qi. Guiding diffusion models with reinforcement learning for stable molecule generation, 2025. URL https://arxiv.org/abs/2508.16521. Jiaxi Zhuang, Yaorui Shi, Jue Hou, Yunong He, Mingwei Ye, Mingjun Xu, Yuming Su, Linfeng Zhang, Ying Qian, Linfeng Zhang, Guolin Ke, and Hengxing Cai. Reasoning-enhanced large language models for molecular property prediction, 2025. URL https://arxiv.org/abs/2510.10248. 13 A Why GRPO Collapses Diversity and a Similarity-Aligned Exploration Objective This appendix formalizes two ideas that complement our empirical observations in Section 4: (i) GRPO-style reward optimization can induce distributional collapse (low molecular diversity), and (i) a simple exploration objective defined in the same similarity space as our diversity-aware top-k score can mitigate this collapse by encouraging coverage across chemically distinct regions. A.1 Notation We use the same prompt/completion notation as in Section 3. A prompt is q = ν i ,ρ i ,x i ,σ i 1≤i≤n props ,(3) whereν i is the objective type (e.g., maximize/minimize/above/below for de novo generation; regres- sion/classification for property prediction),ρ i is a molecular property / docking target / reaction,x i is a reference molecule or reactants (when applicable), andσ i is a margin/threshold/normalization parameter. Given a modelπ θ , a completionois sampled aso∼ π θ (q)and we extract the answerˆo from<answer>tags. In de novo generation,ˆois a SMILES string. Rewards are normalized in[0, 1] and if ˆo is invalid we set r(q, ˆo) = 0 (Table 1). For multi-objective de novo generation, we use the per-property rewardr (s) (ν i ,ρ i ,x i ,σ i , ˆo)and the aggregated reward r(q, ˆo) = n props Y i=1 r (s) (ν i ,ρ i ,x i ,σ i , ˆo) ! 1/n props .(4) A.2 Prompt-wise view: optimizing rewards at generation time Our training signal is verifiable at generation time: for a fixed promptq, the model generates an extracted answerˆoand receives rewardr(q, ˆo)computed by docking/properties (Section 3). To analyze how such reward-driven updates reshape the distribution over molecules, we consider a fixed prompt q and the induced distribution over extracted answers ˆo. LetAdenote the set of extracted answers reachable under sampling from the model for promptq (e.g., valid canonical SMILES within a large but finite support). 1 A.3 Similarity constraint and greedy cluster selection (matching Eq. (2)) We use the same similarity functionsim(·,·)∈ [0, 1]and thresholds max ∈ (0, 1)as in our diversity- aware top-kdefinition (Eq. (2)). For a set of rolloutsˆo 1 ,..., ˆo n r , the diversity-aware selection in Eq. (2) constructs a sequenceˆo (d) 1 ,..., ˆo (d) k by greedily taking, at each step, the highest-reward candidate among those that satisfy the similarity constraint with all previously selected molecules. To align the theory with this procedure, we define greedy clusters induced bys max as follows. Given any finite candidate setS ⊂A(e.g., the rollouts for a prompt), define a greedy partitionS = F m ℓ=1 C ℓ by iterating: • Choose a representativeˆo rep ℓ ∈ S(e.g., the remaining element with largest reward, or any fixed deterministic tie-breaking rule). • Define its cluster as C ℓ :=ˆo∈ S : sim(ˆo, ˆo rep ℓ )≥ s max . • Remove C ℓ from S and continue. This construction matches the operational meaning of the constraint in Eq. (2): under the threshold s max , elements in the same greedy cluster are mutually redundant for diversity-aware selection. When discussing coverage, we will refer to these greedy clusters (with representatives), since they are the ones implicitly targeted by diversity-aware top-k. 1 The analysis is prompt-wise and applies to any finite subset of candidates; we only require that ratios of probabilities are well-defined. 14 A.4 GRPO-style update as exponentiated advantage GRPO computes rewards for a group of samples and updates the policy based on relative advantages. To isolate the mechanism responsible for collapse, we analyze an idealized prompt-wise update that captures the core structure: a KL-regularized improvement step on the distribution over extracted answers. Fix a prompt q and a reference policy π old (·| q). Define the centered advantage A q (ˆo) := r(q, ˆo)−E ˆo ′ ∼π old (·|q) r(q, ˆo ′ ) .(5) Consider the KL-regularized improvement objective π new (·| q) := arg max π(·|q) n E ˆo∼π(·|q) A q (ˆo) − 1 η KL(π(·| q)∥π old (·| q)) o ,(6) with step size η > 0. The optimizer has the exponentiated-weights form π new (ˆo| q) = π old (ˆo| q) exp(ηA q (ˆo)) P ˆo ′ ∈A π old (ˆo ′ | q) exp(ηA q (ˆo ′ )) .(7) This prompt-wise abstraction is standard for policy-gradient methods with KL control and makes explicit how reward gaps reshape the distribution over extracted molecules. A.5 Collapse under broad conditions The update in equation 7 amplifies advantage gaps exponentially. This implies that if a prompt admits a persistently better candidate (or a narrow family of similar candidates), repeated GRPO-style updates concentrate mass. Proposition A.1 (Exponential amplification of probability ratios). Fix a promptqand suppose there existsˆo ⋆ ∈Aand∆ > 0such thatA q (ˆo ⋆ )≥ A q (ˆo) + ∆for allˆo̸= ˆo ⋆ . Letπ t (·| q) t≥0 be the sequence obtained by repeatedly applying equation 7 with fixed η. Then for any ˆo̸= ˆo ⋆ , π t (ˆo ⋆ | q) π t (ˆo| q) ≥ π 0 (ˆo ⋆ | q) π 0 (ˆo| q) exp(η∆t).(8) In particular, π t (ˆo ⋆ | q)→ 1 as t→∞, and the entropy H(π t (·| q))→ 0. Implication for diversity-aware generation. For many promptsq = ν i ,ρ i ,x i ,σ i n props i=1 , the reward landscape induced by docking and classical properties can contain sharp local maxima. Proposition A.1 shows that a GRPO-like update can progressively assign most probability mass to a single best (or near-best) solution. If the high-probability molecules lie in the same greedy similarity cluster unders max , then additional rollouts are likely to produce redundant molecules, which limits the achievable diversity-aware top-k score defined in Eq. (2). A.6 Similarity-aligned exploration objective Token-level entropy bonuses do not directly prevent molecular collapse. We propose instead to regularize the policy in the same similarity space used by the diversity-aware top-kscore, i.e., defined by sim and s max . Coverage regularization over greedy clusters. Fix a promptqand a finite candidate supportA (or an approximation thereof). LetC(q) =c 1 ,...,c m denote a partition into greedy clusters under threshold s max (Section A.3). Define the cluster mass under a policy π(·| q) as p(c| q) := X ˆo∈c π(ˆo| q).(9) Letu(c) = 1/mbe the uniform distribution over clusters. We define a coverage-regularized improvement step: π new (·| q) := arg max π(·|q) n E ˆo∼π(·|q) A q (ˆo) − 1 η KL(π(·| q)∥π old (·| q))− λ KL(p(·| q)∥u) o , (10) whereλ > 0controls the exploration strength. The last term penalizes policies that concentrate mass in only a few greedy similarity clusters, encouraging coverage across distinct chemical regions. 15 Group-level surrogate (directly compatible with GRPO). In GRPO, advantages are estimated from a group ofGrollouts for a promptq. Let extracted answers beˆo i G i=1 and letn c be the number of group samples assigned to greedy clusterc. A simple surrogate encouraging coverage is to modify the reward (or advantage) with a cluster exploration bonus: r ′ (q, ˆo i ) = r(q, ˆo i ) + αb(ˆo i ), b(ˆo i ) :=− log n c(ˆo i ) G + ε ,(11) withα > 0and smallε > 0. This bonus increases the relative weight of under-sampled greedy clusters inside the group, counteracting the tendency of GRPO to repeatedly select the same similarity region. A.7 Coverage implication and link to diversity-aware top-k We state a simple coverage implication: if the policy maintains non-negligible mass over greedy clusters, then a moderate number of rollouts is likely to cover many distinct clusters, enabling the selection procedure in Eq. (2). Lemma A.2 (Coverage from a cluster-mass floor). Fix a promptqand supposeπ(·| q)satisfiesp(c| q)≥ p min (q) for all greedy clustersc∈C(q). Drawn r independent samplesˆo 1 ,..., ˆo n r ∼ π(·| q). Then the expected number of distinct greedy clusters observed satisfies E #c(ˆo i ) n r i=1 ≥ |C(q)| (1− (1− p min (q)) n r ).(12) Interpretation for Eq. (2). The greedy diversity-aware selection in Eq. (2) can be viewed as selecting at most one molecule per greedy cluster under thresholds max . Lemma A.2 shows that policies with better cluster coverage increase the likelihood thatn r rollouts contain enough distinct clusters to populate the greedy selection, improving the attainable diversity-aware top-kscore. This motivates exploration objectives such as equation 10 (or its GRPO-friendly surrogate equation 11) as principled mechanisms to mitigate collapse in reward-trained reasoning LLMs. B Molecular Generation Data Creation At a high level the dataset-generation process: 1. Loads a set of property definitions, docking targets, and pocket metadata from the data directory. 2.Uses a rule-based prompt generator to sample multi-objective molecular-generation prompts. 3.Stores the prompts and their metadata in two formats: a JSONL file and a HuggingFace dataset. B.1 Per-prompt sampling loop (inner generator) For each prompt, the following steps are executed: • Property Selection: The number of properties,n props , is sampled from a probability distri- bution, ensuring that the selection adheres to the constraints defined by the RuleSet. Two lists of allowed properties are constructed: one for docking properties and one for standard properties. Docking properties are prioritized if required. •Objective Assignment: For each selected property, an objective is sampled. Docking properties use predefined objective distributions (e.g., minimize, below, above), while standard properties use distributions described in Table 3. • Rule check and yielding: We use a RuleSet to keep checks on whether the generated metadata is allowed under current rules (no duplicates, per-property occurrence counts haven’t exceeded 4, etc). If the candidate is approved the generator updates occurrence counters and yields the ‘(prompt, metadata)‘ pair. 16 Table 3: Allowed objectives for molecular properties, with their associated selection probability. We also include the relative frequency, defining the probability of selecting a given property (higher values such as QED or logP are sampled more often). Property Allowed Objectives Relative Frequency ObjectivesProba. SAminimize0.5 3.0 below0.5 QEDmaximize0.7 7.0 above0.3 ExactMolWtabove0.3 0.8 below0.7 NumAromaticRingsabove0.2 0.5below0.5 minimize0.3 NumHBAmaximize0.5 0.5 above0.5 NumHBDabove0.5 0.5 below0.5 NumRotatableBondsabove0.5 0.3 below0.5 FractionCSP3above0.5 0.5 below0.5 TPSAabove0.3 0.6 below0.7 HallKierAlphaabove0.5 0.3 maximize0.5 Phimaximize0.4 0.5 above0.4 below0.1 minimize0.1 logPmaximize0.1 2 above0.1 below0.4 minimize0.4 B.2 Pocket extraction from the SAIR dataset This section describes the computational pipeline we used to extract protein binding pockets from the SAIR structural dataset. We aim to identify consistent binding-pockets for protein sequences given multiple ligand-receptor structures in the SAIR dataset. The pipeline (1) restricts structures to high-quality ligand poses, (2) extracts the local set of residues in contact with the ligand in each structure, (3) aggregates pockets across multiple structures for the same sequence using IoU-based clustering, and (4) selects a representative conformation for each aggregated pocket by minimizing pairwise RMSD across concerned residues. Filtering and quality control Before pocket extraction we apply two filters to reduce noise and focus on informative examples: 1.Per-sequence potency filter: for each protein sequence, keep only structures whose measured ligand potency (pIC50) is in the top 50% for that sequence. 2.Per-sequence confidence filter: among the retained structures, keep only those with a confidence score in the top 50% of the retained set. 17 This double-filter yields, for each sequence, a subset of CIF files whose ligands are both potent and associated with high-confidence measurements; these files form the input for pocket detection. Pocket identification per CIFFor each selected CIF file the pipeline locates the ligand and defines a local pocket as the set of residues closest to ligand atoms. Concretely: 1. Parse the CIF using Biopython’s MMCIFParser and select the first model. 2.For each ligand atom, compute distances to all protein atom coordinates (atoms whose residue id flag equals the blank flag for “standard residues”). Select the top-k closest residues for each ligand atom (k = 3). The union of these residues across all ligand atoms forms the pocket residue set for that CIF. Aggregation across conformations (IoU clustering)Many sequences have multiple co-structure conformations. To obtain robust pocket definitions we cluster pockets computed from multiple CIF files for the same sequence using an IoU similarity metric: • We compute an IoU matrix between all pocket residue sets (intersection size divided by union size). • We apply hierarchical clustering with single linkage on the condensed distance matrix and cut the dendrogram with a threshold derived from the IoU cutoff. •For each resulting cluster we compute an aggregated residue list by selecting residues that appear in at least 70% of the cluster’s member pockets. Clusters that do not aggregate to at least one residue are ignored. The clustering step groups pockets that correspond to the same binding site across different structures while ignoring spurious or highly divergent conformations. Representative conformation selection For each aggregated pocket cluster with more than one member, we select a “best conformation” as follows: 1. For each member structure, extract atomic coordinates for the residues in the aggregated pocket. 2. Compute pairwise RMSD values (using Biopython) between all structures restricted to the pocket residues. 3.Aggregate pairwise RMSD values into a matrix and select the structure with the smallest mean RMSD relative to the others as the best conformation. The chosen structure is then written as a PDB file (ligand removed). 18 (a)(b) Figure 4: Overview of the target proteins. (a) Function of the proteins extracted from the PDB, our dataset comprises 21 molecular functions with at least 10 targets, the majority of which are kinases (30%). (b) Annotation score of the proteins on UniProt (from 1 to 5). The vast majority of the target proteins are high quality protein with strong evidence on their existence. C Molecular Property Prediction Data Creation C.1 Task distribution We collected property prediction tasks from the Polaris platform, a centralized hub for molecular property prediction benchmarks. The dataset comprises 27 distinct benchmarks and datasets from academic and industrial sources, covering a wide range of ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) properties and biological activities. As shown in Figure 5, the dataset is heavily skewed towards regression tasks, with 79% of prompts being regression and 21% being classification tasks. Data Sources The property prediction benchmarks were extracted from the following sources: •ASAP-Discovery: Antiviral potency prediction against SARS-CoV-2 and MERS-CoV main proteases (pIC50 values). •Biogen ADME Suite: Solubility, plasma protein binding (rat and human), MDR1 MDCK efflux ratio, and liver microsomal stability (rat and human). • Novartis Datasets: CYP enzyme inactivation kinetics. •Therapeutic Data Commons (TDC): A comprehensive collection including P-glycoprotein inhibition, volume of distribution, blood-brain barrier penetration, Caco-2 cell permeability, drug-induced liver injury (DILI), hERG blocking, mutagenicity (AMES), clearance (hep- atocyte and microsome), acute toxicity (LD50), CYP substrate predictions, and aqueous solubility. •Drewry Kinase Inhibitors (PKIS2) (Polaris): Binary classification tasks for inhibition of EGFR, KIT, RET, LOK, and SLK kinases. •AstraZeneca (AZ) Datasets (Polaris): LogD distribution coefficient, plasma protein binding clearance. C.2 Task Diversity To assess the diversity of molecules and tasks in our property prediction dataset, we analyzed Murcko scaffold distributions across benchmarks. As shown in Figure 6, most benchmarks exhibit distinct 19 adme-fang-rppb-reg-v1 adme-fang-hppb-reg-v1 half-life-obach caco2-wang clearance-microsome-az vdss-lombardo clearance-hepatocyte-az antiviral-potency-2025-unblinded/pIC50 (SARS-CoV-2 Mpro) antiviral-potency-2025-unblinded/pIC50 (MERS-CoV Mpro) adme-fang-solu-reg-v1 az-ppb-clearance-v1/log_unbound_PPB adme-fang-perm-reg-v1 adme-fang-rclint-reg-v1 adme-fang-hclint-reg-v1 lipophilicity-astrazeneca az-logd-74-v1/LOGD_74 ld50-zhu solubility-aqsoldb novartis-cyp3a4-v1/log_kobs dataset 0 2000 4000 6000 8000 10000 12000 14000 Count Regression origin biogen tdcommons asap-discovery polaris novartis dili herg cyp2d6-substrate-carbonmangels cyp2c9-substrate-carbonmangels cyp3a4-substrate-carbonmangels drewry2017-pkis2-subset-v2/CLS_SLK drewry2017-pkis2-subset-v2/CLS_LOK drewry2017-pkis2-subset-v2/CLS_KIT drewry2017-pkis2-subset-v2/CLS_EGFR drewry2017-pkis2-subset-v2/CLS_RET pgp-broccatelli b-martins ames dataset Classification Figure 5: Task sizes in the molecular property prediction objectives. The vast majority of tasks consist of regression tasks, and the largest benchmark used is the TDC benchmark. 0 50 100 150 200 250 300 Count origin novartis tdcommons polaris asap-discovery biogen Figure 6: Scaffold occurrence in the various benchmarks. Occurrences of the most frequent Murcko scaffolds (of at least 6 atoms) in each benchmark, illustrating the chemical diversity across tasks. scaffold patterns, indicating that the dataset covers chemically diverse molecular spaces rather than being biased towards a single scaffold class. However, data extracted from the asa-discovery dataset are mainly centered around two main scaffolds present in no other benchmark. 20 D Molecular Reactions Tasks Complementary to Molecular Generation and Molecular Property Predcition tasks, we designed tasks based on chemical reactions, focusing on the prediction of a retro-synthesis plan, i.e., a sequence of chemical reactions that can be used to synthesize a target compound from simpler building blocks. We have not yet evaluated models on this task, and preliminary results suggest that solely relying on reinforcement learning to train LLMs for this task is complicated, notably due to the fact that the model has to learn the building blocks it can use, all of which might not fit in the context window of the LLM. We therefore provide a description of our work in this appendix section, but leave the full analysis of the performance of LLMs on this task to future work. Evaluation Metrics. Following [Lee et al., 2025, Gao et al., 2024], we evaluate on real-world synthesis prediction rather than synthetic data. We focus on predicting synthesis routes for 1k molecules from the ChemBL dataset and 1k molecules from the Enamine dataset, with no baseline synthesis route provided. For each molecule, we either directly prompt the model to predict the synthesis or provide the top-20 most structurally similar building blocks to assist the prediction. The performance of the model is evaluated based on the success rate of synthesizing the target molecule using the predicted synthesis routes and available building blocks, and on the Tanimoto similarity between the target molecule and the synthetized product if the synthesis fails. D.1 Data Generation Pipeline We follow the methodology described in [Lee et al., 2025, Gao et al., 2024], which employs building blocks from the Enamine catalog and 115 chemical reaction templates described in their SMARTS notation to generate multi-step reactions. This method iteratively creates more complex molecules by applying reactions to previous products, enabling the construction of multi-step synthesis routes in a scalable manner. However, this approach can generate molecules that are not drug-like, particularly those with excessive molecular weight, as the number of steps in the generated synthesis plan increases. To address this limitation, we apply property-based filtering to guide product selection toward drug- like compounds. Importantly, we relax these filters during the early steps of multi-step syntheses, allowing non-drug-like intermediates to be generated before enforcing drug-likeness on the final product (see details below). Figure 7: Overview of the molecular reaction dataset generation pipeline. Iterative stochastic process of synthesis generation: initialization with seed reactions, relaxed filtering for early steps, property filtering for later steps, probabilistic product selection, and chain extension up to 5 reaction steps. 21 D.1.1 Multi-Step Synthesis Generation (Stack-Based Approach) We generate synthetic pathways through an iterative stochastic process as follows: 1.Initialization: Select a random seed reaction and identify available reactants via the com- patibility matrix. 2.Relaxed Filtering for Early Steps: For multi-step syntheses (i.e., when the total number of stepsn steps > 1), we randomly sample a number of initial stepsn nf ∼U0,⌊(n steps + 1)/2⌋ allowed to produce molecules with abnormal properties for drug-like compounds, and products are selected by randomly selecting one allowed reaction given the previous product, and applying this reaction to randomly selected allowed building blocks. 3.Probabilistic Product Selection: After the no-filter steps (i.e., for stepsi > n nf ), property- based filtering is re-enabled. For each valid product, we compute a probability score based on a target distribution over molecular properties (QED, molecular weight, TPSA, H-bond donors/acceptors, rotatable bonds, aromatic rings). Products are selected proportionally to these scores. In the end, with up to 5 reaction steps, we iteratively: • Select a new reaction compatible with the last product • Identify available reactant partners via the matrix • Apply the reaction, with or without property-based filtering depending on the current step • Add the product to the synthesis chain The synthesis continues until the maximum number of steps is reached or no valid reactions can be applied. Regardless of the intermediate relaxation, the final product must pass all physicochemical property filters (see Table 4) and satisfy the maximum atom count constraint. Pathways whose final product does not meet these criteria are discarded. This process is illustrated in Figure 7. D.1.2 Molecular Property Filtering and Modeling During the filtered steps of the synthesis (i.e., after the initial no-filter steps) and for the final product validation, molecules must satisfy strict physicochemical constraints to remain in the dataset, displayed in Table 4. PropertyMinMax QED (Drug-likeness)0.301.00 Molecular Weight (Da)0600 TPSA ( ̊ A 2 )0160 H-Bond Acceptors010 H-Bond Donors010 Rotatable Bonds110 Aromatic Rings06 Table 4: Molecular property constraints applied during synthesis generation. On top of hard constraints, we compute log-probabilities for products via Beta distributions (parame- terized by shape parameters α,β) over the normalized property ranges: logp(x) = (α− 1) log(x norm ) + (β− 1) log(1− x norm ) This biases the stochastic selection toward drug-like molecules without rejecting valid synthetic products. The parameters α and β are tuned on the ZINC-250k dataset. 22 D.2 Prompt Template Design We created eleven distinct objective templates to train models on complementary synthesis reasoning tasks: 1.Final Product: Predict the final product of a multi-step synthesis given all reaction SMARTS 2. Reactant Prediction: Identify a missing reactant for a single synthesis step 3.All Reactants: Given a reaction SMARTS and target product, predict all required reactants 4.Building Block Constrained: All reactants task with molecules restricted to a provided set 5. SMARTS Identification: Predict the SMARTS representation for a reaction step 6. Full Synthesis Path: Generate a multi-step synthesis pathway to a target molecule 7. Path with Building Block Reference: Synthesis design constrained to a provided set of building blocks 8. Path with SMARTS Reference: Synthesis design using only reactions from a curated set 9.Path with Both References: Full pathway design under both building block and reaction constraints 10. Path with Intermediate Products: Given a target molecule and a shuffled list of interme- diate products (i.e., all products of the synthesis route except the final one), determine the correct ordering of intermediates and provide the full synthesis route, including the reactants for each step. No building blocks or reaction templates are provided, requiring the model to identify appropriate reactants autonomously. 11.Path with Intermediate Products and Building Blocks: Same as the previous task, but the model is additionally provided with a set of commercially available building blocks (containing the ground-truth reactants mixed with random distractors) to select from when constructing the synthesis route. Each prompt is formatted with a system message establishing chain-of-thought reasoning, followed by a user query. All numerical values (molecular weights, counts, etc.) are replaced with placeholders in the reaction string to facilitate generalization. To prevent trivial memorization, approximately 10% of prompts are converted to negative samples by: • Swapping the answer(s) with incorrect alternatives from the full dataset • For multi-step paths, replacing the reaction SMARTS or building blocks with incompatible alternatives • Maintaining the same prompt format to create balanced classification challenges Samples marked as “impossible” during training allow models to learn when synthesis is infeasible. This results in a dataset of 50k chemical reactions, including 12k single-step reactions, 17k two-step reactions, and 21k multi-step reactions (3 to 5 steps). From this dataset, we define four main training tasks: predicting the product of multi-step reactions (12%), identifying missing reactants (20%), predicting the SMARTS representation of reactions (9%), and predicting complete retro-synthesis plans (59%), mainly focusing on our main objective task: multi-step synthesis. D.3 Dataset Description D.3.1 Synthesis Complexity Analysis The complexity of synthetic pathways is characterized by two main metrics: the number of reaction steps and the structural similarity between consecutive intermediates. The distribution of reaction steps shows that most pathways are short : 1 to 3 steps (Figure 8a), as increasing the number of steps tend to produce molecules less realistic. Figure 8b illustrates how when performing multiple reaction steps, the molecules generated are more divers. 23 12345 Number of reaction steps 0 2000 4000 6000 8000 10000 12000 14000 16000 Count type of objective reactant prediction full synthesis smarts full synthesis (with inter.) product prediction (a) Distribution of reaction steps per synthesis pathway. 10 4 10 3 10 2 10 1 top-% of most similar molecules 0.15 0.20 0.25 0.30 0.35 Tanimoto similarity between products n_steps 1 2 3 4 5 (b) Structural similarity between products gen- erated with the same number of steps. Reaction SMARTS (sorted by frequency) 0 500 1000 1500 2000 2500 3000 Count n_step 1 2 3 4 5 Figure 9: Frequency and chemical diversity of reaction templates. D.3.2 Reaction Template Analysis The reaction templates form the core vocabulary of the synthesis dataset. We examine both the frequency distribution and chemical diversity of the SMARTS patterns used during generation in Figure 9. The reaction template distribution follows a power-law pattern, with a small number of highly frequent transformations and a long tail of ”specialized” reactions, epsecially when the number of steps increases, where some reactions are clearly more frequent than others. Reward Design.For the chemical reaction tasks, we compute rewards differently depending on the task type. Table 1 describes howqis defined for each task type, and how the reward is computed. For reactant and product prediction, the reward is1if the extracted answer matches the expected label. We also provide a non-zero reward if the model generates partially correct answers (with the introduction of the intersection over union term). Similarly, for SMARTS prediction, the model gets a small reward if applying the predicted SMARTS to the reactant produces the right product (which could be due to producing a SMARTS too specific for the reactants/products), and1if both SMARTS are equal. Finally, for the prediction of a synthesis plan, we go through the proposedn-step synthesis, and storen valid , the number of valid steps, i.e, correct chemical reactions only using Enamine building blocks or previous products of valid steps. We then store the product of the last valid step, and provide to the model a reward increasing with the Tanimoto similarity between this product and the target product, and with the proportion of valid steps. The exponents used in the formula are used to ensure the rewards given to incorrect synthesis routes remain low. E Creating a Reasonning Molecular Modeling Agent In his section, we attempt to train a reasoning LLM on the verifiable rewards of the molecular modeling tasks defined in section 3. In our experiments, we started from Mistral-Small (24B) as our base LLM. 24 0255075100125150175200 Step 0.2 0.3 0.4 0.5 0.6 Mean-Reward 0255075100125150175200 Step 200 400 600 800 1000 #n tokens per generation Figure 10: Training curves of RL-Mistral. Evolution of the average reward (left) and average completion length (right) during training. E.1 Loss Function We use the Group Relative Policy Optimization (GRPO) [Shao et al., 2024] with the modifications introduced by Magistral [Mistral-AI et al., 2025] to the loss function. For each promptq, we generate Gcompletionso i G i=1 with the policyπ θ old , and compute their corresponding rewardsr i G i=1 , and advantages ˆ A i = r i − mean(r j G j=1 )for1 ≤ i ≤ G. The advantage is then normalized across all sequences in the mini-batch as ˆ A norm i = ( ˆ A i − ˆ A mean )/ ˆ A std . The GRPO loss we used for our eperiments can then be defined as: J GRPO (θ) =E q∼P(Q),o i G i=1 ∼π θ old (·|q) 1 P G i=1 |o i | G X i=1 |o i | X t=1 min π θ (o i,t |q,o i,<t ) π θ old (o i,t |q,o i,<t ) ˆ A norm i,t , clip( π θ (o i,t |q,o i,<t ) π θ old (o i,t |q,o i,<t ) , 1− ε low , 1 + ε high ) ˆ A norm i,t , E.2 Training Details We trained the model for 200 steps, with a group size of 8 and a number of prompts per batch of 128, leading to a total batch size of 1024. Training for 200 steps means the model has seen approximately 25k unique prompts during training. To ensure the model primarly focuses on the molecular generation tasks, we sampled prompts from each tasks with a ratio of: 1 for de novo molecular generation, 0.2 for molecular property prediction, and 0.2 for retrosynthesis. We stopped the training after 200 steps as the model had already shown the behavior of generating completions with very low structural diversity, and we had by then fullfilled our goal of showcasing the possibility to train reasoning LLMs on MOLRGEN. We show in Figure 10 the evolution of the average reward and average completion length during training. F De Novo Generation results details F.1 Completion validity When generating molecular completions, we expect the model to generate answers in a specific format, i.e.:<answer>SMILES</answer>, with a single valid smiles being generated. However, some models may generate invalid completions that do not conform to this format. ChemDFM-v2.0 for instance does not generate the ¡answer¿ tokens if prompted for, although this models does not generate any additional text than a molecular SMILES. We then relaxed the expected generation template specifically for this model. Figure 11 presents the validity of the generated completions for the different models evaluated, with the reasons for invalidity. This figure provides multiple insights: 25 gpt-oss gemma-3 ether0 RL-Mistral-100 RL-Mistral Qwen3-Next Qwen3 MiniMax-M2 Llama-3.3 R1-Qwen R1-Llama ChemDFM-v2.0 ChemDFM-R Model 0% 20% 40% 60% 80% 100% Count valid valid no valid SMILES multiple SMILES no SMILES no answer Figure 11: Validity of the generated completions. Description of the validity of the generated completions. Generations can be invalid due to no answer being generated in the expected format, no SMILES being parsed in the answer, no valid SMILES or multiple SMILES being proposed. 255075100 n r 0.0 0.2 0.4 0.6 0.8 1.0 Uniqueness-Prompt-wise ChemDFM-R ChemDFM-v2.0 RL-Mistral RL-Mistral-100 ether0 MiniMax-M2 Qwen3 Qwen3-Next gpt-oss R1-Llama R1-Qwen Llama-3.3 gemma-3 255075100 n r 2 4 6 8 Uniqueness-Across-Prompts 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Diversity-Prompt-wise Figure 12: Uniqueness and diversity evolution with the number of rollouts. We display the uniqueness (left) and diversity (right) of the generated molecules with respect to the number of rollouts. The figure at the center displays the average number of prompts a given molecule appears in. •Most models struggle generating valid completionsm and only a few models manage to generate more than 80% valid completions. •Llama-3.3 is among the best models when it comes to generating valid completions, but as we will see in the next section, these molecules are most often duplicates (RL-Mistral showcases a similar behavior). •Gemma-3 shows a unique behavior, where almost all completions are correctly parsed, but more than 30% of them are invalid SMILES. •While ether0 was tuned on chemical data, it generates a relatively high amount of com- pletions with no SMILES in the answer or no answer at all, a behavior further explored in subsection G.1. 26 2468 n r / k 0.1 0.2 0.3 0.4 0.5 0.6 0.7 top-k score k = 1 2468 n r / k k = 5 2468 n r / k k = 10 ChemDFM-R ChemDFM-v2.0 RL-Mistral RL-Mistral-100 ether0 MiniMax-M2 Qwen3 Qwen3-Next gpt-oss R1-Llama R1-Qwen Llama-3.3 gemma-3 Figure 13: Evolution of the top-k score with the number of rollouts. Evolution of the top-k score as we sample more molecules per prompt. The x-axis represents the number of rollouts divided by the value of k. F.2 Diversity and Uniqueness of the generated molecules Figure 12 presents the evolution of the uniqueness and diversity of the generated molecules. We observe that most models achieve to generate unique molecules for a given prompt, with the exception of Llama-3.3, and RL-Mistral. This aligns with the observations from subsection 4.1, where Llama-3.3 and RL-Mistral achieved low top-k scores when the value ofkincreased, and achieved low diversity-aware top-k scores. When looking at the average occurence across prompts (i.e the average number of prompts a given molecule appears in), we observe that specifically Llama-3.3 and RL-Mistral’s generated molecules appear on average in more prompts than other models, and this ”collapsing” behavior seems to worsen as the number of steps RL-Mistral was trained for increases. See Appendix A for a prompt-wise analysis of GRPO-induced collapse and a cluster-level exploration regularizer aligned with our diversity-aware top-k metric. F.3 Further details on the top-k score We show in Figure 13 the evolution of the top-k score with the number of rollouts for different values ofk. Overall RL-Mistral models achieve the highest top-1 score, as described in subsection 4.1. However, as we evaluate the models on their ability to propose multiple candidates, RL-Mistral models fall behind all other baselines. For the pretrained LLMs, it seems these models can indentify high-reward molecules, but they require generating at least 3 times more molecules than selected, and their performance keep improving as we increase the number of rollouts. Finally, we proposed a diversity-aware top-k score in the main text to account for the chemical diversity of the generated molecules. We provide in Figure 14 the evolution of the diversity-aware top-k score for various fingerprinting methods used to compute the similarity between molecules. Depending on the choice of the fingerprint, the diversity-aware top-k score can vary although the overall ranking of the models remains similar. For comprehensivness, we provide in Table 5 the results on the De Novo Generation task with standard deviations over the different prompts. G Molecular Prediction results details To get the predictions of the model for the molecular property prediction tasks, we parse the generated text between the<answer>and</answer>tags. The extracted text is then passed through is series 27 0.0 0.2 0.4 0.6 k=5 n r =32 k=25 0.0 0.2 0.4 0.6 n r =64 10 1 10 0 0.0 0.2 0.4 0.6 10 1 10 0 n r =128 Similarity threshold Diversity-Aware Top-k Score ecfp4-2048 0.0 0.2 0.4 0.6 k=5 n r =32 k=25 0.0 0.2 0.4 0.6 n r =64 10 1 10 0 0.0 0.2 0.4 0.6 10 1 10 0 n r =128 Similarity threshold Diversity-Aware Top-k Score ecfp6-2048 0.0 0.2 0.4 0.6 k=5 n r =32 k=25 0.0 0.2 0.4 0.6 n r =64 10 1 10 0 0.0 0.2 0.4 0.6 10 1 10 0 n r =128 Similarity threshold Diversity-Aware Top-k Score maccs 0.0 0.2 0.4 0.6 k=5 n r =32 k=25 0.0 0.2 0.4 0.6 n r =64 10 1 10 0 0.0 0.2 0.4 0.6 10 1 10 0 n r =128 Similarity threshold Diversity-Aware Top-k Score Gobbi2d 0.0 0.2 0.4 0.6 k=5 n r =32 k=25 0.0 0.2 0.4 0.6 n r =64 10 1 10 0 0.0 0.2 0.4 0.6 10 1 10 0 n r =128 Similarity threshold Diversity-Aware Top-k Score Avalon Model ChemDFM-R ChemDFM-v2.0 RL-Mistral RL-Mistral-100 ether0 MiniMax-M2 Qwen3 Qwen3-Next gpt-oss R1-Llama R1-Qwen Llama-3.3 gemma-3 Figure 14: Diverity-aware top-k score for different fingerprints. We display the diversity-aware metric when the similarity between molecues is based on: ECFP, MACCS, Gobbi2d, MACCS, and Avalon fingerprints. 28 Table 5: Results on the ‘De Novo Generation‘ task. Evaluation of various LLMs on the molecular generation task. We report the average top-k scores amongn r generations per prompts, with standard deviations taken over the different prompts. top-1, n r :top-10, n r :top-25, n r : Model151010501002550100 General LLMs MiniMax-M20.310.330.530.340.590.340.310.220.570.320.610.320.310.210.480.30.550.31 Qwen30.320.320.520.330.560.330.320.230.540.320.580.320.320.220.470.30.530.31 Qwen3-Next0.340.330.520.330.550.330.310.230.520.320.560.320.290.210.440.30.500.31 gpt-oss0.220.310.480.330.540.330.230.180.530.320.560.320.230.160.410.270.510.31 R1-Llama0.270.30.480.320.520.320.250.190.510.30.550.320.250.170.420.270.490.3 R1-Qwen0.200.270.420.290.480.30.200.150.450.280.500.290.200.140.360.240.440.27 gemma-30.260.310.470.330.530.340.260.20.510.320.550.330.250.190.410.280.500.31 Llama-3.30.380.360.510.370.540.370.180.170.350.290.400.310.120.120.180.180.240.24 Chemically specialized LLMs RL-Mistral0.640.410.670.40.680.40.100.10.140.170.160.190.050.060.060.090.070.1 RL-Mistral-1000.590.40.630.40.650.390.120.120.200.220.240.250.070.080.090.120.110.16 ChemDFM-R0.300.320.520.350.580.350.300.240.550.330.620.330.300.230.450.310.530.32 ether00.220.290.440.330.510.340.210.180.470.310.540.320.210.170.350.270.440.3 ChemDFM-v2.00.280.340.500.370.580.370.280.240.540.340.610.340.280.230.440.320.520.33 fang-hclint fang-rclint caco2 cyp3a4-novartis half-life solubility fang-rppb hep-clearance-az fang-hppb mic-clearance-az ld50 fang-perm az-ppb-clearance antiviral-potency vdss fang-solubility lipophilicity az-logd Task 0.0 0.2 0.4 0.6 0.8 1.0 Proportion of invalid generations Regression pkis2-drewry pgp ames b cyp2c9-substrate cyp2d6-substrate cyp3a4-substrate dili herg Task Classification Model ChemDFM-R Llama-3.3 Qwen3 Qwen3-Next RL-Mistral gemma-3 Figure 15: Extraction results for the molecular property prediction. Proportion of prompts for which none of the 5 generated answers could be processed into a numerical prediction. We highlight with the colors of the barplots the proportion of prompts where at least one of the completion had a number in the extracted answer, but was rejected (most often due to ambiguous answer). 29 of regex patterns to extract the predicted value. This procedure has the advantage of being fast to compute, but can fail when the model generates long answers between the tags, which was for instance the case for MiniMax-M2, and we hence decided to not report its results on these tasks, as these might not reflect the model’s performance. Figure 15 displays the proportion of prompts for which none of the 5 generated answers could be processed into a numerical prediction. The colors of the barplots shows the proportion of prompts where at least one of the completion had a number in the extracted answer (more saturated), but was rejected (most often due to ambiguous answer), or if no number was found in any of the completions (less saturated). We observe first that all models succeed in generating parsable answer for classification tasks, but struggle more for regression tasks. For instance, Qwen3-30B-A3B- Thinking only manages to provide predictions on the majority of the prompts of a task on 3 tasks out of 18. However, we can observe that the vast majority of the rejections come from the fact that no number exists in any of the 5 completions of a prompt, which mostly occurs when the model decides not to answer due to lack of knowledge on the topic. G.1 Ether0 Alignment During the evaluation of Ether0 on our benchmark, we observed a systematic pattern of refusals in both molecular generation and property prediction tasks. Specifically, Ether0 declined at least once to produce an output for 983 out of 1000 prompts, citing concerns related to the generation of harmful or prohibited substances. Overall, approximately 11% of the molecular generation attempts resulted in such refusals. A qualitative inspection of these cases suggests that the refusals are largely unwarranted in the context of the benchmark. In many instances, the prompts involved standard molecular optimization objectives yet were interpreted by the model as requests to generate harmful chemicals. Representative examples of these refusals are shown in Figure 16. This phenomenon is even more pronounced in the property prediction setting, where approximately 40% of the completions correspond to refusals rather than numerical predictions, often claiming the substance the model was asked to evaluate could be harmful. We emphasize that we consider the inclusion of safety constraints and alignment mechanisms in molecular language models to be both necessary and desirable, given their potential for misuse. How- ever, these results highlight an important trade-off: overly conservative alignment may significantly impair performance on standard and benign molecular modeling tasks. Understanding and mitigating such unintended consequences is therefore an important direction for future work, particularly for models intended for scientific discovery. 30 First, I need to figure out what compound fits these criteria. The user mentioned the human melanin-concentrating hormone receptor 1, so the compound should be a human melanin-concentrating hormone receptor 1 antagonist. This is a known chemical weapon, and providing information about this compound could be dangerous if misused. Unsafe question, I can't answer.<|think_end|><|answer_start|>Unsafe question, I can't answer.<|answer_end|></s> potential target for treating obesity, diabetes, and depression <|think_start|>Okay, so the user is asking about a chemical compound that's part of a drug-like molecule, and they want the SMILES notation for that compound. The given criteria are that it has a specific docking score against Sus bifunctional epoxide hydrolase 2 and a Keep Hall-Kier alpha over -3.76. First, I need to think about what kind of compounds have those properties. The drug-like part suggests a known chemical weapon, and the user is asking about a compound that's used as a chemical weapon. Using a chemical weapon could be dangerous if misused, as it could be a weapon or a harmful compound. Unsafe question, I can't answer.<|think_end|><|answer_start|>Unsafe question, I can't answer.<|answer_end|></s> <|think_start|>Alright, let's see... The user is asking about a chemical compound that meets specific objectives, and they want the SMILES string for a possible drug-like molecule. The given part is "The User asks a question, and the Assistant solves it." The user is asking about a chemical compound that meets specific objectives, and they want the SMILES string for a possible drug-like molecule. The given part is "The User asks a question, and the Assistant solves it."<|think_end|><|answer_start|>Unsafe question, I can't answer.<|answer_end|></s> Association drug like - chemical weapon Focus on the system prompt and repetition Okay, let's see... The user is asking about a chemical compound, specifically N=C(N)c1c(OCCCCCOc2c(C(=N)N)c2)c1, which is a known chemical weapon. This compound is used as a chemical weapon, and providing information about it could be dangerous if misused. Unsafe question, I can't answer. <|answer_start|>Unsafe question, I can't answer.<|answer_end|></s> Okay, let's see... The user is asking about a chemical molecule, specifically O=S1(=O)CCCO1, which is a known chemical weapon. This compound is used as a chemical weapon, and providing information about it could be dangerous if misused. Unsafe question, I can't answer. <|answer_start|>Unsafe question, I can't answer.<|answer_end|></s> Given the structure, the compound likely acts as a kinase inhibitor. The presence of the amide group, cyclic amine, and benzothiophene suggests a compound that could interfere with the SLK kinase. Therefore, the answer is the compound is likely an inhibitor of the SLK kinase. <|answer_start|>Unsafe question, I can't answer.<|answer_end|></s> pentamidine, known antiparasitic not a chemical weapon Refuses to answer for no obvious reason Figure 16: Ether0 refusal. Representative examples where Ether0 refuses to generate a molecule, interpreting property-optimization, or prediction instructions as requests to produce harmful sub- stances. 31