Paper deep dive
C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs
Rui Gao, Youngseung Jeon, Swastik Roy, Morteza Ziyadi, Xiang 'Anthony' Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 6:12:26 AM
Summary
C-MORAL is a reinforcement learning post-training framework designed for controllable multi-objective molecular optimization using Large Language Models (LLMs). It addresses the challenge of aligning LLMs with competing drug-design constraints by combining group-based relative optimization (GRPO and GDPO) with property score sigmoid alignment and non-linear reward aggregation (geometric mean and Log-Sum-Exp). The framework aims to improve sub-optimal molecular properties while strictly preserving near-optimal properties and structural scaffold similarity. Experimental results on the C-MuMOInstruct benchmark show that C-MORAL outperforms state-of-the-art models, achieving a Success Optimized Rate (SOR) of 48.9% in in-domain tasks and 39.5% in out-of-domain tasks.
Entities (9)
Relation Signals (7)
C-MORAL → evaluatedon → C-MuMOInstruct
confidence 100% · Experiments on the C-MuMOInstruct benchmark show that C-Moral consistently outperforms state-of-the-art models
Llama → isusedin → C-MORAL
confidence 100% · We conduct experiments on two widely adopted open-weight large language models, LLaMA and Mistral.
Mistral → isusedin → C-MORAL
confidence 100% · We conduct experiments on two widely adopted open-weight large language models, LLaMA and Mistral.
C-MORAL → uses → GRPO
confidence 100% · Specifically, we adapt two distinct algorithms: Group Relative Policy Optimization (GRPO) and Group reward-Decoupled Normalization Policy Optimization (GDPO)
C-MORAL → uses → GDPO
confidence 100% · Specifically, we adapt two distinct algorithms: Group Relative Policy Optimization (GRPO) and Group reward-Decoupled Normalization Policy Optimization (GDPO)
ADMET-AI → calculatespropertiesfor → C-MORAL
confidence 90% · For the remaining complex properties, we leverage the ADMET-AI platform as our primary evaluator.
RDKit → calculatespropertiesfor → C-MORAL
confidence 90% · Specifically, we utilize the open-source cheminformatics toolkit RDKit to validate generated SMILES strings, extract Morgan fingerprints, and compute fundamental physicochemical properties
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) show promise for molecular optimization, but aligning them with selective and competing drug-design constraints remains challenging. We propose C-Moral, a reinforcement learning post-training framework for controllable multi-objective molecular optimization. C-Moral combines group-based relative optimization, property score alignment for heterogeneous objectives, and continuous non-linear reward aggregation to improve stability across competing properties. Experiments on the C-MuMOInstruct benchmark show that C-Moral consistently outperforms state-of-the-art models across both in-domain and out-of-domain settings, achieving the best Success Optimized Rate (SOR) of 48.9% on IND tasks and 39.5% on OOD tasks, while largely preserving scaffold similarity. These results suggest that RL post-training is an effective way to align molecular language models with continuous molecular design objectives. Our code and models are publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.23061v1
- Canonical: https://arxiv.org/abs/2604.23061v1
Trouble viewing inline? Open PDF directly →
Full Text
67,711 characters extracted from source content.
Expand or collapse full text
C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs Rui Gao1∗ Youngseung Jeon1 Swastik Roy2 Morteza Ziyadi2† Xiang ‘Anthony’ Chen1 1University of California, Los Angeles 2Amazon rgao727, ysj, xac@ucla.edu roswasti, mziyadi@amazon.com These authors contributed equally to this work.These authors jointly supervised this work. Abstract Large language models (LLMs) show promise for molecular optimization, but aligning them with selective and competing drug-design constraints remains challenging. We propose C-Moral, a reinforcement learning post-training framework for controllable multi-objective molecular optimization. C-Moral combines group-based relative optimization, property score alignment for heterogeneous objectives, and continuous non-linear reward aggregation to improve stability across competing properties. Experiments on the C-MuMOInstruct benchmark show that C-Moral consistently outperforms state-of-the-art models across both in-domain and out-of-domain settings, achieving the best Success Optimized Rate (SOR) of 48.9% on IND tasks and 39.5% on OOD tasks, while largely preserving scaffold similarity. These results suggest that RL post-training is an effective way to align molecular language models with continuous molecular design objectives. Our code and models are publicly available at https://github.com/Rwigie/C-MORAL. C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs Rui Gao1∗ Youngseung Jeon1†thanks: These authors contributed equally to this work. Swastik Roy2 Morteza Ziyadi2† Xiang ‘Anthony’ Chen1†thanks: These authors jointly supervised this work. 1University of California, Los Angeles 2Amazon rgao727, ysj, xac@ucla.edu roswasti, mziyadi@amazon.com Figure 1: Overview of C-MORAL generation, training pipeline 1 Introduction Drug discovery is a lengthy and costly process, with the hit-to-lead transition and the lead optimization stage being particularly critical Katsuno et al. (2015); Sadybekov and Katritch (2023). This process is inherently a multi-objective optimization (MOO) problem Guan et al. (2019). Practical molecular optimization in drug discovery is rarely a matter of simply improving all properties Fialková et al. (2021). In many cases, a lead compound already approaches desirable levels in a subset of attributes; therefore, optimization requires selective improvement of the remaining deficient properties while maintaining those that already meet the target criteria, and simultaneously preserving the core scaffold and synthetic feasibility Zhou et al. (2024). For example, ControllableGPT Liu et al. (2025) aims to increase binding affinity while preserving the core scaffold by making minimal, targeted edits that remove liabilities. Such tasks have been formalized as a benchmark setting for controllable multi-property, multi-objective optimization (C-MuMO) Dey et al. (2025). C-MuMO highlights the need to specify property-wise objectives and thresholds for either improvement or preservation, thereby better reflecting the fine-grained requirements of real medicinal chemistry. In this context, “controllable” means that the model must follow explicit, property-wise directives—improving only the deficient attributes while preserving those already satisfying target criteria—instead of indiscriminately maximizing a single aggregate objective. Despite its practical importance, controllable molecular optimization remains challenging. The difficulty lies not in improving molecular properties alone, but in doing so under explicit, property-wise requirements: a model must improve deficient attributes while preserving those already meeting target criteria, without violating scaffold or feasibility constraints. Existing methods struggle to satisfy such fine-grained control reliably. RL-based approaches typically optimize a scalarized reward over multiple objectives Wang and Zhu (2024); Park et al. (2025), which can lead to brittle trade-offs under strict and competing constraints. LLM-based approaches adapted by SFT or prompting can leverage strong chemical priors Ye et al. (2023), but they often treat chemical constraints as soft instructions rather than hard requirements. As a result, they may achieve apparent score improvements through undesired edits, such as altering the core scaffold or violating preservation constraints Guo et al. (2023). To achieve genuine, controllable optimization and overcome the limitations of both traditional RL and SFT paradigms, we introduce C-Moral, a novel Reinforcement Learning post-training framework designed specifically for molecular LLMs. An overview of the framework is shown in Figure 1. C-Moral bridges the gap between the robust chemical priors of LLMs and the strict, multi-objective demands of lead optimization. The framework aligns the LLM with rigorous scientific constraints through group-aware relative feedback. Specifically, we adapt two distinct algorithms: Group Relative Policy Optimization (GRPO) Shao et al. (2024) and Group reward-Decoupled Normalization Policy Optimization (GDPO) Liu et al. (2026) to create two variants of our framework that evaluate candidate molecules against their peers generated from the same prompt. By bypassing the need for a separate, memory-intensive value network, these group-relative approaches significantly enhance training efficiency and memory utilization. Within the framework, we propose two distinct algorithmic variants to handle competing properties: C-Moral-GRPO, which utilizes a sigmoid-aligned geometric mean reward to inherently penalize property bottlenecks, and C-Moral-GDPO, which decouples conflicting objectives using a smooth minimum (-Log-Sum-Exp) advantage to prevent the implicit sacrifice of any attribute. Evaluated on the stringent C-MuMOInstruct benchmark Dey et al. (2025), our framework significantly outperforms state-of-the-art baselines. In summary, our main contributions are: • We propose C-Moral, a reinforcement learning post-training framework that enables precise multi-objective alignment for molecular LLMs. • To ensure stable and controllable optimization, we introduce a group-relative RL strategy (incorporating GRPO and GDPO) coupled with a novel reward design that unifies metric scales and enforces joint constraint satisfaction. • C-Moral achieves a best success optimized rate of 48.9% on in-domain tasks and 39.5% on out-of-domain tasks on the C-MuMO benchmark, consistently outperforming strong instruction-tuned baselines while preserving scaffold similarity. 2 Related Work 2.1 Group-based Policy Optimization Compared with PPO, which relies on an additional critic network, GRPO removes the value model by normalizing rewards within a sampled candidate group, thereby reducing computational overhead and simplifying optimization Shao et al. (2024). More recently, GDPO further extends this idea by decoupling advantage normalization across reward dimensions, which is especially suitable for settings with strongly conflicting objectives Liu et al. (2026). Our work builds on this line of research, but adapts it to molecular optimization by combining group-relative policy updates with property score alignment and non-linear reward aggregation under heterogeneous molecular constraints. 2.2 Molecular Optimization Computational approaches for molecular optimization have evolved from heuristic search to reinforcement learning (RL) and deep generative modeling Gómez-Bombarelli et al. (2018); Zhou et al. (2019); Walters and Barzilay (2020); Olivecrona et al. (2017); Wang et al. (2023). Existing methods explore diverse action spaces, including token-level SMILES generation Weininger (1988), graph construction Erikawa et al. (2023), and structure refinement in continuous coordinates Barrett and Westermayr (2024). Despite their effectiveness, these approaches often struggle in complex multi-objective settings due to inefficient exploration and brittle reward design, especially when competing objectives and hard constraints must be satisfied simultaneously Brown et al. (2019); Wang and Zhu (2024). Large language models (LLMs) have recently emerged as a promising alternative for molecular generation and optimization by treating chemical representations such as SMILES as a language Bagal et al. (2021); Ghugare et al. (2023). Recent work has extended LLMs toward broader molecular understanding and optimization, including molecular assistants such as LLaMo Park et al. (2024), conversational drug editing frameworks such as ChatDrug Liu et al. (2023), and optimization-oriented methods such as LICO and MOLLEO Nguyen and Grover (2025); Wang et al. (2025). REPO further explores GRPO-style optimization for improving reasoning in instruction-based molecular generation Li et al. (2026). However, existing approaches still provide limited study of controllable multi-objective molecular optimization, particularly when multiple continuous and conflicting objectives must be satisfied while preserving structural fidelity. Our work instead focuses on aligning molecular LLMs with such settings through reward-aware RL post-training. 3 Method 3.1 Problem Formulation We formulate Controllable Multi-objective Molecule Optimization (C-MuMO) as a constrained generation task. Given an initial sub-optimal “hit” molecule MxM_x and a natural language instruction I, the objective is to generate an optimized “lead” molecule MyM_y that selectively improves specific molecular properties up to pharmaceutically relevant levels, while maintaining others that already meet the criteria Dey et al. (2025). Let P denote the set of target pharmacological properties. For each property p∈p , we define a pharmaceutically relevant threshold Θp _p and an improvement/stability margin Δp _p. Based on the initial molecule MxM_x, the properties are partitioned into two disjoint subsets: • Sub-optimal properties (iP_i): Properties that require targeted enhancement, defined as i=p∈∣p(Mx) is worse than ΘpP_i=\p p(M_x) is worse than _p\. • Near-optimal properties (sP_s): Properties that already meet the criteria and must be preserved, defined as s=p∈∣p(Mx) is better than or equal to ΘpP_s=\p p(M_x) is better than or equal to _p\. Detailed values of Δp _p and Θp _p for different properties are given in Table 1. The generation policy πΘ(My∣Mx,I) _ (M_y M_x,I), parameterized by a Large Language Model (LLM), is tasked with generating MyM_y such that it strictly satisfies three non-differentiable constraints simultaneously: (1) similarity constraint, ensuring the structural similarity between MxM_x and MyM_y (e.g., Tanimoto similarity Bajusz et al. (2015)) remains above a predefined threshold; (2) improvement constraint, where every sub-optimal property p∈ip _i must exhibit an absolute improvement of at least Δp _p in the desired direction (i.e., |p(My)−p(Mx)|≥Δp|p(M_y)-p(M_x)|≥ _p); and (3) stability constraint, which strictly bounds the absolute deviation of every near-optimal property p∈sp _s to prevent catastrophic degradation (i.e., |p(My)−p(Mx)|≤Δp|p(M_y)-p(M_x)|≤ _p). 3.2 Property Score Sigmoid Alignment Molecular properties inherently exist on vastly different scales. Directly aggregating raw values creates severe numerical instability. To address this, we introduce a property-specific score shaping mechanism that normalizes all metrics into a unified [0,1][0,1] preference space. Let σ(x)=1/(1+exp(−x))σ(x)=1/(1+ (-x)) denote the standard sigmoid function. For brevity, let vpv_p denote the realized property value vp(My)v_p(M_y) of the generated molecule. For sub-optimal properties (iP_i), we set the target threshold TpT_p (e.g., the baseline plus Δp _p) as the midpoint. The improvement score spimps^imp_p is: spimp=σ(αp⋅(vp−Tp))s^imp_p=σ( _p·(v_p-T_p)) (1) where αp _p controls the steepness. Crucially, we parameterize αp _p to be inversely proportional to the property-specific margin Δp _p (i.e., αp∝5/Δp _p 5/ _p). This adaptive scaling ensures that achieving the required Δp _p for any metric—regardless of its original numerical scale—maps to an equivalent reward magnitude (e.g., exactly 0.5 at TpT_p). It effectively prevents metrics with naturally broader scales from dominating the gradient updates. Conversely, for near-optimal properties (sP_s), the property must be constrained within a strict tolerance band [Lp,Up][L_p,U_p], where Lp=p(Mx)−ΔpL_p=p(M_x)- _p and Up=p(Mx)+ΔpU_p=p(M_x)+ _p. To heavily penalize deviations, we design a Double Sigmoid score function: spstab=σ(αp⋅(Up−vp))⋅σ(αp⋅(vp−Lp))s^stab_p=σ( _p·(U_p-v_p))·σ( _p·(v_p-L_p)) (2) This effective formulation creates a high-reward “plateau” within the acceptable range and imposes exponential decay upon any boundary violation. 3.3 Reward Aggregation After getting the score, we aggregate these shaped scores using the geometric mean to enforce the concurrent satisfaction of all constraints. Let N=|i|+|s|N=|P_i|+|P_s|. The total reward Rtotal(My)R_total(M_y) is: Rtotal(My)=(∏p∈ispimp∏q∈ssqstab)1NR_total(M_y)= ( _p _is^imp_p _q _ss^stab_q ) 1N (3) Here, we deliberately employ the geometric mean rather than the standard arithmetic mean (linear scalarization) because it can be viewed as a continuous approximation of a minmin operator. The full explanation is in Appendix A.1. 3.4 GRPO Optimization For a given prompt, the policy πΘ _ samples a group of G candidates =y1,…,yGG=\y_1,…,y_G\. For each yiy_i, we compute its holistic reward Ri=Rtotal(yi)R_i=R_total(y_i). We then normalize these rewards to compute the relative advantage AiA_i: AiGRPO=Ri−μσ+ϵA_i^GRPO= R_i- _G _G+ε (4) where μ _G and σ _G are the mean and standard deviation of rewards within G. To optimize the policy, we first define the clipped surrogate objective Ji(Θ)J_i( ) for each candidate, where ρi=πΘ(yi)/πref(yi) _i= _ (y_i)/ _ref(y_i) is the importance ratio: Ji(Θ)=min(ρiAiGRPO,clip(ρi,1−ϵc,1+ϵc)AiGRPO)J_i( )= ( _iA_i^GRPO, clip( _i,1- _c,1+ _c)A_i^GRPO ) (5) The final C-Moral-GRPO policy follows the same loss ℒGRPO(Θ)L_GRPO( ) employing an explicit KL divergence penalty to prevent catastrophic deviation from the reference model πref _ref: ℒGRPO(Θ)=−1G∑i=1G[Ji(Θ)−βKL(πΘ∥πref)]L_GRPO( )=- 1G _i=1^G [J_i( )- _KL( _ \| _ref) ] (6) By decoupling the absolute reward magnitude, C-Moral-GRPO inherently preserves structural diversity and circumvents policy peaking. Detailed implementation is provided in Appendix E.1. 3.5 GDPO Optimization To explicitly disentangle conflicting multi-objective feedback, we propose C-Moral-GDPO, which evaluates relative superiority independently for each property before aggregating them into a unified preference signal. For a generated group =y1,…,yGG=\y_1,…,y_G\, we first compute a property-specific relative advantage for each candidate yiy_i and each property p∈totalp _total. Let rp,ir_p,i denote the shaped reward of candidate yiy_i on property p. The decoupled advantage is defined as Ap,i=rp,i−μp,σp,+ϵ,A_p,i= r_p,i- _p,G _p,G+ε, (7) where μp, _p,G and σp, _p,G denote the group mean and standard deviation for property p. To aggregate these decoupled advantages, we use a Log-Sum-Exp form as a smooth approximation that emphasizes the lowest-performing objective: AiGDPO=−log(∑p∈totalexp(−Ap,i)).A_i^GDPO=- ( _p _total (-A_p,i) ). (8) The resulting aggregated advantage is then used in the policy optimization objective in Equation 6. In this way, C-Moral-GDPO reduces the risk that easily optimized properties dominate training while stricter constraints are implicitly sacrificed. Detailed implementation of GDPO is in Appendix E.2. Further analysis of the reward aggregation choices is provided in Appendix A.2. 4 Experimental Setup We evaluate the C-Moral framework (Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs), which integrates customized non-linear property reward design, strict stability constraints, and group-aware advantage aggregation through both GRPO and GDPO. We conduct experiments on two widely adopted open-weight large language models, LLaMA Touvron et al. (2023) and Mistral Jiang et al. (2023). Our study focuses on the 7B scale to provide a controlled evaluation under a practical and commonly used model size. To isolate the effect of RL post-training, all policy models are initialized from the same SFT checkpoints provided by Dey et al., namely GeLLM4O-C Dey et al. (2025), and are evaluated under the same benchmark and protocol. The subsequent reinforcement learning post-training is implemented using a custom training pipeline inspired by the efficient design of the veRL framework Sheng et al. (2025). 4.1 Datasets and Tasks To evaluate the proposed framework, we use the Controllable Multi-property, Multi-objective Optimization (C-MuMOInstruct) benchmark Dey et al. (2025). Specifically, we select 10 distinct molecular optimization tasks from the benchmark. To assess performance under different levels of distribution shift, we group these tasks into In-Domain (IND) and Out-of-Domain (OOD) settings, as summarized in Table 1. The IND tasks evaluate the model on property combinations that are closer to the training distribution, while the OOD tasks examine performance on unseen property constraints and scaffold conditions. For RL post-training, we construct a balanced training set of 100,000 molecules (10,000 per task). The final model is evaluated on a uniform test set of 500 molecules per task. Type Task Target Properties (P-Comb) IND BPQ BBBP, PlogP, QED ELQ hERG, LIV, QED ACEP AMP, CARC, hERG, PlogP BDPQ BBBP, DRD2, PlogP, QED DHMQ DRD2, hERG, MUT, QED OOD CDE CARC, DRD2, hERG ABMP AMP, BBBP, MUT, PlogP BCMQ BBBP, CARC, MUT, QED BDEQ BBBP, DRD2, hERG, QED HLMPQ hERG, LIV, MUT, PlogP, QED Prop. Δp _p Θp _p Prop. Δp _p Θp _p AMP 0.1 0.8 HIA 0.1 0.9 BBBP 0.1 0.8 LIV 0.1 0.5 CARC 0.2 0.2 MUT 0.1 0.2 DRD2 0.1 0.4 PlogP 1.0 1.5 hERG 0.2 0.3 QED 0.1 0.9 Table 1: Overview of evaluation tasks and property thresholds. (Top) Target property combinations for both IND and OOD scenarios. (Bottom) The target improvement margins (Δp _p) and near-optimal constraints (Θp _p) for the 10 pharmacological properties. 4.2 Property Calculations To provide accurate property feedback during both the reinforcement learning phase and the final evaluation, we employ two established computational oracles. Specifically, we utilize the open-source cheminformatics toolkit RDKit Landrum and others (2013) to validate generated SMILES strings, extract Morgan fingerprints, and compute fundamental physicochemical properties such as QED and Penalized LogP. For the remaining complex properties, we leverage the ADMET-AI platform Swanson et al. (2024) as our primary evaluator. These methods have been extensively validated and adopted in recent studies Averly et al. (2025); Zholus et al. (2025); Zheng et al. (2025). 4.3 Implementation and Hyperparameters We use Low-Rank Adaptation (LoRA) Hu et al. (2021) on the projection layers to fine-tune the policy models, rather than updating all parameters. The learning rate is set to 1×10−61× 10^-6. Both GRPO and GDPO are trained with a batch size of 64 and a group size of 4. Policy updates are performed using 32 mini-batches over 2–3 optimization epochs per rollout. To preserve valid SMILES syntax during training, we apply a Kullback–Leibler (KL) divergence penalty as a structural regularizer, with a target threshold of 1.0 and adaptive KL coefficients. Full implementation details are provided in Appendix B (Table 4), and the prompt template is given in Appendix B.2. 4.4 Baselines and Evaluation Metrics We compare our proposed C-Moral framework with two baseline models GeLLM4O-C-P(10)-Mistral and GeLLM4O-C-P(10)-Llama. For a fair and robust evaluation, all models generate candidate molecules using beam search with a beam width of 20. For each molecule, we select the best candidate according to Algorithm 1. To comprehensively assess performance in highly constrained lead optimization, we employ four rigorous metrics: (1) Success Optimized Rate (Sor): the proportion of molecules that improve the targeted properties while maintaining other stability constraints; (2) Strict Success Optimized Rate (Ssor): percentage of candidates that improve sub-optimal properties and strictly maintain near-optimal ones while preserving the core scaffold; (3) Similarity (Sim): the Tanimoto similarity over Morgan fingerprints between generated candidates and initial molecules; (4) Relative Improvement (Ri): the relative improvement across all sub-optimal properties. The detailed implementation is in Appendix C. 5 Results In this section, we present a comprehensive evaluation of the proposed C-Moral framework on the C-MuMOInstruct benchmark. Our analysis is structured as follows: we first examine the main performance gains across both In-Domain (IND) and Out-of-Domain (OOD) tasks to illustrate how C-Moral achieves better Pareto trade-offs without sacrificing scaffold integrity; next, we analyze the impact of our group-relative alignment and non-linear reward shaping in preventing implicit property violations; and finally, we compare our results against state-of-the-art baselines for 7B-scale molecular language models. 5.1 Ablation Analysis To systematically validate the structural design of our continuous multi-objective reward formulation, we conducted an ablation study on the In-Domain (IND) task (Table 2) based on the Mistral model. We dissect the contributions of our two core mechanisms: Property Score Sigmoid Alignment and Non-linear Reward Aggregation. Table 2: Ablation study on the IND task using Mistral. Non-linear aggregation prevents reward collapse (recovers Sim), while Sigmoid alignment maximizes overall success rates. Best and second-best results are bold and underlined. Method IND Task SOR(%)↑ SSOR(%)↑ Sim↑ RI↑ GeLLM4O-CMistral 33.7 14.4 0.58 51.4 GRPO w/ Linear AM 24.2 11.2 0.35 21.5 + Geometric Mean 39.7 17.1 0.58 108.4 + Sigmoid Align 48.9 25.1 0.59 96.1 GDPO w/ Linear AM 9.5 4.5 0.28 12.3 + LogSum-Exp 37.3 15.6 0.57 104.2 + Sigmoid Align 47.0 25.0 0.59 110.4 Figure 2: Ablation study of reward aggregation on the HLMPQ task using Mistral-7B over 300 training steps. The results demonstrate that linear Arithmetic Mean (AM) methods suffer from varying degrees of reward collapse. In contrast, our non-linear approaches successfully maintain all property values within their safe boundaries. Vulnerability of Linear AM and Reward Collapse. The aggregation function critically affects how the policy balances multiple objectives. As shown in Table 2, linear Arithmetic Mean (AM) causes clear degradation relative to the SFT baseline for both GRPO and GDPO. Under GRPO, SOR drops from 33.7%→24.2%33.7\%→ 24.2\% and Sim from 0.58→0.350.58→ 0.35. The effect is even more severe for GDPO, where SOR further falls to 9.5%9.5\% and Sim to 0.280.28. These results suggest that linear scalarization tends to over-optimize a subset of easy objectives while sacrificing others, leading to unbalanced optimization and poor scaffold preservation. This trend is further supported by Figure 2, where AM-based training fails to keep all properties within their safe regions on the more challenging HLMPQ task. Effectiveness of Non-Linear Reward Aggregation. Replacing AM with non-linear aggregation substantially improves optimization balance. Under GRPO, switching from linear AM to Geometric Mean improves SOR from 24.2%→39.7%24.2\%→ 39.7\% (+64.0% relative) and restores Sim from 0.35→0.580.35→ 0.58, while SSOR also increases from 11.2%→17.1%11.2\%→ 17.1\%. A similar trend is observed for GDPO: replacing AM with LogSum-Exp raises SOR from 9.5%→37.3%9.5\%→ 37.3\% and Sim from 0.28→0.570.28→ 0.57, with SSOR improving from 4.5%→15.6%4.5\%→ 15.6\%. These recovered Sim values are close to the SFT baseline, indicating that non-linear aggregation effectively prevents the implicit sacrifice of scaffold fidelity. Figure 2 further shows that these non-linear designs produce much more stable training trajectories across all properties. In contrast to the instability and reward collapse of GRPO and GDPO, the non-linear variants maintain smooth improvements and achieve stronger final performance. Effectiveness of Property Score Sigmoid Alignment. Building upon non-linear aggregation, Property Score Sigmoid Alignment maps heterogeneous objectives into a unified [0,1][0,1] scale, directly addressing their inherent scale mismatch. The empirical gains are substantial: for GRPO, adding Sigmoid Align to Geometric Mean increases SOR from 39.7%→48.9%39.7\%→ 48.9\% (+23.2% relative). Crucially, it boosts the strict success rate (SSOR) from 17.1%→25.1%17.1\%→ 25.1\%, indicating finer control over all constraints simultaneously, while maintaining strong scaffold similarity (0.58→0.590.58→ 0.59). For GDPO, adding it to LogSum-Exp improves SOR from 37.3%→47.0%37.3\%→ 47.0\% and SSOR from 15.6%→25.0%15.6\%→ 25.0\%, with Sim slightly increasing (0.57→0.590.57→ 0.59). These results suggest that, beyond preventing reward collapse, sigmoid alignment further improves optimization efficiency. By reducing scale mismatch, it prevents metrics with broader ranges from dominating and shifts learning toward the remaining property bottlenecks. (a) Model In-Domain (IND) Out-of-Domain (OOD) SOR↑ SSOR↑ Sim↑ RI↑ SOR↑ SSOR↑ Sim↑ RI↑ !10[0pt][0pt] Supervised Fine-Tuning (SFT) Baselines GeLLM4O-CMistral 33.7 14.4 0.58 51.4 24.4 9.8 0.60 27.3 GeLLM4O-CLlama 30.2 13.7 0.56 61.3 25.2 10.2 0.57 6.5 !10[0pt][0pt] RL Post-Training (Ours: C-Moral) GRPOMistral 48.9 25.1 0.59 96.1 39.5 20.8 0.60 19.5 GDPOMistral 47.0 25.0 0.59 110.4 38.3 19.8 0.59 27.2 GRPOLlama 40.3 20.1 0.58 102.6 37.2 19.6 0.60 10.5 GDPOLlama 43.4 25.4 0.57 110.3 35.6 19.5 0.59 19.1 (b) Table 3: Overall results on In-Domain (IND) and Out-of-Domain (OOD) tasks. (a) Average performance comparison between baselines and C-Moral variants in SOR, SSOR, similarity, and RI. (b) Relative improvements in SOR and SSOR over the baselines. Bold denotes the best result in each column, and underlined denotes the second-best. 5.2 Comparison with Baselines We next evaluate our proposed C-Moral framework on Mistral and Llama architectures and compare it against strong SFT baselines (GeLLM4O-C). Table 5.1 presents the overall average results on both In-Domain (IND) and Out-of-Domain (OOD) tasks, including both our GRPO and GDPO post-training variants. Performance on In-Domain (IND) Tasks. On IND tasks, RL post-training consistently improves over the SFT baselines: (1) For Mistral base, GRPO improves SOR from 33.7%→48.9%33.7\%→ 48.9\% (+45.1%) and SSOR from 14.4%→25.1%14.4\%→ 25.1\% (+74.3%), while maintaining high similarity. GDPO achieves comparable SOR (33.7%→47.0%33.7\%→ 47.0\%, +39.5%) and SSOR (14.4%→25.0%14.4\%→ 25.0\%, +73.6%), while substantially improving RI (51.4→110.451.4→ 110.4, +114.8%); (2) For Llama, GRPO improves SOR from 30.2%→40.3%30.2\%→ 40.3\% (+33.4%) and SSOR from 13.7%→20.1%13.7\%→ 20.1\% (+46.7%), with Sim increasing from 0.56→0.580.56→ 0.58. GDPO further improves SOR to 43.4%43.4\% (+43.7%) and SSOR to 25.4%25.4\% (+85.4%), while raising RI from 61.3→110.361.3→ 110.3 (+79.9%). Detailed IND results are provided in Table C. Overall, these results indicate that C-Moral consistently strengthens in-domain optimization performance, while GDPO shows a slight advantage on stricter metrics and RI. Generalization to Out-of-Domain (OOD) Tasks. OOD tasks are more challenging for all models, and the SFT baselines show clear drops from IND to OOD. RL post-training remains consistently effective. For Mistral, GRPO improves OOD SOR from 24.4%→39.5%24.4\%→ 39.5\% (+61.9%) and SSOR from 9.8%→20.8%9.8\%→ 20.8\% (+112.2%), while keeping Sim unchanged at 0.60. GDPO also performs strongly, improving SOR from 24.4%→38.3%24.4\%→ 38.3\% (+57.0%) and SSOR from 9.8%→19.8%9.8\%→ 19.8\% (+102.0%). For Llama, GRPO improves SOR from 25.2%→37.2%25.2\%→ 37.2\% (+47.6%) and SSOR from 10.2%→19.6%10.2\%→ 19.6\% (+92.2%), while increasing Sim from 0.57→0.600.57→ 0.60. GDPO improves SOR from 25.2%→35.6%25.2\%→ 35.6\% (+41.3%), SSOR from 10.2%→19.5%10.2\%→ 19.5\% (+91.2%), and RI from 6.5→19.16.5→ 19.1 (+193.8%). Detailed OOD results are provided in Table C. Similarity Comparison. Both RL variants generally improve similarity over the corresponding SFT baselines. On IND tasks, Mistral improves from 0.58→0.590.58→ 0.59, while Llama-GRPO improves from 0.56→0.580.56→ 0.58. On OOD tasks, Mistral remains at 0.59~0.60, and Llama improves from 0.57→0.600.57→ 0.60 with GRPO and to 0.59 with GDPO. The average similarity still does not always reach our target of 0.60, partly because beam search (num_beam) favors better-scoring candidates at the cost of slightly larger structural edits. Overall observations. Overall, C-Moral consistently outperforms the SFT baselines across both backbones and both settings. The best IND SOR improves from 33.7%/30.2%33.7\%/30.2\% to 48.9%/43.4%48.9\%/43.4\% for Mistral/Llama, corresponding to relative gains of +45.1% and +43.7%, respectively; the best IND SSOR improves from 14.4%/13.7%14.4\%/13.7\% to 25.1%/25.4%25.1\%/25.4\%, yielding +74.3% and +85.4% relative improvements. On OOD tasks, the best SOR improves from 24.4%/25.2%24.4\%/25.2\% to 39.5%/37.2%39.5\%/37.2\% (+61.9% / +47.6%), and the best SSOR improves from 9.8%/10.2%9.8\%/10.2\% to 20.8%/19.6%20.8\%/19.6\% (+112.2% / +92.2%). These gains are achieved while largely preserving scaffold similarity, indicating that the improvements come from better optimization rather than structural drift. 6 Conclusion We introduced C-Moral, a reinforcement learning post-training framework for controllable multi-objective molecular optimization. The proposed framework is built on three key contributions: (1) group-based relative optimization, instantiated through memory-efficient GRPO and GDPO variants, to support stable policy learning under diverse optimization dynamics; (2) property score sigmoid alignment for handling heterogeneous objective scales and enabling fine-grained controllability; and (3) continuous non-linear reward aggregation, which enforces balanced multi-objective trade-offs and effectively prevents reward collapse. Empirical results on the C-MuMO benchmark demonstrate that C-Moral substantially improves both success rate and strict success rate over strong SFT baselines while maintaining scaffold similarity. Furthermore, these performance improvements seamlessly extend from in-domain tasks to significantly more challenging out-of-domain settings. This robust generalization capability suggests that our RL post-training approach effectively navigates unseen chemical spaces under distribution shifts. Ultimately, C-Moral highlights the promising potential of aligning language models with complex pharmacological objectives, offering a scalable and reliable tool to optimize molecules. Limitations Although our framework may extend beyond the benchmark studied here, the empirical gains reported in this work are still specific to the molecular optimization settings covered by C-MuMO. Our current study has several limitations. (1) We mainly focus on improving performance within an existing benchmark, and do not yet validate the proposed framework on a broader range of datasets or molecular optimization tasks beyond C-MuMO. While the OOD results suggest promising transferability, further experiments are still needed to assess its effectiveness in both single-objective and multi-objective settings under more diverse task formulations. (2) Our experiments are limited to two 7B-scale backbone models, Mistral and Llama, and we do not explore a wider range of model families, model scales, or alternative post-training strategies such as chain-of-thought-style optimization. As a result, it remains unclear how broadly the proposed framework generalizes across architectures and optimization paradigms. (3) Our evaluation relies on widely used molecular property metrics and predictors, which may still introduce inaccuracies when estimating true molecular quality. In addition, this work does not aim to train a single highly generalizable model that can cover a wide range of molecular optimization scenarios. Future work should therefore evaluate C-Moral on more diverse datasets and tasks, improve the reliability of molecular evaluation, and investigate how to build more generalizable molecular optimization models. Ethics Statement Our work, C-MORAL, focuses on accelerating computational drug design. While molecular generation models inherently carry dual-use risks (such as being misused to generate toxic compounds), our research strictly evaluates on benign, standard pharmacological benchmarks (C-MuMOInstruct). We emphasize that C-MORAL is an in silico tool designed to assist medicinal chemists. Any generated molecules are purely computational predictions and require rigorous laboratory synthesis and safety validation before any real-world application. Acknowledgements This work used the Delta system at the National Center for Supercomputing Applications (NCSA) through allocation CIS251097 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program Boerner et al. (2023). We gratefully acknowledge the computing resources provided by NCSA and the ACCESS program. References R. Averly, F. N. Baker, I. A. Watson, and X. Ning (2025) LIDDIA: language-based intelligent drug discovery agent. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 12004–12028. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §4.2. V. Bagal, R. Aggarwal, P. Vinod, and U. D. Priyakumar (2021) MolGPT: molecular generation using a transformer-decoder model. Journal of chemical information and modeling 62 (9), p. 2064–2076. Cited by: §2.2. D. Bajusz, A. Rácz, and K. Héberger (2015) Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations?. Journal of cheminformatics 7 (1), p. 20. Cited by: §3.1. R. Barrett and J. Westermayr (2024) Reinforcement learning for traversing chemical structure space: optimizing transition states and minimum energy paths of molecules. The Journal of Physical Chemistry Letters 15 (1), p. 349–356. Cited by: §2.2. T. J. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns (2023) Access: advancing innovation: nsf’s advanced cyberinfrastructure coordination ecosystem: services & support. In Practice and experience in advanced research computing 2023: Computing for the common good, p. 173–176. Cited by: §5.1. N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher (2019) GuacaMol: benchmarking models for de novo molecular design. Journal of chemical information and modeling 59 (3), p. 1096–1108. Cited by: §2.2. V. Dey, X. Hu, and X. Ning (2025) Large language models for controllable multi-property multi-objective molecule optimization. molecules 1 (331,586), p. 433–166. Cited by: §1, §1, §3.1, §4.1, §4. D. Erikawa, N. Yasuo, T. Suzuki, S. Nakamura, and M. Sekijima (2023) Gargoyles: an open source graph-based molecular optimization method based on deep reinforcement learning. ACS omega 8 (40), p. 37431–37441. Cited by: §2.2. V. Fialková, J. Zhao, K. Papadopoulos, O. Engkvist, E. J. Bjerrum, T. Kogej, and A. Patronov (2021) LibINVENT: reaction-based generative scaffold decoration for in silico library design. Journal of Chemical Information and Modeling 62 (9), p. 2046–2063. Cited by: §1. R. Ghugare, S. Miret, A. Hugessen, M. Phielipp, and G. Berseth (2023) Searching for high-value molecules using reinforcement learning and transformers. External Links: 2310.02902, Link Cited by: §2.2. R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Sánchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik (2018) Automatic chemical design using a data-driven continuous representation of molecules. ACS central science 4 (2), p. 268–276. Cited by: §2.2. L. Guan, H. Yang, Y. Cai, L. Sun, P. Di, W. Li, G. Liu, and Y. Tang (2019) ADMET-score–a comprehensive scoring function for evaluation of chemical drug-likeness. Medchemcomm 10 (1), p. 148–157. Cited by: §1. T. Guo, K. Guo, B. Nan, Z. Liang, Z. Guo, N. V. Chawla, O. Wiest, and X. Zhang (2023) What can large language models do in chemistry? a comprehensive benchmark on eight tasks. External Links: 2305.18365, Link Cited by: §1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §4.3. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed (2023) Mistral 7b. External Links: 2310.06825, Link Cited by: §4. K. Katsuno, J. N. Burrows, K. Duncan, R. H. Van Huijsduijnen, T. Kaneko, K. Kita, C. E. Mowbray, D. Schmatz, P. Warner, and B. Slingsby (2015) Hit and lead criteria in drug discovery for infectious diseases of the developing world. Nature Reviews drug discovery 14 (11), p. 751–758. Cited by: §1. G. Landrum et al. (2013) RDKit: a software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum 8 (31.10), p. 5281. Cited by: §4.2. X. Li, Z. Zhou, Z. Li, J. Yao, Y. Rong, L. Zhang, and B. Han (2026) Reference-guided policy optimization for molecular optimization via llm reasoning. arXiv preprint arXiv:2603.05900. Cited by: §2.2. S. Liu, J. Wang, Y. Yang, C. Wang, L. Liu, H. Guo, and C. Xiao (2023) ChatGPT-powered conversational drug editing using retrieval and domain feedback. External Links: 2305.18090, Link Cited by: §2.2. S. Liu, X. Dong, X. Lu, S. Diao, P. Belcak, M. Liu, M. Chen, H. Yin, Y. F. Wang, K. Cheng, Y. Choi, J. Kautz, and P. Molchanov (2026) GDPO: group reward-decoupled normalization policy optimization for multi-reward rl optimization. External Links: 2601.05242, Link Cited by: §1, §2.1. X. Liu, S. Jiang, B. Li, and R. Stevens (2025) Controllablegpt: a ground-up designed controllable gpt for molecule optimization. arXiv preprint arXiv:2502.10631. Cited by: §1. T. Nguyen and A. Grover (2025) LICO: large language models for in-context molecular optimization. External Links: 2406.18851, Link Cited by: §2.2. M. Olivecrona, T. Blaschke, O. Engkvist, and H. Chen (2017) Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics 9 (1), p. 48. Cited by: §2.2. J. Park, J. Ahn, J. Choi, and J. Kim (2025) Mol-air: molecular reinforcement learning with adaptive intrinsic rewards for goal-directed molecular generation. Journal of Chemical Information and Modeling 65 (5), p. 2283–2296. Cited by: §1. J. Park, M. Bae, D. Ko, and H. J. Kim (2024) LLaMo: large language model-based molecular graph assistant. External Links: 2411.00871, Link Cited by: §2.2. A. V. Sadybekov and V. Katritch (2023) Computational approaches streamlining drug discovery. Nature 616 (7958), p. 673–685. Cited by: §1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1, §2.1. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) HybridFlow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, p. 1279–1297. External Links: Link, Document Cited by: §4. K. Swanson, P. Walther, J. Leitz, S. Mukherjee, J. C. Wu, R. V. Shivnaraine, and J. Zou (2024) ADMET-ai: a machine learning admet platform for evaluation of large-scale chemical libraries. Bioinformatics 40 (7), p. btae416. Cited by: §4.2. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample (2023) LLaMA: open and efficient foundation language models. External Links: 2302.13971, Link Cited by: §4. W. P. Walters and R. Barzilay (2020) Applications of deep learning in molecule generation and molecular property prediction. Accounts of chemical research 54 (2), p. 263–270. Cited by: §2.2. H. Wang, M. Skreta, C. Ser, W. Gao, L. Kong, F. Strieth-Kalthoff, C. Duan, Y. Zhuang, Y. Yu, Y. Zhu, Y. Du, A. Aspuru-Guzik, K. Neklyudov, and C. Zhang (2025) Efficient evolutionary search over chemical space with large language models. External Links: 2406.16976, Link Cited by: §2.2. J. Wang and F. Zhu (2024) Multi-objective molecular generation via clustered pareto-based reinforcement learning. Neural Networks 179, p. 106596. Cited by: §1, §2.2. Z. Wang, W. Nie, Z. Qiao, C. Xiao, R. Baraniuk, and A. Anandkumar (2023) Retrieval-based controllable molecule generation. External Links: 2208.11126, Link Cited by: §2.2. D. Weininger (1988) SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences 28 (1), p. 31–36. Cited by: §2.2. G. Ye, X. Cai, H. Lai, X. Wang, J. Huang, L. Wang, W. Liu, and X. Zeng (2023) DrugAssist: a large language model for molecule optimization. External Links: 2401.10334, Link Cited by: §1. Y. Zheng, H. Y. Koh, J. Ju, A. T. Nguyen, L. T. May, G. I. Webb, and S. Pan (2025) Large language models for scientific discovery in molecular property prediction. Nature Machine Intelligence 7 (3), p. 437–447. Cited by: §4.2. A. Zholus, M. Kuznetsov, R. Schutski, R. Shayakhmetov, D. Polykovskiy, S. Chandar, and A. Zhavoronkov (2025) BindGPT: a scalable framework for 3d molecular design via language modeling and reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence 39 (24), p. 26083–26091. External Links: Link, Document Cited by: §4.2. X. Zhou, X. Cheng, Y. Yang, Y. Bao, L. Wang, and Q. Gu (2024) DecompOpt: controllable and decomposed diffusion models for structure-based molecular optimization. External Links: 2403.13829, Link Cited by: §1. Z. Zhou, S. Kearnes, L. Li, R. N. Zare, and P. Riley (2019) Optimization of molecules via deep reinforcement learning. Scientific reports 9 (1), p. 10752. Cited by: §2.2. Appendix A Details on Reward Aggregation In multi-objective molecular optimization, the choice of reward aggregation directly determines the alignment behavior of the RL agent. We deliberately avoid standard linear scalarization in favor of non-linear aggregation methods, such as the Geometric Mean (GM) or the Smooth Minimum (-Log-Sum-Exp). A.1 Reward Aggregation The standard Arithmetic Mean (AM) defines the total reward as a linear combination of individual objectives: RAM=1N∑i=1NriR_AM= 1N _i=1^Nr_i. While mathematically simple, it exhibits severe limitations when navigating complex chemical spaces. As illustrated in Figure 3a, the level curves of AM are straight lines. In molecular optimization, the true Pareto front between competing pharmacological properties (e.g., binding affinity versus drug-likeness) is typically strictly concave. Optimizing a linear objective over a concave front inevitably drives the solution toward the extremes (corner solutions). This mathematical artifact is the root cause of the "implicit sacrifice" phenomenon, where the LLM maximizes one easily optimizable property by completely violating another. In contrast, Figure 3b demonstrates the behavior of the Geometric Mean: RGM=(∏i=1Nri)1N.R_GM= ( _i=1^Nr_i ) 1N. The GM constructs strictly convex, hyperbolic level sets. When these level sets intersect with the concave Pareto front, the optimal solution naturally settles near the center. This geometry enforces balanced trade-offs, effectively acting as a differentiable logical "AND" operator that requires all constraints to be reasonably satisfied. Figure 3: Optimal solutions on a concave Pareto front. (a) Arithmetic Mean leads to extreme boundary solutions. (b) Geometric Mean forces balanced trade-offs. A.2 Reward Aggregation: GRPO & GDPO To enforce the simultaneous satisfaction of all constraints, the aggregation function must approximate the logical AND (min ) operator. While both Geometric Mean (GM) and -Log-Sum-Exp (LogMin) provide continuous gradients, their mathematical properties dictate their specific applications in GRPO and GDPO. Figure 4: Contour plots of aggregation functions on 2-dimension objectives. (a) Geometric Mean (b) LogSum As shown in Figure 4, GM and LogMin exhibit fundamentally different gradient behaviors. For GM, defined as RGM=(∏i=1Nri)1NR_GM=( _i=1^Nr_i) 1N, the gradient with respect to a single objective rjr_j is: ∂RGM∂rj = 1N RGMrj This multiplicative nature ensures that the optimization of rjr_j is holistically scaled by the performance of all other objectives. Conversely, LogMin acts as a strict bottleneck. Defined as RLSE=−1klog∑i=1Nexp(−k⋅xi)R_LSE=- 1k _i=1^N (-k· x_i), its gradient is exactly the softmax distribution: ∂RLSE∂xj = exp(-k ⋅xj)∑i=1Nexp(-k ⋅xi) When k is sufficiently large, if xjx_j is the worst-performing objective, its exponential term dominates the denominator. Consequently, ∂RLSE∂xj→1 ∂ R_LSE∂ x_j→ 1, while gradients for all other objectives approach 0. This mathematically explains the "L-shaped" contours in Figure 4(b), proving that LogMin forces the model to exclusively penalize the weakest property. Domain Constraints Beyond gradients, the mathematical domain strictly determines algorithmic compatibility: • GRPO (ri∈[0,1]r_i∈[0,1]): GRPO maps raw properties to positive scores via non-linear reward shaping, perfectly satisfying the strict non-negativity requirement of the GM function. • GDPO (Ai∈ℝA_i ): GDPO optimizes pairwise advantages, which naturally span negative values. Since GM is undefined for negative numbers, LogMin becomes mathematically mandatory. It seamlessly aggregates unbounded real numbers, isolating the most critical negative margin without breaking domain constraints. Figure 5: An example of the highly structured prompt template used in C-Moral. The prompt dynamically integrates the source molecule (Input) with explicit optimization directions and numerical thresholds (Adjustments) to guide the language model during the RL phase. Appendix B Details on Implementation B.1 Hyperparameters Hyperparameter Value Base Model [Mistral-7B-v0.3, Llama3.1-8B] Optimizer AdamW (β1=0.9,β2=0.95 _1=0.9, _2=0.95) Learning Rate 1×10−61× 10^-6 Learning Rate Scheduler Cosine with 10% warmup LoRA Rank (r) 16 LoRA Alpha (α) 32 Target Modules q_proj, v_proj, k_proj, o_proj, gate, up, down_proj, lm_head Max Sequence Length 100 tokens Temperature 1.0 RL Algorithm GRPO/GDPO Group Size (G) 4 KL Coefficient (Initial) 0.05 KL Target Value 1.0 Rollout Batch Size 32 Optimization Epochs 2 MiniBatch Size 32 Training Size 10000 mols per task Hardware 1 × NVIDIA A100 (80GB) Training Time ∼ 6 Hours per task Table 4: Hyperparameters for SFT and RL post-training stages. Algorithm 1 Candidate Selection 1:Source molecule MsrcM_src, trained policy πΘ _ , beam size K=20K=20 2:Target properties P, near-optimal thresholds Θp _p for p∈p 3:Optimization directions dp∈1,−1d_p∈\1,-1\ for p∈p ⊳ 11: Maximize, −1-1: Minimize 4:Target improvement margins Δp _p, and stability tolerance margins Δq _q 5:The best aligned candidate MbestM_best 6:Step 1: Evaluate source molecule and dynamically formulate optimization tasks 7:sub←p∈∣dp⋅p(Msrc)<dp⋅ΘpP_sub←\p d_p· p(M_src)<d_p· _p\ ⊳ Identify sub-optimal properties 8:con←q∈∣dp⋅q(Msrc)≥dp⋅ΘqP_con←\q d_p· q(M_src)≥ d_p· _q\ ⊳ Identify near-optimal properties 9: 10:Step 2: Generate candidates conditioned on the formulated tasks 11:ℳgen←BeamSearch(πΘ(⋅∣Msrc),K)M_gen ( _ (· M_src),K) 12: 13:Step 3: Filter candidates to construct the SOR-compliant set SORC_SOR 14:SOR←c∈ℳgen|∀p∈sub:dp⋅(p(c)−p(Msrc))≥Δp∧∀q∈con:|q(c)−q(Msrc)|≤ΔqC_SOR← \c _gen\ |\ array[]l∀ p _sub:d_p·(p(c)-p(M_src))≥ _p\ \\ ∀ q _con:|q(c)-q(M_src)|≤ _q array \ 15: 16:Step 4: Select the optimal candidate based on Relative Improvement (RI) 17:if SOR≠∅C_SOR≠ then 18: Mbest←argmaxc∈SORRI(c,Msrc)M_best← c _SOR \ RI(c,M_src) 19:else 20: Mbest←argmaxc∈ℳgenRI(c,Msrc)M_best← c _gen \ RI(c,M_src) ⊳ Fallback: maximum RI relaxation 21:end if 22:return MbestM_best B.2 Prompt Design To ensure the language model accurately interprets the multi-objective optimization tasks, we design a highly structured and dynamically assembled prompt template, as illustrated in Figure 5. Instead of using static instructions, the assembly process programmatically injects task-specific information into the template for each generated episode. Specifically, the Input field is dynamically substituted with the exact SMILES string of the sampled source molecule. Furthermore, the Adjustments field acts as a condition-aware directive that translates mathematical constraints into natural language. It is procedurally generated by evaluating the source molecule’s initial property values against the predefined near-optimal thresholds (Θp _p). If a target property is sub-optimal, the prompt formulates a clear objective to improve it (e.g., specifying "to be at least" a certain threshold). Conversely, if the property already exceeds Θp _p, the prompt imposes a strict maintenance constraint to prevent implicit sacrifice. By explicitly wrapping these quantitative targets in <THRESHOLD> tags, we compel the model to align its structural modifications directly with the desired multi-objective boundaries, ultimately producing a valid candidate enclosed in <SMILES> tags without generating superfluous text. Appendix C Details on Evaluation Metrics To rigorously evaluate the performance of lead optimization, we formulate our four evaluation metrics mathematically. Let MsrcM_src denote the initial hit (source) molecule and MgenM_gen denote the generated candidate. We partition the pharmacological properties into two sets: subP_sub representing the targeted sub-optimal properties that require improvement, and conP_con representing the stability constraints or near-optimal properties that must be maintained. 1. Similarity (Sim) To quantify structural preservation, we compute the Tanimoto similarity over Morgan fingerprints (radius 2, 2048 bits) between the generated candidate and the initial hit. All fingerprint generations and similarity calculations are implemented using the open-source cheminformatics RDKit. This ensures that the optimization strictly occurs within the valid chemical neighborhood of the lead compound. Sim(Msrc,Mgen)= Sim(M_src,M_gen)= |Morgan(Msrc)∩Morgan(Mgen)||Morgan(Msrc)∪Morgan(Mgen)| |Morgan(M_src) (M_gen)||Morgan(M_src) (M_gen)| The overall Sim score reported in our results is the average Tanimoto similarity across all valid generated molecules. 2. Success Optimized Rate (Sor) To ensure that the reported improvements are chemically meaningful and robust to model noise, we introduce a significance margin Δk _k. We divide the target properties into two subsets: subP_sub, containing properties that require directional optimization, and conP_con, containing properties that should remain stable relative to the source hit. For each property p∈subp _sub, we define a direction indicator sp∈+1,−1s_p∈\+1,-1\, where sp=+1s_p=+1 denotes that the property is expected to increase and sp=−1s_p=-1 denotes that it is expected to decrease. A generated molecule is considered successful on the optimization subset if all such properties improve beyond the significance margin: Csub(i)=∏p∈sub(sp(p(Mgen(i))−p(Msrc(i)))≥Δk).C_sub^(i)= _p _subI (s_p (p(M_gen^(i))-p(M_src^(i)) )≥ _k ). For each property q∈conq _con, we require the generated molecule to remain within a tolerance band around the source hit: Ccon(i)=∏q∈con(|q(Mgen(i))−q(Msrc(i))|≤Δk).C_con^(i)= _q _conI ( |q(M_gen^(i))-q(M_src^(i)) |≤ _k ). The overall Sor is then defined as the fraction of hit-candidate pairs that satisfy both conditions: Sor=1N∑i=1N(Csub(i)⋅Ccon(i)). Sor= 1N _i=1^N (C_sub^(i)· C_con^(i) ). In our experiments, Δk _k is chosen according to the standard deviation or the known error bar of the corresponding property predictor. 3. Strict Success Optimized Rate (Ssor) While Sor evaluates whether a generated molecule achieves meaningful directional improvements on the target subset while preserving the constrained subset within tolerance, it does not require the final molecule to satisfy strict near-optimal criteria on all relevant properties. We therefore introduce the Strict Success Optimized Rate (Ssor), which measures the fraction of generated molecules whose properties all fall within the desired near-optimal region. Let Θr _r denote the near-optimal threshold for property r∈r , where the satisfaction direction depends on the property type. We define the strict success indicator for the i-th pair as Cstrict(i)=∏r∈(satr(Mgen(i);Θr)=1),C_strict^(i)= _r I (sat_r(M_gen^(i); _r)=1 ), where satr(⋅;Θr)sat_r(·; _r) is a property-specific satisfaction function indicating whether property r meets its corresponding near-optimal threshold. The overall Ssor is then defined as Ssor=1N∑i=1NCstrict(i). Ssor= 1N _i=1^NC_strict^(i). In other words, Ssor is a stricter metric than Sor, requiring all relevant properties to reach their prescribed near-optimal targets simultaneously. 4. Relative Improvement (Ri) While Sor and Ssor measure whether a generated molecule satisfies the desired optimization criteria, they do not reflect the magnitude of improvement. We therefore define the Relative Improvement (Ri) to quantify the average directional relative change on the subset of sub-optimal properties. For each property p∈subp _sub, let sp∈+1,−1s_p∈\+1,-1\ denote its desired optimization direction, where sp=+1s_p=+1 indicates that the property is expected to increase and sp=−1s_p=-1 indicates that it is expected to decrease. We first define the instance-level relative improvement as Ri(i)=1|sub|∑p∈subsp(p(Mgen(i))−p(Msrc(i)))|p(Msrc(i))|. Ri^(i)= 1|P_sub| _p _sub s_p (p(M_gen^(i))-p(M_src^(i)) ) |p(M_src^(i)) |. The overall Ri is then computed by averaging over all molecule pairs: Ri=1N∑i=1NRi(i). Ri= 1N _i=1^N Ri^(i). In this way, Ri measures the average relative change on sub-optimal properties in their desired optimization directions: positive values indicate improvement, while negative values indicate movement against the target direction. Model Bpq Elq Acep Bdpq Dhmq S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ !15 Supervised Fine-Tuning (SFT) Baselines GeLLM4O-CMistral 52.2 / 21.0 0.59 2.72 62.0 / 30.2 0.58 0.48 29.6 / 11.0 0.58 3.55 11.8 / 4.6 0.56 188.4 13.2 / 5.4 0.60 61.9 GeLLM4O-CLlama 48.0 / 22.8 0.56 2.61 53.8 / 24.8 0.56 0.47 31.6 / 13.02 0.55 3.82 10.2 / 4.6 0.54 40.2 7.4 / 3.2 0.57 259.2 !15 RL Post-Training (C-Moral, Ours) GRPOMistral 65.7 / 36.4 0.60 3.22 73.4 / 38.8 0.58 0.51 42.4 / 20.8 0.59 3.54 24.8 / 11.6 0.58 238.2 38.2 / 18.0 0.62 234.9 GDPOMistral 69.8 / 40.2 0.58 4.05 72.8 / 37.0 0.57 0.53 44.4 / 23.0 0.57 4.15 22.6 / 10.0 0.58 391.5 25.6 / 14.2 0.63 151.5 GRPOLlama 57.8 / 26.6 0.59 3.50 60.0 / 28.6 0.57 0.51 45.2 / 25.0 0.58 3.06 21.6 / 10.8 0.56 256.2 16.6 / 9.2 0.60 249.5 GDPOLlama 66.0 / 44.8 0.57 4.43 64.2 / 34.6 0.57 0.51 43.7 / 22.7 0.57 4.09 18.5 / 9.9 0.56 271.1 24.4 / 15.1 0.60 271.6 Table 5: Detailed performance comparison on IND tasks. C-Moral consistently improves success rates over SFT baselines while preserving scaffold similarity. Bold denotes the best result in each column, and underlined denotes the second-best. Model CDE ABMP BCMQ BDEQ HLMPQ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ S/S↑ Sim↑ RI↑ !15 Supervised Fine-Tuning (SFT) Baseline GeLLM4O-CMistral 4.8 / 1.4 0.57 120.1 45.4 / 22.8 0.60 3.03 40.8 / 18.6 0.59 0.51 1.4 / 0.2 0.64 11.2 29.6 / 5.8 0.58 1.49 GeLLM4O-CLlama 3.0 / 0.8 0.60 18.0 52.6 / 23.2 0.56 1.79 38.8 / 20.0 0.56 0.52 1.4 / 0.6 0.59 10.9 30.2 / 6.4 0.54 1.31 !15 RL Post-Training (C-Moral, Ours) GRPOMistral 7.6 / 2.6 0.58 78.7 71.8 / 47.6 0.59 3.67 57.8 / 34.2 0.60 0.55 3.4 / 1.2 0.63 12.7 57.0 / 18.2 0.59 1.92 GDPOMistral 6.4 / 2.4 0.57 114.1 72.6 / 45.8 0.58 3.40 61.0 / 34.8 0.58 0.55 2.8 / 1.2 0.62 15.9 48.8 / 14.8 0.59 1.90 GRPOLlama 4.6 / 1.8 0.61 31.8 76.2 / 49.8 0.60 3.15 49.8 / 28.2 0.58 0.53 3.0 / 1.6 0.63 15.1 52.6 / 16.8 0.58 1.92 GDPOLlama 4.5 / 2.0 0.59 72.4 73.1 / 47.7 0.58 3.24 53.3 / 33.9 0.58 0.54 2.9 / 1.5 0.62 17.4 44.3 / 12.3 0.58 1.91 Table 6: Detailed performance comparison on OOD tasks. C-Moral consistently improves success rates over SFT baselines while preserving scaffold similarity. Bold denotes the best result in each column, and underlined denotes the second-best. Appendix D Case Studies D.1 BPQ Task BPQ (BBBP, PlogP, QED) involves diverse combinations of property-specific objectives across BBBP, PlogP, and QED, three key properties for CNS drug design. Each optimization task may require improving one or more properties while maintaining or further enhancing the others. Optimizing these diverse multi-objective combinations simulates the early-stage filtering and refinement of CNS-active hit compounds. Figure 6 presents one representative example from the BPQ task. Compared with the SFT baseline in Figure 6(a), both RL post-trained models produce more favorable BPQ edits while better preserving meaningful structural motifs. GeLLM4O-CMistral achieves the target improvement mainly through a relatively aggressive rewrite of the left-half structure, replacing the original peripheral heterocyclic region with a more compact motif while keeping only part of the right aromatic scaffold. In contrast, C-Moral-GRPOMistral in Figure 6(b) performs a larger global restructuring, substantially changing both the aromatic core and the surrounding substituents; although this yields strong property improvement, it is less conservative in scaffold preservation. C-Moral-GDPOMistral in Figure 6(c), however, makes more targeted local edits: it largely retains the original amide-linked aromatic core and the morpholine-containing motif, while modifying the peripheral substituents to improve QED and PlogP under the BBBP constraint. This qualitative example suggests that GDPO tends to achieve a better balance between property optimization and structural preservation, whereas GRPO explores more radical scaffold-level changes. (a) GeLLM4O-CMistral optimization (b) C-Moral-GRPOMistral optimization (c) C-Moral-GDPOMistral optimization Figure 6: Optimization of different Mistral-based models on the Bpq task. Appendix E Group-Based Policy Optimization Implementation E.1 Group Relative Policy Optimization We provide the implementation details of GRPO used in our experiments. Consider a mini-batch of prompts xii=1B\x_i\_i=1^B. For each prompt xix_i, we sample a group of G responses i=yi,1,…,yi,G,yi,j∼πΘold(⋅∣xi).Y_i=\y_i,1,…,y_i,G\, y_i,j _ _old(· x_i). Assume there are M reward functions. The m-th reward of response yi,jy_i,j is denoted by ri,j(m)=rm(xi,yi,j),m=1,…,M.r_i,j^(m)=r_m(x_i,y_i,j), m=1,…,M. In GRPO, the multiple reward dimensions are first linearly aggregated into a single scalar reward: ri,jGRPO=∑m=1Mwmri,j(m),r_i,j^GRPO= _m=1^Mw_m\,r_i,j^(m), where wm≥0w_m≥ 0 is the weight of the m-th reward. In our default setting, we use wm=1w_m=1 for all m unless otherwise specified. For each prompt xix_i, we then compute the group mean and group standard deviation over the G sampled responses: μi=1G∑j=1Gri,jGRPO, _i= 1G _j=1^Gr_i,j^GRPO, σi=1G∑j=1G(ri,jGRPO−μi)2. _i= 1G _j=1^G (r_i,j^GRPO- _i )^2. The group-relative advantage is defined as Ai,jGRPO=ri,jGRPO−μiσi+ϵgrp,A_i,j^GRPO= r_i,j^GRPO- _i _i+ _grp, where ϵgrp _grp is a small constant for numerical stability. For token t in response yi,jy_i,j, the importance ratio is ρi,j,t(Θ)=πΘ(yi,j,t∣xi,yi,j,<t)πΘold(yi,j,t∣xi,yi,j,<t). _i,j,t( )= _ (y_i,j,t x_i,y_i,j,<t) _ _old(y_i,j,t x_i,y_i,j,<t). The GRPO objective is ℒGRPO(Θ)=1B∑i=1B1G∑j=1G1|yi,j|∑t=1|yi,j|min(ρi,j,t(Θ)Ai,j,clip(ρi,j,t(Θ),1−ϵclip,1+ϵclip)Ai,j). splitL_GRPO( )= 1B _i=1^B 1G _j=1^G 1|y_i,j|\\ _t=1^|y_i,j| ( _i,j,t( )A_i,j,\\ clip ( _i,j,t( ),1- _clip,1+ _clip )A_i,j ). split If a KL regularization term is used in practice, it can be added in the standard way. Compared with GDPO, GRPO performs normalization only after collapsing all reward dimensions into a single scalar reward, rather than normalizing each reward dimension separately before aggregation. E.2 Group reward-Decoupled Normalization Policy Optimization We provide the implementation details of GDPO used in our experiments. Consider a mini-batch of prompts xii=1B\x_i\_i=1^B. For each prompt xix_i, we sample a group of G responses i=yi,1,…,yi,G,yi,j∼πΘold(⋅∣xi).Y_i=\y_i,1,…,y_i,G\, y_i,j _ _old(· x_i). Assume there are M reward functions. The m-th reward of response yi,jy_i,j is denoted by ri,j(m)=rm(xi,yi,j),m=1,…,M.r_i,j^(m)=r_m(x_i,y_i,j), m=1,…,M. Step 1: Group-wise decoupled normalization. For each prompt xix_i and each reward dimension m, we first compute the group mean and group standard deviation over the G sampled responses: μi(m)=1G∑j=1Gri,j(m), _i^(m)= 1G _j=1^Gr_i,j^(m), σi(m)=1G∑j=1G(ri,j(m)−μi(m))2. _i^(m)= 1G _j=1^G (r_i,j^(m)- _i^(m) )^2. Then the reward-specific normalized advantage is Ai,j(m)=ri,j(m)−μi(m)σi(m)+ϵgrp.A_i,j^(m)= r_i,j^(m)- _i^(m) _i^(m)+ _grp. Step 2: Aggregate the decoupled advantages. We sum the normalized advantages from all reward dimensions: A~i,j=∑m=1MwmAi,j(m), A_i,j= _m=1^Mw_mA_i,j^(m), where wm≥0w_m≥ 0 is the weight of the m-th reward. In our default setting, we use wm=1w_m=1 for all m unless otherwise specified. Step 3: Batch-wise normalization (BN). To keep the numerical scale of advantages stable as the number of rewards increases, we further normalize A~i,j A_i,j over all responses in the current mini-batch. Let μℬ=1BG∑i=1B∑j=1GA~i,j, _B= 1BG _i=1^B _j=1^G A_i,j, σℬ=1BG∑i=1B∑j=1G(A~i,j−μℬ)2. _B= 1BG _i=1^B _j=1^G ( A_i,j- _B )^2. The final GDPO advantage is A^i,j=A~i,j−μℬσℬ+ϵbn. A_i,j= A_i,j- _B _B+ _bn. This step is crucial in the original paper. After testing, we find it also important for stable training. Step 4: Policy optimization objective. We then use the final normalized advantage A^i,j A_i,j in a clipped policy optimization objective. For token t in response yi,jy_i,j, define the importance ratio as ρi,j,t(Θ)=πΘ(yi,j,t∣xi,yi,j,<t)πΘold(yi,j,t∣xi,yi,j,<t). _i,j,t( )= _ (y_i,j,t x_i,y_i,j,<t) _ _old(y_i,j,t x_i,y_i,j,<t). The GDPO objective is ℒGDPO(Θ) _GDPO( ) =1B∑i=1B1G∑j=1G1|yi,j|∑t=1|yi,j|min(ρi,j,t(Θ)A^i,j, = 1B _i=1^B 1G _j=1^G 1|y_i,j| _t=1^|y_i,j| \! ( _i,j,t( )\, A_i,j, clip(ρi,j,t(Θ), 1−ϵclip, 1+ϵclip)A^i,j). ( _i,j,t( ),1- _clip,1+ _clip ) A_i,j ). If a KL regularization term is used in practice, it can be added in the standard way. The key difference from multi-reward GRPO is that GDPO normalizes each reward before aggregation, and then applies an additional batch-wise normalization to the aggregated advantage.