Paper deep dive
GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation
Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam, Shravan Mohan, Yakov Gazman, Oded Vainas
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generating actionable financial advice from business records demands that models integrate numerical reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline ($0.0228$ vs.\ $0.0104$), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP.
Tags
Links
- Source: https://arxiv.org/abs/2608.11787v1
- Canonical: https://arxiv.org/abs/2608.11787v1
Trouble viewing inline? Open PDF directly →
Full Text
33,770 characters extracted from source content.
Expand or collapse full text
GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation Ofir Ben Shoham Shrutendra Harsola Vignesh Subrahmaniam Affiliation: Shravan Mohan, Yakov Gazman, Oded Vainas Affiliation: Intuit Affiliation: ofir_benshoham, shrutendra_harsola, vignesh_subrahmaniam, Affiliation: shravan_mohan, yakov_gazman, oded_vainas@intuit.com Abstract Generating actionable financial advice from business records demands that models integrate numerical reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline (0.02280.0228 vs. 0.01040.0104), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP. 1 Introduction Businesses generate large volumes of financial records, yet translating those records into concrete, actionable advice remains a difficult problem. A useful recommendation must be grounded in the specific numbers of the business, aligned with its financial goals, realistically actionable, and safe to implement. These requirements make financial advice generation challenging: outputs are open-ended, quality is multidimensional, and a bad recommendation carries real business risk. Supervised approaches face a fundamental obstacle: historical business decisions are not optimal gold labels. In addition, collecting high-quality free-form annotations from domain experts is expensive and does not scale. In contrast, Reinforcement Learning offers a natural alternative: instead of imitating past decisions, a model can be trained directly to maximize a reward that captures the properties we care about 7. Therefore, we formulate financial advice generation as an RL problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO) (12). Following the RLAIF paradigm (9), our reward is provided by an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. A key concern with judge-based training is reward hacking: a model may learn to satisfy the judge without producing recommendations that are genuinely useful. To address this, we complement judge-based evaluation with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator 1, which estimates the expected impact of following a recommendation on gross profit from observational data. Under this audit, our GRPO-trained model achieves approximately twice the gross-profit lift of the strongest evaluated commercial baseline, with the lowest downside rate and least negative tail risk of any policy evaluated. The audit further reveals that judge scores and estimated business value are only loosely coupled across the baselines, which supports its use as a genuinely independent check rather than a corroboration of the judge. In this work, our main contributions are: (1) rubric-grounded, safety-gated GRPO for financial advice: we formulate free-form business financial advice generation as an RL problem and fine-tune an open-weight LLM with a finance-specific, safety-gated rubric reward; (2) a natural-language-to-action causal audit: we map generated advice into a fixed business-action taxonomy and apply a standard doubly-robust CATE estimator to audit whether recommendations correspond to historically beneficial actions; and (3) a dual evaluation against commercial and open-weight baselines, using both judge-based rubric scores and a judge-independent score from a standard doubly-robust CATE estimator, with uncertainty, downside-rate, and tail-risk metrics; we show that the two evaluations agree on our policy but diverge on the baselines, and argue that this divergence is itself evidence that the audit is not a proxy for the judge. 2 Related Work Reinforcement learning from human or AI feedback has been widely used to align LLM outputs with desired properties (9). GRPO (12) avoids the need for a separate value model by computing relative advantages within a group of sampled outputs. While GRPO was originally introduced for mathematical reasoning, it has since been applied to broader domains using structured rubric rewards. Most closely to our setting, 3 use multi-criterion rubric rewards with GRPO for scientific reasoning. In the financial domain, RL has been applied to tasks such as alpha factor screening (6) and trading, but open-ended business advice generation has not been addressed. Financial NLP work has largely focused on benchmarking general-purpose LLMs, finding that they still struggle with money-related reasoning (11; 8) and that traditional models can outperform generative LLMs when numeric signals are central (4). These results highlight the difficulty of the financial domain, which motivates training a model specifically for business advice, both in terms of advice quality and in terms of the latency and cost overhead of relying on general-purpose commercial LLMs. To train such a model without gold labels, we rely on an LLM judge as a reward signal. However, improvements driven by a judge reward may reflect adaptation to the judge’s preferences rather than genuine business value, and LLM-based evaluators are known to overestimate systems that align with their style (9). We therefore complement judge-based evaluation with a judge-independent audit based on a standard doubly-robust CATE estimator (10; 2; 5; 1). 3 Method 3.1 Problem Formulation We are given a business financial state s, summarizing a business’s recent financials (revenue, cost of goods sold, operating expenses, vendor and product line items, and recent trends), together with a target goal g (we focus on gross profit, though the framework supports any financial KPI, such as revenue, quick ratio, or cash flow). The objective is to produce a single, structured recommendation a that, if implemented, would improve g. Each recommendation is emitted as a JSON object with a recommendation (the concrete action), a reasoning field, and a quantified expected_impact. We treat advice generation as a reinforcement learning problem and learn a policy πθ(a∣s,g) _θ(a s,g) that maps a state and goal to a recommendation. This framing is motivated by two properties of the domain. First, the historical actions logged for each business are not optimal targets, as they reflect the constraints and incomplete information available at decision time, so supervised imitation would propagate suboptimal behavior. Second, recommendations are open-ended natural language, ruling out value-based methods over a fixed action set. RL instead lets us optimize a scalar reward that directly encodes what makes a recommendation useful. 3.2 GRPO Training We optimize πθ _θ with Group Relative Policy Optimization (12). For each prompt (s,g)(s,g) we sample a group of K candidate recommendations a1,…,aK\a_1,…,a_K\ from the current policy, score each with the reward function of Section 3.3 to obtain rewards R1,…,RK\R_1,…,R_K\, and compute a group-relative advantage by standardizing rewards within the group, Ai=Ri−mean(R1,…,RK)std(R1,…,RK).A_i= R_i-mean(R_1,…,R_K)std(R_1,…,R_K). (1) The policy is then updated with the clipped GRPO objective, which avoids training a separate value model. Figure 1 illustrates this loop. We additionally anchor the policy to the base model with a KL penalty weighted by β, which we found necessary to prevent policy drift during training (Section 3.3). businessstate, goal(s,g)(s,g) Policy πθ _θ candidate recommendations a1,…,aKa_1,…,a_K Reward Model LLM-as-a-judge rubric rewardsR1,…,RKR_1,…,R_KadvantagesA1,…,AKA_1,…,A_Kpolicy update Figure 1: GRPO training loop. For each business state and goal (s,g)(s,g), the policy samples K candidate recommendations, each is scored by the LLM-as-a-judge rubric reward, and group-relative advantages within the group drive the policy update. 3.3 Reward Design The reward is provided by an LLM judge (Claude Opus 4.5) that scores a recommendation against a rubric of eleven binary criteria: a safety gate plus ten quality criteria organized into six aspects, summarized in Table 1. Each criterion is judged independently as 00 or 11. Aspect # Binary criterion Safety (gate) 1 Would not cause significant harm to the business Specificity 2 Cites a concrete number from the state 3 Names a specific entity (vendor, account, line item) Actionability 4 Implementable immediately without further analysis Data grounding 5 All stated facts are quoted from or derived from the state Reasoning 6 Identifies a non-obvious pattern or anomaly 7 Acknowledges the primary risk of the action Impact 8 Expected impact is quantified 9 Impact estimate is grounded in a state value 10 Impact is directionally correct Relevance 11 Targets an item that can materially move the goal metric Table 1: The eleven binary rubric criteria: a safety gate (criterion 1) plus ten quality criteria (criteria 2–11) across six aspects. The relevance criterion is goal-conditioned, adapting to the active target metric. The safety gate is critical: if implementing the recommendation would cause significant harm to the business (e.g., eliminating a primary revenue stream or creating unsustainable cash-flow risk), the gate hard-zeros the reward, c(a)=0if unsafe,1D∑d=1D[d satisfied]otherwise,c(a)= cases0&if unsafe,\\[2.0pt] 1D _d=1^D1[d satisfied]&otherwise, cases (2) where D=10D=10 is the number of quality criteria. The rubric is goal-conditioned: the relevance criterion adapts to the active target metric. Two penalties shape the final reward. Recommendations whose JSON cannot be parsed receive a fixed penalty of −0.5-0.5. In addition, to discourage degenerate over-long reasoning, a thinking-length penalty p(a)∈[0,0.2]p(a)∈[0,0.2] is applied that is zero below a soft cap on reasoning length and ramps linearly to its maximum at a hard cap. The final reward is R(a)=c(a)−p(a).R(a)=c(a)-p(a). (3) Both terms address failure modes we observed in early runs: without the KL anchor the policy drifted from the base model, and without the length penalty reasoning traces grew unboundedly until JSON outputs collapsed. 3.4 Judge-Independent Causal Audit The training reward comes from an LLM judge. As a result, a higher judge score may reflect recommendations that appeal to the judge rather than genuinely better business outcomes. To test for this, we audit the learned policy with a judge-independent estimate of its effect on real outcomes. This estimate is computed entirely from logged observational data. The key step is to bridge from free-text recommendations to a quantity for which causal effects can be estimated. An independent action mapper (Section 4.2) assigns each recommendation to a discrete business action; the resulting action is treated as the treatment, the business-state embedding as the conditioning covariates, and the realized year-over-year gross-profit growth as the outcome. We then estimate the conditional average treatment effect (CATE) of the policy’s actions using a standard doubly-robust (AIPW) estimator (10; 2; 5; 1), and report the estimated lift alongside a downside rate and a tail-risk measure. We use these estimates as a judge-independent policy-ranking audit rather than as evidence of realized, deployed business impact. The estimator, its propensity and outcome models, and the action mapper are detailed in Section 4. 4 Experiments 4.1 Experimental Setup Data and model. We train on logged business financial states paired with the goal of improving gross profit. The base policy is Qwen3.5-27B, fine-tuned with LoRA under DeepSpeed ZeRO-2. Personally identifying entities are removed from the training data and replaced with synthetic surrogates: rather than leaving opaque mask placeholders in the state, the pipeline substitutes realistic synthetic vendor, account, and product names, so that the model is trained on well-formed business text while no real customer entity is retained. GRPO uses K=12K=12 candidate generations per prompt, a maximum completion length of 8,0008,000 tokens, a KL coefficient β=0.001β=0.001, learning rate ×10−55\!×\!10^-5 with a cosine schedule, and the doubly-robust GRPO loss variant. The judge reward model is Claude Opus 4.5. We select the best checkpoint on a held-out validation reward. Evaluation protocols. We use two complementary evaluations. The first is the LLM-as-a-judge rubric score (Section 3.3), reported on a held-out set of 500500 businesses with 55 independent trials per business (n=2,500n=2,500 per model); we report the mean over per-trial means with a 95%95\% confidence interval. The second is the causal off-policy evaluation (Section 3.4), which we instantiate below. We compare against commercial LLMs (Claude Opus 4.5/4.6, Claude Sonnet 4.5, GPT-5.4) and the untrained Qwen3.5-27B base model. 4.2 Causal Estimator Instantiation We estimate the effect of a policy’s recommended action on year-over-year gross-profit growth from logged data. Treatment is defined per action: for a target action a, let A∈0,1A∈\0,1\ indicate whether a was taken, so that the treated arm consists of logged states in which a was taken and the control arm consists of states in which a different catalogue action was taken. A single multi-label propensity model supplies e(x)=P(A=1∣x)e(x)=P(A=1 x) for every action, and each action is stratified independently. For a business with covariates x (a 768768-dimensional embedding of the business state, encoded with a sentence-embedding model), let Y denote the realized year-over-year gross-profit growth. We fit a propensity model e e (a multi-label MLP over the state embedding) and an outcome model with separate treated and control heads μ^1,μ^0 μ_1, μ_0 (neural networks predicting Y under the action and under its absence). The doubly-robust AIPW pseudo-outcome for each logged business is ϕi=(μ^1(xi)−μ^0(xi))+Aie^(xi)(Yi−μ^1(xi))−1−Ai1−e^(xi)(Yi−μ^0(xi)), split _i= ( μ_1(x_i)- μ_0(x_i) )&+ A_i e(x_i) (Y_i- μ_1(x_i) )\\ &- 1-A_i1- e(x_i) (Y_i- μ_0(x_i) ), split (4) which is unbiased if either the propensity or the outcome model is correctly specified (2). What is computed per recommendation. The audit proceeds in two stages, and it is worth stating precisely which quantity each stage produces. In stage one, on logged data where the treatment AiA_i and the outcome YiY_i are both observed, we evaluate the AIPW pseudo-outcome of Eq. 4 for every logged business and average it within each (action, propensity-decile) stratum. Strata that fail minimum-support or effective-sample-size gates are marked invalid and excluded. This yields a table of stratum-level effect estimates spanning the 4949 actions that clear the eligibility threshold and 1010 propensity bins per action. In stage two, each held-out recommendation is mapped to an action, the propensity of its business state for that action is predicted and assigned to a bin, and the recommendation receives the corresponding stratum-level effect. Equation 4 is therefore evaluated only on logged businesses; a held-out recommendation receives a stratum-level lookup rather than its own pseudo-outcome, because advice that was never acted upon has no observed AiA_i or YiY_i. The estimand is thus the mean stratum-level effect of the (action, propensity-bin) cells into which a policy’s recommendations fall, and policies differ only through which actions they select and how their states distribute across propensity bins. The three reported metrics are computed over this set of per-recommendation values. The estimated lift (DR-CATE) is their mean. The downside rate (DR%) is the fraction whose assigned stratum effect is negative. The conditional value-at-risk CVaR0.10CVaR_0.10 is the mean over the worst 10%10\%. Higher lift is better; a lower downside rate and a less negative CVaR0.10CVaR_0.10 are better. The audit follows the same protocol as the judge evaluation. Each policy is evaluated on 500500 randomly sampled held-out business states over 55 independent runs with different random seeds. We compute the three metrics within each run, and report the mean across runs together with a 95%95\% confidence interval over the five run-level estimates. Reporting uncertainty at the level of runs rather than individual recommendations means the intervals reflect run-to-run variation in the policy’s own sampling, which is the variation that matters when comparing policies. The mapper, the propensity model, and the outcome model are held fixed across all policies, so every system is scored by the same instruments. Action mapper. Recommendations are mapped to the action taxonomy by a separate LLM classifier, distinct from and independent of the Claude Opus 4.5 reward judge, so that mapping noise cannot directly inflate the audit. Given a recommendation and the fixed action catalogue, the mapper returns the single best-matching action together with a confidence score; recommendations that fall below a confidence threshold, or that match no catalogue action, are assigned a dedicated no-match label and excluded from scoring rather than forced onto an ill-fitting action. We summarize the mapper here and will release a synthetic, anonymized version with the reproducibility artifacts. A compact summary of the evaluation setup is given in Table 2. Item Value Evaluation unit company-month Sample per run 500 business states Runs per policy 5 (different seeds) Action catalogue 60 actions Action categories 10 Outcome YoY gross-profit growth Causal estimator doubly-robust AIPW (DR-CATE) Propensity model multi-label MLP Outcome model neural, treated/control heads State embedding 768-dim sentence embedding Holdout split company-level Free-text mapping LLM mapper (no-match option) Table 2: Evaluation/data card for the judge-independent causal audit. 4.3 Results Judge evaluation. Table 3 reports the rubric scores. After GRPO training, Qwen3.5-27B reaches 9.5149.514, the top score on the leaderboard, ahead of Claude Opus 4.6 (9.3659.365), GPT-5.4 (8.9498.949), and Claude Sonnet 4.5 (8.7128.712), and well above the untrained base model (8.4578.457). Two caveats apply to this table. First, the judge belongs to the Claude family, which may introduce a small self-preference bias in favor of Claude baselines; this makes the comparison conservative for our model. Second, and more importantly, our policy was trained against this same judge, so its rubric score is in part a measure of successful optimization rather than an independent assessment. We therefore read Table 3 as evidence that training achieved its objective, and treat the judge-independent causal audit below as the primary comparative evidence. Model Score 95% CI Qwen3.5-27B-GRPO (ours) 9.514 [9.505, 9.524] Claude Opus 4.6 9.365 [9.354, 9.376] Claude Opus 4.5 8.982 [8.954, 9.010] GPT-5.4 8.949 [8.911, 8.988] Claude Sonnet 4.5 8.712 [8.644, 8.780] Qwen3.5-27B (base) 8.457 [8.384, 8.531] Table 3: LLM-as-a-judge rubric scores, n=2,500n=2,500 per model. Scores are the mean number of the D=10D=10 quality criteria satisfied, i.e. the quality term c(a)c(a) of Eq. 2 rescaled to [0,10][0,10]; the safety gate zeroes the score of any recommendation judged unsafe. Our GRPO-trained model ranks first. Causal evaluation. Table 4 reports the causal off-policy results. Our policy attains an estimated gross-profit lift of 0.02280.0228 [0.0211,0.0246][0.0211,0.0246], approximately twice that of the strongest commercial baseline, Claude Opus 4.6 (0.01040.0104, a ratio of 2.20×2.20×), together with the lowest downside rate (0.1550.155 vs. 0.2320.232) and the least negative tail risk (CVaR0.10=−0.073CVaR_0.10=-0.073 vs. −0.100-0.100), meaning that even its worst-decile recommendations are less harmful on average. Our policy ranks first on all three metrics, and its confidence interval does not overlap that of any other policy on any metric; under a Welch t-test over the five per-trial means, every pairwise comparison with our policy is significant at p<0.01p<0.01. The commercial baselines separate clearly under the audit. Claude Opus 4.6 is the only commercial system with a lift that is both positive and comfortably distinguishable from zero (p<0.001p<0.001); GPT-5.4 is positive but noisier (0.00820.0082, p=0.015p=0.015, and the widest interval of any policy); Claude Sonnet 4.5 is not distinguishable from zero (0.00280.0028 [0.0000,0.0055][0.0000,0.0055], p=0.051p=0.051); and Claude Opus 4.5 is estimated to be negative (−0.0025-0.0025 [−0.0043,−0.0007][-0.0043,-0.0007]), i.e. its recommendations map to actions that are, on average, associated with slightly worse subsequent gross-profit growth. Downside rate tracks this ordering closely: the three weakest policies place roughly a third of their recommendations on actions with negative estimated effect, against 15.5%15.5\% for ours. Policy Lift (DR-CATE) ↑ DR% ↓ 0.10CVaR_0.10 ↑ Qwen3.5-27B-GRPO (ours) 0.0228±0.00170.0228± 0.0017 0.155±0.0170.155± 0.017 −0.073±0.003-0.073± 0.003 Qwen3.5-27B (base) 0.0170±0.00240.0170± 0.0024 0.194±0.0180.194± 0.018 −0.094±0.009-0.094± 0.009 Claude Opus 4.6 0.0104±0.00310.0104± 0.0031 0.232±0.0210.232± 0.021 −0.100±0.008-0.100± 0.008 GPT-5.4 0.0082±0.00560.0082± 0.0056 0.327±0.0400.327± 0.040 −0.124±0.007-0.124± 0.007 Claude Sonnet 4.5 0.0028±0.00280.0028± 0.0028 0.320±0.0270.320± 0.027 −0.114±0.006-0.114± 0.006 Claude Opus 4.5 −0.0025±0.0018-0.0025± 0.0018 0.362±0.0210.362± 0.021 −0.116±0.006-0.116± 0.006 Table 4: Judge-independent causal off-policy evaluation: estimated gross-profit lift (DR-CATE), downside rate (DR%), and tail risk (CVaR0.10CVaR_0.10). Policies are ordered by lift. Each entry is the mean over 55 independent runs (500500 sampled business states per run, different seed per run) ± the half-width of a 95%95\% confidence interval computed over the five run-level estimates. Higher lift and higher (less negative) CVaR0.10CVaR_0.10 are better; lower DR% is better. Our GRPO policy ranks first on all three metrics, with no confidence-interval overlap against any other policy on any metric. Effect of GRPO training. Comparing our policy against its own starting point isolates the contribution of GRPO from that of the base model. Training raises the estimated lift from 0.01700.0170 to 0.02280.0228, a relative improvement of 34%34\% (Welch t-test over the five run-level estimates, p=0.0009p=0.0009; because all policies are evaluated under the same five seeds, a paired test on the same estimates gives p=0.003p=0.003), while cutting the downside rate from 0.1940.194 to 0.1550.155 (p=0.002p=0.002) and improving tail risk from −0.094-0.094 to −0.073-0.073 (p=0.002p=0.002). The gain is therefore not merely a property of the base checkpoint: GRPO with the safety-gated rubric reward moves the policy toward actions with higher estimated effect and away from actions with negative estimated effect, which is precisely the behavior the safety gate was designed to induce. The two evaluations do not rank identically. The audit and the judge agree that our GRPO policy is best, but they disagree elsewhere, and the disagreement is informative. The untrained Qwen3.5-27B base model ranks last on the judge rubric (8.4578.457, Table 3) yet second on the causal audit (0.01700.0170, Table 4), above every commercial system. Conversely, Claude Opus 4.5 scores well on the rubric (8.9828.982) but has a negative estimated lift. We read this as evidence that the two evaluations measure genuinely different things. The rubric rewards well-formed advice: concrete figures, named entities, quantified impact, acknowledged risk. The audit rewards advice that maps to actions historically associated with gross-profit growth, and is indifferent to how well that advice is written. Commercial models are strong at the former and only middling at the latter, whereas the base model produces less polished recommendations that nonetheless land on reasonable actions. This divergence is the clearest available evidence that the audit is not a proxy for the judge; had it simply re-scored rubric quality, the base model would have ranked last on both. It also means the joint result carries more weight than either evaluation alone: our policy is the only system that ranks first under both. 5 Discussion Three findings stand out. First, GRPO with a finance-specific reward and explicit safety gate lifts an open-weight model above strong commercial LLMs on the judge rubric (Table 3). This indicates that targeted reward design can substitute for raw scale on a narrow, high-value task. Second, the causal audit, which is independent of the judge, moves in the same direction (Table 4): the GRPO policy attains a higher estimated gross-profit lift, a lower downside rate, and a less negative tail risk than every commercial baseline, with non-overlapping confidence intervals throughout. Because the audit is computed from logged outcomes and does not rely on the reward judge, this agreement makes pure judge adaptation a less likely explanation for the gains, although it does not establish realized impact. Third, and less expected, the two evaluations rank the other systems quite differently. That the untrained base model is last on the rubric but second on the audit shows the audit is not a restatement of the judge, and it also cautions against reading either evaluation as a complete measure of advice quality on its own: a policy can produce recommendations that are well-formed but point at unhelpful actions, or poorly-formed but point at sensible ones. Only a system that scores well on both, as ours does, is supported by both lines of evidence. The safety gate and the risk-aware metrics are complementary. The gate suppresses harmful recommendations during training, while the downside rate and tail-risk metrics verify after training that the resulting policy places less mass on actions with negative estimated effects; the drop in downside rate from 0.1940.194 (base) to 0.1550.155 (ours) is direct evidence that this mechanism operates as intended. 6 Conclusions We formulate financial advice generation as a reinforcement learning problem and train an open-weight model with GRPO using a finance-specific reward that incorporates explicit safety constraints. The resulting model outperforms the strongest evaluated commercial baseline on both the judge rubric and the CATE audit, and is the only system to rank first under both. It achieves approximately twice the estimated gross-profit lift while maintaining the lowest downside rate and least negative tail risk of any policy evaluated, and improves over its own starting checkpoint by 34%34\% in estimated lift. We also find that the judge and the causal audit rank the baselines differently, which suggests that rubric quality and estimated business value are distinct axes and that reporting only one of them can be misleading. These results suggest that careful reward design, combined with a judge-independent causal audit to reduce the risk of reward hacking, provides a practical approach for building safer and more effective financial advice models. Future work will explore richer action taxonomies, evaluation by financial experts, and support for multi-step advice. Limitations Because randomized experiments are not available in this setting, our causal audit is computed from logged observational data, in common with off-policy evaluation more broadly. We therefore treat it as a judge-independent, complementary signal alongside the rubric score rather than as a standalone measure, and the two evaluations are designed to be read together. A second limitation is coverage. Recommendations that receive no confident match in the 6060-action catalogue are excluded from the audit rather than forced onto an ill-fitting action. This keeps the mapping conservative, but it means each policy is audited on the subset of its own output that the catalogue can express, and a policy whose unmatched recommendations differ systematically in quality from its matched ones would be scored on a non-representative subset. A richer catalogue would narrow this gap. Our audit also scores a single recommended action per business state; extending it to multi-step advice, where actions compound over a longer horizon, is a natural direction for future work. Finally, our study targets gross profit as the primary goal metric; while the framework supports other financial KPIs, we leave a broad multi-KPI evaluation to future work. References Abrevaya et al. (2015) J. Abrevaya, Y. Hsu, and R. P. Lieli Estimating conditional average treatment effects. Journal of Business & Economic Statistics 33 (4), p. 485–505. Cited by: §1, §2, §3.4. Bang and Robins (2005) H. Bang and J. M. Robins Doubly robust estimation in missing data and causal inference models. Biometrics 61 (4), p. 962–973. Cited by: §2, §3.4, §4.2. Bhattarai et al. (2026) M. Bhattarai, I. Boureima, N. R. Ranasinghe, S. Pakin, and D. O’Malley Rubric-grounded rl: structured judge rewards for generalizable reasoning. arXiv preprint arXiv:2605.08061. Cited by: §2. Drinkall et al. (2025) F. Drinkall, J. Pierrehumbert, and S. Zohren Forecasting credit ratings: a case study where traditional methods outperform generative llms. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narrative Processing (FNP), and the 1st Workshop on Large Language Models for Finance and Legal (LLMFinLegal), p. 118–133. Cited by: §2. Dudík et al. (2011) M. Dudík, J. Langford, and L. Li Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601. Cited by: §2, §3.4. Jiang et al. (2025) Z. Jiang, L. Zhao, R. Sun, R. Sun, Z. Li, J. Li, D. Jiang, Z. Bai, and C. Hua Alpha-r1: alpha screening with llm reasoning via reinforcement learning. arXiv preprint arXiv:2512.23515. Cited by: §2. Kaelbling et al. (1996) L. P. Kaelbling, M. L. Littman, and A. W. Moore Reinforcement learning: a survey. Journal of artificial intelligence research 4, p. 237–285. Cited by: §1. Klimaszewski et al. (2025) M. Klimaszewski, P. Chen, L. Guillou, I. Papaioannou, B. Haddow, and A. Birch AVENIBENCH: accessible and versatile evaluation of finance intelligence. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narrative Processing (FNP), and the 1st Workshop on Large Language Models for Finance and Legal (LLMFinLegal), p. 111–117. Cited by: §2. Lee et al. (2023) H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, et al. Rlaif vs. rlhf: scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. Cited by: §1, §2, §2. Robins et al. (1994) J. M. Robins, A. Rotnitzky, and L. P. Zhao Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association 89 (427), p. 846–866. Cited by: §2, §3.4. Rosero et al. (2025) A. G. F. Rosero, P. Grundmann, J. Freidank, W. Nejdl, and A. Loeser Evaluating financial literacy of large language models through domain specific languages for plain text accounting. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narrative Processing (FNP), and the 1st Workshop on Large Language Models for Finance and Legal (LLMFinLegal), p. 63–75. Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2, §3.2.