Paper deep dive
Reinforcement Learning with Conditional Expectation Reward
Changyi Xiao, Caijun Xu, Yixin Cao
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 6:16:22 AM
Summary
The paper introduces Conditional Expectation Reward (CER), a novel reinforcement learning mechanism for large language models that eliminates the need for handcrafted, domain-specific verifiers. By using the model itself as an implicit verifier, CER calculates the expected likelihood of generating a reference answer conditioned on the model's own generated answer. This provides a soft, graded reward signal that is effective across both mathematical and general reasoning domains, outperforming or matching traditional rule-based verification methods.
Entities (5)
Relation Signals (3)
Conditional Expectation Reward → evaluatedon → MATH-7.5K
confidence 95% · We evaluate the performance of CER across both mathematical and general reasoning domains... train models on two datasets: the mathematical dataset MATH-7.5K
Conditional Expectation Reward → evaluatedon → WebInstruct
confidence 95% · We evaluate the performance of CER across both mathematical and general reasoning domains... the general-domain dataset WebInstruct
Conditional Expectation Reward → generalizes → Reinforcement Learning with Verifiable Rewards
confidence 90% · To generalize RLVR to general domains, we propose the CER
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.10624v1
- Canonical: https://arxiv.org/abs/2603.10624v1
Trouble viewing inline? Open PDF directly →
Full Text
46,531 characters extracted from source content.
Expand or collapse full text
Reinforcement Learning with Conditional Expectation Reward Changyi Xiao Caijun Xu Yixin Cao Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism. The code is available at https://github.com/changyi7231/CER. Machine Learning, ICML 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated strong effectiveness in incentivizing the reasoning capabilities of large language models, which relies on a verifier to provide accurate and checkable reward signals during learning (Zhou et al., 2025). Such a verifier evaluates the correctness of a generated answer with respect to a given reference answer, and is typically implemented using carefully designed, domain-specific rules that enable deterministic verification (Guo et al., 2025; He et al., 2025). RLVR is particularly useful in mathematical reasoning tasks (Guo et al., 2025; Team et al., 2025), where answers admit canonical or easily normalized representations, allowing correctness to be verified reliably through exact matching or symbolic equivalence checks (Hugging Face, 2025). However, RLVR remains difficult to extend to broader reasoning domains such as physics, chemistry, finance, and other domains with open-form answers (Ma et al., 2025; Zhou et al., 2025; Liu et al., 2025). In these domains, valid answers often exhibit diverse surface forms and substantial semantic variation, making it challenging to specify exhaustive verification rules. Consequently, constructing reliable verifiers becomes costly or even infeasible, which substantially constrains the applicability of RLVR to narrowly scoped tasks with well-defined answer spaces. Moreover, rule-based verifiers typically provide binary feedback, assigning rewards only to strictly equivalent answers while treating all other answers as equally incorrect. As a result, they are unable to assign positive rewards to answers that are partially correct, thereby providing limited learning signals during optimization. To address these issues, we propose Conditional Expectation Reward (CER) to extend RLVR to general reasoning domains. Rather than relying on external verification rules or auxiliary verifier models, CER uses the large language model itself as an implicit verifier. By exploiting the model’s internal consistency with respect to a reference answer, CER provides a model-intrinsic reward signal that remains applicable even when explicit verification is unavailable. Specifically, CER measures the expected probability of generating the reference answer conditioned on the model’s generated answer, thereby producing a soft, graded reward signal to verify the generated answer. The underlying intuition is that when a generated answer is identical to, or strongly consistent with, the reference answer, the model will assign a higher conditional probability to reproducing the reference answer given that generation. We further show that CER can be theoretically interpreted as a smooth relaxation of the exact-match criterion, yielding reward values that reflect varying degrees of consistency between the generated and reference answers. This characteristic is particularly well suited to general reasoning domains, where partial correctness, semantic overlap, and multiple valid surface realizations are common. We finally conduct experiments to demonstrate the effectiveness of CER. We show that CER achieves great performance on general domains, both mathematical and non-mathematical domains. These findings highlight CER as a general and robust reward mechanism for RLVR, offering a practical solution for extending reinforcement learning to a wide range of reasoning domains. 2 Conditional Expectation Reward We first introduce RLVR, followed by the definition and theoretical properties of CER. We then present the empirical formulation of CER and the corresponding training objective, and finally describe an efficient procedure for computing CER. RLVR RLVR is a reinforcement learning paradigm in which the reward signal is objectively and automatically checkable by a verifier (Zhang et al., 2025). Specifically, for a given question q with a unique reference answer a∗=a∗(q)a^*=a^*(q), the policy model πθ _θ autoregressively generates a solution s and a final answer a to address the question. Here, the solution s does not include the final answer a. Owing to the autoregressive factorization, the policy model πθ _θ satisfies πθ(a,s|q)=πθ(s|q)πθ(a|s,q). _θ(a,s|q)= _θ(s|q) _θ(a|s,q). (1) This process yields a quadruple (q,s,a,a∗)(q,s,a,a^*). An illustrative example is provided in the following box. Question q: What is the value of x in the equation 2x+3=7?2x+3=7? Solution s: Solve the equation step by step. First, subtract 33 from both sides: 2x+3=7⇒ 2x=4.2x+3=7\; \;2x=4. Next, divide both sides by 22: x=2.x=2. Therefore, the value of x is Answer a: 2 Reference answer a∗a^*: 2 RLVR is formulated as the optimization of the following expected reward: ℒf(θ)=q∼,(s,a)∼πθ(⋅|q)[f(a,a∗(q))], _f(θ)=E_q ,(s,a) _θ(·|q)[f(a,a^*(q))], (2) where D is the distribution of q, and f(a,a∗(q))f(a,a^*(q)) is a reward that evaluates the correctness of the generated answer a with respect to the reference answer a∗a^* associated with question q, which is computed by a verifier f(⋅,⋅)f(·,·). In practice, such a verifier is often implemented as a set of carefully designed rules (Hugging Face, 2025). Rule-based verifiers are particularly effective in domains such as mathematics and code generation, where answers admit unambiguous representations and equivalence can be precisely defined. However, extending rule-based verification to general reasoning domains remains challenging (Ma et al., 2025; Zhou et al., 2025). In these domains, valid answers are often free-form and exhibit substantial variation. Consequently, it is difficult to design a rule-based verifier that is both complete and accurate, which limits the direct applicability of RLVR beyond domains with well-structured and formally verifiable answer spaces. We illustrate this limitation with a concrete example. Consider the following question, for which multiple answers may be semantically equivalent despite differing in surface form. A typical rule-based verifier would only treat a1a_1 as correct, while assigning zero reward to other valid answers such as a2a_2 and a3a_3. This behavior collapses semantically correct but lexically different answers into the same category as incorrect ones, leading to overly sparse and noisy reward signals. Such rigid verification discourages exploration of diverse yet correct answers and hampers effective learning in general reasoning settings. Question q: Is quantum physics deterministic? Reference answer a∗a^*: No Answer a1a_1: No Answer a2a_2: Quantum physics is not deterministic. Answer a3a_3: No, quantum physics is not deterministic; it is probabilistic. Definition To generalize RLVR to general domains, we propose the CER, which leverages the policy model itself as an implicit verifier, without relying on external verifiers or models. Instead of explicitly checking answer correctness, CER evaluates the internal consistency of the policy model with respect to a reference answer, thereby enabling applicability to general domains. For a quadruple (q,s,a,a∗)(q,s,a,a^*), we define CER as: ρ(a,a∗):= ρ(a,a^*):= s′∼πθ(⋅|q)[πθ(a∗|s′,q)|A=a] _s _θ(·|q) [ _θ(a^*|s ,q)\, |\,A=a ] = = s′∼πθ(⋅|q,a)[πθ(a∗|s′,q)]. _s _θ(·|q,a) [ _θ(a^*|s ,q) ]. (3) CER measures the expected likelihood of generating the reference answer a∗a^* given the condition that the model has generated an answer a. The intuition is that if the generated answer a is identical to, or strongly correlated with, the reference answer a∗a^*, then the policy model should assign a higher conditional probability to generating a∗a^* after having produced a. By optimizing the policy model with respect to CER, the model is encouraged to place higher probability mass on answers that are internally consistent with the reference answer, thereby implicitly guiding the generation process toward a∗a^* without requiring explicit verification. We illustrate CER with an example from training. For the following given question, the model generates three distinct answers, 14,13,94\14,13,94\. The answer 1414 receives the highest CER value (0.752), as it exactly matches the reference answer. The answer 1313 attains the second-highest CER value (0.313), reflecting its numerical proximity to the reference answer 1414. In contrast, the answer 9494 receives a near-zero CER value (0.00004), as it is numerically distant from the reference answer. Question q: How many positive multiples of 77 that are less than 10001000 end with the digit 33? Reference answer a∗a^*: 14 Generated Answers and CER (a,ρ(a,a∗))(a,ρ(a,a^*)): (14,0.752),(13,0.313),(94,0.00004)\(14,0.752),(13,0.313),(94,0.00004)\ Properties We summarize several fundamental properties of CER, which demonstrate the effectiveness of CER. • Boundedness. 0≤ρ(a,a∗)≤1.0≤ρ(a,a^*)≤ 1. Since πθ(a∗|s′,q)∈[0,1] _θ(a^*|s ,q)∈[0,1] for all (q,s′)(q,s ), the weighted sum of these probabilities also lies in [0,1][0,1]. As a result, CER provides a bounded and well-scaled reward signal that is suitable for stable optimization. • Minimum. ρ(a,a∗)=0⟺πθ(a∗|s′,q)=0 ρ(a,a^*)=0 _θ(a^*|s ,q)=0 for all (q,s′) with πθ(s′|q,a)>0. all (q,s ) with _θ(s |q,a)>0. ρ(a,a∗)ρ(a,a^*) is zero exactly when the policy assigns probability 0 to a∗a^* on any (q,s′)(q,s ) that appear in the conditional distribution defined by A=aA=a. In this case, once the policy has generated a, it is impossible for the model to regenerate a∗a^* under the same posterior distribution over (q,s′)(q,s ). • Maximum. ρ(a,a∗)=1⟺πθ(a∗|s′,q)=1 ρ(a,a^*)=1 _θ(a^*|s ,q)=1 for all (q,s′) with πθ(s′|q,a)>0. all (q,s ) with _θ(s |q,a)>0. ρ(a,a∗)ρ(a,a^*) reaches its maximum value only when the policy assigns probability 11 to the reference answer a∗a^* for every (q,s′)(q,s ) that can occur under the conditional distribution defined by A=aA=a. In this case, the policy cannot produce any alternative answer with positive probability. Consequently, ρ(a,a∗)=1⟹a=a∗ρ(a,a^*)=1 a=a^* • Self-Consistency. Theorem 1 (Exact-Match Case). If a=a∗a=a^*, then ρ(a∗,a∗)= ρ(a^*,a^*)= s′∼πθ(⋅|q)[πθ(a∗|s′,q)|A=a∗] _s _θ(·|q) [ _θ(a^*|s ,q)\, |\,A=a^* ] = = s′∼πθ(⋅|q,a∗)[πθ(a∗|s′,q)] _s _θ(·|q,a^*) [ _θ(a^*|s ,q) ] ≥ \;≥\; s∼πθ(⋅|q)[πθ(a∗|s,q)]. _s _θ(·|q) [ _θ(a^*|s,q) ]. with equality if and only if πθ(a∗|s,q) _θ(a^*|s,q) is constant over all (q,s)(q,s) such that πθ(s|q)>0 _θ(s|q)>0. See Appendix A for the proof. This shows that conditioning on the event that the policy has generated a∗a^* strictly increases the posterior predictive probability of regenerating a∗a^*, unless the policy assigns an identical likelihood to a∗a^* across all (q,s)(q,s) pairs with πθ(s|q)>0 _θ(s|q)>0. Therefore, CER exhibits a self-consistency amplification effect via posterior reweighting toward higher probability assigned to the reference answer in the exact-match case. • Equivalence. Theorem 2 (Value Equivalence). ℒρ(θ) _ρ(θ) =q∼,(s,a)∼πθ(⋅|q)[ρ(a,a∗(q))] =E_q ,(s,a) _θ(·|q)[ρ(a,a^*(q))] =q∼,(s,a)∼πθ(⋅|q)[(a=a∗(q))], =E_q ,(s,a) _θ(·|q)[I(a=a^*(q))], i.e., the expected CER objective is equivalent in value to the exact-match objective, where (a=a∗(q))I(a=a^*(q)) indicates whether a exactly matches a∗(q)a^*(q). Thus, ρ(a,a∗)ρ(a,a^*) can be interpreted as a soft generalization of the hard exact-match reward (a=a∗)I(a=a^*), while preserving the same expected value. CER yields a continuous-valued reward rather than a binary signal. This property allows CER to provide graded feedback that reflects varying degrees of consistency between the generated answer a and the reference answer a∗a^*, which is particularly beneficial in general domains where partial correctness or semantic similarity may be present. In summary, these properties show that CER is a well-behaved and principled reward function. It is bounded and properly scaled, admits clear minimum and maximum conditions, and exhibits a self-consistency amplification effect when the generated answer matches the reference answer. Moreover, CER is value-equivalent in expectation to the exact-match objective while providing a continuous, graded reward signal, thereby serving as a soft generalization of exact-match rewards in general domains. Empirical CER We next derive an empirical form of CER, as the definition in Eq. (2) is intractable due to the summation over all possible outcomes under πθ(a∗|s′,q) _θ(a^*|s ,q). To obtain a computable approximation, we apply Bayes’ rule and Monte Carlo sampling to derive an empirical estimator of CER: ρ(a,a∗)= ρ(a,a^*)= s′∼πθ(⋅|q)[πθ(a∗|s′,q)|A=a] _s _θ(·|q) [ _θ(a^*|s ,q)\, |\,A=a ] = = ∑s′πθ(s′|q,a)πθ(a∗|s′,q) _s _θ(s |q,a)\, _θ(a^*|s ,q) = = ∑s′πθ(s′|q)πθ(a|s′,q)πθ(a∗|s′,q)∑s′πθ(s′|q)πθ(a|s′,q) _s _θ(s |q) _θ(a|s ,q) _θ(a^*|s ,q) _s _θ(s |q) _θ(a|s ,q) ≈ ≈ ∑j=1Mπθ(a|sj,q)πθ(a∗|sj,q)∑j=1Mπθ(a|sj,q),sj∼πθ(⋅|q). _j=1^M _θ(a|s_j,q)\, _θ(a^*|s_j,q) _j=1^M _θ(a|s_j,q),s_j _θ(·|q). (4) The third line applies Bayes’ rule to rewrite πθ(s′|q,a) _θ(s |q,a) in terms of quantities compatible with the autoregressive factorization of large language models. The final line further applies Monte Carlo estimation by drawing M independent samples sj∼πθ(⋅|q)s_j _θ(·|q) to produce an empirical estimate of CER. The resulting empirical CER is a normalized likelihood-weighted average, where each term πθ(a∗|sj,q) _θ(a^*|s_j,q) is weighted by πθ(a|sj,q) _θ(a|s_j,q). This weighting captures the joint consistency of a and a∗a^* under the same conditional context, so that samples for which the policy assigns high probability to both a and a∗a^* contribute more to the estimator, leading to a larger value of CER. Objective We finally define the objective based on the empirical CER. For a quadruple (q,s,a,a∗)(q,s,a,a^*), the reward function is defined as: R(q,s,a,a∗)=∑j=1Mπθ(a|sj,q)πθ(a∗|sj,q)∑j=1Mπθ(a|sj,q), R(q,s,a,a^*)= _j=1^M _θ(a|s_j,q) _θ(a^*|s_j,q) _j=1^M _θ(a|s_j,q), where sj∼πθ(⋅|q). s_j _θ(·|q). (5) Using this reward function, we define the training objective as ℒρ(θ) _ρ(θ) =q∼,(s,a)∼πθ(⋅|q)[ρ(a,a∗)] =E_q ,(s,a) _θ(·|q)[ρ(a,a^*)] ≈q∼,(s,a)∼πθ(⋅|q)[R(q,s,a,a∗)]. _q ,(s,a) _θ(·|q)[R(q,s,a,a^*)]. (6) Then the corresponding policy gradient is given by ∇θℒρ(θ) _θL_ρ(θ) ≈ ≈ q∼,(s,a)∼πθ(⋅|q)[R(q,s,a,a∗)∇θlogπθ(a,s|q)] _q ,(s,a) _θ(·|q)[R(q,s,a,a^*) _θ _θ(a,s|q)] ≈ ≈ q∼[1N∑i=1NR(q,si,ai,a∗)∇θlogπθ(ai,si|q)], _q [ 1N _i=1^NR(q,s_i,a_i,a^*) _θ _θ(a_i,s_i|q)], where (si,ai)∼πθ(⋅|q). (s_i,a_i) _θ(·|q). (7) For each q, we sample N independent (si,ai)(s_i,a_i) from πθ(⋅|q) _θ(·|q) for estimating the gradient. The reward R(q,si,ai,a∗)R(q,s_i,a_i,a^*) is treated as a fixed scalar with respect to θ during optimization to detach it from gradient computation for stable policy learning (Ziegler et al., 2019; Ouyang et al., 2022). Figure 1: An illustration of CER computation, where RN(⋅·) denotes row normalization. The left panel depicts the generation process of the quadruple (q,si,ai,a∗)(q,s_i,a_i,a^*), while the right panel shows the CER computation for the quadruple, corresponding to Eq. (8). Efficiency We now describe an efficient procedure for computing CER by reusing samples, avoiding redundant computations and adjusting the hyperparameter. As shown in Eq. (2), computing CER requires sampling M independent solutions sjj=1M\s_j\_j=1^M from πθ(⋅|q) _θ(·|q). However, CER can be seamlessly integrated into policy gradient without additional sampling. Specifically, for each question q, we already sample N independent sii=1N\s_i\_i=1^N from πθ(⋅|q) _θ(·|q) to estimate the policy gradient. These same samples can be directly reused for reward computation by setting sjj=1M:=sii=1N\s_j\_j=1^M:=\s_i\_i=1^N. As a result, CER incurs no extra sampling cost. To understand the computation of CER more intuitively, we further show that the CER computing can be expressed in a tensorized form. Let M:=NM:=N and sjj=1M:=sii=1N\s_j\_j=1^M:=\s_i\_i=1^N, and define ∈[0,1]N R∈[0,1]^N with entries i=R(q,si,ai,a∗) R_i=R(q,s_i,a_i,a^*), ∈[0,1]N×M W∈[0,1]^N× M with entries ij=πθ(ai|sj,q) W_ij= _θ(a_i|s_j,q), and ∈[0,1]M P∈[0,1]^M with entries j=πθ(a∗|sj,q) P_j= _θ(a^*|s_j,q). The reward vector R can then be written as =−1, R= D^-1 W P, (8) where D is a diagonal matrix whose entries are the row sums of W. See Figure 1 for an illustration. Although this approach avoids additional sampling, directly computing Eq. (8) still requires M(N+1)M(N+1) forward passes to compute the entries of W and P. We further reduce this cost by eliminating redundant computations. In particular, if two sampled answers ai1a_i_1 and ai2a_i_2 are identical, then their corresponding rewards satisfy R(q,si1,ai1,a∗)=R(q,si2,ai2,a∗)R(q,s_i_1,a_i_1,a^*)=R(q,s_i_2,a_i_2,a^*), and thus the reward only needs to be computed once for each unique answer. Besides, the hyperparameter M controls a trade-off between computational efficiency and reward estimation accuracy. Larger values of M yield more accurate estimates of CER at the cost of increased computation, while smaller values improve efficiency with a loss in precision. We can adjust M to achieve a balance between performance and efficiency. Table 1: The performance of models trained on a general-domain dataset. Methods MATH500 AMC23 AIME2024 AIME2025 MMLU-Pro SuperGPQA Average Qwen3‑4B‑Base Base 62.6 40.5 10.6 8.1 42.0 21.0 30.8 Exact-Match 78.0 61.4 22.3 20.6 62.9 33.5 46.5 Rule-based 84.5 65.8 21.7 18.5 62.3 32.6 47.6 VeriFree 83.4 62.2 19.6 16.7 58.7 29.1 44.9 General-verifier 84.6 64.1 21.9 17.7 63.7 34.2 47.7 CER 81.6 67.7 22.8 21.3 63.8 35.2 48.7 Rule+CER 85.6 66.6 22.5 19.9 64.1 35.2 49.0 Qwen3‑8B‑Base Base 73.9 53.1 14.6 12.3 51.9 27.0 38.9 Exact-Match 82.4 66.6 25.4 20.4 66.2 36.2 49.5 Rule-based 86.0 72.0 26.7 21.0 66.6 37.7 51.7 VeriFree 86.0 61.4 22.3 19.6 65.4 35.5 48.4 General-verifier 84.8 74.1 25.0 21.7 67.3 37.7 51.8 CER 87.2 72.3 25.8 20.6 69.7 38.4 52.3 Rule+CER 85.2 76.4 23.5 20.6 71.0 38.3 52.5 Table 2: The performance of models trained on a mathematical dataset. Methods MATH500 AMC23 AIME2024 AIME2025 MMLU-Pro SuperGPQA Average Qwen3‑4B‑Base Base 62.6 40.5 10.6 8.1 42.0 21.0 30.8 Exact-match 81.2 57.3 17.7 15.6 46.7 24.2 40.5 Rule-based 84.2 63.1 22.9 21.5 62.5 32.2 47.7 VeriFree 81.5 62.7 19.8 18.1 59.9 30.9 45.5 General-verifier 83.6 63.0 19.8 19.0 58.5 30.9 45.7 CER 84.1 63.6 24.8 20.0 60.8 32.1 47.6 Rule+CER 85.0 67.5 23.3 20.8 61.2 31.3 48.2 Qwen3‑8B‑Base Base 73.9 53.1 14.6 12.3 51.9 27.0 38.9 Exact-match 80.0 63.4 16.5 13.5 61.4 33.5 44.7 Rule-based 86.7 70.2 26.3 22.7 65.8 35.1 51.1 VeriFree 85.0 68.4 22.9 20.2 62.3 32.4 48.5 General-verifier 86.0 69.4 26.7 20.6 64.3 34.9 50.3 CER 87.2 70.9 23.8 23.1 64.8 35.0 50.8 Rule+CER 87.3 72.0 26.5 21.0 65.6 36.0 51.4 3 Experiments We first describe the experimental settings in Section 3.1. We then present the main experimental results in Section 3.2 to evaluate the effectiveness of the proposed method. Next, we analyze the computational efficiency of CER in Section 3.3. Finally, in Section 3.4, we provide a detailed visualization of the CER computation process to offer further insights into its behavior and properties. 3.1 Settings Datasets We evaluate the performance of CER across both mathematical and general reasoning domains. Accordingly, we train models on two datasets: the mathematical dataset MATH-7.5K (Hendrycks et al., 2021) and the general-domain dataset WebInstruct (Ma et al., 2025). For WebInstruct, we retain only non-mathematical questions at the university difficulty level to focus on general-domain beyond mathematics, yielding a dataset of 50K questions. WebInstruct includes domains such as physics, chemistry, biology, finance and so on. This subset spans a wide range of disciplines, such as physics, chemistry and biology. Evaluation We evaluate the models on four mathematical datasets, MATH500 (Lightman et al., 2023), AMC23 (Art of Problem Solving, 2025b), AIME2024 and AIME2025 (Art of Problem Solving, 2025a), and two general-domain datasets, SuperGPQA (Du et al., 2025) and MMLU-Pro (Wang et al., 2024). Performance is reported using the pass@1 metric. For the mathematical datasets, pass@1 is computed using a rule-based verifier (Hugging Face, 2025). For the general-domain datasets, which consist of multiple-choice questions, pass@1 is computed via exact matching. For each mathematical dataset, we conduct 16 evaluation runs and report the average performance. Baselines We compare CER with several baseline verifiers. These include the exact-match verifier, which checks whether the generated answer exactly matches the reference answer; a model-based verifier, General-verifier (Ma et al., 2025), which employs an external large language model to assess answer correctness; and a perplexity-based verifier, VeriFree (Zhou et al., 2025), which uses the perplexity of the reference answer for verification, a rule-based verifier (Hugging Face, 2025), which verifies the correctness by utilizing handcrafted rules. Hyperparameter settings We set the batch size of questions to 3232, the number of solutions N to 1616, M in Eq. (2) to 16, the learning rate to 10−610^-6, epoch to 1 for WebInstruct dataset and epoch to 5 for MATH-7.5K dataset. For training, we use temperature =1.0=1.0 and top-p=1.0p=1.0, while for evaluation we use temperature =0.6=0.6, top-p=0.95p=0.95 and top-k=20k=20. The maximum question length is set to 20482048 tokens, and the maximum output length is set to 40964096 tokens for training and 81928192 for evaluation. We utilize RLOO (Kool et al., 2019) as the optimization method. 3.2 Results CER demonstrates strong generality across domains. On the general-domain training dataset (Table 2), CER achieves the highest average performance among all compared methods for both Qwen3-4B-Base and Qwen3-8B-Base (except the combined method Rule+CER). In particular, CER consistently outperforms exact-match rewards and the perplexity-based rewards VeriFree, and exceeds the performance of rule-based verifiers and learned verifiers General-verifier across most evaluation benchmarks. The advantage of CER is especially pronounced on general-domain evaluation datasets such as MMLU-Pro and SuperGPQA, where it achieves consistent performance gains. Notably, this advantage holds without relying on domain-specific handcrafted rules or models. When trained on the mathematical dataset (Table 2), CER attains performance comparable to rule-based rewards and outperform learned verifier approaches. Despite the absence of an external verifier, CER maintains strong results across mathematical benchmarks, indicating that it does not overfit to a specific domain. Taken together, these results suggest that CER can serve as a unified reward formulation applicable to both general-domain and mathematics-oriented reasoning tasks. CER substantially improves over exact-match rewards. As established in Theorem 2, CER can be viewed as a soft generalization of the hard exact-match reward. While exact-match provides a binary signal that only distinguishes perfectly correct answers from all others, CER assigns continuous-valued rewards that reflect partial correctness to the reference answer. Empirically, this difference translates into consistent gains over exact-match training across both datasets and model scales. The graded feedback provided by CER yields denser and more informative learning signals, which is particularly beneficial when correct answers admit surface-level variations. CER is complementary to rule-based rewards. We further investigate a simple yet effective strategy for combining CER with rule-based rewards, in which the final reward is defined as the average of the CER score and the rule-based reward. As shown in Tables 2 and 2, the combined approach (Rule+CER) generally achieves better performance than either method used in isolation. This demonstrates that integrating them yields a more informative training signal. On the general-domain training dataset, CER enhances rule-based methods by providing graded rewards: rule-based schemes assign positive reward only to strictly equivalent answers and treat all other outputs as equally incorrect. As a result, they fail to differentiate partially correct answers, leading to sparse and uninformative learning signals. CER alleviates this limitation by assigning non-binary rewards that better reflect answer quality. On the mathematical dataset, rule-based rewards in turn complement CER. In this domain, rule-based methods more reliably capture mathematical equivalence, thereby correcting errors introduced by imperfect similarity estimation in CER. Overall, these results highlight the complementary strengths of CER and rule-based rewards and motivate their combined use across different domains. Figure 2: This figure illustrates the computation of CER as defined in Eq. (8). The left panel shows the question, the reference answer, and the 16 generated answers. The right panel depicts the components: the reward vector R (left column), the row-normalized matrix −1 D^-1 W (central block), and the reference-likelihood vector P (right column). 3.3 Efficiency We analyze the computational efficiency of various rewards and their corresponding performance. In CER, the computational cost is governed by the hyperparameter M in Eq. (2), which specifies the number of samples used to estimate the reward. By tuning M, CER enables a flexible mechanism to balance runtime efficiency and reward fidelity. Table 3 reports runtime and average performance across methods, with all experiments conducted on four NVIDIA H100 GPUs. For CER, increasing M improves performance at the cost of higher runtime overhead. Empirically, CER exhibits a smooth and controllable trade-off, enabling practitioners to select M that balances efficiency and performance under given computational constraints. Exact-match rewards incur the lowest overhead but yield inferior performance. CER with smaller M and rule-based rewards achieve reasonable performance while remaining efficient, whereas CER with large M, VeriFree, and General-verifier incur higher runtime costs due to multiple large language model queries during reward computation. Table 3: The average performance across six datasets and corresponding runtime for each method. Model Performance Runtime Exact-match 46.5 45.2h Rule 47.6 54.7h VeriFree 44.9 58.7h General-verifier 47.7 57.5h CER (M=1) 46.4 47.0h CER (M=2) 47.7 52.2h CER (M=4) 48.0 55.6h CER (M=8) 48.2 59.3h CER (M=16) 48.7 67.4h 3.4 Visualization of CER Computing To better illustrate the computing of CER, we visualize the components involved in Eq. (8). Recall that i R_i denotes the CER associated with the quadruple (q,si,ai,a∗)(q,s_i,a_i,a^*), ij=πθ(ai|sj,q) W_ij= _θ(a_i|s_j,q) represents the likelihood of generating answer aia_i conditioned on solution sjs_j and question q, and D is the diagonal matrix whose entries are the row sums of W, j=πθ(a∗|sj,q) P_j= _θ(a^*|s_j,q) measures the likelihood of producing the reference answer a∗a^* given (sj,q)(s_j,q). Figure 2 presents a training example. The left panel shows the question q, the reference answer a∗a^*, and the 16 generated answers ai\a_i\. The right panel visualizes the vectors R and P, together with −1 D^-1 W. Specifically, the left column corresponds to the reward vector R, the central block depicts the normalized matrix −1 D^-1 W with each row summing to one, and the right column corresponds to the vector P. Several observations can be drawn from this visualization. First, CER effectively captures surface-level variation and semantic similarity among answers, which is particularly important for questions with free-form answers. In this example, the 16 generated answers contain 10 unique surface forms that nevertheless share similar semantics, such as “No, quantum physics is generally considered non-deterministic.” and “No, quantum physics is not deterministic.” CER assigns positive rewards to all such semantically consistent answers, whereas exact-match or rule-based methods would assign positive reward only to the strictly matching answer “No” This demonstrates that CER provides richer and more informative reward signals for general-domain reasoning tasks. Second, answers that receive higher CER rewards tend to exhibit stronger alignment with solutions that also assign high likelihood to the reference answer. Since most entries of P are relatively large, this alignment can be examined by inspecting the sparsity patterns of the normalized matrix −1 D^-1 W. From top to bottom, the rows of the normalized matrix become increasingly sparse, which results in smaller CER values. This trend is consistent with the formulation in Eq. (2), where reduced overlap with other solutions leads to a lower reward. Third, the visualization suggests that increasing the value of M in Eq. (2) can improve performance. A larger M can yield a denser and more stable normalized matrix, leading to a more accurate estimation of CER and, consequently, improved performance. Finally, the figure also illustrates that identical answers receive identical CER rewards. For example, since a1=a2a_1=a_2, the corresponding rows in the normalized matrix are identical, which results in equal CER values for a1a_1 and a2a_2. This property reflects the consistency of CER with respect to repeated answer instances. 4 Related Work RLVR RLVR (Lambert et al., 2024; Guo et al., 2025; Team et al., 2025) has emerged as a prominent paradigm for improving the reasoning performance of large language models. RLVR relies on rule-based verifiers to provide accurate and stable reward signals, such as the math-verify library (Hugging Face, 2025) for mathematical reasoning tasks (Guo et al., 2025) and the SandboxFusion toolbox (Cheng et al., 2024) for code generation (Luo et al., 2025; He et al., 2025). These rule-based verification methods are particularly effective in domains where answers admit unambiguous representations and deterministic equivalence rules can be readily constructed. However, their applicability is limited in general reasoning domains, where valid answers are often open-form and exhibit substantial surface variation. In contrast, CER aims to extend RLVR to such general domains. General Domains Existing verification methods applicable to general reasoning domains can be broadly categorized into model-based verifiers and perplexity-based verifiers. Model-based verifiers employ a fine-tuned large language model to assess the correctness of a generated answer with respect to a reference answer. For instance, Kimi-k1.5 (Team et al., 2025) fine-tunes a model on large-scale verification data to endow it with verification capabilities. General-Verifier (Ma et al., 2025) further develops a generative, model-based verifier trained specifically for chain-of-thought answer verification, enabling more nuanced and context-aware judgments. Perplexity-based verifiers, in contrast, define reward signals based on the likelihood or perplexity of the reference answer under the large language model. VeriFree (Zhou et al., 2025) combines perplexity-based rewards with variance reduction techniques to construct a training objective. Building on this line of work, Nover (Liu et al., 2025) introduces length normalization to mitigate length bias, while RLPR (Yu et al., 2025) reformulates perplexity as a sum of token-level probabilities to further address sensitivity to answer length. In contrast to both model-based and perplexity-based approaches, CER leverages self-consistency between generated answers and the reference answer to produce soft, graded, and model-intrinsic reward signals. This design enables reliable feedback without requiring additional verifier models or handcrafted rules, making CER applicable across a wide range of general reasoning domains. 5 Conclusion In this paper, we propose CER as a general framework for extending RLVR beyond domains that rely on strict, rule-based verification. By leveraging the large language model itself as an implicit verifier, CER produces soft, graded reward signals that reflect partial correctness and semantic consistency, thereby overcoming the limitations of binary rule-based feedback. Our theoretical analysis shows that CER can be viewed as a smooth relaxation of exact-match evaluation, providing a principled connection to conventional verifiable rewards. Empirically, we demonstrate that CER is effective across both mathematical and general-domain reasoning tasks. Together, these results indicate that CER offers a flexible and broadly applicable mechanism for guiding reinforcement learning in large language models, enabling more general and robust reasoning capabilities. Impact Statement This paper seeks to advance the state of machine learning by introducing new insights and techniques. Progress in this field can enable improvements across many domains, but the work presented here is foundational. Consequently, no specific positive or negative impacts are uniquely attributable to this work. References Art of Problem Solving (2025a) External Links: Link Cited by: §3.1. Art of Problem Solving (2025b) External Links: Link Cited by: §3.1. Y. Cheng, J. Chen, J. Chen, L. Chen, L. Chen, W. Chen, Z. Chen, S. Geng, A. Li, B. Li, et al. (2024) FullStack bench: evaluating llms as full stack coders. arXiv preprint arXiv:2412.00535. Cited by: §4. X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, et al. (2025) Supergpqa: scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739. Cited by: §3.1. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §4. J. He, J. Liu, C. Y. Liu, R. Yan, C. Wang, P. Cheng, X. Zhang, F. Zhang, J. Xu, W. Shen, et al. (2025) Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312. Cited by: §1, §4. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §3.1. Hugging Face (2025) Math-verify: a robust mathematical expression evaluation system. Note: GitHub repository and Python packagehttps://github.com/huggingface/Math-Verify, version 0.8.0 Cited by: §1, §2, §3.1, §3.1, §4. W. Kool, H. van Hoof, and M. Welling (2019) Buy 4 REINFORCE samples, get a baseline for free!. Cited by: §3.1. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. (2024) Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §4. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. In The Twelfth International Conference on Learning Representations, Cited by: §3.1. W. Liu, S. Qi, X. Wang, C. Qian, Y. Du, and Y. He (2025) NOVER: incentive training for language models via verifier-free reinforcement learning. arXiv preprint arXiv:2505.16022. Cited by: §1, §4. M. Luo, S. Tan, R. Huang, A. Patel, A. Ariyak, Q. Wu, X. Shi, R. Xin, C. Cai, M. Weber, et al. (2025) Deepcoder: a fully open-source 14b coder at o3-mini level. Notion Blog. Cited by: §4. X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen (2025) General-reasoner: advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652. Cited by: §1, §2, §3.1, §3.1, §4. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §2. K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025) Kimi k1. 5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: §1, §4, §4. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. (2024) Mmlu-pro: a more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems 37, p. 95266–95290. Cited by: §3.1. T. Yu, B. Ji, S. Wang, S. Yao, Z. Wang, G. Cui, L. Yuan, N. Ding, Y. Yao, Z. Liu, et al. (2025) RLPR: extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506.18254. Cited by: §4. K. Zhang, Y. Zuo, B. He, Y. Sun, R. Liu, C. Jiang, Y. Fan, K. Tian, G. Jia, P. Li, et al. (2025) A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv:2509.08827. Cited by: §2. X. Zhou, Z. Liu, A. Sims, H. Wang, T. Pang, C. Li, L. Wang, M. Lin, and C. Du (2025) Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493. Cited by: §1, §1, §2, §3.1, §4. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving (2019) Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. Cited by: §2. Appendix A Appendix A.1 Proof Theorem 1 (Exact-Match Case). If a=a∗a=a^*, then ρ(a∗,a∗)= ρ(a^*,a^*)= s′∼πθ(⋅|q)[πθ(a∗|s′,q)|A=a∗] _s _θ(·|q) [ _θ(a^*|s ,q)\, |\,A=a^* ] = = s′∼πθ(⋅|q,a∗)[πθ(a∗|s′,q)] _s _θ(·|q,a^*) [ _θ(a^*|s ,q) ] ≥ \;≥\; s∼πθ(⋅|q)[πθ(a∗|s,q)]. _s _θ(·|q) [ _θ(a^*|s,q) ]. with equality if and only if πθ(a∗|s,q) _θ(a^*|s,q) is constant over all (q,s)(q,s) such that πθ(s|q)>0 _θ(s|q)>0. Proof. Since ρ(a,a∗)ρ(a,a^*) is defined for a generated answer a, we always have the probability Pr(A=a)>0 (A=a)>0. In particular, if a=a∗a=a^*, then Pr(A=a∗)>0 (A=a^*)>0, so conditioning on A=a∗A=a^* is well defined. By Bayes’ rule, s′∼πθ(⋅|q)[πθ(a∗|s′,q)|A=a∗] _s _θ(·|q)\! [ _θ(a^*|s ,q)|A=a^* ] =s∼πθ(⋅|q)[πθ(a∗|s,q)(A=a∗)]Pr(A=a∗) = E_\,s _θ(·|q)\! [ _θ(a^*|s,q)\,I(A=a^*) ] (A=a^*) =s∼πθ(⋅|q)[πθ(a∗|s,q)Pr(A=a∗|q,s)]s∼πθ(⋅|q)[Pr(A=a∗|q,s)] = E_\,s _θ(·|q)\! [ _θ(a^*|s,q)\, (A=a^*|q,s) ]E_s _θ(·|q)\! [ (A=a^*|q,s) ] =s∼πθ(⋅|q)[πθ(a∗|s,q)2]s∼πθ(⋅|q)[πθ(a∗|s,q)]. = E_s _θ(·|q)\! [ _θ(a^*|s,q)^2 ]E_s _θ(·|q)\! [ _θ(a^*|s,q) ]. Hence, ρ(a∗,a∗)=s∼πθ(⋅|q)[πθ(a∗|s,q)2]s∼πθ(⋅|q)[πθ(a∗|s,q)].ρ(a^*,a^*)= E_s _θ(·|q)\! [ _θ(a^*|s,q)^2 ]E_s _θ(·|q)\! [ _θ(a^*|s,q) ]. Since πθ(a∗|s,q)≥0 _θ(a^*|s,q)≥ 0, Jensen’s inequality (or equivalently [X2]≥[X]2E[X^2] [X]^2) implies ρ(a∗,a∗)≥s∼πθ(⋅|q)[πθ(a∗|s,q)],ρ(a^*,a^*)\;≥\;E_s _θ(·|q) [ _θ(a^*|s,q) ], which proves the desired inequality. Equality holds if and only if πθ(a∗|s,q) _θ(a^*|s,q) is constant over all (q,s)(q,s) such that πθ(s|q)>0 _θ(s|q)>0. ∎ Theorem 2 (Value Equivalence). ℒρ(θ) _ρ(θ) =q∼,(s,a)∼πθ(⋅|q)[ρ(a,a∗(q))] =E_q ,(s,a) _θ(·|q)[ρ(a,a^*(q))] =q∼,(s,a)∼πθ(⋅|q)[(a=a∗(q))], =E_q ,(s,a) _θ(·|q)[I(a=a^*(q))], i.e., the expected CER objective is equivalent in value to the exact-match objective, where (a=a∗(q))I(a=a^*(q)) indicates whether a exactly matches a∗(q)a^*(q). Proof. By definition, ℒρ(θ) _ρ(θ) =q∼,(s,a)∼πθ(⋅|q)[ρ(a,a∗(q))] =E_q ,(s,a) _θ(·|q)[ρ(a,a^*(q))] =q∼[∑s,aπθ(s,a|q)ρ(a,a∗(q))]. =E_q \! [ _s,a _θ(s,a|q)\,ρ(a,a^*(q)) ]. Using the definition of ρ, ℒρ(θ) _ρ(θ) =q∼[∑s,aπθ(s,a|q)∑s′πθ(s′|q,a)πθ(a∗(q)|s′,q)] =E_q \! [ _s,a _θ(s,a|q) _s _θ(s |q,a)\, _θ(a^*(q)|s ,q) ] =q∼[∑s′πθ(a∗(q)|s′,q)∑s,aπθ(s,a|q)πθ(s′|q,a)]. =E_q \! [ _s _θ(a^*(q)|s ,q) _s,a _θ(s,a|q)\, _θ(s |q,a) ]. For fixed q, we have ∑s,aπθ(s,a|q)πθ(s′|q,a)=∑aπθ(a|q)πθ(s′|q,a)=πθ(s′|q), _s,a _θ(s,a|q)\, _θ(s |q,a)= _a _θ(a|q)\, _θ(s |q,a)= _θ(s |q), where the first equality marginalizes out s and the second follows from the law of total probability. Therefore, ℒρ(θ) _ρ(θ) =q∼[∑s′πθ(s′|q)πθ(a∗(q)|s′,q)] =E_q \! [ _s _θ(s |q)\, _θ(a^*(q)|s ,q) ] =q∼[∑s′,aπθ(s′,a|q)(a=a∗(q))] =E_q \! [ _s ,a _θ(s ,a|q)\,I(a=a^*(q)) ] =q∼,(s,a)∼πθ(⋅|q)[(a=a∗(q))]. =E_q ,(s,a) _θ(·|q)[I(a=a^*(q))]. This shows that the expected CER objective is equivalent in value to the exact-match objective. ∎