Paper deep dive
Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs
Hadi Mohammadi, Tamás Kozák, Anastasia Giachanou
Models: Qwen2.5-14B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 1:00:55 AM
Summary
This paper evaluates the effectiveness of Group Relative Policy Optimization (GRPO) versus Direct Preference Optimization (DPO) in enhancing the faithfulness of Chain-of-Thought (CoT) reasoning in Qwen2.5 LLMs. The study finds that while both methods improve with model scale, GRPO significantly outperforms DPO in faithfulness metrics at larger scales (14B), despite exhibiting less stability at smaller scales.
Entities (5)
Relation Signals (3)
Qwen2.5-14B-Instruct → trainedwith → GRPO
confidence 95% · GRPO-14B reaches the highest scores across the metrics
GRPO → improvesfaithfulnessof → Chain-of-Thought
confidence 90% · GRPO shows greater potential for improving faithfulness metrics
DPO → improvesfaithfulnessof → Chain-of-Thought
confidence 90% · DPO provides more consistent improvements across model sizes
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Chain-of-thought (CoT) reasoning has emerged as a powerful technique for improving the problem-solving capabilities of large language models (LLMs), particularly for tasks requiring multi-step reasoning. However, recent studies show that CoT explanations often fail to reflect the model's actual reasoning process, as models may produce coherent yet misleading justifications or modify answers without acknowledging external cues. Such discrepancies undermine the reliability of CoT-based methods for safety supervision and alignment monitoring, as models can generate plausible but deceptive rationales for incorrect answers. To better understand this limitation, we evaluate two optimization methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), in their ability to improve CoT faithfulness. Our experiments show that GRPO achieves higher performance than DPO in larger models, with the Qwen2.5-14B-Instruct model attaining the best results across all evaluation metrics. Both approaches exhibit positive correlations between model size and performance, but GRPO shows greater potential for improving faithfulness metrics, albeit with less stable behavior at smaller scales. These results suggest that GRPO offers a promising direction for developing more transparent and trustworthy reasoning in LLMs.
Tags
Links
- Source: https://arxiv.org/abs/2512.22631
- Canonical: https://arxiv.org/abs/2512.22631
Trouble viewing inline? Open PDF directly →
Full Text
18,241 characters extracted from source content.
Expand or collapse full text
Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs Hadi Mohammadi 1 , Tamás Kozák 1 , and Anastasia Giachanou 1 Utrecht University, Department of Information and Computing Sciences, Utrecht, The Netherlands h.mohammadi@u.nl, tamas.kozi@proton.me, a.giachanou@u.nl Abstract. Chain-of-thought (CoT) reasoning that has emerged as a powerful technique can improve the problem-solving capabilities of large language models (LLMs), particularly for tasks requiring multi-step rea- soning. However, recent studies show that CoT explanations often fail to reflect the model’s actual reasoning process, as models may produce coherent yet misleading justifications or modify answers without ac- knowledging external cues. Such discrepancies undermine the reliability of CoT-based methods for safety supervision and alignment monitoring, as models can generate plausible but deceptive rationales for incorrect answers. To better understand this limitation, we evaluate two optimiza- tion methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), in their ability to improve CoT faith- fulness. Our experiments show that GRPO achieves higher performance than DPO in larger models, with the Qwen2.5-14B-Instruct model at- taining the best results across all evaluation metrics. Both approaches exhibit positive correlations between model size and performance, but GRPO shows greater potential for improving faithfulness metrics, albeit with less stable behavior at smaller scales. These results suggest that GRPO offers a promising direction for developing more transparent and trustworthy reasoning in LLMs. Keywords: Chain-of-Thought· Faithfulness· GRPO· DPO· LLM Alignment· Explainability 1 Introduction Chain-of-thought (CoT) reasoning [12] has advanced the ability of large lan- guage models (LLMs) to solve complex, multi-step problems. While standard LLMs perform strongly on tasks tasks such as summarization, information ex- traction, and translation [5], their internal decision-making processes remain largely opaque. Models trained with explicit reasoning prompts (e.g., PaLM 540B) can decompose problems into intermediate steps [12], yet the faithfulness of these reasoning traces, the extent to which they reflect the model’s genuine internal reasoning, remains uncertain [1]. Recent work shows that CoT explanations can mislead. Models sometimes produce fluent but unfaithful justifications or change answers when exposed to arXiv:2512.22631v1 [cs.CL] 27 Dec 2025 2H. Mohammadi et al. hidden cues without acknowledging them [1,6]. For instance, when incorrect hints appear in XML metadata, state-of-the-art models such as Claude 3.7 Sonnet may shift from correct to incorrect answers while offering persuasive but deceptive reasoning chains [1]. This behaviour undermines the reliability of CoT-based alignment and safety monitoring, as models can appear transparent while con- cealing unfaithful reasoning. Improving faithfulness (i.e., ensuring that CoT explanations correspond to a model’s true internal reasoning) is therefore crucial for building transparent and trustworthy AI systems. However, most current fine-tuning approaches focus on improving performance or preference alignment rather than reasoning fidelity. Two recent optimization techniques, Direct Preference Optimization (DPO) [8] and Group Relative Policy Optimization (GRPO) [10], offer complementary strategies for addressing this challenge. DPO simplifies alignment by replacing the reinforcement learning from hu- man feedback (RLHF) pipeline with a direct preference objective that trains on pairs of preferred and rejected responses. This removes the need for reward or value models, reducing computation while maintaining strong human alignment. GRPO, in contrast, uses group-based normalization to estimate relative advan- tages within sampled outputs, removing value networks and lowering memory use while staying competitive with PPO [9]. Although both DPO and GRPO aim to improve model alignment and reasoning quality, their comparative effects on CoT faithfulness have not yet been systematically evaluated. Understanding how these methods differ in promoting faithful reasoning is essential for developing more interpretable and reliable LLMs. In this work, we empirically compare DPO and GRPO across multiple model scales (1.5B–14B parameters) using the Qwen2.5-Instruct family [7]. Our goal is to determine whether GRPO can enhance the faithfulness of CoT reason- ing relative to DPO, and how these methods scale with model capacity. Our results show that GRPO achieves superior performance at larger scales, while DPO provides more consistent improvements across model sizes. These results offer new insights into the trade-offs between optimization efficiency, reasoning faithfulness, and model scalability. 2 Methodology We conducted a controlled comparison of DPO and GRPO to examine how each method affects reasoning accuracy and faithfulness in large language models. 2.1 Datasets Training and evaluation were based on the GSM8K dataset [3], a benchmark of 8.5K grade-school math word problems that require explicit multi-step reasoning. The dataset is widely used for evaluating chain-of-thought (CoT) performance. Each example includes a question, a detailed reasoning process, and a final nu- Title Suppressed Due to Excessive Length3 merical answer. For DPO, we used the GSM8K Preference Dataset 2.1 1 , which contains 1.2K examples in a triplet format (prompt, preferred response, rejected response). To ensure that GRPO training used a comparable amount of data, we sampled approximately 15% of the original GSM8K training set, producing a similar number of training instances. All models were evaluated on a fixed set of 200 test questions drawn from the GSM8K test split using a deterministic random seed. 2.2 Models We fine-tuned four Qwen2.5-Instruct models [7] with parameter sizes of 1.5B, 3B, 7B, and 14B, using both DPO and GRPO. This produced a total of eight fine-tuned models. The Qwen2.5 family was selected because it spans multiple open-source model sizes trained on the same data distribution (approximately 18 trillion tokens), enabling controlled scale comparisons. All experiments used the instruction-tuned variants, which provide a stable baseline for further fine- tuning. Models were loaded in 4-bit precision to reduce memory consumption, allowing the 14B version to fit on a single high-memory GPU. Fine-tuning was implemented using LoRA adapters for parameter-efficient updates, which also kept GPU requirements consistent across scales. For comparability reasons, both DPO and GRPO used uniform hyperparam- eters across scales, although this choice may not be optimal for every model size. Each model was trained using the AdamW optimiser with a learning rate of 5e-6 and a maximum sequence length of 1,024 tokens, which covered 90% of examples without truncation. We pre-fine-tuned on∼80 examples to learn XML-like response structure (<start_working_out>, <end_working_out>, <SOLUTION> tags). For each ques- tion, models generated 4 responses evaluated via four reward functions: (1) For- mat compliance: reward for correctly producing structured XML-style output; (2) Approximate format match: partial reward for near-correct formatting; (3) Answer correctness: tiered reward based on whether the predicted numerical re- sult matched the ground truth; (4) Numerical validation: additional reward for consistent numerical reasoning within the CoT trace. GRPO computes advantages within each group of responses and updates model parameters relative to the group mean. We trained for one epoch with a 90–10 train–validation split. DPO training used the same base configuration but optimised the model to classify the preferred response in each pair as more likely than the rejected one, using a binary cross-entropy objective. LoRA parameters were set as follows: rank was set to 32 and alpha to 64 to stabilise convergence. Each model was trained for three epochs, saving checkpoints at each epoch and selecting the best one based on validation loss. Although both approaches share a common fine-tuning setup, GRPO focuses on relative advantage estimation within generated outputs, while DPO optimises for explicit human preference 1 https://huggingface.co/datasets/valerielucro/gsm8k_preference_dataset_2.1 4H. Mohammadi et al. alignment. This difference in objective functions forms the basis of our compar- ison. 2.3 Evaluation Metrics We evaluated the fine-tuned models using five complementary metrics that cap- ture both accuracy and reasoning faithfulness: (1) Greedy Accuracy [11]: evalu- ates correctness using deterministic decoding with the most probable token at each step; (2) Self-Consistency [11]: samples multiple reasoning paths and uses majority voting on the final answer; (3) Consistency Ratio: measures the propor- tion of generations that yield the most common answer, providing a proxy for stability. NLI-based Faithfulness; (4) NLI-based [2]: assesses whether the gen- erated reasoning logically entails the reference explanation using a pretrained natural language inference model; (5) LLM-as-a-Judge [4]: uses a larger model (Gemma 3 27B) to rate reasoning coherence and correctness on a 1–5 scale, normalised to [0, 1]. 3 Results 3.1 Performance Analysis Table 1 summarises performance across all model sizes and optimisation meth- ods. Both DPO and GRPO show strong positive correlations between model scale and performance. The Qwen2.5-14B models achieve the best overall results for both accuracy and faithfulness metrics. GRPO-14B reaches the highest scores across the metrics. At smaller scales, results are less stable. The 3B models show a marked performance drop compared to the 1.5B versions (GRPO: from 0.250 to 0.055; DPO: from 0.315 to 0.165). This likely reflects hyperparameter sensitiv- ity rather than a fundamental capacity limit, as performance recovers strongly at 7B and peaks at 14B. The pattern suggests that uniform hyperparameters, while ensuring fairness, may not be optimal across scales. Figure 1 shows average scores across scales. Both methods exhibit dramatic performance drops at 3B compared to 1.5B: GRPO greedy accuracy falls from 0.250 to 0.055 (−78%), DPO falls from 0.315 to 0.165 (−48%). This suggests 3B represents a critical capacity threshold where uniform hyperparameters be- come suboptimal. Possible explanations: (1) capacity mismatch requiring differ- ent LoRA rank/alpha ratios; (2) learning rate (5e-6) inappropriate for 3B scale causing instability; (3) Qwen2.5-3B architectural differences interacting poorly with training approaches. The dramatic recovery at 7B (GRPO: 0.720, DPO: 0.435) indicates this is hyperparameter sensitivity rather than fundamental lim- itation. Figure 2 presents comprehensive metric comparisons. At 1.5B, both meth- ods perform similarly. At 3B, both degrade with GRPO showing more severe drops. At 7B, GRPO advantages emerge across all metrics, particularly in faith- fulness. At 14B, both achieve peak performance with GRPO outperforming in faithfulness while maintaining comparable accuracy. Title Suppressed Due to Excessive Length5 Table 1: Performance comparison across methods and scales. Highest values are highlighted with bold Method Size Accuracy & Reliability Faithfulness (B) Grdy Self- Cons. NLI LLM Acc. Cons. RatioJudge DPO1.5 0.315 0.4450.6410.313 0.134 DPO3 0.165 0.3150.5880.150 0.074 DPO7 0.435 0.5800.6540.209 0.250 DPO14 0.740 0.8850.8820.314 0.576 GRPO 1.5 0.250 0.3100.5120.476 0.120 GRPO 3 0.055 0.1200.1970.079 0.235 GRPO 7 0.720 0.8100.8130.470 0.374 GRPO 14 0.755 0.885 0.902 0.491 0.748 Fig. 1: Overall performance: (a) Average across scales showing 3B collapse and recovery; (b) 14B peak performance demonstrating GRPO’s faithfulness advan- tages. 3.2 Faithfulness and Reliability Analysis Greedy accuracy, self-consistency, and consistency ratio show strong correlations across all models (Pearson r > 0.85). This result indicates that the measures capture similar aspects of reliability. Models that answer correctly also tend to do so consistently, producing stable reasoning paths across generations. Faithfulness behaves differently. DPO’s NLI scores remain almost unchanged as model size increases (from 1.5B: 0.313 to 14B: 0.314), while its LLM-Judge scores rise steadily from 0.134 to 0.576. GRPO, in contrast, improves on both measures: NLI increases from 0.476 to 0.491, and LLM-Judge from 0.120 to 0.748—a fivefold gain. These patterns show that NLI and LLM-Judge capture distinct aspects of faithfulness. NLI tests whether a model’s reasoning logically supports its conclusions, while LLM-Judge reflects the overall quality, coherence, and human alignment of the reasoning process. At the 14B scale, GRPO out- 6H. Mohammadi et al. Fig. 2: Model-scale comparison: (a) 1.5B baseline similarity; (b) 3B performance collapse; (c) 7B GRPO advantages emerge in faithfulness metrics. performs DPO by +29.9% in LLM-Judge and +56.4% in NLI, suggesting that GRPO produces reasoning that is both more coherent and logically grounded. Qualitative inspection supports these findings. GRPO learns from multiple reward signals that evaluate structure, correctness, and numerical validity. This setup encourages systematic reasoning rather than memorisation of answer pat- terns. DPO, on the other hand, trains on preference pairs and focuses on human- like responses. It improves stylistic quality but may not always strengthen in- ternal reasoning consistency. As a result, GRPO tends to generate reasoning that reflects its internal decision process more faithfully, while DPO produces smoother but sometimes less reliable explanations. These results imply that group-relative optimisation can better align a model’s expressed reasoning with its underlying decision process [1,6]. However, this ad- vantage appears only at larger scales, where GRPO has enough capacity to sta- bilise training. The comparison also reveals a practical trade-off: GRPO achieves higher faithfulness but requires careful tuning and greater computational re- sources, whereas DPO provides steadier improvements that remain accessible on smaller models. 4 Conclusion In this paper, we compared GRPO and DPO for improving faithfulness in chain- of-thought reasoning across Qwen2.5 models ranging from 1.5B to 14B param- eters. GRPO achieved the best results at larger scales: the 14B model reached 0.755 greedy accuracy and 0.885 self-consistency, with a +56.4% gain in NLI faithfulness and +29.9% in LLM-Judge scores compared to DPO. Both methods benefited from increased model size, but GRPO displayed higher potential for faithful reasoning alongside less stable behaviour at smaller scales. These results show that GRPO delivers stronger faithfulness at the cost of scale sensitivity and tuning effort, whereas DPO provides steadier, more accessible gains. Title Suppressed Due to Excessive Length7 Future work should focus on adaptive hyperparameter strategies for smaller models, richer evaluation frameworks for reasoning faithfulness, and more effi- cient training methods that preserve transparency under resource constraints. References 1. Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S.R., Leike, J., Kaplan, J., Perez, E.: Reasoning models don’t always say what they think. https://arxiv.org/abs/2505.05410 (2025), arXiv preprint arXiv:2505.05410 2. Chen, Y., Eger, S.: MENLI: Robust evaluation metrics from natural language infer- ence. https://arxiv.org/abs/2208.07316 (2022), arXiv preprint arXiv:2208.07316 3. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., Schulman, J.: Training veri- fiers to solve math word problems. https://arxiv.org/abs/2110.14168 (2021), arXiv preprint arXiv:2110.14168 4. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, S., Zhang, K., Wang, Y., Gao, W., Ni, L., Guo, J.: A survey on LLM-as-a- judge. https://arxiv.org/abs/2411.15594 (2024), arXiv preprint arXiv:2411.15594 5. Kleidermacher, H.C., Zou, J.: Science across languages: Assessing LLM multi- lingual translation of scientific papers. https://arxiv.org/abs/2502.17882 (2025), arXiv preprint arXiv:2502.17882 6. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukoši ̄ut ̇e, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCandlish, S., Kundu, S., Kadavath, S., Yang, S., Henighan, T., Maxwell, T., Telleen-Lawton, T., Hume, T., Hatfield-Dodds, Z., Kaplan, J., Brauner, J., Bowman, S.R., Perez, E.: Measur- ing faithfulness in chain-of-thought reasoning. https://arxiv.org/abs/2307.13702 (2023), arXiv preprint arXiv:2307.13702 7. Qwen Team: QWen2.5 technical report. https://arxiv.org/abs/2412.15115 (2024), arXiv preprint arXiv:2412.15115 8. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. https: //arxiv.org/abs/2305.18290 (2023), arXiv preprint arXiv:2305.18290 9. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. https://arxiv.org/abs/1707.06347 (2017), arXiv preprint arXiv:1707.06347 10. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y., Guo, D.: DeepSeekMath: Pushing the limits of mathematical rea- soning in open language models. https://arxiv.org/abs/2402.03300 (2024), arXiv preprint arXiv:2402.03300 11. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. https://arxiv.org/abs/2203.11171 (2022), arXiv preprint arXiv:2203.11171 12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. https://arxiv.org/abs/2201.11903 (2022), arXiv preprint arXiv:2201.11903