Paper deep dive
FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning
Yuxi Sun, Aoqi Zuo, Haotian Xie, Wei Gao, Mingming Gong, Jing Ma
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/14/2026, 2:23:55 AM
Summary
FACT-E is a causality-inspired framework designed to evaluate the trustworthiness of Chain-of-Thought (CoT) reasoning in Large Language Models. It addresses the issue of unfaithful intermediate reasoning steps by combining two metrics: CoT-to-Answer Consistency and Intra-Chain Faithfulness. The latter uses exogenous noise as an instrumental variable in a contrastive design to mitigate LLM self-affirmation bias and spurious correlations, ensuring that reasoning trajectories are both logically sound and supportive of the correct final answer.
Entities (6)
Relation Signals (3)
FACT-E → evaluates → Chain-of-Thought
confidence 100% · FACT-E, a causality-inspired framework for evaluating CoT quality.
Intra-Chain Faithfulness → partof → FACT-E
confidence 95% · FACT-E consists of two modules: CoT-to-Answer Consistency... and Intra-Chain Faithfulness.
FACT-E → testedon → GSM8K
confidence 95% · Experiments on GSM8K, MATH, and CommonsenseQA show that FACT-E improves reasoning-trajectory selection.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful intermediate steps. Existing self-evaluation approaches are prone to inherent biases: the model may confidently endorse coherence even when the step-to-step implication is not valid, leading to unreliable faithfulness evaluation. We propose FACT-E, a causality-inspired framework for evaluating CoT quality. FACT-E uses controlled perturbations as an instrumental signal to separate genuine step-to-step dependence from bias-driven artifacts, producing more reliable faithfulness estimates (\textit{intra-chain faithfulness}). To select trustworthy trajectories, FACT-E jointly considers \textit{intra-chain faithfulness} and \textit{CoT-to-answer consistency}, ensuring that selected chains are both faithful internally and supportive of the correct final answer. Experiments on GSM8K, MATH, and CommonsenseQA show that FACT-E improves reasoning-trajectory selection and yields stronger in-context learning exemplars. FACT-E also reliably detects flawed reasoning under noisy conditions, providing a robust metric for trustworthy LLM reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2604.10693v1
- Canonical: https://arxiv.org/abs/2604.10693v1
Trouble viewing inline? Open PDF directly →
Full Text
87,004 characters extracted from source content.
Expand or collapse full text
FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning Yuxi Sun1 Aoqi Zuo2 Haotian Xie1 Wei Gao3 Mingming Gong2 Jing Ma1 1Hong Kong Baptist University 2The University of Melbourne 3Singapore Management University csyxsun, jingma@comp.hkbu.edu.hk azuo@student.unimelb.edu.au weigao@smu.edu.sg mingming.gong@unimelb.edu.au Corresponding author. Abstract Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful intermediate steps. Existing self-evaluation approaches are prone to inherent biases: the model may confidently endorse coherence even when the step-to-step implication is not valid, leading to unreliable faithfulness evaluation. We propose FACT-E, a causality-inspired framework for evaluating CoT quality. FACT-E uses controlled perturbations as an instrumental signal to separate genuine step-to-step dependence from bias-driven artifacts, producing more reliable faithfulness estimates (intra-chain faithfulness). To select trustworthy trajectories, FACT-E jointly considers intra-chain faithfulness and CoT-to-answer consistency, ensuring that selected chains are both faithful internally and supportive of the correct final answer. Experiments on GSM8K, MATH, and CommonsenseQA show that FACT-E improves reasoning-trajectory selection and yields stronger in-context learning exemplars. FACT-E also reliably detects flawed reasoning under noisy conditions, providing a robust metric for trustworthy LLM reasoning. FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning Yuxi Sun1 Aoqi Zuo2 Haotian Xie1 Wei Gao3 Mingming Gong2 Jing Ma1†thanks: Corresponding author. 1Hong Kong Baptist University 2The University of Melbourne 3Singapore Management University csyxsun, jingma@comp.hkbu.edu.hk azuo@student.unimelb.edu.au weigao@smu.edu.sg mingming.gong@unimelb.edu.au 1 Introduction The paradigm of Chain-of-Thought (CoT) prompting has fundamentally enhanced the reasoning capabilities of Large Language Models (LLMs) (Wei et al., 2022; Yu et al., 2025; Fu et al., 2025a). However, a critical challenge persists in discerning the reliability of reasoning trajectories (Sun et al., 2025b). Models frequently generate rationales that yield correct results and appear superficially persuasive (Cui et al., 2024; Turpin et al., 2024; Lanham et al., 2023), yet are fundamentally intra-chain unfaithful, characterized by broken logical dependencies between intermediate steps or the inclusion of inaccurate content. Detecting such intra-chain unfaithfulness is thus crucial for improving the robustness and trustworthiness of model-generated reasoning. Figure 1: Motivating example illustrating the limitation of LLM self-assessment on CoT evaluation. Two reasoning chains appear fluent and coherent, yet CoT 1 contains successive intermediate steps that are not logically necessary for subsequent reasoning. Conventional method (e.g., self-reflect) assigns similarly high quality scores to both chains, failing to detect this breakdown, whereas FACT-E evaluates the unfaithfulness in a chain and successfully identifies CoT 2 as more trustworthy. Existing methodologies for recognizing faithful and filtering trustworthy reasoning traces generally fall into two paradigms. (1) LLM-as-Judge Methods leverage the model itself as an evaluator. Techniques such as self-correction and self-reflection operate in a black-box manner to assess whether a generated CoT supports the final answer (Kadavath et al., 2022a; Xi et al., 2024; Madaan et al., 2023). Another line of work decomposes CoT into sub-questions and verifies intermediate steps against corresponding sub-answers Radhakrishnan et al. (2023); Zhu et al. (2023). Crucially, this paradigm is inherently answer-centric: it relies primarily on changes in the final answer as supervision, overlooking the internal dependencies among intermediate reasoning steps, which implicitly assume that such surface-level correction implies logical validity. 2) Causal-based Methods have subsequently emerged, employing causal interventions to evaluate reasoning quality. Some approaches assess whether a reasoning chain is faithful to the final answer by perturbing inputs or constructing counterfactuals Yang et al. (2025); Xiong et al. (2025). Others leverage causal measures such as the Probability of Necessity and Sufficiency (PNS) to identify redundant or non-influential steps, or task LLMs with autonomously generating causal graphs to support structured reasoning (Yu et al., 2025; Hüyük et al., 2025; Fu et al., 2025a). Despite their causal underpinnings, the efficacy of these methods is hindered by the inherent biases of LLM-based evaluators (Fu et al., 2025a; Yu et al., 2025). This dependence induces a closed-loop feedback fallacy (Huang et al., 2023; Zheng et al., 2023), wherein an LLM may confidently validate the faithfulness between its generated reasoning steps despite the absence of a rigorous logical entailment (Jiang et al., 2024a; McKenna et al., 2023; Zheng et al., 2023; Huang et al., 2023). Take the question in Figure 1 as an example, in CoT 1, the second step (“The key property of exponentiation to note here is that any number raised to the power of 0 is 1.”) logically deviates from the first step in the reasoning path (“The expression 122354235231^2^235423523 involves raising 1 to the power of 22354235232^235423523”). However, traditional LLM evaluation methods (e.g., self-reflection (Kadavath et al., 2022b)) may struggle to distinguish logically sparse reasoning between these two successive steps in a CoT. As a result, they assign similar quality scores to CoT 1 and CoT 2, despite the substantial difference in their internal coherence. This issue may stem from spurious correlation between the LLM’s assessment and its internal bias (e.g., LLMs demonstrate a persistent self-affirmation bias, consistently assigning positive evaluations to their own generations with negligible variance (Huang et al., 2023)). Consequently, such spurious correlations can make the model overconfident or cause it to neglect evaluating the faithfulness between segments, regardless of their true relationship. In this work, we propose a novel causal view based on a Structural Causal Model (SCM) (Pearl, 2009) to support the LLM self-assessment of intra-chain faithfulness via a contrastive design. To mitigate spurious correlations between the LLM’s faithfulness assessment and its internal biases, we introduce external noise as an instrumental variable, yielding a more reliable faithfulness evaluation. In addition to intra-chain faithfulness, we also consider answer correctness as a complementary dimension of CoT quality. Accordingly, we introduce FACT-E (Faithfulness and Consistency Tandem Estimation), a causality-inspired framework for CoT quality estimation. FACT-E consists of two modules: CoT-to-Answer Consistency, which verifies that the reasoning chain supports the correct final answer, and Intra-Chain Faithfulness, which leverages causal insights to refine the LLM’s faithfulness judgments. We empirically validate FACT-E efficacy across mathematical and commonsense reasoning tasks. Our results show that selecting reasoning paths based on FACT-E scores substantially improves answer accuracy and enhances in-context learning. Furthermore, experiments under noisy conditions demonstrate that our approach effectively identifies process-level failures, thereby improving the robustness and controllability of LLM reasoning. Our contributions are mainly threefold: • We leverage causality to obtain more reliable intra-chain faithfulness evaluations by introducing external noise as an instrumental variable, mitigating the impact of unobserved LLM biases in self-assessment. • We propose FACT-E, a novel CoT evaluation framework that jointly considers (i) answer correctness implied by the CoT (CoT-to-Answer Consistency) and (i) faithfulness between successive CoT segments (Intra-Chain Faithfulness). • We conduct experiments across three representative tasks: (1) Improving Answer Accuracy by selecting higher-quality CoT; (2) Enhancing In-Context Learning by using optimized chains as exemplars; and (3) Noise Detection by identifying flawed reasoning. The results show that FACT-E achieves competitive performance against strong baselines. 2 Task Formulation with A Causal View In this section, we formally formulate the task of measuring the trustworthiness of a reasoning chain from a causal perspective. Figure 2: Structural causal graphs for CoT quality estimation. (a) depicts the process of answering a query Q with CoT S. (b) illustrates the traditional self-assessment approaches, where the LLM evaluates the faithfulness score Y between ≤iS_≤ i and >iS_>i, denoted by XiX_i, mediated by LLM’s self-evaluation M. The LLM’s internal bias Z is an unobserved confounder that affects all variables. (c) FACT-E introduces exogenous noise E as an instrumental variable to obtain a more reliable faithfulness evaluation for CoTs. 2.1 Problem Definition Given a query Q, the LLM is prompted to generate a set of reasoning chains S. To effectively filter and select trustworthy reasoning chains from a pool of post-hoc candidates, our goal is to estimate a reliability score ℛR_S for each candidate chain ∈S as a quality measure, denoted as LLM(Q,)→ℛ∈[0,1]LLM(Q,S) _S∈[0,1]. A higher ℛR_S indicates that the reasoning process is not only correct in its final outcome but also faithful among its intermediate steps. We model these two aspects below. 2.2 CoT-to-Answer Consistency A correct answer is a prerequisite for a high-quality CoT. Accordingly, we first model the chain’s consistency with the correct outcome. Definition 1 (CoT-to-Answer Consistency). An evaluative metric that quantifies the trustworthiness of a CoT candidate by estimating the probability that the reasoning chain consistently leads to the correct final answer. Following recent work (Wu et al., 2024; Fu et al., 2025a; Zhang et al., 2024), we model the reasoning process using Structural Causal Model (SCM) as Q→AQ → A, where the CoT S acts as a mediator that transmits the influence of the query Q to the final answer A. Besides, since the reasoning process can be causally dependent on LLM’s internal knowledge U, we model this process in Figure 2(a). A robust ℛR_S must capture whether the mediation is consistently reaching the correct A. 2.3 Intra-Chain Faithfulness Merely ensuring that a CoT yields the correct answer is insufficient for a comprehensive reliability assessment. A correct answer can often be reached through flawed or hallucinated reasoning steps. To capture the reliability of the intermediate steps beyond the final answer, we introduce Intra-Chain Faithfulness. Definition 2 (Intra-Chain Faithfulness). A measurement of a reasoning path’s quality, focusing on the logical dependencies between steps and their content correctness. A failure of faithfulness occurs when a CoT appears superficially coherent but relies on fragile connections or exhibits erroneous intermediate steps. Formally, let =s1,s2,…,sLS=\s_1,s_2,…,s_L\ denotes a CoT consisting of L steps. To evaluate the faithfulness degree in the chain, we decompose the reasoning process at step i into successive two segments, ≤iS_≤ i and >iS_>i, denoted by XiX_i, where the segment ≤i=s1,…,siS_≤ i=\s_1,…,s_i\ precedes the subsequent segment >i=si+1,…,sLS_>i=\s_i+1,…,s_L\. In Figure 2(b), we denote the faithfulness score between ≤iS_≤ i and >iS_>i by Y. The mediator M represents the LLM’s evaluation on their faithfulness. Ideally, a robust faithfulness estimation is achieved when the mediation process M remains unbiased. However, in practice, LLM evaluation is affected by its unobservable internal bias Z (e.g., self-affirmation bias). The overall faithfulness score of a CoT is aggregated as the faithfulness score across all split indices i from 11 to L−1L-1. 3 Methodology 3.1 Quantify Intra-Chain Faithfulness Ideally, LLM assessment should act as an objective judge of faithfulness. However, in practice, the internal bias may exist. Specifically, two major sources of bias arise: (1) Self-affirmation bias, where the LLM exhibits an inherent tendency to positively evaluate its own outputs (Huang et al., 2023; Zheng et al., 2023); and (2) Statistical shortcuts, where the LLM relies on shallow heuristics learned during pre-training, such as lexical overlap or frequent co-occurrence patterns. As a result, the LLM may hallucinate strong logical connections based on statistical familiarity, even when the underlying reasoning progression is flawed (Zheng et al., 2023; Xiong et al., 2025; Jiang et al., 2024b). As illustrated in Figure 2(b), there exists the spurious correlation between LLM assessment on the faithfulness degree between intermediate steps in a CoT due to the unobservable LLM’s internal bias. To address this issue, we introduce an exogenous instrumental variable E. Since we cannot directly intervene on the CoT generation process, we approximate causal interventions by using E to construct counterfactual segments and inject them into the chain. Concretely, E denotes randomly injected perturbations that disrupt logical dependencies (e.g., omitting steps) and corrupt content correctness (e.g., introducing operation errors). Appendix A.3 lists a non-exhaustive set of noise configurations. Since these perturbations modify >iS_>i and thus the faithfulness relation between ≤iS_≤ i and >iS_>i, E causally influences both XiX_i and M. Figure 2(c) illustrates this intervention process. By intervening on E, we observe corresponding changes in the assessed faithfulness Y through the mediator M. The Average Causal Effect (ACE) directly characterizes faithfulness by measuring its sensitivity to structured perturbations. For each noise type ej∈ℰe_j , given XiX_i and its inference process M, we define ACE as: ACE(ej,Xi) (e_j,X_i) =[Y∣Xi,M,do(E=∅)] =E[Y X_i,M,do(E= )] −[Y∣Xi,M,do(E=ej)]. -E[Y X_i,M,do(E=e_j)]. Intuitively, this quantity aligns with intuition: the ACE measures the relatively increased confidence of LLM evaluation M on the faithfulness Y of XiX_i with respect to the XiX_i injected with noises eje_j. We propose to measure the average causal effect across all types of intervened perturbations ℰE: ACE(ℰ,Xi)=1|ℰ|∑ej∈ℰACE(ej,Xi).ACE(E,X_i)= 1|E| _e_j ACE(e_j,X_i). Estimation with contrastive design. To estimate ACE, we introduce a contrastive design between the original and injected-noise reasonings. Specifically, we implement ℰE as functional interventions on the LLM’s rollout process, instead of static noise injection. For each segmentation point i, we define a perturbed counterpart Xi(ej)=(≤i,>i′)X_i^(e_j)=(S_≤ i,S _>i), where >i′S _>i is a counterfactual rollout trajectory generated by LLM conditioned on both prefix ≤iS_≤ i and a specific logical perturbation ej∈ℰe_j . Table 1 illustrates such a perturbed rollout, where a specific operation error is injected while maintaining the original prefix. The ACE thus quantifies the relative confidence increase on the original reasoning continuation >iS_>i compared to its perturbed counterpart >i′S _>i, denoted by ℱSF_S: ℱ _S =1|ℰ|(L−1)∑i=1L−1∑ej∈ℰACE(ej,Xi) = 1|E|(L-1) _i=1^L-1 _e_j ACE(e_j,X_i) ≈1|ℰ|(L−1)∑i=1L−1∑ej∈ℰ[(≤i,>i)≻(≤i,>i′(ej))], ≈ 1|E|(L-1) _i=1^L-1 _e_j 1\! [(S_≤ i,S_>i) (S_≤ i,S (e_j)_>i) ], where L denotes the number of steps in the CoT, and [⋅]1[·] is an indicator function that equals 11 if the original >iS_>i is preferred over the perturbed >i′S _>i given the same prefix ≤iS_≤ i, and 0 otherwise. This contrastive design serves two critical purposes. First, it effectively neutralizes self-affirmation bias. Since both trajectories are self-generated by ℳM and share an identical prefix and stylistic context, the model cannot rely on stylistic familiarity or surface-level patterns—features that typically lead LLMs to assign inflated scores to their own outputs. Second, the injection of logical noise eje_j acts as a “stress test.” Even when the reasoning remains linguistically fluent and professional, its logical chain is intentionally fractured. This forces the LLM to look past superficial patterns and focus on the actual reasoning steps. Consequently, our method improves the intra-chain faithfulness score in reflecting true logical integrity rather than mere plausible-sounding text. Question Suppose sinD=0.7 D=0.7 in the right triangle DEFDEF (where ∠E=90∘ E=90 and EF=7EF=7). What is the length of DEDE? (≤i,>i)(S_≤ i,S_>i) We are given that sinD=0.7 D=0.7 and need to find the length of DEDE. In the right triangle DEFDEF, EFEF is the side opposite angle D and DEDE is the hypotenuse. By definition, sinD=oppositehypotenuse D= oppositehypotenuse. Thus, sinD=EFDE D= EFDE, i.e., 0.7=7DE0.7= 7DE. Solving for DEDE, we obtain DE=70.7=10DE= 70.7=10. (≤i,>i′)(S_≤ i,S _>i) We are given that sinD=0.7 D=0.7 and need to find the length of DEDE. In the right triangle DEFDEF, EFEF is the side opposite angle D and DEDE is the hypotenuse. By definition, sinD=oppositehypotenuse D= oppositehypotenuse. Thus, sinD=DEDF D= DEDF, leading to DE7=0.7 DE7=0.7. Solving for DEDE, we get DE=0.7×7=4.9DE=0.7× 7=4.9. Table 1: Illustrative example of a perturbed rollout >i′S _>i generated from the same prefix ≤iS_≤ i. The injected logical error leads to an incorrect continuation despite identical contextual and stylistic conditions. 3.2 Quantify CoT-to-Answer Consistency Definition 1 on CoT-to-Answer Consistency quantifies the model’s confidence on a given reasoning chain S yielding the correct final answer. Formally, given a query q and a candidate reasoning chain S, the LLM is prompted independently N times to judge whether S is sufficient to reach the correct answer. For each trial n, we define J(n)∈0,1J^(n)∈\0,1\ as the model’s binary judgment, where J(n)=1J^(n)=1 indicates that the model judges S yields the correct answer, and J(n)=0J^(n)=0 otherwise. We denote P(J(n)=1∣q,)P(J^(n)=1 q,S) as the conditional probability of the positive judgment (e.g., the logit of “True” token), and compute the CoT-to-Answer Consistency score by averaging these probabilities across the trials where the model explicitly validates the reasoning: ≈ _S≈ 1N∑n=1NP(J(n)=1∣q,)⋅J(n)=1, 1N _n=1^NP(J^(n)=1 q,S)·1\J^(n)=1\, where ⋅1\·\ is the indicator function. A high C_S indicates consistent and confident prediction that the reasoning chain S provides a viable path to the correct answer, independent of explicit verification of intermediate logical validity. 3.3 Tandem Estimation A high-quality CoT should not only arrive at the correct answer but also ensure that its intermediate steps are faithful. When faithfulness (ℱF_S) is low, it indicates the presence of loose logical connections or errors within the reasoning, in such cases, the final answer, even if correct, is more likely a byproduct of chance or internal bias than of logical necessity. Conversely, a low consistency score (C_S) indicates that the reasoning, however internally coherent, ultimately fails to solve the task. To bridge these two aspects, we define a reliable score ℛR_S by scaling observed correctness with reasoning faithfulness, thereby ensuring that the final score reflects only outcomes grounded in logical integrity: ℛ=ℱS×S≈1N(L−1)∑i=1L−1∑n=1N _S=F_S×C_S≈ 1N(L-1) _i=1^L-1 _n=1^N [Xi(∅)≻Xi(ℰ)]⋅P(J(n)=1∣q,S)⋅J(n)=1, 1_[X_i^( ) X_i^(E)]· P(J^(n)=1 q,S)·1\J^(n)=1\, where L denotes the total number of steps in S, N is the number of answer-sampling trials, and Xi=(≤i,>i)X_i=(S_≤ i,S_>i) represents the pair of reasoning segments split at position i. Algorithm 1 Select the trustworthy CoT from the candidates set via FACT-E (Lightweight) 1:=((j),(j))j=1KS=\(S^(j),a^(j))\_j=1^K; Question q; LLM ℳM; Noise set ℰE; The number of sampling trials N 2:Optimal CoT optS_opt, Answer opta_opt, Score ℛmaxR_ 3:SC←∅SC← 4:for each candidate ((j),(j))∈(S^(j),a^(j)) do 5: // Step 1: CoT-to-Answer Consistency 6: (j)←1N∑n=1NP(J(n)=1∣,(j))⋅J(n)=1C_S^(j)← 1N _n=1^NP(J^(n)=1 ,S^(j))·1\J^(n)=1\ 7: if (j)=0C_S^(j)=0 then continue 8: end if 9: // Step 2: Intra-Chain Faithfulness 10: Sample (j)⊂1,…,Lj−1T^(j)⊂\1,…,L_j-1\ where Lj=|(j)|L_j=|S^(j)| and |(j)|=min(N,Lj−1)|T^(j)|= (N,L_j-1) 11: for each t∈(j)t ^(j) do 12: # Generate counterfactual path via rollout 13: >t′(e)←ℳ(⋅∣InjectNoise(≤t(j),e)),∀e∈ℰS (e)_>t (· (S^(j)_≤ t,e)),∀ e 14: # Estimation via Contrastive Design 15: Pfaith(j,t)←e∈ℰ[[(≤t(j),>t(j))≻(≤t(j),>t′(e))]]P_faith^(j,t) _e [1_[(S^(j)_≤ t,S^(j)_>t) (S^(j)_≤ t,S (e)_>t)] ] 16: end for 17: # Aggregate Faithfulness Score 18: ℱ(j)←1|(j)|∑t∈(j)Pfaith(j,t)F_S^(j)← 1|T^(j)| _t ^(j)P_faith^(j,t) 19: // Step 3: FACT-E score 20: ℛ(j)←(j)⋅ℱ(j)R_S^(j) _S^(j)·F_S^(j) 21: SC←SC∪((j),(j),ℛ(j))SC ∪\(S^(j),a^(j),R_S^(j))\ 22:end for 23:# Select the optimized CoT and its answer 24:(opt,opt,ℛmax)←argmax(,,ℛ)∈SCℛ(S_opt,a_opt,R_ )← _(S,a,R_S) R_S 25:return opt,opt,ℛmaxS_opt,a_opt,R_ 3.4 Algorithms Evaluating ℱF_S at every step in a long reasoning chain can incur substantial computational overhead. We thus adopt a lightweight estimation strategy for ℛR_S using a fixed-checkpoint approach, where the number of checkpoints is set to N to match the number of sampling trials for estimating C_S. Algorithm 1 outlines the overall selection procedure. Specifically, for each CoT candidate, we first estimate its CoT-to-Answer Consistency (C_S). Candidates with zero consistency are discarded. For the remaining candidates, we sample N random intermediate positions to estimate Intra-Chain Faithfulness. The optimal reasoning trace optS_opt is then selected by jointly maximizing answer correctness and intermediate steps faithfulness. The standard (non-lightweight) version of the algorithm is provided in Algorithm 2, and task-specific settings for error injection are detailed in Appendix A.3. Gpt-4o-mini DeepSeek-V3 Method Math-500 CommonsenseQA GSM-8K Math-500 CommonsenseQA GSM-8K CoT 78.69(-4.253%) 83.00(+0.029%) 92.40(-0.154%) 85.52(-4.587%) 83.80(-1.671%) 96.00(-0.243%) Denoise 83.06(+0.117%) 83.30(+0.329%) 92.20(-0.354%) 84.70(-5.407%) 84.60(-0.871%) 95.40(-0.843%) Polish 83.33(+0.387%) 80.00(-2.971%) 92.60(+0.046%) 94.80(+4.693%) 85.20(-0.271%) 95.60(-0.643%) Refelect 80.60(-2.343%) 83.20(+0.229%) 92.10(-0.454%) 87.43(-2.677%) 85.80(+0.329%) 95.80(-0.443%) Consistency 82.79(-0.153%) 82.10(-0.871%) 92.80(+0.246%) 93.17(+3.063%) 85.00(-0.471%) 96.40(+0.157%) FACT-E(Lightweight) 85.52(+2.577%) 85.20(+2.229%) 93.98(+1.426%) 90.32(+0.213%) 89.00(+3.529%) 97.20(+0.957%) FACT-E(Standard) 86.61(+3.667%) 84.00(+1.029%) 91.80(-0.754%) 94.81(+4.703%) 84.90(-0.571%) 97.30(+1.057%) Qwen3 ChatGpt CoT 93.17(+0.154%) 83.00(-0.257%) 93.20(+0.171%) 52.18(+3.430%) 61.60(-7.471%) 77.60(-0.800%) Denoise 92.08(-0.936%) 83.60(+0.343%) 93.20(+0.171%) 42.62(-6.130%) 72.20(+3.129%) 76.80(-1.600%) Polish 92.90(-0.116%) 81.60(-1.657%) 92.40(-0.629%) 41.00(-7.750%) 63.60(-5.471%) 77.00(-1.400%) Refelect 92.90(-0.116%) 82.60(-0.657%) 92.60(-0.429%) 48.09(-0.660%) 69.60(+0.529%) 78.40(0.000%) Consistency 92.90(-0.116%) 82.80(-0.457%) 93.20(+0.171%) 51.09(+2.340%) 71.60(+2.529%) 80.60(+2.200%) FACT-E(Lightweight) 94.26(+1.244%) 83.60(+0.343%) 93.20(+0.171%) 52.18(+3.430%) 72.70(+3.629%) 81.40(+3.000%) FACT-E(Standard) 92.90(-0.116%) 85.60(+2.343%) 93.40(+0.371%) 54.09(+5.340%) 72.20(+3.129%) 77.00(-1.400%) Table 2: Accuracy comparison (%) across three benchmarks for four LLMs. Values in parentheses denote the relative change with respect to the average performance of all methods for each model-dataset pair. Best results are shown in Bold, second-best results are underlined. 4 Experimental Setup In this section, we describe our experimental setup. We assess our method by addressing the following research questions: (1) RQ1: Can FACT-E identify trustworthy reasoning trajectories to improve answer accuracy? (2) RQ2: Can the selected CoTs serve as superior exemplars that enhance in-context learning? (3) RQ3: Can FACT-E effectively detect and filter rationale noise to safeguard performance? Evaluation setup and metric for RQ1 & RQ2. We design two experiments to evaluate RQ1 and RQ2: (1) Trustworthy reasoning path selection. For each test set =(test,test)Q=\(x_test,y_test)\, we generate K distinct CoT candidates (i,i)i=1K\(T_i,y_i)\_i=1^K via sampling. Our method selects the best reasoning path (∗,∗)(T^*,y^*) with the highest ℛR_s. Overall performance is measured by accuracy: Acc()=1||∑[∗=test]Acc(Q)= 1|Q| [y^*=y_test]. (2) In-context learning (ICL) evaluation. We further evaluate whether the automatically selected chains can serve as higher-quality ICL exemplars. Given a prompt set =[x1,1∗,y1,…,xk,k∗,yk]P=[x_1,T^*_1,y_1,…,x_k,T^*_k,y_k], we vary the number of exemplars K∈5,10,15K∈\5,10,15\. The ICL accuracy is defined as: AccICL()=1||∑(test,test)∈[fθ(,test)=test],Acc_ICL(P)= 1|Q| _(x_test,y_test) I\! [f_θ(P,x_test)=y_test ], where fθ(,test)f_θ(P,x_test) denotes the model prediction conditioned on the exemplars. Benchmarks for RQ1 & RQ2. To evaluate across domains and difficulty levels, we use GSM-8k (Cobbe et al., 2021) and MATH-500 (Hendrycks et al., 2021) for mathematical reasoning, alongside CommonsenseQA (Talmor et al., 2019). Evaluation for RQ3. Given a test set =(test,test)Q=\(x_test,y_test)\ and a set of CoT prompting exemplars =(i,i,i)i=1KP=\(x_i,T_i,y_i)\_i=1^K, where a subset of rationales ′iT _i (with indices ranging from 11 to K) contains injected noise, we construct a noisy prompt set as noise=[(1,′1,1),…,(n,′n,n)]P_noise=[(x_1,T _1,y_1),…,(x_n,T _n,y_n)]. The accuracy of denoising method ℳM using language model fθf_θ is defined as Acc(ℳ,,noise)=1||∑(test,test)∈[ℳ(noise,test)=test]Acc(M,Q,P_noise)= 1|Q| _(x_test,y_test) I [M(P_noise,x_test)=y_test ]. We evaluate robustness on two benchmarks from the NoRa suite (Zhou et al., 2024), including Math-11 (NoRa-Math) and Commonsense (NoRa-Commonsense), each designed to test resilience to different types of reasoning noise. The detailed experimental process is shown in Algorithm 3. Method MATH-500 CommonsenseQA GSM-8K 4o-mini DeepSeek-V3 Qwen3 ChatGpt 4o-mini DeepSeek-V3 Qwen3 ChatGpt 4o-mini DeepSeek-V3 Qwen3 ChatGpt Base 82.79 92.35 90.16 46.99 83.00 86.00 81.40 49.60 93.20 94.60 93.20 74.20 Denoise 82.65 (-0.14) 92.62 (+0.27) 85.79 (-4.37) 47.54 (+0.55) 82.80 (-0.20) 86.60 (+0.60) 81.00 (-0.40) 51.60 (+2.00) 93.00 (-0.20) 93.60 (-1.00) 93.80 (+0.60) 77.00 (+2.80) Polish 83.42 (+0.63) 94.81 (+2.46) 92.08 (+1.92) 49.73 (+2.74) 81.80 (-1.20) 84.80 (-1.20) 81.00 (-0.40) 52.22 (+2.62) 91.60 (-1.60) 91.60 (-3.00) 93.60 (+0.40) 71.80 (-2.40) Reflect 83.35 (+0.56) 93.44 (+1.09) 88.25 (-1.91) 51.64 (+4.65) 81.80 (-1.20) 86.00 (+0.00) 80.80 (-0.60) 53.20 (+3.60) 91.00 (-2.20) 93.80 (-0.80) 92.80 (-0.40) 76.20 (+2.00) Consist. 82.89 (+0.10) 90.98 (-1.37) 91.26 (+1.10) 46.99 (+0.00) 83.80 (+0.80) 82.00 (-4.00) 81.80 (+0.40) 55.60 (+6.00) 92.00 (-1.20) 94.40 (-0.20) 92.40 (-0.80) 71.20 (-3.00) FACT-E 85.52 (+2.73) 93.44 (+1.09) 92.62 (+2.46) 53.28 (+6.29) 84.00 (+1.00) 86.20 (+0.20) 82.40 (+1.00) 63.40 (+13.80) 92.20 (-1.00) 95.00 (+0.40) 94.80 (+1.60) 77.40 (+3.20) Table 3: Accuracy (%) of in-context learning with five demonstration examples (K=5K=5). Best results are shown in bold with darker shading, and second-best results are underlined with lighter shading. Values in parentheses denote the absolute change relative to Base. LLM backbones and baseline methods. We conduct evaluations using four LLM backbones from open-source to closed-source, including DeepSeek-V3, Qwen3-14B, Gpt-4o-mini, and ChatGpt (Gpt-3.5-turbo). For all models, we set the temperature parameter τ to 0 and the number of sampling trials N=3N=3, and we conduct experiments in other N settings shown in §5. To ensure stable results, we evaluate 500 questions per task and repeat each experiment three times. We compare against five representative baseline methods. CoT (Wei et al., 2022), Polish (Xi et al., 2023) and Reflect (Kadavath et al., 2022b) fall under the self-correction paradigm, aiming to improve generation quality through prompt rephrasing and iterative reflection. Denoise (Zhang et al., 2023) adopts a mask-reconstruction strategy that requires the model to recover masked content, while Consistency (Wang et al., 2022) aggregates multiple sampled outputs to improve robustness. These baselines represent mainstream denoising and self-improvement techniques. The detailed descriptions of baselines are provided in Appendix A.7. 5 Experimental Results FACT-E can select trustworthy chains and improve answer accuracy. As shown in Table 2, FACT-E exhibits clear advantages across 12 experimental configurations involving four LLMs and three benchmarks. The standard version of FACT-E achieves the best or second-best performance in 8 out of 12 cases. In particular, FACT-E (standard) shows substantial improvements on the Math-500 benchmark, outperforming the average baseline by 5.340% on ChatGpt and 4.703% on DeepSeek-V3. A closer inspection reveals that FACT-E effectively identifies trustworthy CoT trajectories, achieving 54.09% accuracy on MATH-500, compared to self-correction baselines such as POLISH (41.00%) and DENOISE (42.62%). Moreover, the lightweight variant of FACT-E remains highly competitive: across the same 12 configurations, FACT-E (lightweight) attains the best result in 6 cases and the second-best in 5 cases. This indicates that even with stochastic checkpoint sampling, FACT-E preserves strong discriminative power while significantly reducing computational overhead. Selected trustworthy CoTs enhance ICL. As reported in Table 3, FACT-E achieves the best performance in 8 out of 12 task configurations, with particularly notable gains on ChatGpt (e.g., from 49.60% to 63.40% on CommonsenseQA and from 46.99% to 53.28% on MATH-500). Compared with competing baselines, FACT-E exhibits substantially higher stability across different models and benchmarks. For example, while POLISH performs competitively on MATH using DeepSeek-V3 and Qwen3, its performance degrades using ChatGpt (on MATH) and even falls below the Base method using Gpt-4o-mini (on CommonsenseQA). These observations suggest that methods relying solely on self-assessment are sensitive to the underlying model’s capabilities, whereas FACT-E remains robust across architectures. # Answer CoT (Chain of Thought) Score Question: Suppose sinD=0.7 D=0.7 in the diagram below. What is DEDE? [asy] pair D,E,F; F = (0,0); D = (sqrt(51),7); E = (0,7); draw(D–E–F–D); draw(rightanglemark(D,E,F,15)); label("D",D,NE); label("E",E,NW); label("F",F,SW); label("77",(E+F)/2,W); [/asy] 1 51 51 ✓ We are given that sinD=0.7 D=0.7 and we need to find the length of DEDE. In the right triangle DEFDEF, we know that EFEF is the vertical side and has a length of 77. The sine of angle D is defined as the ratio of the length of the opposite side to the hypotenuse. Therefore, we can express this relationship as: sinD=EFDF D= EFDF. Substituting the known values, we have: 0.7=7DF0.7= 7DF To find DFDF, we can rearrange the equation: DF=70.7DF= 70.7. Calculating this gives:DF=10DF=10Now, we can use the Pythagorean theorem to find DEDE. In triangle DEFDEF:DF2=DE2+EF2DF^2=DE^2+EF^2 Substituting the known values:102=DE2+7210^2=DE^2+7^2This simplifies to:100=DE2+49100=DE^2+49Now, isolating DE2DE^2:DE2=100−49=51DE^2=100-49=51 Taking the square root gives:DE=51DE= 51 Thus, the length of DEDE is 51 51. 0.998 2 1010 ✗ We are given that sinD=0.7 D=0.7 and we need to find the length of DEDE. In the right triangle DEFDEF, we know that EFEF is the opposite side to angle D and DEDE is the hypotenuse. The sine function is defined as the ratio of the length of the opposite side to the hypotenuse. Therefore, we can express this relationship as: sinD=EFDE D= EFDE From the diagram, we see that EF=7EF=7. Substituting the known values into the sine equation gives us: 0.7=7DE0.7= 7DE To find DEDE, we can rearrange this equation: DE=70.7DE= 70.7 Calculating this gives: DE=70.7=10DE= 70.7=10 Thus, the length of DEDE is 1010. 0 Question: What is 1(2235423523)1^(2^235423523)? 1 11 ✓ To solve the expression 1(2235423523)1^(2^235423523), we need to understand the properties of exponents. The base of the exponent is 1. According to the rules of exponents, any number raised to any power is equal to 1, as long as the base is 1. Therefore, regardless of the value of the exponent, 11 raised to any power will always equal 11. 0.999 2 11 ✓ The expression 1(2235423523)1^(2^235423523) involves raising 1 to the power of 22354235232^235423523. The key property of exponentiation to note here is that any number raised to the power of 0 is 1, and any number raised to any power (including very large numbers) is still 1 if the base is 1. Therefore, regardless of the exponent, since the base is 1, the result will always be 1. 0.596 Table 4: Question-answer evaluation results. Flawed portions of the CoT are highlighted in red. Correct and incorrect answers are marked with ✓ and ✗, respectively. Figure 3: Performance (%) on noisy-rationale benchmarks. Results are shown for Qwen3 and DeepSeek-V3 on NoRa-Commonsense, and Gpt-4o-mini on NoRa-Math/Commonsense, under in-context learning with different numbers of noisy rationale demonstrations. FACT-E for noisy CoT detection. To evaluate robustness against rationale noise, we test FACT-E with 1–3 and 1–5 noisy exemplars (i.e., different K) in the prompt (see Algorithm 3), simulating real-world errors in demonstrations. As shown in Figure 3, FACT-E consistently achieves top accuracy on NoRa-Commonsense using Qwen3 and DeepSeek-V3. Although Gpt-4o-mini is slightly less effective on Commonsense, it demonstrates leading performance in NoRa-Math. These results underscore FACT-E’s efficacy in identifying and mitigating intermediate reasoning errors. Case study. As illustrated in Table 5, FACT-E effectively distinguishes CoT candidates that contain flawed reasoning processes. In the first example, FACT-E differentiates between two reasoning paths: although CoT 1 yields the correct final answer, CoT 2 produces an incorrect result due to the erroneous trigonometric derivation “sinD=EF/DEsinD=EF/DE”. FACT-E assigns this chain a score of 0, correctly indicating that it cannot reliably support the correct conclusion. Additional examples further demonstrate FACT-E’s ability to identify reasoning paths whose intermediate steps lack causal validity despite arriving at the correct answer. In the second case, the highlighted red text corresponds to redundant reasoning where the transition across “and” lacks a rigorous causal dependency, resulting in a lower score of 0.596. These fine-grained evaluations show that FACT-E provides a more nuanced characterization of reasoning quality beyond final-answer correctness. Additional case studies are provided in Table A.7. Analysis of the number of sampling trials (N). Experimental results within Gpt-4o-mini across datasets indicate that accuracy generally improves with more sampling trials, although the gains are not strictly monotonic. As shown in Figures 4(a) (lightweight) and 4(b) (standard), accuracy often increases substantially between the second and third trials, followed by saturation or minor fluctuations at four or five trials. This suggests that three trials typically capture most of the performance gains, while additional iterations yield diminishing returns. Figure 4: Performance (%) of lightweight FACT-E (a) and standard FACT-E (b) varying the number of sampling trials (N) on three benchmarks. 6 Related Works Previous work on improving CoT reasoning (Wei et al., 2023; Kojima et al., 2023) has mainly focused on prompt design (Zhou et al., 2022; Wang et al., 2023); structured search frameworks (Sel et al., 2023; Yu et al., 2023) such as Tree-of-Thought (Yao et al., 2023) and Graph-of-Thought (Besta et al., 2024); and fine-tuned language models in specific domain (Jiang et al., 2023; Sun et al., 2025a). These methods aim to improve final-answer accuracy, often treating accuracy gains as indirect evidence of better reasoning. However, they do not directly address a key practical question (Shen et al., 2025): given a specific query and a generated CoT, how can the quality of that reasoning trace be reliably assessed? Existing approaches to CoT evaluation broadly fall into two categories (Wei et al., 2022; Kojima et al., 2023). LLM-based assessment methods, including self-correction, self-reflection, and self-refinement (Kadavath et al., 2022a; Madaan et al., 2023; Xi et al., 2024), rely on the model’s own judgments to evaluate or improve its reasoning. While effective in some settings, these methods assume reliable self-evaluation and are therefore sensitive to model biases. Causality-based methods attempt to assess reasoning quality by analyzing dependencies among intermediate steps, for example using Probability of Necessity and Sufficiency (PNS) or Average Causal Effect (ACE) (Yu et al., 2025; Fu et al., 2025b). However, these approaches, depend heavily on LLM self-assessment, lack principled uncertainty quantification and face scalability limitations. In contrast, our work focuses on rigorously evaluating CoT reasoning by disentangling step-level faithfulness dependencies while explicitly addressing confounding effects (i.e., internal bias), providing a more reliable and scalable framework for LLM reasoning evaluation. 7 Conclusion We address the inherent bias in LLM self-evaluation by introducing a causal framework based on Structural Causal Models, named FACT-E. By leveraging constructed noise as an instrumental variable to estimate the Average Causal Effect, our approach isolates the true causal influence of intermediate reasoning steps by effectively mitigating spurious correlations arising from internal model biases (e.g., self-affirmation bias), enabling a more reliable estimation of reasoning faithfulness. Limitations Our approach, standard FACT-E, aims to assess all the steps of CoT. While it requires prompt LLMs multiple times, leading to higher inference costs compared to simpler prompting approaches. To mitigate the cost, we introduce lightweight FACT-E, which reduces the number of prompting requests while maintaining competitive performance. We compare the LLMs’ overhead during inference using different strategies, shown in Tables 5. LLMs are inherently susceptible to a broad spectrum of cognitive biases (Jiang et al., 2024a; Xiong et al., 2025; Zheng et al., 2023). While it is impossible to account for every potential artifact, our framework specifically targets self-affirmation bias and shortcut bias, both of which significantly distort self-assessment tasks (Xiong et al., 2025; Zheng et al., 2023). Additionally, we refrain from employing multi-LLM systems for CoT verification. While cross-model evaluation is common, it often introduces significant uninterpretable variables, such as inter-model sycophancy or shared parametric biases, which can lead to a false sense of consensus (Du et al., 2023). Ethics Statement All evaluations were conducted using open-source datasets. Our data sources are all from objective and neutral facts and do not contain any personal information and offensive comments directed at individuals or particular groups. Our study on mitigating bias in LLMs acknowledges the ethical implications of data-driven biases in AI, particularly their impact on performance. All experiments were conducted using publicly available datasets, and no human participants were involved. Acknowledge This work is supported by the National Natural Science Foundation of China Young Scientists Fund (No. 62206233). MG was supported by the Australian Government through the ARC Discovery Projects (DP240102088). References Besta et al. (2024) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17682–17690. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cui et al. (2024) Yingqian Cui, Pengfei He, Xianfeng Tang, Qi He, Chen Luo, Jiliang Tang, and Yue Xing. 2024. A theoretical understanding of chain-of-thought: Coherent reasoning and error-aware demonstration. arXiv preprint arXiv:2410.16540. Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning. Fu et al. (2025a) Jiarun Fu, Lizhong Ding, Hao Li, Pengqi Li, Qiuning Wei, and Xu Chen. 2025a. Unveiling and causalizing cot: A causal pespective. arXiv preprint arXiv:2502.18239. Fu et al. (2025b) Jiarun Fu, Lizhong Ding, Hao Li, Pengqi Li, Qiuning Wei, and Xu Chen. 2025b. Unveiling and causalizing cot: A causal pespective. Preprint, arXiv:2502.18239. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798. Hüyük et al. (2025) Alihan Hüyük, Xinnuo Xu, Jacqueline Maasch, Aditya V. Nori, and Javier González. 2025. Reasoning elicitation in language models via counterfactual feedback. Preprint, arXiv:2410.03767. Jiang et al. (2024a) Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Jose Taylor, and Dan Roth. 2024a. A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4722–4756. Jiang et al. (2024b) Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Jose Taylor, and Dan Roth. 2024b. A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4722–4756, Miami, Florida, USA. Association for Computational Linguistics. Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Computational Linguistics. Kadavath et al. (2022a) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022a. Language models (mostly) know what they know. Preprint, arXiv:2207.05221. Kadavath et al. (2022b) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022b. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916. Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, and 1 others. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702. Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. McKenna et al. (2023) Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Hosseini, Mark Johnson, and Mark Steedman. 2023. Sources of hallucination by large language models on inference tasks. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2758–2774. Pearl (2009) Judea Pearl. 2009. Causality: Models, Reasoning, and Inference. Cambridge University Press. Radhakrishnan et al. (2023) Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, and 1 others. 2023. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768. Sel et al. (2023) Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. 2023. Algorithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint arXiv:2308.10379. Shen et al. (2025) Xu Shen, Song Wang, Zhen Tan, Laura Yao, Xinyu Zhao, Kaidi Xu, Xin Wang, and Tianlong Chen. 2025. Faithcot-bench: Benchmarking instance-level faithfulness of chain-of-thought reasoning. arXiv preprint arXiv:2510.04040. Sun et al. (2025a) Yuxi Sun, Wei Gao, Hongzhan Lin, Jing Ma, and Wenxuan Zhang. 2025a. Explainable ethical assessment on human behaviors by generating conflicting social norms. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 166–184, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics. Sun et al. (2025b) Yuxi Sun, Aoqi Zuo, Wei Gao, and Jing Ma. 2025b. CausalAbstain: Enhancing multilingual LLMs with causal reasoning for trustworthy abstention. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14060–14076, Vienna, Austria. Association for Computational Linguistics. Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158. Turpin et al. (2024) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2024. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems (NeurIPS). Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS). Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903. Wu et al. (2024) Junda Wu, Tong Yu, Xiang Chen, Haoliang Wang, Ryan Rossi, Sungchul Kim, Anup Rao, and Julian McAuley. 2024. DeCoT: Debiasing chain-of-thought for knowledge-intensive tasks in large language models via causal intervention. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14073–14087, Bangkok, Thailand. Association for Computational Linguistics. Xi et al. (2024) Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Self-polish: Enhance reasoning in large language models via problem refinement. Preprint, arXiv:2305.14497. Xi et al. (2023) Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Jia Liu, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2023. Self-polish: Enhance reasoning in large language models via problem refinement. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11383–11406. Xiong et al. (2025) Zidi Xiong, Shan Chen, Zhenting Qi, and Himabindu Lakkaraju. 2025. Measuring the faithfulness of thinking drafts in large reasoning models. arXiv preprint arXiv:2505.13774. Yang et al. (2025) Sohee Yang, Sang-Woo Lee, Nora Kassner, Daniela Gottesman, Sebastian Riedel, and Mor Geva. 2025. How well can reasoning models identify and recover from unhelpful thoughts? arXiv preprint arXiv:2506.10979. Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. Yu et al. (2023) Junchi Yu, Ran He, and Rex Ying. 2023. Thought propagation: An analogical approach to complex reasoning with large language models. arXiv preprint arXiv:2310.03965. Yu et al. (2025) Xiangning Yu, Zhuohan Wang, Linyi Yang, Haoxuan Li, Anjie Liu, Xiao Xue, Jun Wang, and Mengyue Yang. 2025. Causal sufficiency and necessity improves chain-of-thought reasoning. Preprint, arXiv:2506.09853. Zhang et al. (2024) Chen Zhang, Lanning Zhang, and Dexiang Zhou. 2024. Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19533–19541. Zhang et al. (2023) Zhen Zhang, Guanhua Zhang, Bairu Hou, Wenqi Fan, Qing Li, Sijia Liu, Yang Zhang, and Shiyu Chang. 2023. Certified robustness for large language models with self-denoising. arXiv preprint arXiv:2307.07171. Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623. Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and 1 others. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. Zhou et al. (2024) Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. 2024. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? Advances in Neural Information Processing Systems, 37:123846–123910. Zhu et al. (2023) Wang Zhu, Jesse Thomason, and Robin Jia. 2023. Chain-of-questions training with latent answers for robust multistep question answering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8845–8860, Singapore. Association for Computational Linguistics. Appendix Appendix A More Details of FACT-E A.1 The standard version of our method Algorithm 1 outlines a lightweight version of FACT-E, which utilizes a fixed checkpoint to estimate ℛR_S. To ensure a fair comparison, the number of checkpoints is set to N, matching the sampling budget used for estimating C_S. In contrast, the standard version of FACT-E is designed to verify all sequential dependencies between steps; comprehensive details for this version are provided in Algorithm 2. A.2 Algorithm of detecting noise CoT method As detailed in Algorithm 3, our framework employs FACT-E to identify and filter noisy rationales from the constructed noisy prompt set noiseP_noise. The process operates through two distinct phases: a rigorous denoising pipeline to curate a high-quality exemplar set cleanP_clean, followed by a final inference step. Phase 1 aims to detect noisy examples. The method evaluates each candidate example (i,(i),i)∈noise(x_i,S^(i),y_i) _noise through a three-step assessment: (1) CoT-to-Answer Consistency (C_S): We first verify the external alignment between the provided rationale and its label. Specifically, we calculate the conditional probability of the ground truth label iy_i given the fixed reasoning chain (i)S^(i), denoted as (i)←P(i|i,(i))C_S^(i)← P(y_i|x_i,S^(i)). Examples with zero consistency ((i)=0C_S^(i)=0) are immediately pruned as they fail to derive the correct answer. (2) Intra-Chain Faithfulness (ℱF_S): To efficiently assess the causal validity of the reasoning steps, we sample a subset of indices T rather than iterating through the entire chain. For each sampled step t, we generate counterfactual trajectories >t′(e)S (e)_>t via noise injection. A faithfulness score is then estimated by checking if the model strictly prefers the fixed original chain (i)S^(i) over the perturbed version >t′(e)S (e)_>t, ensuring that the rationale is robust to causal interventions. (2) Filtering via FACT-E Score: A final reliability score is derived by combining external consistency and internal faithfulness: ℛ(i)←(i)⋅ℱ(i)R_S^(i) _S^(i)·F_S^(i). Only rationales satisfying the reliability threshold (ℛ(i)≥τR_S^(i)≥τ) are retained in the cleaned set cleanP_clean, where τ is determined using a small set of held-out validation data for NoRa-Math and NoRa-Commonsense, respectively, with values typically ranging from 0.6 to 0.7. Phase 2 aims to do the final inference. Finally, the LLM utilizes the refined context cleanP_clean to perform few-shot inference on the test question testx_test. This approach ensures that the model conditions only on derivationally sound and logically robust exemplars, thereby enhancing performance on benchmarks such as NoRa-Math and NoRa-Commonsense. Algorithm 2 Select the trustworthy CoT via FACT-E (Standard) 1:=((j),(j))j=1KS=\(S^(j),a^(j))\_j=1^K; Question q; LLM ℳM; Noise set ℰE; The number of sampling trials N 2:Optimal CoT optS_opt, Answer opta_opt, Score ℛmaxR_ 3:SC←∅SC← 4:for each candidate ((j),(j))∈(S^(j),a^(j)) do 5: // Step 1: CoT-to-Answer Consistency 6: (j)←1N∑n=1NP(J(n)=1∣,(j))⋅J(n)=1C_S^(j)← 1N _n=1^NP(J^(n)=1 ,S^(j))·1\J^(n)=1\ 7: # Skip candidates that cannot lead to the correct answer 8: if (j)=0C_S^(j)=0 then 9: continue 10: end if 11: // Step 2: Intra-Chain Faithfulness 12: Let Lj=|(j)|L_j=|S^(j)| 13: for t=1t=1 to Lj−1L_j-1 do 14: # Generate counterfactual path via rollout 15: >t′(e)←ℳ(⋅∣InjectNoise(≤t(j),e)),∀e∈ℰS (e)_>t (· (S^(j)_≤ t,e)),∀ e 16: # Estimation via Contrastive Design 17: Pfaith(j,t)←e∈ℰ[[(≤t(j),>t(j))≻(≤t(j),>t′(e))]]P_faith^(j,t) _e [1_[(S^(j)_≤ t,S^(j)_>t) (S^(j)_≤ t,S (e)_>t)] ] 18: end for 19: # Aggregate Faithfulness Score 20: ℱ(j)←1Lj−1∑t=1Lj−1Pfaith(j,t)F_S^(j)← 1L_j-1 _t=1^L_j-1P_faith^(j,t) 21: // Step 3: FACT-E score 22: ℛ(j)←(j)⋅ℱ(j)R_S^(j) _S^(j)·F_S^(j) 23: SC←SC∪((j),(j),ℛ(j))SC ∪\(S^(j),a^(j),R_S^(j))\ 24:end for 25:# Select the optimized CoT and its answer 26:(opt,opt,ℛmax)←argmax(,,ℛ)∈SCℛ(S_opt,a_opt,R_ )← _(S,a,R_S) R_S 27:return opt,opt,ℛmaxS_opt,a_opt,R_ Algorithm 3 Detecting and filtering Noisy CoT to Enhance Answering (RQ3) 1:Noisy set noise=(i,(i),i)i=1nP_noise=\(x_i,S^(i),y_i)\_i=1^n; Test question testx_test; LLM ℳM; Noise set ℰE; Sampling budget N; Threshold τ 2:Final answer testy_test via cleaned set cleanP_clean 3:clean←∅P_clean← 4:// Phase 1: Detecting Noisy Examples 5:for each example (i,(i),i)∈noise(x_i,S^(i),y_i) _noise do 6: // Step 1: CoT-to-Answer Consistency 7: (i)←P(i∣i,(i))C_S^(i)← P(y_i _i,S^(i)) 8: # Skip examples that cannot support the ground-truth answer 9: if (i)=0C_S^(i)=0 then 10: continue 11: end if 12: // Step 2: Intra-Chain Faithfulness 13: Sample (i)⊂1,…,Li−1T^(i)⊂\1,…,L_i-1\ where Li=|(i)|L_i=|S^(i)| and |(i)|=min(N,Li−1)|T^(i)|= (N,L_i-1) 14: for each t∈(i)t ^(i) do 15: # Generate counterfactual path via rollout 16: >t′(e)←ℳ(⋅∣InjectNoise(≤t(i),e)),∀e∈ℰS (e)_>t (· (S^(i)_≤ t,e)),∀ e 17: # Estimation via Contrastive Design 18: Pfaith(i,t)←e∈ℰ[[(≤t(i),>t(i))≻(≤t(i),>t′(e))]]P_faith^(i,t) _e [1_[(S^(i)_≤ t,S^(i)_>t) (S^(i)_≤ t,S (e)_>t)] ] 19: end for 20: # Aggregate Faithfulness Score 21: ℱ(i)←1|(i)|∑t∈(i)Pfaith(i,t)F_S^(i)← 1|T^(i)| _t ^(i)P_faith^(i,t) 22: // Step 3: Filtering with FACT-E score 23: ℛ(i)←(i)⋅ℱ(i)R_S^(i) _S^(i)·F_S^(i) 24: if ℛ(i)≥τR_S^(i)≥τ then 25: clean←clean∪(i,(i),i)P_clean _clean∪\(x_i,S^(i),y_i)\ 26: end if 27:end for 28:// Phase 2: Final Inference with Cleaned Context 29:test←ℳ(clean,test)y_test (P_clean,x_test) 30:return testy_test A.3 Noise Injection To rigorously evaluate the faithfulness of reasoning, we apply a set of perturbations ℰE to the CoT candidates, as illustrated in Table 6. Specifically, Operation Error and Conceptual Swap target the precision of individual steps by altering operators and substituting concepts, respectively. Misgeneralization and Reordered Logic perturb the inductive and structural integrity of the reasoning path. Finally, Contradiction assesses the model’s ability to maintain logical grounding by introducing premise-violating information. The detailed prompts of noise injected (construct the counterfactual segments of the chain) are shown in Table 10. A.4 More Case Studies Table 8 illustrates additional case studies about the score of FACT-E regarding CoT. CoTs with logical errors (highlighted in red) and incorrect answers receive significantly low scores (e.g., 0.177 and 0.338), ensuring that hallucinatory or flawed reasoning is penalized. Correct answers derived from rigorous, error-free reasoning steps achieve the highest scores (approx. 0.8), validating the metric’s ability to select optimal CoTs. Crucially, the FACT-E distinguishes between "correct answer with flawed logic" and "correct answer with sound logic." In the third case, while both paths yield the correct result, the mathematically rigorous chain scores higher (0.7995) than the one containing minor logical defects (0.5992). Similarly, in the third coordinate conversion task, FACT-E identifies that the deduction following "so" does not maintain a strict causal relationship with the preceding steps, assigning a score of 0.5992 compared to the more logically sound CoT 2, which scores 0.7995. A.5 LLM Inference Overhead For efficiency, as shown in the following Table 5, the lightweight variant has complexity linear in the checkpoint budget is N, where N is the number of iterations, whereas the standard variant scales with the number of inspected split points. In practice, the exact number of LLM calls also depends on how perturbation types are batched into prompts. Method # LLM inference requests Polish 7 Consistency 3 Reflect 6 Denoise 7 Ours-light 7 Ours-standard 3⋅ℓ(c)+13· (c)+1 Table 5: The number of LLM inference requests per query (when set iteration N=3N=3). ℓ(c) (c) denotes the length of CoT. Noise Type Description (MATH Task) Description (Commonsense) Operation Error Modification of a specific mathematical operation or procedural step. Modification of a specific logical step or operative element within the reasoning. Conceptual Swap Substitution of distinct mathematical or logical concepts. Substitution of semantically or logically related entities, properties, or concepts. Misgeneralization Erroneous extrapolation from a specific concept to an invalid general rule. Improper extension of a specific concept or heuristic to an invalid or broader context. Reordered Logic Permutation of the sequential order of reasoning steps. Permutation of the sequential or causal order of reasoning steps. Contradiction Introduction of an inconsistency with established premises or prior conclusions. Introduction of an assertion that conflicts with established facts, premises, or prior logic. Table 6: Unified Definitions of Noise Types across MATH and Commonsense Tasks. A.6 Ablative Study The ablation results in Figure 5 further confirm the effectiveness of combining Intra-Chain Faithfulness and CoT-to-Answer Consistency. Figure 5: Ablation study of FACT-E. A.7 Baseline Methods We conduct evaluations using four LLM backbones, including Deepseek-v3, Qwen3-14B, Gpt-4o-mini, and Gpt-3.5-turbo. For all models, we set the temperature parameter τ to 0. To ensure stable results, we evaluate 500 questions per task and repeat each experiment three times. We compare against five representative baseline methods. Polish (Xi et al., 2023) and Reflect (Kadavath et al., 2022b) fall under the self-correction paradigm, aiming to improve generation quality through prompt rephrasing and iterative reflection. Denoise (Zhang et al., 2023) adopts a mask-reconstruction strategy that requires the model to recover masked content, while Consistency (Wang et al., 2022) aggregates multiple sampled outputs to improve robustness. These baselines represent mainstream denoising and self-improvement techniques; detailed descriptions are provided in Appendix A.7. • Self-Polish (Polish) (Xi et al., 2023) enhances the quality of reasoning chains by teaching large language models (LLMs) to eliminate noisy information, restructure logical sequences, and reorganize local conditions into coherent reasoning steps. In our implementation, we (1) prompt the LLM to independently refine each noisy chain-of-thought (CoT) exemplar without external guidance, repeating this refinement process three times, and (2) aggregate these rephrased demonstrations to construct the final context for downstream task reasoning. • Self-Denoise (Denoise) (Zhang et al., 2023) improves LLM robustness by preprocessing input prompts with random masking, requiring the model to reconstruct the masked content. This process reduces noise and mitigates incoherent reasoning. Our implementation involves (1) applying random masks to noisy rationales in each demonstration, (2) prompting the LLM to infer and complete the masked sections for each example, and (3) utilizing the reconstructed CoT demonstrations for subsequent task reasoning. This procedure is repeated three times, with the most frequent answer selected as final. • Self-Consistency (Consistency) (Wang et al., 2022) enhances reasoning performance through output sampling and majority voting, without modifying the input. In our approach, we execute the same reasoning task three times and select the answer that appears most frequently across all runs. • Self-Reflect (Reflect) (Kadavath et al., 2022b) enhances LLM reasoning by encouraging the model to explicitly critique and revise its own intermediate outputs. In our implementation, we (1) prompt the LLM to generate an initial CoT for a given task, (2) instruct the model to self-reflect on the generated reasoning—identifying potential flaws, gaps, or errors, (3) direct the LLM to revise its CoT based on this reflection, and (4) use the improved rationale as input for final answer prediction. This reflective process can be iterated to further refine reasoning quality. In our approach, we execute the same reasoning task three times and select the answer that appears most frequently across all runs. # Answer CoT (Chain of Thought) Score Question: Suppose sinD=0.7 D=0.7 in the diagram below. What is DEDE? [asy] pair D,E,F; F = (0,0); D = (sqrt(51),7); E = (0,7); draw(D–E–F–D); draw(rightanglemark(D,E,F,15)); label("D",D,NE); label("E",E,NW); label("F",F,SW); label("77",(E+F)/2,W); [/asy] 1 6.866.86 ✗ We are given that sinD=0.7 D=0.7 and we need to find the length of DEDE. In the right triangle DEFDEF, we know that DEDE is the side opposite angle D and EFEF is the side adjacent to angle D. The length of EFEF is given as 77. Using the definition of sine, we have: sinD=oppositehypotenuse=DEDF D= oppositehypotenuse= DEDF. We can also express DFDF in terms of DEDE and EFEF using the Pythagorean theorem: DF=DE2+EF2DF= DE^2+EF^2. Substituting EF=7EF=7 into the equation gives: DF=DE2+72=DE2+49DF= DE^2+7^2= DE^2+49. Now substituting this into the sine equation: 0.7=DEDE2+490.7= DE DE^2+49. To eliminate the square root, we can square both sides: 0.72=DE2DE2+490.7^2= DE^2DE^2+49. Calculating 0.720.7^2: 0.49=DE2DE2+490.49= DE^2DE^2+49. Cross-multiplying gives: 0.49(DE2+49)=DE20.49(DE^2+49)=DE^2. Expanding this: 0.49DE2+24.01=DE20.49DE^2+24.01=DE^2. Rearranging terms: DE2−0.49DE2=24.01DE^2-0.49DE^2=24.01. This simplifies to: 0.51DE2=24.010.51DE^2=24.01. Dividing both sides by 0.510.51: DE2=24.010.51≈47.0DE^2= 24.010.51≈ 47.0. Taking the square root to find DEDE: DE≈47.0≈6.86DE≈ 47.0≈ 6.86. 0.338 Question: BC¯ BC is parallel to the segment through A, and AB=BCAB=BC. What is the number of degrees represented by x? [asy] draw((0,0)–(10,0)); draw((0,3)–(10,3)); draw((2,3)–(8,0)); draw((2,3)–(4,0)); label("A",(2,3),N); label("B",(4,0),S); label("C",(8,0),S); label("124∘124 ",(2,3),SW); label("x∘x ",(4.5,3),S); [/asy] 1 124∘124 ✗ We start by analyzing the given information. We have two parallel lines, one at the top (where point A is located) and one at the bottom (where points B and C are located). The angle at point A is given as 124∘124 . Since BC¯ BC is parallel to the line through A, the angle at A and the angle x at point B are corresponding angles. According to the properties of parallel lines, corresponding angles are equal. Therefore, we can set up the equation: x=124∘x=124 . Thus, the value of x is directly equal to the angle at point A. 0.398 2 28∘28 ✓ In the given problem, we have two parallel lines represented by the segments through points A and B and C. Since BC¯ BC is parallel to the segment through A, we can use the properties of alternate interior angles. The angle at point A is given as 124∘124 . Since ABAB is equal to BCBC, triangle ABCABC is isosceles with AB=BCAB=BC. Therefore, the angles at points B and C must be equal. Let the angle at point B be x. The angle at point C will also be x. The sum of the angles in triangle ABCABC must equal 180∘180 . Thus, we can set up the equation: 124∘+x+x=180∘124 +x+x=180 This simplifies to: 124∘+2x=180∘124 +2x=180 Subtracting 124∘124 from both sides gives: 2x=180∘−124∘=56∘2x=180 -124 =56 Dividing both sides by 22 yields: x=56∘2=28∘x= 56 2=28 . Thus, the value of x is 28∘28 . 0.798 3 56∘56 ✗ In the given problem, we have two parallel lines, and we know that BC¯ BC is parallel to the segment through A. The angle at point A is given as 124∘124 . Since ABAB is equal to BCBC, triangle ABCABC is isosceles with AB=BCAB=BC. The angle x is the angle at point B which is opposite to the angle at point A. By the properties of parallel lines and the angles formed, we can use the fact that the angle at A and the angle at B are supplementary because they are on the same side of the transversal line ACAC. Thus, we can set up the equation: x+124∘=180∘x+124 =180 Now, we can solve for x: x=180∘−124∘=56∘x=180 -124 =56 Therefore, the value of x is 56∘56 . 0.177 Question: Convert the point (0,3)(0,3) in rectangular coordinates to polar coordinates. Enter your answer in the form (r,θ),(r,θ), where r>0r>0 and 0≤θ<2π.0≤θ<2π. 1 (3,π2)(3, π2) ✓ To convert from rectangular coordinates (x,y)(x,y) to polar coordinates (r,θ),(r,θ), we use the formulas r=x2+y2r= x^2+y^2 and θ=arctan(yx).θ= ( yx ). Given the point (0,3),(0,3), we have r=02+32=3r= 0^2+3^2=3 and θ=arctan(30)=π2.θ= ( 30 )= π2. However, we need to adjust θ to be in the interval [0,2π),[0,2π), so the final answer is (3,π2).(3, π2). 0.5992 2 (3,π2)(3, π2) ✓ To convert a point from rectangular coordinates (x,y)(x,y) to polar coordinates (r,θ)(r,θ), we use the formulas r=x2+y2r= x^2+y^2 and θ=arctan(yx)θ= ( yx ). Given the point (0,3)(0,3), we have x=0x=0 and y=3y=3. Calculating r=02+32=3r= 0^2+3^2=3 and θ=arctan(30)=π2θ= ( 30 )= π2 (since arctan is undefined at x=0x=0 but we know the point lies on the positive y-axis). Therefore, the point (0,3)(0,3) in rectangular coordinates is equivalent to the point (3,π2)(3, π2) in polar coordinates. 0.7995 Table 7: Question and Answer Evaluation Results, the flawed part of CoT is highlighted in red. The correct and wrong answers are shown in ✓ and ✗, respectively. Appendix B Further Analysis As illustrated in the Figure 6, performance universally degrades across all base models as difficulty increases from Lvl-1 to Lvl-4. Notably, Ours exhibits superior robustness, effectively mitigating the "performance cliff" observed in other baselines. On DeepSeek-V3, Ours achieves an accuracy of 92.31% at the highest difficulty (Lvl-4), outperforming the CoT baseline (79.69%) by a substantial margin of 12.62%. While standard enhancement methods like Polish or Consistency yield gains on simpler tasks, their efficacy diminishes as logical complexity peaks. Ours consistently maintains a flatter decay curve by inserting mechanisms, e.g., error-correction/verification. This is particularly evident in the ChatGPT experiments, where Ours preserves its performance edge even when other methods drop below the 33% accuracy threshold at Lvl-4. Across the full Math500 dataset, Ours achieves state-of-the-art or competitive results on all evaluated LLMs. Compared to the sampling-heavy Consistency baseline, Ours improves total accuracy by 9.52% on GPT-4o-mini and 4.03% on Deepseek V3, demonstrating superior reasoning efficiency and reliability across varying model scales. Figure 6: Analysis of different levels of MATH-500. 5 Examples 10 Examples 15 Examples Method DeepSeek-V3 Qwen3-14B DeepSeek-V3 Qwen3-14B DeepSeek-V3 Qwen3-14B Standard CoT 92.35 90.16 92.35 91.53 92.62 92.62 Denoise 92.62 85.79 94.54 89.34 92.08 90.16 Polish 94.81 92.08 92.62 89.07 91.80 90.98 Reflect 93.44 88.25 94.54 87.98 90.44 90.71 Consistency 90.98 91.26 93.17 90.98 93.00 89.89 Ours 93.44 92.62 93.20 92.35 93.44 93.26 Table 8: Performance comparison of DeepSeek-V3 and Qwen3-14B on MATH-500 across different numbers of prompting examples. We conducted extended analysis on all in-context learning experiments in § 5, specifically examining how performance changes as the number of demonstration examples increases, as illustrated in Table 8. Based on the experimental results, the proposed method ("Ours") demonstrates strong and consistent performance across both DeepSeek-V3 and Qwen3-14B models, achieving the highest or competitive accuracy in all example-count settings. While increasing the number of in-context examples does not uniformly improve performance—and sometimes even degrades it, particularly for Qwen3-14B under methods like Polish and Reflect—our approach remains robust, showing no noticeable decline. DeepSeek-V3 generally outperforms Qwen3-14B in most scenarios, though Qwen3-14B benefits markedly from our method, especially with 15 examples where it reaches 93.26% accuracy. These findings highlight the effectiveness and generalizability of our approach compared to existing prompting strategies. Appendix C Prompts We provide the prompts of FACT-E in Table 10, the prompts of baselines are shown in 9 and 11. Appendix D LLM Usage Claim In this paper, LLMs are utilized exclusively for the purpose of aiding and polishing writing. Their application is strictly confined to improving linguistic clarity, coherence, grammar, and style within textual content. No additional functionalities are incorporated. Method Prompt Template zero shot Please answer the following question without any explanation. Please format your response as follows: Answer: Final numeric answer Question: question Answer: In-context Learning Following the given examples and think step by step to solve the following question. First provide the reasoning process (CoT), then give the final numeric answer. Please format your response as follows: CoT: Step-by-step reasoning Answer: Final numeric answer Following the examples below: After reviewing the following examples, solve the new problem in the same way: Few_shot_examples Now solve the following question: Question: question One Few-shot Example Question: "Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?" CoT: Step 1: Calculate the original value of the house: $80,000 Step 2: Calculate the increase in value due to repairs: 150% of $80,000 = 1.5 × 80,000 = $120,000 Step 3: Calculate the total selling price of the house: $80,000 + $120,000 = $200,000 Step 4: Calculate the total cost incurred by Josh: $80,000 (purchase) + $50,000 (repairs) = $130,000 Step 5: Calculate the profit: $200,000 (selling price) - $130,000 (total cost) = $70,000 Answer: 70000 Table 9: Baseline prompts Module Prompt Template C_S Question: question CoT: cot Answer: ground_truth Task: Determine whether the provided Chain of Thought (CoT) logically deduces the correct answer for the given question. Respond with "True" if the reasoning leads to the answer, or "False" if it does not. The counterfactual chain generation You are given a math question and its corresponding reasoning chain. This reasoning chain is divided into two parts: - The steps before step t, called ‘Chain before step t‘. - The steps after step t, called ‘Chain after step t‘. Your task is to generate a completely alternative reasoning chain after step t, directly reflecting the following error: error The alternative reasoning chain must: 1. Start exactly where the chain before step t ends, preserving earlier logic. 2. Modify the original continuation to reflect the specified error type. 3. Be logically coherent up to step t and introduce the assigned error naturally. 4. End with a final boxed answer, if the original did. Input Format: Question: question Chain before step t: before_step_flip Chain after step t: after_step_flip Output Format: Contrastive Chain After Step t: ℱF_S Choose the better option directly, without explaining your reasoning. Question: "question" Previous reasoning (partial chain of thought): before_step_flip Now evaluate which of the following two options is a more logical, coherent, and fluent continuation of the previous reasoning. The better option should follow naturally from the previous steps and maintain consistency in mathematical logic and style. Option A: before_step_flip,after_step_flip Option B: contrastive_cot_entry[’cot’] Answer Choice: [A/B/NA] Table 10: Prompts used in our method. Method Stage Prompt Content and Some examples of demo Consistency — Base Prompt: Think step by step to solve the following question. First provide the reasoning process (CoT), then give the numeric final answer. Format: CoT: Step-by-step reasoning Answer: Final numeric answer Example: CoT: f(−2)+f(−1)+f(0)=3(−2)−2−2−2+3(−1)−2−1−2+3(0)−20−2=−8−4+−5−3+−2−2=2+53+1=143f(-2)+f(-1)+f(0)= 3(-2)-2-2-2+ 3(-1)-2-1-2+ 3(0)-20-2= -8-4+ -5-3+ -2-2=2+ 53+1= 143 Answer: 143 143 Question: [Question Content] Reflect 1 (Same as the Base Prompt above) 2 Reflection Prompt: Based on the Chain-of-Thought (COT) reasoning and the answer you just provided, please reconsider the following question. Confirm the correctness of your prior answer, and then answer it again, also using the Chain-of-Thought (COT) format followed by the final answer. Denoise 1 (Same as the Base Prompt above) 2 Masking Process: Replace specific tokens within the CoT reasoning with [MASK] to create a denoising objective. An Example: Question: A curve is parameterized by (x,y)=(t3+7,−3t2−6t−5).(x,y)=(t^3+7,-3t^2-6t-5). Find the point the curve passes through at t=2.t=2. To find the point on [MASK] curve at [MASK] t=2t=2, [MASK] need [MASK] substitute t=[MASK][MASK]t=[MASK][MASK] into the parameterization [MASK] for x [MASK] y. [MASK] parameterization is given by: [MASK] x=t3+7x=t^3+7 y=−3t2−[MASK]−5y=-3t^2-[MASK]-5 First, we [MASK]x [MASK] when t[MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] + 7 = 8 + 7 = 15. Next, [MASK] calculate [MASK]y when [MASK][MASK]2[MASK][MASK]2 [MASK] y=[MASK]−[MASK]−[MASK]=−3(4)−12−5=[MASK]−12[MASK]5[MASK][MASK]y=[MASK]-[MASK]-[MASK]=-3(4)-12-5=[MASK]-12[MASK]5[MASK][MASK]. Thus, [MASK] point on [MASK] curve at [MASK] t=[MASK]t=[MASK] is ([MASK],y)=(15,−29)([MASK],y)=(15,-29). 3 Inference Template: Instruction: Please reconstruct and improve the following reasoning, then solve the question. Question: question Reasoning: masked_cot Task: Complete the reasoning by filling in the masked parts ([MASK]), then provide the final answer. Format: CoT: Step-by-step reasoning Answer: Final numeric answer Polish 1 (Same as the Base Prompt above) 2 Polish Template: Context: Question: question Original CoT: CoT Original Answer: answer Instruction: Based on your previous answer and CoT to this question, please rewrite new versions of the CoT to be more understandable and more relevant to the question. Don’t omit any useful information, especially the numbers, and maintain their original meaning when polysemous words appear. Format: CoT: Step-by-step reasoning Answer: Final numeric answer Example: CoT: f(−2)+f(−1)+f(0)=3(−2)−2−2−2+3(−1)−2−1−2+3(0)−20−2=143f(-2)+f(-1)+f(0)= 3(-2)-2-2-2+ 3(-1)-2-1-2+ 3(0)-20-2= 143 Answer: 143 143 Table 11: The configuration of baseline methods. Consistency samples the base prompt N times to reach a consensus, while Reflect and Polish utilize a sequential two-stage process for self-correction and refinement, respectively. Denoise incorporates a token recovery task to stabilize the model’s logical derivation.