Paper deep dive
A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models
Vivek Shukla, Varun Shukla, Atul, Divya Mishra, Mehul Kumar Das
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/3/2026, 10:45:10 AM
Summary
The paper introduces the Reasoning-Answer Faithfulness Score (RAFS), a reference-free metric designed to detect silent reasoning failures in Large Language Models. Unlike traditional evaluation methods that rely on final answer accuracy against a reference, RAFS evaluates the consistency between the generated reasoning trace and the final answer. It combines four components: step validity, reasoning-to-answer entailment with counterfactual sensitivity, answer consensus, and conditional reasoning stability. The framework aims to distinguish between correct answers derived from invalid reasoning (silent failures) and incorrect answers derived from valid reasoning (extraction errors), providing an auditable warning signal for deployment.
Entities (10)
Relation Signals (9)
Reasoning-Answer Faithfulness Score → detects → Silent Reasoning Failures
confidence 95% · RAFS is intended to complement mathematical answer accuracy with an auditable warning signal for silent reasoning failures
Reasoning-Answer Faithfulness Score → measures → Reasoning-Answer Consistency
confidence 95% · RAFS is an automatic measure... that reflects the extent to which generated reasoning trace R and final answer A are in agreement
Reasoning-Answer Faithfulness Score → comprises → Reasoning-to-Answer Entailment
confidence 90% · RAFS combines step validity, reasoning to answer entailment and counterfactual sensitivity...
Reasoning-Answer Faithfulness Score → comprises → Conditional Reasoning Stability
confidence 90% · RAFS combines... and conditional reasoning stability.
Reasoning-Answer Faithfulness Score → comprises → Answer Consensus
confidence 90% · RAFS combines... answer consensus...
Reasoning-Answer Faithfulness Score → comprises → Step Validity
confidence 90% · RAFS combines step validity, reasoning to answer entailment and counterfactual sensitivity, answer consensus, and conditional reasoning stability.
DeBERTa-Large → isusedfor → Entailment Probability Estimation
confidence 88% · a frozen DeBERTa-large model fine-tuned on MNLI [12] supplies directional entailment probabilities
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mathematical chain of thought (CoT) evaluation is commonly reduced to whether the final answer matches a reference. This conflates producing a correct conclusion with producing a valid derivation an invalid chain can accidentally reach the right answer, while a valid calculation can be followed by a transcription error. We call this mismatch the reasoning answer consistency gap. This framework paper introduces the Reasoning Answer Faithfulness Score (RAFS), a reference free, instance level diagnostic of whether an emitted mathematical trace is locally credible, supports its answer, and is stable under resampling and targeted counterfactual interventions. RAFS combines step validity, reasoning to answer entailment and counterfactual sensitivity, answer consensus, and conditional reasoning stability. It evaluates transcript level agreement, not a models private computation and not factual correctness outside the tested mathematical setting. We retain a preregistered, results blind confirmatory study on GSM8K and MATH, with hypotheses, admissibility rules, calibration, and tests fixed before confirmatory outcomes are inspected. A separate feasibility pilot is specified to verify end to end execution and estimate interven tion coverage before that freeze numerical pilot claims are re ported only when trace level artifacts are available. We formalize four reasoning answer outcomes, justify the non compensatory aggregator, instantiate semantic trace distance, quantify compute and abstention tradeoffs, and define verifier independence and power analyses. RAFS is intended to complement mathematical answer accuracy with an auditable warning signal for silent reasoning failures and answer extraction errors
Tags
Links
- Source: https://arxiv.org/abs/2607.26102v1
- Canonical: https://arxiv.org/abs/2607.26102v1
Trouble viewing inline? Open PDF directly →
Full Text
52,372 characters extracted from source content.
Expand or collapse full text
A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models Vivek Shukla Allenhouse Institute of Technology Kanpur, India vivekshukla0552@gmail.com Varun Shukla Allenhouse Institute of Technology Kanpur, India varun.shuklaa@gmail.com Atul Allenhouse Institute of Technology Kanpur, India atulverma15704@gmail.com Divya Mishra Allenhouse Institute of Technology Kanpur, India divyamishra03125@gmail.com Mehul Kumar Das Allenhouse Institute of Technology Kanpur, India dasmehulkumar08@gmail.com Abstract Mathematical chain-of-thought (CoT) evaluation is commonly reduced to whether the final answer matches a reference. This conflates producing a correct conclusion with producing a valid derivation: an invalid chain can accidentally reach the right answer, while a valid calculation can be followed by a transcription error. We call this mismatch the reasoning–answer consistency gap. This framework paper introduces the Reasoning–Answer Faithfulness Score (RAFS), a reference-free, instance-level diagnostic of whether an emitted mathematical trace is locally credible, supports its answer, and is stable under resampling and targeted counterfactual interventions. RAFS combines step validity, reasoning-to-answer entailment and counterfactual sensitivity, answer consensus, and conditional reasoning stability. It evaluates transcript-level agreement, not a model’s private computation and not factual correctness outside the tested mathematical setting. We retain a preregistered, results-blind confirmatory study on GSM8K and MATH, with hypotheses, admissibility rules, calibration, and tests fixed before confirmatory outcomes are inspected. A separate feasibility pilot is specified to verify end-to-end execution and estimate intervention coverage before that freeze; numerical pilot claims are reported only when trace-level artifacts are available. We formalize four reasoning–answer outcomes, justify the non-compensatory aggregator, instantiate semantic trace distance, quantify compute and abstention tradeoffs, and define verifier-independence and power analyses. RAFS is intended to complement mathematical answer accuracy with an auditable warning signal for silent reasoning failures and answer-extraction errors I Introduction Chain-of-thought prompting can improve performance on multi-step tasks by eliciting intermediate natural-language calculations or deductions before a final answer [1]. The same traces are also increasingly presented as explanations: a user is invited to trust not only the answer, but the apparent path by which it was obtained. These two roles are not equivalent. A trace can be useful for generation yet fail to be a valid justification, and a fluent justification can be produced after an answer has effectively been selected [3]. Treating final-answer accuracy as a proxy for reasoning quality therefore creates a consequential blind spot. Consider a model that makes two compensating arithmetic errors and nevertheless outputs the reference answer. Exact-match accuracy deems a response as correct even though the derivation would be incorrect with a slight perturbation. This falls under the category of reasoning false positive. The answer is correct, but the reasoning is incorrect. We refer to such instances as silent reasoning failures. Outcome-only evaluations disguise the reasoning. On the other hand, a model may perform an accurate derivation, but copy the incorrect answer from the multiple-choice options or perform an incorrect rounding in the final step. This would be classed as a reasoning false negative. Even though most of the reasoning is correct, the answer is still incorrect. Combining the two axes into one bit obscures the models capability, and gives poor training feedback, while possibly making an incorrect output appear correct, which is a dangerous position to be in. This paper restricts its claims to mathematical reasoning, where arithmetic and algebraic steps are comparatively auditable. Even in this restricted setting, the hidden defect may reverse an answer under a small change to the problem, while rejecting a sound calculation because of formatting misidentifies the component requiring repair. Existing accuracy benchmarks cannot distinguish these cases; answer self-consistency aggregates outcomes without necessarily validating their chains, and step verifiers may judge each step without measuring whether the final answer follows from the trace. The confirmatory GSM8K/MATH study remains preregistered and results-blind: its hypotheses, model families, annotation protocol, intervention rules, calibration choices, baselines, and tests are fixed before confirmatory outcomes are examined. A small, disjoint feasibility pilot may be used only to establish that the pipeline runs, measure intervention coverage and cost, and debug implementation. It may not be used to select confirmatory hypotheses or report confirmatory performance. boundary retains falsifiability and lets operational gaps be seen before an expensive annotation study.The Reasoning-Answer Faithfulness Score (RAF score) is an automatic measure of [0, 100] that reflects the extent to which generated reasoning trace R and final answer A are in agreement, and is calculated without gold answer requirement for system deployment. RAF score considers local process credibility, global answer support, resampled answer consensus, and reasoning stability. Strategic intervention can determine if changing a certain pivotal step leads to changes in the answer to a directionally expected form, therefore, the score on polished, causally inert explanations is reduced. RAF score should not be seen as a providence oracle since high agreement, in multiple sampled reasoning traces, can occur with the same misconception. It is more a fair, calibrated assessment of the observable answer and explanation.The innovation is threefold: • Reasoning correctness and answer correctness are construed as distinct dimensions which allows the disagreement of the two, rather than answer error, to take precedence in the evaluation. • A RAF score is an integrated, reference-free metric of support that incorporates step-wise validation, counterfactual sensitivity,and a stretch of reasoning uncertainty. • A score, type of failure, the earliest suspect step,and an abstention decision are given with a deployment focused detector, allowing verification or regeneration requests, versus accepted answer in a binary sense. I Related Work I-A Outcome-Based and Self-Consistent Reasoning CoT prompting and self-consistency decoding showed that sampling and aggregating multiple reasoning pathways can improve final-answer accuracy [2]. Thus, GSM8K and MATH became popular benchmarks for assessing mathematical reasoning. However, exact match only compares the answer A with a reference, and majority voting only assesses agreement among answers. Neither establishes that an individual reasoning trace R is valid or that A follows from R. Dataset contamination and template memorization further obscure the interpretation of high benchmark scores [8]. I-B Faithfulness and Reasoning-Trace Evaluation Prior work shows that CoT explanations can omit influential prompt features or rationalize biased answers. Intervention-based tests such as truncating or corrupting a chain probe whether later predictions depend on earlier reasoning [5]. Parcalabescu and Frank argue that many purported faithfulness tests more precisely measure output-level self-consistency, not correspondence to inaccessible internal computation [10]. We adopt this distinction: RAFS measures observable reasoning–answer faithfulness and uses interventions as evidence, not proof, of causal mediation. Reference-free metrics score semantic alignment, logicality, informativeness, and fluency, while direct evaluation frameworks reconstruct or judge reasoning paths. These approaches improve process visibility, but they do not jointly encode the four-quadrant outcome taxonomy and the stability of the R→AR\!→\!A relation across samples. I-C 2024–2025 Process Supervision Recent work increasingly evaluates steps rather than outcomes. Causal objectives and inferential bridging analyze the relevance among context, CoT, and answer. ProcessBench provides expert labels for locating the earliest error in mathematical solutions and reports limited generalization of several process reward models (PRMs) to harder problems [20]. ThinkPRM instead performs generative, step-wise verification with substantially fewer process labels than conventional discriminative PRMs. Studies of contemporary reasoning models further suggest that explicit CoTs may fail to disclose prompt influences and may be unreliable monitoring targets. RAFS complements these directions. Unlike a PRM, it is not a new reward model architecture; unlike an outcome verifier, it does not reduce a trace to answer likelihood; unlike a similarity-only metric, it tests whether pivotal reasoning changes propagate to A. Its contribution is a model-agnostic aggregation protocol and diagnostic taxonomy that can wrap a critic model, symbolic checker, NLI model, or specialized PRM. TABLE I: Component-level comparison. A checkmark denotes an explicit target of the method; “partial” denotes related evidence that does not instantiate the RAFS component. Method PRP_R SRAS_RA CAC_A DRD_R Primary object measured ProcessBench ✓ – – – Expert benchmark for step-error detection and earliest-error localization. ThinkPRM ✓ – – – Generative step-wise process verification; it can supply qtq_t but does not test answer dependence. ROSCOE partial partial – partial Reference-free semantic alignment, logicality, informativeness, and fluency; no controlled R→AR\!→\!A intervention. FRODO – ✓ – – Causal/counterfactual rationale influence as a training and evaluation objective. RAFS (ours) ✓ ✓ ✓ ✓ Instance-level aggregation plus four-quadrant diagnosis, coverage-aware abstention, and suspect-step output. I Problem Formulation Let x be a problem, R=(r1,…,rT)R=(r_1,…,r_T) a natural-language reasoning trace divided into T atomic steps, and A the extracted final answer. When references are available for evaluation, let A∗A^* denote the gold answer and V∗∈0,1V^*∈\0,1\ denote expert judgment that the chain is a valid derivation for x. Define answer correctness YA=[A≡A∗]Y_A=1[A≡ A^*] and reasoning correctness YR=V∗Y_R=V^*. The pair (YR,YA)(Y_R,Y_A) induces four outcomes (Fig. 1). This separation is useful for formal mathematical and logical tasks in machine learning [17], including structured algebraic settings in which intermediate polynomial operations must remain auditable [14]. A incorrectAA correctRR correctRR incorrectExtraction failureRR right, A wrongreasoning false negativeFaithful successRR right, A rightOvert failureRR wrong, A wrongSilent failureRR wrong, A rightreasoning false positiveFinal-answer correctness (YAY_A)Reasoning correctness (YRY_R) Figure 1: The reasoning–answer consistency gap. Accuracy observes only the horizontal axis and therefore merges faithful success with silent failure, and extraction failure with overt failure. The evaluation objective is not merely to estimate YAY_A, but to estimate whether R is trustworthy and supports A. At deployment, A∗A^* and V∗V^* are unavailable. We therefore construct observable proxy signals from a verifier ensemble, sampled solutions, and controlled interventions. RAFS estimates the consistency event F(x,R,A)=[R is credible] 1[R⊧A],F(x,R,A)=1[R is credible]\,1[R A], (1) while separately retaining uncertainty. On labeled test sets, RAFS is evaluated against YR=YAY_R=Y_A and, more importantly, against the two off-diagonal error classes. IV Reasoning–Answer Faithfulness Score IV-A Step Validity An atomic-step verifier g receives (x,r1:t)(x,r_1:t) and returns qt=Pg(vt=1∣x,r1:t),q_t=P_g(v_t=1 x,r_1:t), (2) where vtv_t indicates that step t follows from the problem and preceding steps. The verifier may combine an LLM critic, deterministic arithmetic checks, contradiction detection, correctness/informativeness evaluation, and verifier-benchmark supervision [13, 15]. To penalize a single fatal step while remaining differentiable, we use a mixture of geometric mean and minimum: PR=ρexp(1T∑t=1Tlog(qt+ϵ))+(1−ρ)mintqt.P_R=ρ ( 1T _t=1^T (q_t+ε) )+(1-ρ) _tq_t. (3) The earliest t below a calibrated threshold is returned as the suspect step. IV-B Reasoning-to-Answer Support Let e=Pg(R⊧A∣x)e=P_g(R A x) be the verifier’s probability that the final answer is entailed by the complete trace. Entailment alone is vulnerable to post-hoc rationalization. We therefore identify a set J of pivotal steps using leave-one-step-out influence or verifier attribution, and produce candidate counterfactual chains R(j)R^(j) that alter the conclusion of step j. This intervention design is motivated by causal-rationale and inferential-bridging approaches [18, 19]. The intended perturbation direction (e.g., increasing a computed quantity or reversing a Boolean claim) is recorded before the answer head is queried. Let A(j)A^(j) be the answer obtained when an answer head reads an admissible R(j)R^(j), and let Δj∈[0,1] _j∈[0,1] measure whether the answer changes in that preregistered direction. We distinguish intervention validity from whether the edited claim is true: a useful counterfactual may deliberately make the target claim false, but it must isolate that change. A candidate is admitted only if it passes all of the following preregistered rubric items: 1. Atomicity and locality: rjr_j contains one designated proposition cjc_j, and the edit changes only cjc_j. All steps rir_i for i≠ji≠ j, all non-target propositions within rjr_j, and the problem x are held fixed; the original answer span is masked from the edit generator. 2. Bounded form change: after answer-span masking and token normalization, the normalized token-level Levenshtein distance satisfies dedit(rj,rj(j))=Lev(rj,rj(j))max(|rj|,|rj(j)|)≤δedit,d_edit(r_j,r_j^(j))= Lev(r_j,r_j^(j)) (|r_j|,|r_j^(j)|)≤ _edit, (4) where δedit _edit is fixed in the preregistration (default 0.200.20). The edit may not insert a new premise, justification, or solution path. 3. Semantic isolation: a claim differencer must identify exactly one changed proposition, cj→cj′c_j→ c_j . The replacement preserves entities, units, variable bindings, quantifiers not under intervention, and the claim’s semantic type (numeric, relational, Boolean, or categorical); it introduces no collateral contradiction beyond the intended change to cjc_j and its direct implications. 4. Well-formedness and non-leakage: the edited step is grammatical, type- and unit-consistent, and interpretable in the original context. It may not mention, paraphrase, or otherwise leak the expected counterfactual answer. 5. Independent validation: a symbolic checker is used when the claim is executable; otherwise two validators, not used to generate the edit, judge locality and semantic validity. Both must accept. Disagreement, parser failure, or an ambiguous predicted answer direction rejects the candidate. The claim differencer and validators operate before A(j)A^(j) is observed. This ordering prevents an intervention from being retained because it happened to produce the desired response. Let Jval⊆J_val J be the steps with admitted interventions. Counterfactual sensitivity is IRA=1|Jval|∑j∈JvalΔj,SRA=λe+(1−λ)IRA.I_RA= 1|J_val| _j∈ J_val _j, S_RA=λ e+(1-λ)I_RA. (5) A chain that verbally entails A but has no measurable effect on answer generation receives limited support. Rejected interventions are logged with a failure code and are not counted as negative evidence. If fewer than the preregistered minimum mminm_ interventions are admissible (default mmin=2m_ =2), IRAI_RA is marked unavailable and RAFS abstains rather than imputing counterfactual support. IV-C Consensus and Conditional Stability Sample K independent pairs (Rk,Ak)k=1K\(R_k,A_k)\_k=1^K at nonzero temperature. For normalized answer classes A, let p(a)=K−1∑k[Ak=a]p(a)=K^-1 _k1[A_k=a]. Normalized answer consensus is CA=1−H(p)log(max(2,||)),H(p)=−∑a∈p(a)logp(a).C_A=1- H(p) ( (2,|A|)), H(p)=- _a p(a) p(a). (6) High answer consensus can coexist with diverse or contradictory explanations. Semantic reasoning metrics motivate comparing normalized claims rather than surface wording [11]. We therefore compute conditional reasoning stability among traces supporting the modal answer a^=argmaxap(a) a= _ap(a). Concretely, z(R)z(R) is the ordered multiset of atomic claims after canonicalizing numbers, units, and variable names. For claims c,c′c,c , a frozen DeBERTa-large model fine-tuned on MNLI [12] supplies directional entailment probabilities, and dc(c,c′)=1−12[p(c⇒c′)+p(c′⇒c)]d_c(c,c )=1- 12[p(c c )+p(c c)]. We define d(z(Ri),z(Rj))d(z(R_i),z(R_j)) as the minimum-cost monotone alignment of their claim sequences under dcd_c, with cost one for an unmatched claim, normalized by the longer sequence. Bidirectional entailment distinguishes contradiction or a missing proof obligation from paraphrase; cosine embedding distance is cheaper but can place contradictory claims close together. Executable numeric claims are canonicalized by symbolic equivalence before NLI scoring. With this fixed d∈[0,1]d∈[0,1], DR=1−2|a^|(|a^|−1)∑i<j∈a^d(z(Ri),z(Rj)).D_R=1- 2|K_ a|(|K_ a|-1) _i<j _ ad(z(R_i),z(R_j)). (7) If fewer than two traces support a a, we set DR=0D_R=0 and force abstention. IV-D Composite Score and Calibration For weights =(wP,wS,wC,wD) w=(w_P,w_S,w_C,w_D) on the simplex, RAFS is the weighted geometric mean RAFS=100exp(∑m∈P,S,C,Dwmlog(sm+ϵ)),RAFS=100 ( _m∈\P,S,C,D\w_m (s_m+ε) ), (8) where (sP,sS,sC,sD)=(PR,SRA,CA,DR)(s_P,s_S,s_C,s_D)=(P_R,S_RA,C_A,D_R). The geometric form is intentionally non-compensatory: excellent fluency or consensus cannot fully erase a near-zero validity or support score. We fit ρ, λ, w, and decision thresholds on a held-out development split using expert four-quadrant labels. Calibration uses isotonic regression or temperature scaling, never the test labels. The aggregator is chosen to match an explicit failure semantics: every positively weighted component represents necessary evidence. Consequently, sm→0s_m→ 0 must imply RAFS→0→ 0. An arithmetic mean violates this boundary condition because three strong components can compensate for a failed process-validity or reasoning-support signal. Hard minimum gating satisfies the boundary condition, but it is non-smooth and allows a single miscalibrated component to dominate the decision. Under continuity, monotonicity, the zero-boundary condition, and constant relative sensitivity ∂logf/∂logsm=wm∂ f/∂ s_m=w_m, integration yields logf=∑mwmlogsm+c f= _mw_m s_m+c. The normalization f(1,…,1)=1f(1,…,1)=1 then gives Eq. (8). Figure 2 illustrates this distinction by varying one weak component while fixing the remaining three components at 0.90.9 and assigning equal weights. The arithmetic mean remains approximately 0.680.68 even when the weak component is zero, demonstrating undesirable compensation. Hard minimum gating follows the weakest component exactly until it reaches 0.90.9, making it maximally sensitive to a single estimate. The geometric mean approaches zero with the failed component but changes smoothly, which is the intended RAFS behavior. Arithmetic, minimum-gated, and learned logistic aggregators remain preregistered ablations rather than being excluded by construction. 00.10.10.20.20.30.30.40.40.50.50.60.60.70.70.80.80.90.91100.20.20.40.40.60.60.80.811Weakest component value s1s_1Aggregate scoreGeometricArithmeticHard minimum Figure 2: Analytical response of candidate aggregators when s2=s3=s4=0.9s_2=s_3=s_4=0.9 and weights are equal. The arithmetic mean remains high when one necessary signal fails; minimum gating is maximally sensitive; the geometric mean provides smooth, non-compensatory behavior. Two calibrated thresholds define the deployment states: accept when RAFS ≥τh≥ _h and uncertainty is low; review/regenerate when RAFS ≤τl≤ _l; and abstain otherwise. Without a gold answer, high answer consensus paired with low PRP_R or SRAS_RA is flagged as a silent-failure candidate. Low consensus with high PRP_R and high entailment instead suggests answer-extraction or decoding instability. Algorithm 1 Reference-Free RAFS Calculation 1:problem x, primary pair (R,A)(R,A), samples K, verifier g 2:score s, state, suspect step t∗t^* 3:segment R into atomic steps (r1,…,rT)(r_1,…,r_T) 4:for t=1t=1 to T do 5: qt←g(x,r1:t)q_t← g(x,r_1:t); run applicable symbolic checks 6:end for 7:compute PRP_R using Eq. (3); t∗←mint:qt<τvt^*← \t:q_t< _v\ 8:e←g(R⊧A∣x)e← g(R A x); identify pivotal steps J 9:for all j∈Jj∈ J do 10: generate candidate R(j)R^(j) and record the predicted response direction 11: validate Eq. (4) and rubric items 1–5 before querying the answer head 12: if R(j)R^(j) is admissible then 13: add j to JvalJ_val; then measure answer response Δj _j 14: end if 15:end for 16:if |Jval|<mmin|J_val|<m_ then 17: return Abstain with intervention-coverage flag 18:end if 19:compute SRAS_RA using Eq. (5) 20:sample (Rk,Ak)k=1K\(R_k,A_k)\_k=1^K and normalize answers 21:compute CAC_A and DRD_R using Eqs. (6)–(7) 22:s←100exp(∑mwmlog(sm+ϵ))s← 100 ( _mw_m (s_m+ε)) 23:state ← calibrated Accept, Review, or Abstain 24:return (s,state,t∗)(s,state,t^*) V Detection Architecture Figure 3 shows the operational pipeline, complementing direct reasoning-path evaluation with explicitly separated evidence branches [16]. The generator first produces a primary trace and answer. A parser separates atomic claims, calculations, and the answer span. Three analysis branches then run in parallel: (i) a process verifier scores each prefix and invokes symbolic tools when possible; (i) an answer-support module tests entailment and counterfactual dependence; and (i) a sampler estimates answer entropy and within-answer reasoning stability. The aggregator returns RAFS together with decomposed scores, so a user can tell whether rejection arose from a local invalid step, a trace–answer mismatch, or broad sampling uncertainty. Problem xxCoT generator(R,A)(R,A)Step and answernormalizationProcess branchstep scores q1:Tq_1:Tsymbolic checksSupport branchR⊧AR Acounterfactual editsSampling branchanswer entropyreasoning stabilityRAFSaggregatorEq. (8)Accept / review / abstainfailure type + suspect step Ground-truth-free evidence Figure 3: RAFS detection architecture. Independent evidence branches make the final decision auditable and allow deterministic tools or learned verifiers to be exchanged without changing the metric. The pipeline has two safeguards. First, cross-family independence means that no judge, edit validator, or answer head shares a base-model family or instruction-tuning lineage with the trace generator: for example, DeepSeek judges Llama traces, Llama judges Mistral traces, and Mistral judges DeepSeek traces. Merely changing a system prompt, adapter, quantization, or checkpoint size within Llama does not count as independent. Symbolic checks are family-independent. The confirmatory ablation holds prompts, traces, calibration size, and decision thresholds fixed and compares (i) same-checkpoint, (i) different checkpoint in the same family, and (i) cross-family judging; it reports silent-failure AUPRC, recall at 90% precision, calibration, and judge–generator error correlation. Second, all critic probabilities are calibrated on trace- and step-level labels. If symbolic execution is available, it overrides a conflicting soft arithmetic judgment; for open-ended claims, disagreement increases uncertainty rather than being resolved by unjustified majority vote. VI Experimental Design The confirmatory evaluation uses GSM8K and MATH as complementary arithmetic and competition-mathematics benchmarks [6, 7]. VI-A Feasibility Pilot and Confirmatory Boundary Before freezing the confirmatory study, we reserve 100 disjoint items (50 GSM8K and 50 MATH) and one 7–8B generator for an implementation-only pilot. The pilot must archive every emitted trace, atomic segmentation, calibrated qtq_t, intervention candidate and rejection code, answer-head response, component score, latency, and human quadrant label. Its mandatory report includes the median and interquartile range of qtq_t (plus the lower-tail mass below τv _v), candidate admission by each rubric item, the instance fraction with |Jval|<2|J_val|<2, and the four-quadrant counts with exact binomial intervals. It also compares geometric, arithmetic, minimum-gated, and logistic aggregation with nested cross-validation internal to the pilot. Since these comparisons are descriptive, they don’t alter Eq. (8) or the confirmatory hypotheses. The current source manuscript does not contain pilot artifacts or executable model outputs that allow verification of such quantities. Consequently, we do not produce numerical ’pilot results’ based on illustrative scores. This paragraph will be replaced with archived data and the citation of the item-level evidence table if this version is to be claimed as having the feasibility evidence. Until that time, the pilot serves as a pre-confirmatory gate rather than as evidence. It is better to have this clear limitation, rather than allow the hand-assigned values of Fig. 5 to be mistaken for observations. VI-B Models and Benchmarks We suggest assessment of three families of open-weight models with different training recipes and capabilities: Llama 3.1/3.2 Instruct, Mistral 7B or Mixtral Instruct, and DeepSeekMath-7B or a distilled DeepSeek-R1 variant [23, 24]. Models are assessed in their publicly available inference configurations at 7–8B with the option of including a larger checkpoint to assess scaling. The primary data sets are GSM8K and MATH, which feature problem solving for multi-step grade school arithmetic and advanced topics including algebra, counting, geometry, intermediate algebra, number theory, precalculus, and probability, presented at a competition level of difficulty. For each problem, the primary (R, A) evidence is generated via greedy decoding, and the evidence is generated via K = 16 samples with temperature sampling. Answers are normalized via task-specific parsers. Reasoning is segmented at the boundaries of sentences and transitions of equations, and is then audited manually for a stratified subset. Experiments use fixed prompts, decoding budgets, and random seeds across models. No benchmark solution or gold answer is exposed to RAFS at inference; references are used only for post-hoc accuracy and quadrant labels. VI-C Four-Quadrant Annotation and Power We increase the annotation target from 1,200 to 2,400 responses, stratified across model, dataset, and answer correctness. Two mathematically qualified annotators independently identify the earliest invalid step, legitimate recovery, trace–answer entailment, and quadrant; a third adjudicates disagreements. We report Cohen’s κ for reasoning validity and Krippendorff’s α for error location. To reduce hindsight bias, annotators first inspect R with the answer span masked, then judge R⊧AR A after unmasking it. The increase is driven by the expected rarity of silent failures. At 10% prevalence, n=1,200n=1,200 yields only about 120 positives and unstable tail precision; n=2,400n=2,400 yields about 240. A preregistration-time Monte Carlo calculation (10% prevalence, baseline AUPRC 0.300.30, paired score correlation 0.600.60, one-sided paired bootstrap test at α=.05α=.05) gave 81% power for an AUPRC increase of 0.0680.068 over 500 simulated studies (200 bootstrap replicates, seed 19). These distributional assumptions are planning inputs, not pilot findings. If the feasibility pilot’s exact 95% prevalence interval implies fewer than 200 silent positives in 2,400 annotations, sampling is adaptively enriched with answer-correct/low-process-score cases until at least 200 silent positives are adjudicated; inverse-probability weights recover population AUPRC. The enrichment rule and weights are fixed before confirmatory labels are opened. VI-D Baselines and Ablations RAFS is compared with: (1) exact-match answer accuracy; (2) answer self-consistency; (3) mean LLM-as-a-judge step score; (4) ROSCOE-style reference-free metrics; (5) a PRM score; and (6) entailment between the full chain and answer. Ablations remove PRP_R, IRAI_RA, CAC_A, or DRD_R; replace the geometric mean with arithmetic, hard-minimum, and learned logistic aggregation; vary K∈4,8,16,32K∈\4,8,16,32\; and compare same-checkpoint, within-family, and cross-family judges. A targeted ablation compares random perturbations with pivotal-step interventions to test whether semantic precision is necessary. VI-E Evaluation Metrics and Hypotheses The primary endpoint is macro-F1 over the four quadrants. Because silent failures are the central risk, we also report their area under the precision–recall curve (AUPRC), recall at 90% precision, and expected calibration error. Step localization is measured by exact earliest-error accuracy and distance from the expert index. Selective prediction is evaluated by risk–coverage curves: as low-RAFS cases are deferred, the error rate among accepted responses should decrease. We test four preregistered hypotheses. H1: answer accuracy overestimates valid reasoning by merging silent failures with faithful successes. H2: RAFS improves silent-failure AUPRC over answer consensus and mean step score. H3: counterfactual sensitivity contributes most strongly when answers are correct but reasoning is invalid. H4: conditional reasoning stability detects shared-answer rationalization that answer entropy alone misses. Confidence intervals are obtained by problem-level bootstrap; paired model comparisons use permutation tests with Holm correction. TABLE I: Planned reporting matrix. No numerical result is asserted before execution of the preregistered experiments. Method 4-way F1 Silent AUPRC ECE ↓ Exact match – baseline – Answer self-consistency planned planned planned Step judge / PRM planned planned planned Entailment only planned planned planned RAFS (ours) planned planned planned VII Coverage, Cost, and Deployment Modes VII-A Intervention Coverage and Expected Abstention The five-item rubric intentionally sacrifices coverage for easily interpretable interventions. Before pilot measurements exist, deployment capacity will rely on sensitivity analysis, not reported empirical rates. If we attempt to admit three pivotal candidates with each candidate assuming a 55 percent admission probability, the candidate rejection rate is 45 percent. The binomial probability of two or fewer admissions is then calculated to be 42.5 percent. Attempting to admit five independent candidates lowers this planning parameter to 13.1 percent. Independence is a loose assumption; the correlated validator failures may result in a higher abstention rate. Therefore, this feasibility pilot provides both candidate and instance coverage, with bootstrapped rejection codes for locality, edit distance, semantic isolation, leakage, parser failure, validator disagreement, and ambiguous direction, with proposed abstention rates determined to be a 40 percent rejection rate for selective audits of a small number of high-consequence mathematical solutions. This abstention rate is not acceptable for an always-on benchmark scorer. Operational policy will attempt to admit candidates sequentially, up to five. If two admissions are realized, the policy will apply a coverage abstention, as will the policy if the coverage threshold is not met. Coverage abstention will be distinguished from a low-score review, in that coverage abstention indicates a lack of evidence for causation, not unfaithfulness of the trace. 22334455667700.20.20.40.40.60.60.80.81115% planning thresholdCandidate interventions attemptedP(|Jval|<2)P(|J_val|<2) Figure 4: Planning sensitivity of coverage abstention to the candidate budget, assuming independent admission probability 0.550.55 and mmin=2m_ =2. This is an analytical planning curve, not an observed pilot result. VII-B Call and Latency Budget Consider the defaulted K=16K=16. Three candidate edits, two validators per edit, and two admitted interventions. As an example, let us consider a single primary solution. Computing k=16 resamples will likely involve a batched step-verification call, support/pivotal-analysis calls, edit generations, validations, and counterfactual answer-head calls. Note that prefix-by-prefix verification (without batching) requires T - 1 calls, where T is the total number of tokens in your answer. Most of the calls within the sampling and validation branches can be executed in parallel. Therefore, the total (wall-clock) time of the latency will be determined by four to seven waves of calls, rather than 31. During the sampling and validation calls within RAFS, token consumption (estimation) is expected to be between 12 to 25 times greater than a base inference, as many judge outputs are relatively short. The pilot measures the uncached tokens, along with median/95th latency, along with the system’s hardware, batch size, and peak memory measures. Full RAFS is designed for expert review and dataset auditing. For these applications, the cost of a missed silent failure is greater than dozens of short verifier calls. RAFS-lite uses K=4K=4, one batched process judgment and trace–answer entailment, and the same NLI stability calculation, but skips counterfactual generation and reports (PR,e,CA,DR)(P_R,e,C_A,D_R) with a separately calibrated lite score. This version retains no claims of counterfactual faithfulness and consists of seven calls, at a minimum. Ambiguous and high-impact cases are to be escalated to full RAFS. A basic screening mode which performs only symbolic checks and answer normalization is to be used in lieu of full RAFS. VIII Illustrative Silent-Failure Case Figure 5 illustrates a failure that exact match cannot expose. The problem asks for profit after buying an item for $40 and selling it for $55. The trace incorrectly computes 55−40=1055-40=10, then introduces an unsupported “$5 adjustment” and outputs $15. The answer happens to match the reference, but neither local validity nor global entailment is acceptable. A pivotal-step intervention that corrects the subtraction eliminates the need for the invented adjustment; the model’s original answer mechanism is therefore unstable. RAFS flags the trace even though YA=1Y_A=1. Question: An item is bought for $40 and sold for $55. What is the profit? Step 1: Profit equals selling price minus purchase price. q1=.98q_1=.98 Step 2: 55−40=1055-40=10. q2=.03q_2=.03 Step 3: Add a $5 “transaction adjustment,” giving $15. q3=.08q_3=.08 Final answer: $15. Exact match: ✓ PRP_R low; R⊧AR A unsupported; intervention unstable RAFS: Review—silent-failure candidate at Step 2 Figure 5: Illustrative silent failure. The numerical RAFS is intentionally omitted until weights and verifier outputs are calibrated; the decomposed evidence already identifies the failure mode. This example also shows why an answer-only perturbation is insufficient. If a judge sees the correct reference, it may overlook the invalid chain through outcome bias. RAFS instead requires agreement among local checks, trace-level support, and intervention behavior. In a real evaluation, the displayed qtq_t values would be calibrated verifier outputs rather than hand-assigned illustrative values. IX RAFS-Guided Correction Detection is helpful only when the uncertainty can be acknowledged and corrected. We introduce the correction operator (x,R,A,)C(x,R,A, s), where =(PR,SRA,CA,DR) s=(P_R,S_RA,C_A,D_R), which acts the least selectively based on the supporting diagnostic evidence. The operator subliminally provides the final answer to a new solver only when answer comparison is made, making anchoring to the potentially incorrect response less severe. IX-A Localized Reasoning Repair When PRP_R is low and an earliest suspect step t∗t^* is identified, the system preserves the verified prefix r1:t∗−1r_1:t^*-1 and regenerates the suffix under the requirement that replacement steps state their premises and satisfy available symbolic checks. We draw B candidate suffixes and reject any candidate whose first replacement step fails verification. The highest-RAFS surviving pair is selected only if it improves on the original by a margin δ and reaches the accept threshold; otherwise, the system abstains. Formally, (R′,A′)=argmax(R~,A~)∈ℬ(t∗)RAFS(x,R~,A~),(R ,A )= _( R, A) (t^*)RAFS(x, R, A), (9) subject to r1:t∗−1′=r1:t∗−1r _1:t^*-1=r_1:t^*-1 and qt∗:T′≥τvq _t^*:T≥ _v. Preserving the verified prefix limits unnecessary semantic drift and makes the correction auditable. IX-B Answer-Extraction Repair When process and entailment scores are high but sample answers differ, we consider failures post-derivation. The system freezes R, and a constrained answer extractor is invoked. The output is required to be a deterministic function of the final supported claim. For multiple-choice tasks, this directly includes a value-to-option mapping check. For numerical tasks, checks for sign, units, simplifications, and rounding are included. Most importantly, we do not regenerate reasoning, as this has the potential to replace a valid derivation with a less faithful one. If the corrected A is entailed by R and increases answer consensus, we log this as an extraction correction and not a reasoning improvement. IX-C Independent Re-Solving and Abstention A low SRA with high answer consensus is a typical example of a “silent-failure” behavior. Local edits can be risky because the answer may end up being a binding constraint to all candidate repairs. The system, therefore, employs a different prompt, and checkpoint, and/or a different symbolic route with both R and A concealed, and requests an independent solution.Agreement is determined for the intermediate claims and is considered before answer comparison. If the independent solution is achieved via the same validated route and answer,it replaces the initial explanation. If it is via a different validated route and answer, the case is escalated. IX-D Correction Evaluation Correction is evaluated separately from detection. Metrics include the correction success rate, which is the rate of previously invalid cases which transitioned to faithful successes; the regression rate, which is the rate of faithful successes which were negatively impacted; edit locality, which is measured in the normalized length of the suffix which was edited; and compute overhead. We also consider false reassurances, which are cases in which RAFS increased, but the validity of the reasoning from the expert did not improve.A valid correction must improve the reasoned expert quadrant,not merely optimize the automated score. Other metrics included full-response regeneration, self-refine prompting,answer-only re-extraction, and the proposed failure-type-specific policy. This design is meant to test if RAFS presents an actionable structure, beyond merely ranking the responses. X Analysis and Discussion X-A Interpreting RAFS RAFS is best seen as a formalized structured warning signal. A high score denotes that the observed trace is locally credible, supports the answer, and is stable across the defined sampling and intervention distribution. While a high score verifies the observed trace is stable, it does not support the veracity of the answer, as model family and model verifier can have a shared coherent fallacy. Low scores are more prescriptive. The described components determine whether a user should repair that step, re-run the answer extraction, engage an external solver, or abstain from the task. The four-quadrant model introduces a new dimension to model comparison. Consider two systems having the same exact-match accuracy. A human underlining system that produces fewer incorrect silent failures is, in general, superior, given that its correct responses are more frequently backed up. A system that exhibits a higher number of reasoning false negatives may be process competent and may have a sufficiently capable answer head or a sufficiently constrained answer decoder. RAFS makes these distinctions clear, given that accuracy has a scoring system that has an equal distribution across the systems. X-B Contribution to Explainable and Safe AI Measurable limitations on explainable AI show where explainable AI can be advanced. RAFS evaluates dependency between decision and explanation and develops human oversight. A reviewer could see why a decision is suspicious and if the decision is insensitive to rationale changes. Selective deployment can apply more powerful models, expert systems, and symbolic tools to unreasonably low scoring cases while keeping automation for stable and predictable cases. RAFS can produce multi-objective feedback. During training, silent failures should not be rewarded in the same way as faithful successes just because the two answers are the same. Reasoning false negatives that do not have an obvious outcome should also not be rewarded in the same way as evident failures. In the future, a trainer could reach an answer that is correct in the RAFS sense, optimizing score correctness while achieving suboptimal RAFS, preventing outcome from dominating process.. To avoid reward hacking, however, RAFS should be periodically audited with novel intervention templates and independent human labels. X-C Adversarial Inference-Time Gaming Inference-time gaming is distinct from training-time reward hacking. Recent analyses of reasoning-model disclosure and naturally occurring CoT faithfulness motivate treating this as a separate evaluation threat [21, 22]. A generator that knows the audit may emit verbose, redundant rationalizations whose claims are difficult to atomize, or write an answer-independent explanation engineered to remain superficially coherent under familiar edits. Such behavior can raise judge confidence while reducing intervention admissibility, producing abstention rather than a clean low score. It can also repeat the same claim in several forms so that a local edit appears not to affect the answer. Mitigations include hiding the sampled intervention family from the generator; drawing periodic novel templates that change numeric, relational, and proof-obligation claims; collapsing semantically duplicate claims before selecting pivotal steps; and red-teaming with models fine-tuned to maximize RAFS while preserving invalid solutions. Results are reported both on standard traces and on this adaptive adversarial set. A rise in intervention rejection is itself a monitored signal, and coverage abstention cannot be converted to acceptance by verbosity. These measures reduce predictable gaming but do not establish strategy-proofness. X-D Limitations and Threats to Validity First, natural-language steps are difficult to atomize; a single sentence can contain several claims, and an apparent error can be repaired later. Second, verifier bias can correlate with generator bias. Independent ensembles and symbolic checks mitigate but do not eliminate this problem. Third, counterfactual edits may create unnatural chains, so intervention validity must be screened. Fourth, reasoning diversity is not always bad: two distinct proofs can support the same answer. Our conditional stability representation must compare normalized claims or proof obligations rather than surface wording. Fifth, RAFS evaluates emitted text, not private activations. Even strong counterfactual response is only evidence that the text mediates the tested prediction pathway. Sixth, the planning calculation for abstention assumes independent candidate admissions, whereas rubric failures may be correlated. Finally, all present definitions, validators, and claims are restricted to GSM8K/MATH-style mathematical reasoning. XI Conclusion Final-answer accuracy alone cannot determine whether a mathematical CoT is valid. It hides correct answers supported by invalid chains and misclassifies valid derivations followed by answer-selection errors. We formalized this reasoning–answer consistency gap and introduced RAFS, a reference-free score combining step validity, trace-to-answer support, counterfactual sensitivity, answer consensus, and conditional reasoning stability. The pipeline produces an auditable failure type and abstains under uncertainty or inadequate intervention coverage. The preregistered Llama, Mistral, and DeepSeek study on GSM8K and MATH will test whether RAFS improves silent-failure detection over accuracy, self-consistency, and isolated process scores; until the feasibility artifacts and confirmatory results exist, the contribution is a technically specified and falsifiable mathematical-reasoning framework, not a demonstrated safety guarantee. The component decomposition interprets operational failures and lends itself to more than just a single score. Analyzing process validity can encourage step repair. Low support predictability can encourage independent re-solving. Low support sampling can encourage review or abstention. Most importantly, RAFS captures unsupported correct answers as a unique evaluation outcome. In the absence of RAFS, unsupported correct answers are captured as part of overall correct answer submissions. This framework allows for clear analysis of various interventions and the uncertainty they produce. Because unsupported evidence cannot be confused with positive evidence, this framework promotes evidence value evaluation. This framework allows for clear analysis for interventions with a low final answer score, but a great deal of rational support. XII Future Work Next, we will execute and publish the feasibility pilot, followed by the preregistered confirmatory experiments spanning model scales, decoding strategies, and mathematical difficulty. Any extension to scientific reasoning, medicine, or law will likely need new domain-specific verifiers, retrieval or evidence grounding, expert validity definitions, and new calibration, all of which are absent here. Future work could address the atomization, adaptive interventions, and ensembles of verifiers, and apply adversarially optimized generators. Longitudinal and human review studies could evaluate if RAFS-guided training and calibrated abstention help improve the quality of mathematical decisions. It will be crucial to examine transfer to new problem templates, new proof styles, and significantly longer proofs. Time-efficient implementations should be able to achieve almost full RAFS reliability at the cost of RAFS-lite using adaptive sampling, verifier cascades, and early stopping. Constructing public intervention toolkits, and standardized reporting frameworks will help to evaluate, in a consistent manner, adversarial robustness, verifier families, and dimensions of coverage, calibration, and latency. New benchmarks should also involve controlled silent-failure with goal-directed reasoning and answer extraction tasks with designed failures and self-correcting errors. Such challenge sets would enable repeatable stress testing and reveal whether improvements persist outside naturally occurring benchmark errors. Any later extension to autonomous cloud services or critical-infrastructure cyber defence would require domain-specific state, evidence, and intervention validators beyond the present mathematical setting [9, 4]. References [1] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Proc. NeurIPS, 2022. [2] X. Wang et al., “Self-consistency improves chain of thought reasoning in language models,” in Proc. ICLR, 2023. [3] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting,” in Proc. NeurIPS, 2023. [4] Atul et al., “Federated generative intelligence for explainable and autonomous cyber defence in critical infrastructures,” in Proc. IEEE ISCS, 2025, doi: 10.1109/ISCS69371.2025.11386415. [5] T. Lanham et al., “Measuring faithfulness in chain-of-thought reasoning,” arXiv:2307.13702, 2023. [6] K. Cobbe et al., “Training verifiers to solve math word problems,” arXiv:2110.14168, 2021. [7] D. Hendrycks et al., “Measuring mathematical problem solving with the MATH dataset,” in Proc. NeurIPS Datasets and Benchmarks, 2021. [8] H. Zhang et al., “GSM-Symbolic/GSM1K: Are large language models actually good at arithmetic reasoning?” arXiv:2405.00332, 2024. [9] V. Shukla et al., “Agentic AI framework for autonomous and self-managing cloud services,” in Proc. IEEE SCEECS, 2026, doi: 10.1109/SCEECS68810.2026.11429932. [10] L. Parcalabescu and A. Frank, “On measuring faithfulness or self-consistency of natural language explanations,” in Proc. ACL, 2024. [11] O. Golovneva et al., “ROSCOE: A suite of metrics for scoring step-by-step reasoning,” in Proc. ICLR, 2023. [12] P. He, X. Liu, J. Gao, and W. Chen, “DeBERTa: Decoding-enhanced BERT with disentangled attention,” in Proc. ICLR, 2021. [13] A. Prasad et al., “ReCEval: Evaluating reasoning chains via correctness and informativeness,” in Proc. EMNLP, 2023. [14] V. Shukla, A. Chaturvedi, and M. K. Misra, “On authentication schemes using polynomials over non commutative rings,” Wireless Personal Communications, vol. 118, no. 1, p. 1–9, 2021, doi: 10.1007/s11277-020-08008-4. [15] A. Jacovi et al., “A benchmark for verifiers of reasoning chains,” in Proc. ACL, 2024. [16] G. Bao et al., “Direct evaluation of chain-of-thought in multi-hop reasoning with knowledge graphs,” in Findings of ACL, 2024. [17] A. Awasthi, Pranjal, A. Chaturvedi, V. Shukla, and M. K. Misra, “Mathematics and logics in ML: Application aspects,” in Proc. IEEE International Students’ Conference on Electrical, Electronics and Computer Science, 2024, p. 1–6, doi: 10.1109/SCEECS61402.2024.10482143. [18] D. Paul, R. West, A. Bosselut, and B. Faltings, “Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning,” in Findings of EMNLP, 2024. [19] J. Li et al., “Towards faithful chain-of-thought: Large language models are bridging reasoners,” arXiv:2405.18915, 2024. [20] C. Zheng et al., “ProcessBench: Identifying process errors in mathematical reasoning,” arXiv:2412.06559, 2024. [21] Y. Chen et al., “Reasoning models don’t always say what they think,” arXiv:2505.05410, 2025. [22] Y. Xu et al., “Chain-of-thought reasoning in the wild is not always faithful,” arXiv:2503.08679, 2025. [23] Z. Shao et al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” arXiv:2402.03300, 2024. [24] D. Guo et al., “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” arXiv:2501.12948, 2025.