Paper deep dive
PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs
Linghao Meng, Feng He, Xuan Yang, Junyuan Mao, Pinze Ren, Deqing Mu, Hesen Yang, Qiankun Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/12/2026, 5:54:29 AM
Summary
The paper introduces PHRBench, a benchmark for evaluating post-hallucination reasoning (PHR) in Large Language Models (LLMs). It analyzes how 18 models handle hallucinated premises across four domains, characterizing resolution behaviors as compliance, avoidance, or heuristic correction. Key findings include a scaling tension where larger models suffer greater accuracy drops from hallucinations but exhibit more corrective behavior, and that successful recovery is rare, associated with frequent belief updates, and predictable via prompt features.
Entities (16)
Relation Signals (9)
PHRBench → evaluates → Post-Hallucination Reasoning
confidence 95% · In this work, we introduce PHRBench, a controlled benchmark for behaviorally structured PHR across four domains and 18 large language models.
PHRBench → characterizesbehavior → Hallucination Compliance
confidence 92% · PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction
PHRBench → characterizesbehavior → Hallucination Avoidance
confidence 92% · PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction
PHRBench → characterizesbehavior → Heuristic Correction
confidence 92% · PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction
Successful Recovery → associatedwith → Belief Update Frequency
confidence 90% · successful recovery remains relatively rare and is associated with more frequent belief updates along the reasoning trajectory.
Model Scaling → causes → Accuracy Degradation
confidence 90% · degradation consistently increases with model scale. For example, scaling Qwen2.5 from 1.5B to 72B improves baseline accuracy by 27.6%↑, while the degradation under hallucinated context doubles to 8.0%.
Model Scaling → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post-hallucination reasoning (PHR) mainly characterize changes in final outcomes and aggregate reasoning dynamics, leaving how models resolve hallucinated premises at the response level insufficiently understood. In this work, we introduce PHRBench, a controlled benchmark for behaviorally structured PHR across four domains and 18 large language models. PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction, and defines an insightful trajectory as successful correction that ultimately reaches the correct answer. Across 4820 controlled instances, we find that successful recovery remains relatively rare and is associated with more frequent belief updates along the reasoning trajectory. We further find that properties of the hallucinated prompt contain substantial predictive signal for successful recovery, with a lightweight predictor achieving an AUROC of 0.847. These findings provide a behavioral view of post-hallucination reasoning, characterizing how LLMs resolve erroneous context and when successful recovery is likely to occur.
Tags
Links
- Source: https://arxiv.org/abs/2610.10455v2
- Canonical: https://arxiv.org/abs/2610.10455v2
Trouble viewing inline? Open PDF directly →
Full Text
136,275 characters extracted from source content.
Expand or collapse full text
PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs Linghao Meng1,∗ Feng He2,∗ Xuan Yang1,∗ Junyuan Mao1 Pinze Ren3 Deqing Mu4 Hesen Yang1 Qiankun Li5,† 1National University of Singapore 2Independent Researcher 3Tsinghua University 4Johns Hopkins University 5Nanyang Technological University menglinghao25,xuanyang@u.nus.edu hefengcs@gmail.com cs-qiankun.li@ntu.edu.sg *Equal contribution. †Corresponding author. Abstract Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post-hallucination reasoning (PHR) mainly characterize changes in final outcomes and aggregate reasoning dynamics, leaving how models resolve hallucinated premises at the response level insufficiently understood. In this work, we introduce PHRBench, a controlled benchmark for behaviorally structured PHR across four domains and 1818 large language models. PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction, and defines an insightful trajectory as successful correction that ultimately reaches the correct answer. Across 48204820 controlled instances, we find that successful recovery remains relatively rare and is associated with more frequent belief updates along the reasoning trajectory. We further find that properties of the hallucinated prompt contain substantial predictive signal for successful recovery, with a lightweight predictor achieving an AUROC of 0.8470.847. These findings provide a behavioral view of post-hallucination reasoning, characterizing how LLMs resolve erroneous context and when successful recovery is likely to occur. Code: github.com/menglinghao2025/PHR Dataset: huggingface.co/datasets/menglinghao2025/PHR Homepage: menglinghao2025.github.io/PHR 1 Introduction Despite substantial progress in detection and mitigation methods, large language models continue to produce hallucinations across diverse tasks and settings (Zhang et al., 2025; Farquhar et al., 2024; Li, 2025; Wu et al., 2023; Kalai et al., 2025; Janiak et al., 2025; Tamber et al., 2025; Ganesh et al., 2026). Existing safeguards can reduce such errors, but cannot reliably prevent every hallucinated statement from entering subsequent processing (Xu et al., 2024; Kalai et al., 2025; Janiak et al., 2025). This limitation becomes increasingly consequential as LLM outputs are reused across retrieval systems, tool-augmented pipelines, autonomous agents, and multi-stage generation workflows (Niu et al., 2024; Yao et al., 2022; Wu et al., 2023). In these systems, an output produced at one stage often becomes the context for subsequent reasoning. As illustrated in Figure 1(a), hallucinated information that escapes upstream detection can therefore propagate through the system and act as a premise for downstream decisions (Zhang et al., 2023; Zhong et al., 2024; Jamshidi, 2026; Cui et al., 2026; Hu et al., 2025). System reliability consequently depends not only on preventing hallucinations, but also on whether downstream models can recover once hallucinated information is already present. HIVE (He et al., 2026) introduces post-hallucination reasoning (PHR) to study this downstream reliability problem and shows that hallucination can persistently reshape subsequent reasoning. HIVE (He et al., 2026) further reveals that hallucinated context can either impair or improve downstream performance, indicating that its effects cannot be explained by factuality alone. However, existing PHR analysis mainly characterizes changes in final outcomes and aggregate reasoning dynamics, without systematically distinguishing response-level behaviors toward hallucinated premises (Ganesh et al., 2026; Lee et al., 2024a; Grover et al., 2024; Zhan et al., 2026; Lu et al., 2026b). As previewed in Figure 1(b), the same correct answer may arise from fundamentally different behaviors. A model may accept the hallucinated semantics yet still arrive at the correct answer, bypass the premise without resolving it, or explicitly identify and correct the underlying error. Conversely, a model may identify and correct a hallucinated premise but still fail to derive the correct answer. Final correctness therefore conflates compliance, avoidance, successful correction, and correction failure. Aggregate trajectory statistics provide additional evidence about how reasoning changes, but they still do not identify the resolution behavior taken by an individual response. Without distinguishing resolution behavior from final outcome, a correct answer may be mistaken for successful hallucination resolution. In this paper, we study PHR as a behaviorally structured resolution process through PHRBench, a controlled benchmark of 48204820 instances spanning Chemistry, Biomedicine, Physics, and Code Generation. We evaluate 1818 proprietary and open-source models and characterize each reasoning trajectory independently of final-answer correctness as Hallucination Compliance, Hallucination Avoidance, or Heuristic Correction. We further define an insightful trajectory as one that performs Heuristic Correction and reaches the correct answer. Using this framework, we ask two questions. Q1: How do LLMs resolve hallucinated premises, and how does successful recovery vary with model scale and reasoning dynamics? We analyze behavioral resolution and trajectory dynamics across models and scales. Q2: Can successful recovery be predicted before response generation? We evaluate the predictive signal of prompt-level structural and semantic features. Our results, summarized in Figure 1(c), reveal three main findings. First, scaling creates a tension between contextual vulnerability and corrective capability: hallucinated context reduces accuracy by 7.77.7 percentage points on average, with larger degradation at greater scales within multiple model families, while corrective behavior becomes more frequent. Second, successful recovery remains rare and exhibits a distinct reasoning signature. Insightful trajectories are associated with more frequent belief updates, suggesting more effective revision after conflicts are detected. Third, successful recovery contains substantial ex ante predictive signal. Prompt-level features achieve an AUROC of 0.8470.847, with both structural and semantic factors contributing to prediction. Figure 1: Overview of behaviorally structured post-hallucination reasoning. (a) Hallucinations that escape upstream detection can propagate into downstream reasoning, making recovery critical for reliable LLM systems. (b) Final correctness alone cannot reveal how a model handles hallucinated context. (c) PHRBench reveals a scaling tension between vulnerability and correction, characterizes and predicts successful correction before generation. Our key contributions are as follows: ❶We introduce PHRBench for behaviorally structured post-hallucination reasoning. Across 44 domains and 1818 LLMs, PHRBench contains 48204820 controlled instances and separates behavioral resolution from final-answer correctness through compliance, avoidance, and correction. ❷We reveal a scaling tension and characterize successful recovery. Model scaling reveals a tension between robustness to hallucinated context and corrective capability, while successful recovery exhibits a distinct pattern of reasoning revision. ❸We show that successful recovery contains substantial ex ante predictive signal. Using lightweight structural and semantic features, we show that successful recovery can be anticipated before generation, providing a foundation for intervention mechanisms. 2 Related Work Post-Hallucination Reasoning Transmission. Prior work on post-hallucination reasoning has examined how hallucinated information is transmitted across subsequent generations and interactions (Chen et al., 2024; Gan et al., 2025; Xiong et al., 2026; Zhang et al., 2023; Zhang et al., 2026; Jiang et al., 2026). MMHalSnowball (Zhong et al., 2024) shows that earlier hallucinations can mislead later responses even when visual evidence remains available. M-Snowball (Jiang et al., 2026) examines how hallucinated information accumulates as models increasingly rely on contaminated conversational history. The Woozle Effect (Zhang et al., 2026) investigates how hallucinated information is transmitted across agents and influences participants. Hallucination Cascade (Jamshidi et al., 2026) and Collective Hallucination (Jamshidi, 2026) further analyze how hallucinated content evolves through sequential agent interactions. Together, these studies trace where hallucinated information propagates across downstream multi reasoning processes. Post-Hallucination Reasoning Dynamics. Beyond transmission, another line of work examines how hallucinated outputs are processed in subsequent reasoning (Lu et al., 2026b; Lu et al., 2026a; Dhuliawala et al., 2024; Tyen et al., 2024). Volcano (Lee et al., 2024b) uses self-feedback to revise initial hallucinated responses, while Pelican (Sahu et al., 2024) decomposes generated claims and verifies them through program-of-thought reasoning. A2R (Lee et al., 2024a) uses explicit hallucination assessment to guide iterative refinement. DoT (Grover et al., 2024) navigates hallucinated thoughts to improve subsequent reasoning. SHD (Lu et al., 2026b) models hallucination in long CoT as an evolving state across reasoning steps, while DeepHalluBench (Zhan et al., 2026) audits intermediate hallucinations throughout full research trajectories rather than only on final outputs. These studies characterize PHR through verification, revision, and trajectory-level analysis, but do not distinguish the response-level behaviors through which a model resolves hallucinated content. Behavioral Resolution in Post-Hallucination Reasoning. HIVE formalizes the broader setting of post-hallucination reasoning, showing that hallucinated semantics can alter downstream outcomes and reasoning dynamics (He et al., 2026). However, these analyses do not reveal how an individual response resolves the hallucinated content it encounters. Simultaneously, different resolution behaviors can lead to the same final outcome. Our work therefore studies PHR as a behaviorally structured resolution process, separating response-level resolution behavior from final-answer correctness. 3 PHRBench In this section, we introduce PHRBench, a benchmark designed to systematically evaluate post-hallucination reasoning in LLMs. PHRBench is constructed by first collecting diverse reasoning questions across multiple domains and task formats, and then augmenting them with controlled truthful and hallucinated semantic contexts. This construction enables us to examine not only whether hallucination changes the final answer, but also how it alters the reasoning trajectory. Overall, PHRBench has three key characteristics: ✤ Diverse Reasoning Scenarios. PHRBench ensures diversity along two dimensions. (1) Multiple Domains: PHRBench contains 460460 base questions spanning Chemistry, Biomedicine, Physics, and Code Generation, covering heterogeneous domain knowledge. (2) Multiple Tasks: PHRBench includes both multiple-choice and open-ended questions, allowing us to examine PHR under both constrained answer spaces and free-form generation settings. The questions are curated from established benchmarks with varying levels of task complexity, providing broad reasoning scenarios. ✤ Controlled Semantic Augmentation. PHRBench constructs controlled semantic contexts for each base question to isolate the effect of hallucination on reasoning. (1) Truthful and Hallucinated Contexts: each question is paired with one truthful and multiple hallucinated augmentations, all preserving task relevance and avoiding answer leakage for controlled comparison. (2) Multiple Augmentation Patterns: hallucinated augmentations cover three forms of semantic corruption: Rule Contradiction, State Distortion, and Pseudoscientific Entanglement, corresponding to domain principle violations, misreported task-relevant conditions, and outdated scientific theories. ✤ Reasoning-Oriented Evaluation. PHRBench evaluates PHR from both outcome and reasoning-process perspectives. For outcome performance, we use Accuracy (Acc) to quantify whether hallucinated context changes the final answer. For reasoning trajectories, we introduce four diagnostic metrics: Output Length (OL), Uncertainty Index (UI), Belief Update Frequency (BUF), and Branching Complexity (BC). This analytical framework enables a fine-grained diagnosis of how hallucinated semantics affect both the final answer and the reasoning process leading to it. Detailed definitions of these metrics are provided in Section 4.2. Data Collection and Statistics. Selecting tasks that meaningfully expose PHR is crucial for constructing PHRBench. We therefore focus on scientific reasoning tasks that require models to combine domain knowledge, task-specific conditions, and intermediate inference to derive verifiable outcomes. We consider four domains: Chemistry, Biomedicine, Physics, and Code Generation. The first three emphasize reasoning over scientific knowledge and established principles, while Code Generation extends this setting to computational reasoning. Following this design, we collect 460460 base questions from eight established benchmarks across the four domains. The scientific questions are drawn from MoleculeNet-BBBP (Wu et al., 2018), ChemBench (Mirza et al., 2025), MedMCQA (Pal et al., 2022), MMLU (Hendrycks et al., 2020), SciQ (Welbl et al., 2017), and GPQA (Rein et al., 2023), while coding problems are sourced from HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), covering both multiple-choice and open-ended reasoning formats. Figure 2 illustrates the distribution of reasoning subtypes and the domain-wise composition of the hallucinated augmentations across three construction types. Detailed source statistics and sampling procedures are provided in Appendix A.1. Figure 2: Dataset statistics of PhrBench. (a) Reasoning category distribution of the 460460 base questions. (b) Domain-wise composition of hallucinated augmentations across 3 construction types. Augmentation Construction. Before augmentation, we standardize the collected questions using task-specific templates that preserve the original task semantics and evaluation criteria. For each standardized base question, we construct one truthful augmentation and multiple hallucinated augmentations without revealing the answer. The truthful version remains factually valid, while each hallucinated version introduces a single targeted semantic mutation with the remaining context preserved. The resulting PHRBench contains 460460 base questions, 460460 truthful augmentations, and 39003900 hallucinated augmentations, yielding 48204820 instances in total. Details of templates, quality control, and representative examples are provided in Appendix A.2 and Appendix A.3. 4 Experimental Setup 4.1 Models We evaluate 1818 representative LLMs on PHRBench, covering proprietary and open-source models across different parameter scales, model generations, and training paradigms (Anthropic, 2025; OpenAI, 2024a; OpenAI, 2024b; OpenAI, 2025; Comanici et al., 2025; Google DeepMind, 2024; Yang et al., 2024; Yang et al., 2025; Meta, 2024; Grattafiori et al., 2024; Glm et al., 2024; Jiang et al., 2023). To ensure consistency in our experimental setup, we set the temperature to 0.70.7 and the maximum output size to the maximum supported size for each model. Additionally, for each model and prompt, we independently sample 1010 responses to account for generation stochasticity. See Appendix B.1 for all configurations. 4.2 Evaluation Metrics PHRBench evaluates post-hallucination reasoning using two groups of diagnostic metrics. Outcome Metric characterizes final-answer performance, while Reasoning Trajectory Metrics capture changes in the reasoning process. Outcome Metric. PHRBench uses Accuracy (Acc) to measure the proportion of responses whose final answers match the ground truth. Reasoning Trajectory Metrics. PHRBench uses four metrics to characterize reasoning trajectories: Output Length, Uncertainty Index, Belief Update Frequency, and Branching Complexity. Output Length (OL) measures the total number of tokens generated in a response. Uncertainty Index (UI) measures the average token-level uncertainty along the reasoning trajectory: =−1N∑i=1N∑v∈VP(xv∣x<i)logP(xv∣x<i),UI=- 1N _i=1^N _v∈ VP(x_v x_<i) P(x_v x_<i), (1) where N denotes the generated sequence length, V is the model vocabulary, and P(xv∣x<i)P(x_v x_<i) denotes the probability assigned to token v given the preceding context x<ix_<i. Belief Update Frequency (BUF) measures how frequently next-token distributions shift substantially between consecutive reasoning steps: =1K−1∑k=2K(DKL(Pk∥Pk−1)>τ),BUF= 1K-1 _k=2^KI (D_KL (P_k P_k-1 )>τ ), (2) where K denotes the number of reasoning steps, PkP_k and Pk−1P_k-1 denote consecutive belief distributions, DKLD_KL is the Kullback–Leibler divergence, and τ is the update threshold. Branching Complexity (BC) measures the prevalence of alternative reasoning paths using predefined branching-related lexical cues, normalized by response length: =100N∑k∈Count(k,r),BC= 100N _k Count(k,r), (3) where N is its total word count, K denotes the predefined cue set, and Count(k,r)Count(k,r) is the number of occurrences of cue k in generated response r. Full details are provided in Appendix B.2. 5 Results and Analysis 5.1 Main Results Table 1: Outcome performance and reasoning trajectory statistics under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 57.557.5 48.4↓9.148.4_ [rgb]1,0,0 9.1 100.7100.7 113.2113.2 0.0540.054 0.0710.071 0.1520.152 0.3290.329 1.1331.133 1.1111.111 GPT-4o-mini 55.555.5 48.3↓7.248.3_ [rgb]1,0,0 7.2 71.271.2 72.472.4 0.0720.072 0.0590.059 0.1300.130 0.1860.186 0.7520.752 0.7200.720 GPT-4o 51.851.8 42.3↓9.542.3_ [rgb]1,0,0 9.5 60.060.0 60.860.8 0.0610.061 0.0530.053 0.1130.113 0.2140.214 0.6590.659 0.6180.618 GPT-5.2 61.661.6 44.7↓16.944.7_ [rgb]1,0,0 16.9 43.043.0 45.145.1 0.0070.007 0.0160.016 0.0130.013 0.0260.026 0.5780.578 0.5150.515 Gemini-2.0-Flash 56.256.2 45.1↓11.145.1_ [rgb]1,0,0 11.1 76.276.2 86.586.5 0.0460.046 0.0610.061 0.0920.092 0.1780.178 0.7450.745 0.7850.785 Gemini-2.5-Pro 58.558.5 46.4↓12.146.4_ [rgb]1,0,0 12.1 82.482.4 95.395.3 0.0450.045 0.0680.068 0.1240.124 0.2440.244 0.8520.852 0.8500.850 Open-source Models Qwen2.5-1.5B-Instruct 24.624.6 20.6↓4.020.6_ [rgb]1,0,0 4.0 115.6115.6 117.1117.1 0.2720.272 0.2180.218 0.0910.091 0.0760.076 1.9871.987 1.7481.748 Qwen2.5-7B-Instruct 44.544.5 38.0↓6.538.0_ [rgb]1,0,0 6.5 153.2153.2 147.1147.1 0.3190.319 0.1510.151 0.1480.148 0.1210.121 2.2652.265 1.9391.939 Qwen2.5-32B-Instruct 46.846.8 39.6↓7.239.6_ [rgb]1,0,0 7.2 228.4228.4 215.1215.1 0.2450.245 0.1500.150 0.2850.285 0.3530.353 2.8202.820 2.6442.644 Qwen2.5-72B-Instruct 52.252.2 44.2↓8.044.2_ [rgb]1,0,0 8.0 276.9276.9 262.8262.8 0.1880.188 0.1120.112 0.4140.414 0.4850.485 3.4263.426 3.1503.150 Qwen3-8B 46.046.0 43.8↓2.243.8_ [rgb]1,0,0 2.2 225.8225.8 256.6256.6 0.2170.217 0.0720.072 0.5760.576 0.3920.392 2.8552.855 3.4703.470 Qwen3-32B 48.548.5 41.0↓7.541.0_ [rgb]1,0,0 7.5 296.3296.3 288.4288.4 0.1810.181 0.0650.065 0.6050.605 0.4650.465 3.2323.232 3.8203.820 Qwen3-235B-A22B 58.258.2 47.4↓10.847.4_ [rgb]1,0,0 10.8 365.0365.0 415.2415.2 0.0850.085 0.0450.045 0.8430.843 0.7530.753 4.1554.155 4.8284.828 Llama-3.1-8B-Instruct 34.234.2 31.9↓2.331.9_ [rgb]1,0,0 2.3 210.5210.5 199.9199.9 0.3780.378 0.3220.322 0.3710.371 0.2070.207 2.7522.752 2.2222.222 Llama-3.1-70B-Instruct 48.448.4 40.9↓7.540.9_ [rgb]1,0,0 7.5 285.2285.2 275.4275.4 0.4310.431 0.4190.419 0.5230.523 0.3840.384 3.3563.356 2.9892.989 Llama-3.2-3B-Instruct 31.231.2 29.5↓1.729.5_ [rgb]1,0,0 1.7 167.0167.0 153.4153.4 0.1590.159 0.1370.137 0.3220.322 0.1240.124 2.0762.076 1.4411.441 GLM-4-9B-Chat 32.332.3 22.4↓9.922.4_ [rgb]1,0,0 9.9 127.3127.3 79.079.0 0.1710.171 0.1730.173 0.0500.050 0.0380.038 1.8861.886 1.1221.122 Mistral-7B-Instruct 21.121.1 15.7↓5.415.7_ [rgb]1,0,0 5.4 131.0131.0 122.4122.4 0.1300.130 0.1040.104 0.2620.262 0.3700.370 1.8531.853 1.4721.472 Figure 3: Left. Accuracy Retention across all models and types. Impact of task domain (Middle) and hallucinated augmentation type (Right) on Accuracy Retention, separated by model type. Evaluation Protocol. Following the evaluation protocol described in Section 4, we evaluate all 18 models under both the truthful augmentation and hallucinated augmentation settings, denoted as T and H in Table 1. Comprehensive results are provided in Appendix C. ❶Hallucinated context increasingly degrades stronger models. Hallucinated context consistently reduces final-answer accuracy across all models, with the degradation becoming more pronounced for stronger and larger models. As shown in Table 1, the average accuracy drop is 7.7%7.7\%, with GPT-5.2 showing the largest decrease (16.9%↓16.9\% ), followed by Gemini-2.5-Pro (12.1%↓12.1\% ) and Gemini-2.0-Flash (11.1%↓11.1\% ). Proprietary models also exhibit larger degradation on average than open-source models (11.0%11.0\% vs. 6.1%6.1\%). More importantly, degradation consistently increases with model scale. For example, scaling Qwen2.5 from 1.51.5B to 7272B improves baseline accuracy by 27.6%↑27.6\% , while the degradation under hallucinated context doubles to 8.0%8.0\%. Figure 3 further reveals substantial heterogeneity across tasks and augmentation types. Chemistry shows the lowest average retention across the models, while Physics is comparatively more robust. Across augmentation types, State Distortion causes the strongest degradation, whereas Pseudoscientific Entanglement retains approximately 98%98\% of baseline accuracy. Thus, susceptibility to hallucinated context also depends on how the hallucinated information conflicts with the task. ❷Scaling amplifies reasoning activity under both augmentations. Larger models consistently exhibit larger OL, BUF, and BC. As shown in Table 1, this scaling pattern holds across the Qwen2.5, Qwen3, and Llama-3.1 families under both settings. For example, scaling Qwen2.5 from 1.51.5B to 7272B yields substantial increases in OL of 161.3161.3 under T and 145.7145.7 under H. BUF rises by 0.3230.323 and 0.4090.409, while BC grows by 1.4391.439 and 1.4021.402, respectively. Across models, OL, BUF, and BC also strongly co-vary, with Pearson correlations of 0.9080.908, 0.9770.977, and 0.8560.856, whereas UI shows no consistent scaling pattern. These results indicate that scaling primarily amplifies the extent and complexity of reasoning, while uncertainty follows a comparatively distinct pattern. Takeaway 1. Greater reasoning capacity does not guarantee greater robustness. Instead, stronger models can exhibit richer reasoning dynamics while becoming more vulnerable to hallucinated context, revealing a strong-model trap. 5.2 How Do Models Resolve Hallucinated Premises? To understand how models resolve hallucinated premises, we analyze their responses from three complementary perspectives. First, we identify models’ behavioral resolution and insightful trajectories and then characterize their patterns. Second, we examine the reasoning dynamics associated with different resolution behaviors and insightful trajectories. Finally, we complement these aggregate analyses with qualitative cases to illustrate how behavioral resolution unfolds during reasoning. Resolution Definitions. We categorize each reasoning trajectory into three behavioral modes according to how the model handles the hallucinated premise: Hallucination Compliance, where the model accepts the premise; Hallucination Avoidance, where the model bypasses the premise; and Heuristic Correction, where the model identifies and corrects the premise. This categorization is independent of final-answer correctness. To further capture successful post-hallucination reasoning, we define an insightful trajectory as one that performs heuristic correction and ultimately reaches the correct answer. Detailed definitions and examples are provided in Appendix D.1. ❸Active correction remains rare, but strengthens with model scale. Most models predominantly comply with or avoid hallucinated premises rather than explicitly correcting them. As shown in Table 2, heuristic correction remains relatively infrequent across the 18 models, but increases consistently with model scale. For example, scaling Qwen2.5 from 1.51.5B to 7272B increases heuristic correction by 18.90%↑18.90\% , with similar scaling trends observed for Qwen3 and Llama-3.1. According to Figure 4, insightful trajectories follow the same overall trend but remain less frequent: 1111 of 1818 models produce them in fewer than 10%10\% of responses, while Qwen3-235B-A22B achieves the highest rate at 24.91%24.91\%. The persistent gap between heuristic correction and insightful trajectories indicates that revising a hallucinated semantic does not necessarily lead to a correct final answer. Table 2: Behavioral resolution proportions across evaluated models. Model Comp. (%) Avoid. (%) HC (%) Proprietary Models Claude-Sonnet-4 47.2347.23 33.4933.49 19.2819.28 GPT-4o-mini 32.6232.62 60.5460.54 6.856.85 GPT-4o 45.8745.87 44.0844.08 10.0510.05 GPT-5.2 41.1041.10 55.2155.21 3.693.69 Gemini-2.0-Flash 43.4343.43 49.0949.09 7.487.48 Gemini-2.5-Pro 45.5945.59 42.5542.55 11.8611.86 Open-source Models GLM-4-9B-Chat 19.5719.57 78.4578.45 1.981.98 Mistral-7B-Instruct 47.6647.66 39.3839.38 12.9712.97 Model Comp. (%) Avoid. (%) HC (%) Open-source Models Qwen2.5-1.5B-Instruct 32.1432.14 60.2860.28 7.587.58 Qwen2.5-7B-Instruct 59.5559.55 28.0428.04 12.4112.41 Qwen2.5-32B-Instruct 43.7243.72 36.0836.08 20.2020.20 Qwen2.5-72B-Instruct 39.3439.34 34.1834.18 26.4826.48 Qwen3-8B-Instruct 47.1247.12 27.3227.32 25.5625.56 Qwen3-32B-Instruct 38.7438.74 30.8630.86 30.4030.40 Qwen3-235B-A22B-Instruct 33.1833.18 27.7427.74 39.0839.08 Llama-3.2-3B-Instruct 25.0025.00 68.8868.88 6.126.12 Llama-3.1-8B-Instruct 40.6840.68 46.3946.39 12.9312.93 Llama-3.1-70B-Instruct 39.1539.15 41.6841.68 19.1719.17 Figure 4: Insightful trajectory rates across the evaluated models. ❹Successful resolution is associated with higher belief updating. Different resolution behaviors exhibit distinct reasoning patterns of distributional change under the reference scorer. As shown in Figure 5, heuristic correction produces, on average, about 3030 more reasoning words and a 0.310.31 higher BUF than hallucination compliance across the four representative models. Hallucination avoidance instead exhibits the highest average branching complexity. More importantly, insightful trajectories achieve an average BUF of 0.630.63, compared with only 0.180.18 for non-insightful trajectories. These results suggest that successful resolution is associated with more frequent distributional shifts and longer reasoning trajectories. Full details are provided in Appendix D.2. Figure 5: Reasoning dynamics across behavioral resolution modes and insightful trajectories for four representative models. ❺Insightful trajectories reject conflicting premises and sustain the correction. Qualitative cases show that relevant knowledge alone is insufficient for resolving hallucinated premises. As shown in Figure 6, non-insightful trajectories may access the correct knowledge but ultimately allow the injected premise to override it. Insightful trajectories instead reject the conflicting premise and redirect subsequent reasoning toward the correct answer. This contrast supports effective belief updating as the key distinction of successful resolution. More cases are provided in Appendix D.3. Figure 6: Qualitative comparison of insightful and non-insightful trajectories. Takeaway 2. Scaling increases models’ tendency to correct hallucinated premises, but correction alone is insufficient. Insightful resolution depends on whether detected conflicts trigger effective belief revision that redirects subsequent reasoning. 5.3 Can We Predict Insightful Trajectories? Section 5.2 shows that insightful trajectories are relatively rare but exhibit distinctive reasoning dynamics. In this section, we further examine whether their occurrence can be anticipated from properties of the input prompt before generation. Specifically, we characterize prompts using structural and semantic features, evaluate their predictive power, and analyze the features and interactions that contribute to the prediction. Full details are provided in Appendix E. Prompt-Level Features. We construct 24 features to characterize factors that may trigger an insightful trajectory, covering both prompt structure and hallucination–task semantics. Representative features include Total Prompt Length, Candidate Count, Domain-Rule Violation, Error Locality, and Applicability Mismatch. Full definitions are provided in Appendix E.1. ❻Insightful trajectories are predictable from prompt-level properties. Prompt characteristics contain substantial predictive signal for whether an insightful trajectory will emerge. We extract the 24 features for each hallucinated prompt using an external LLM (Zheng et al., 2023; Li et al., 2025) and train an XGBRegressor (Chen and Guestrin, 2016) to estimate its propensity to induce insightful reasoning. The resulting predictor achieves an AUROC of 0.8470.847 and an accuracy of 0.8630.863, indicating that the occurrence of insightful reasoning is systematically associated with properties available before generation rather than being entirely incidental. ❼Structural and semantic features provide complementary predictive signals. The predictive signal is distributed across both structural and semantic properties of the prompt. As shown in Figure 7(a), Total Prompt Length is the strongest individual predictor, accounting for 20.7%20.7\% of the global SHAP contribution (Lundberg and Lee, 2017; Lundberg et al., 2020), substantially exceeding other structural features. Hallucinated augmentation-specific semantics also contribute substantially, with Domain-Rule Violation and Error Locality accounting for 7.4%7.4\% and 7.2%7.2\%, respectively. This predictive structure is not dominated by strong pairwise interactions. As shown in Figure 7(b), the largest interaction occurs between Total Prompt Length and Question Length (0.0200.020), while nearly all remaining pairwise interactions are below 0.010.01. Together with their non-trivial individual contributions, these weak interactions suggest that the selected features decouple and capture largely complementary aspects of the conditions associated with insightful reasoning. Takeaway 3. Insightful reasoning is predictable before generation. Its emergence reflects complementary structural and hallucination–task semantic properties of the prompt as dominant factors. (a) Global SHAP contribution and directional effects of the most influential prompt-level features. (b) Pairwise SHAP interaction effects among the most influential prompt-level features. Figure 7: SHAP analysis of prompt-level features for predicting insightful trajectories. 6 Conclusion In this paper, we introduce PHRBench, a comprehensive benchmark for behaviorally structured post-hallucination reasoning across four domains and 1818 LLMs. With the evaluation results, our analysis shows that hallucinated context consistently degrades performance, while larger models simultaneously exhibit stronger corrective behavior, revealing a tension between vulnerability and recovery capability. By separating hallucination compliance, hallucination avoidance, and heuristic correction from final outcomes, we further show that successful correction remains relatively rare and is characterized by stronger belief updating. We also find that successful correction can be anticipated from properties of the hallucinated semantics before generation. A lightweight predictor based on structural and semantic features achieves strong predictive performance, with factors such as domain-rule violation and error locality contributing substantially. Together, these results suggest that reliable PHR requires evaluating not only whether a model reaches the correct answer, but also how it resolves hallucinated context and when successful recovery is likely to occur. Acknowledgments We sincerely thank Zhenzhen She for her invaluable discussion and insights on figure visualization and reasoning behavioral definition. References Anthropic (2025) Anthropic Introducing claude 4. Note: https://w.anthropic.com/news/claude-4 Cited by: §4.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §3. Chen et al. (2024) K. Chen, Q. Chen, J. Zhou, H. Yishen, and L. He Diahalu: a dialogue-level hallucination evaluation benchmark for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 9057–9079. Cited by: §2. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §3. Chen and Guestrin (2016) T. Chen and C. Guestrin Xgboost: a scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, p. 785–794. Cited by: §5.3. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §4.1. Cui et al. (2026) Y. Cui, H. Fu, H. Zhang, L. Wang, and C. Zuo Free-mad: consensus-free multi-agent debate. In Findings of the Association for Computational Linguistics: ACL 2026, p. 31977–31997. Cited by: §1. Dhuliawala et al. (2024) S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston Chain-of-verification reduces hallucination in large language models. In Findings of the association for computational linguistics: ACL 2024, p. 3563–3578. Cited by: §2. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: §1. Gan et al. (2025) Z. Gan, Y. Liao, and Y. Liu Rethinking external slow-thinking: from snowball errors to probability of correct reasoning. arXiv preprint arXiv:2501.15602. Cited by: §2. Ganesh et al. (2026) P. Ganesh, R. Shokri, and G. Farnadi Rethinking hallucinations: correctness, consistency, and prompt multiplicity. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6959–6978. Cited by: §1, §1. Glm et al. (2024) T. Glm, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. Chatglm: a family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. Cited by: §4.1. Google DeepMind (2024) Google DeepMind Introducing gemini 2.0: our new ai model for the agentic era. Note: Google Blog External Links: Link Cited by: §4.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.1. Grover et al. (2024) S. Grover, V. Vineet, and Y. S. Rawat Navigating hallucinations for reasoning of unintentional activities. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 9666–9680. Cited by: §1, §2. He et al. (2026) F. He, Z. Wang, Q. Wang, Q. Guan, D. Liu, R. Tang, and Q. Li HIVE: understanding post-hallucination reasoning in vision language models. ECCV. Cited by: §D.1, §1, §1, §2. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: §3. Hu et al. (2025) W. Hu, W. Zhang, Y. Jiang, C. J. Zhang, X. Wei, and Q. Li Removal of hallucination on hallucination: debate-augmented rag. In Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: long papers), p. 15839–15853. Cited by: §1. Jamshidi et al. (2026) S. Jamshidi, A. M. Dakhel, K. W. Nafi, and F. Khomh Hallucination cascade: analyzing error propagation in multi-agent llm systems. arXiv preprint arXiv:2606.07937. Cited by: §2. Jamshidi (2026) S. Jamshidi Collective hallucination in multi-agent llms: modeling and defense. arXiv preprint arXiv:2606.07941. Cited by: §1, §2. Janiak et al. (2025) D. Janiak, J. Binkowski, A. Sawczyn, B. Gabrys, R. Shwartz-Ziv, and T. J. Kajdanowicz The illusion of progress: re-evaluating hallucination detection in llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 34716–34733. Cited by: §1. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. External Links: Link Cited by: §4.1. Jiang et al. (2026) Y. Jiang, X. Jiang, L. Zhang, Z. Wang, Y. Lu, P. Wang, B. Han, F. Zheng, and D. Yang M-snowball: evaluating and mitigating hallucination snowballing in multimodal multi-turn dialogue. arXiv preprint arXiv:2606.00622. Cited by: §2. Kalai et al. (2025) A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang Why language models hallucinate. arXiv preprint arXiv:2509.04664. Cited by: §1. Lee et al. (2024a) D. Lee, E. Park, H. Lee, and H. Lim Ask, assess, and refine: rectifying factual consistency and hallucination in llms with metric-guided feedback learning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2422–2433. Cited by: §1, §2. Lee et al. (2024b) S. Lee, S. H. Park, Y. Jo, and M. Seo Volcano: mitigating multimodal hallucination through self-feedback guided revision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 391–404. Cited by: §2. Li et al. (2025) D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, et al. From generation to judgment: opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 2757–2791. Cited by: §5.3. Li (2025) X. Li A review of prominent paradigms for llm-based agents: tool use, planning (including rag), and feedback learning. In Proceedings of the 31st international conference on computational linguistics, p. 9760–9779. Cited by: §1. Lu et al. (2026a) H. Lu, Y. Liu, J. Xu, G. Nan, Y. Yu, Z. Chen, and K. Wang Auditing meta-cognitive hallucinations in reasoning large language models. Advances in Neural Information Processing Systems 38, p. 162500–162543. Cited by: §2. Lu et al. (2026b) H. Lu, M. Pan, G. Nan, J. Zhuang, Z. Zhao, Z. Sun, K. Wang, Y. Liu, et al. Streaming hallucination detection in long chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, p. 21157–21183. Cited by: §1, §2. Lundberg et al. (2020) S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S. Lee From local explanations to global understanding with explainable ai for trees. Nature machine intelligence 2 (1), p. 56–67. Cited by: §5.3. Lundberg and Lee (2017) S. M. Lundberg and S. Lee A unified approach to interpreting model predictions. Advances in neural information processing systems 30. Cited by: §5.3. Meta (2024) Meta Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. Note: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Cited by: §4.1. Mirza et al. (2025) A. Mirza, N. Alampara, S. Kunchapu, M. Ríos-García, B. Emoekabu, A. Krishnan, T. Gupta, M. Schilling-Wilhelmi, M. Okereke, A. Aneesh, et al. A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry 17 (7), p. 1027–1034. Cited by: §3. Niu et al. (2024) C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, R. Zhong, J. Song, and T. Zhang Ragtruth: a hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 10862–10878. Cited by: §1. OpenAI (2024a) OpenAI GPT-4o mini: advancing cost-efficient intelligence. Note: https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Cited by: §4.1. OpenAI (2024b) OpenAI GPT-4o System Card. arXiv preprint arXiv:2410.21276. External Links: Link Cited by: §4.1. OpenAI (2025) OpenAI Introducing GPT-5.2. Note: https://openai.com/index/introducing-gpt-5-2/ Cited by: §4.1. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, p. 248–260. Cited by: §3. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman Gpqa: a graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022. Cited by: §3. Sahu et al. (2024) P. Sahu, K. Sikka, and A. Divakaran Pelican: correcting hallucination in vision-llms via claim decomposition and program of thought verification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8228–8248. Cited by: §2. Tamber et al. (2025) M. S. Tamber, F. Bao, C. Xu, G. Luo, S. Kazi, M. Bae, M. Li, O. Mendelevitch, R. Qu, and J. Lin Benchmarking llm faithfulness in rag with evolving leaderboards. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 799–811. Cited by: §1. Tyen et al. (2024) G. Tyen, H. Mansoor, V. Cărbune, Y. P. Chen, and T. Mak LLMs cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics: ACL 2024, p. 13894–13908. Cited by: §2. Welbl et al. (2017) J. Welbl, N. F. Liu, and M. Gardner Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, p. 94–106. Cited by: §3. Wu et al. (2023) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. Autogen: enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155. Cited by: §1. Wu et al. (2018) Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande MoleculeNet: a benchmark for molecular machine learning. Chemical science 9 (2), p. 513–530. Cited by: §3. Xiong et al. (2026) Z. Xiong, Y. Lin, W. Xie, P. He, Z. Liu, J. Tang, H. Lakkaraju, and Z. Xiang How memory management impacts llm agents: an empirical study of experience-following behavior. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 623–645. Cited by: §2. Xu et al. (2024) Z. Xu, S. Jain, and M. Kankanhalli Hallucination is inevitable: an innate limitation of large language models. arXiv preprint arXiv:2401.11817. Cited by: §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §4.1. Yao et al. (2022) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §1. Zhan et al. (2026) Y. Zhan, T. Fan, L. Huang, Z. Guo, and C. Huang Why your deep research agent fails? on hallucination evaluation in full research trajectory. arXiv preprint arXiv:2601.22984. Cited by: §1, §2. Zhang et al. (2023) M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534. Cited by: §1, §2. Zhang et al. (2025) Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. Computational Linguistics 51 (4), p. 1373–1418. Cited by: §1. Zhang et al. (2026) Z. Zhang, Y. Zhang, D. Zeng, K. Liu, and J. Zhao Beware of the woozle effect: exploring and mitigating hallucination propagation in multi-agent debate. IEEE Transactions on Audio, Speech and Language Processing. Cited by: §2. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: §5.3. Zhong et al. (2024) W. Zhong, X. Feng, L. Zhao, Q. Li, L. Huang, Y. Gu, W. Ma, Y. Xu, and B. Qin Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11991–12011. Cited by: §1, §2. Appendix Appendix A. Details of PHRBench.A Appendix B. Detailed Experimental Setup.B Appendix C. Detailed Results of Main Experiments.C Appendix D. Detailed Analysis of Behavioral Resolution.D Appendix E. Details of Insightful Trajectory Prediction.E Appendix A Details of PHRBench A.1 Data Collection and Statistics Data Collection. We construct PHRBench by collecting 460460 base questions from eight established benchmarks across four domains: Chemistry, Biomedicine, Physics, and Code Generation. We retain questions that require domain-specific or computational reasoning and organize the collected instances by domain as follows. Chemistry. Chemistry reasoning requires models to infer molecular properties and apply established chemical principles and mechanisms. We collect 5050 molecular-property questions from MoleculeNet-BBBP, 5050 chemistry reasoning questions from ChemBench, and 2020 open-ended chemistry questions from GPQA, yielding 120120 questions in total. Biomedicine. Biomedical reasoning requires models to integrate medical and biological knowledge with task-specific conditions to derive diagnoses, mechanisms, or conclusions. We select 100100 questions from MedMCQA and 2020 open-ended biology questions from GPQA, resulting in 120120 questions. Physics. Physics reasoning requires models to apply established physical laws, conceptual relations, and quantitative constraints. We collect 1414 questions from MMLU College Physics, 3636 from MMLU Conceptual Physics, 5050 from SciQ, and 2020 open-ended physics questions from GPQA, yielding 120120 questions. Code Generation. Code generation requires models to interpret semantic specifications, compose procedural logic, and produce executable solutions. We select 5050 problems from HumanEval and 5050 from MBPP, resulting in 100100 coding problems. Reasoning Category Annotation. After integrating questions from the eight source benchmarks, we further categorize all 460460 instances according to their primary reasoning requirements. The taxonomy follows a hierarchical structure of Domain → Reasoning Subtype. When a source benchmark provides an appropriate native task or subdomain label, we retain that label whenever possible; otherwise, questions are harmonized into a unified domain-specific taxonomy. Importantly, the assigned subtype reflects the primary reasoning process required to solve the question rather than the benchmark from which it originates. This provides a unified characterization of reasoning requirements across heterogeneous source datasets. Table 3 reports the resulting reasoning-category distribution. For reasoning-category annotation, we first preserve native task or subdomain labels when they are compatible with our taxonomy. Questions without suitable native labels are manually mapped to the domain-specific reasoning subtypes according to their primary reasoning requirement. Each question is assigned to a single subtype to ensure a mutually exclusive partition of the 460460 base questions. The resulting taxonomy captures substantial within-domain variation in reasoning requirements and enables subsequent analysis to distinguish effects associated with domain from those associated with specific reasoning patterns. Domain Reasoning Subtype Count Percentage Chemistry Molecular Property Reasoning 50 41.7% Organic Chemistry 28 23.3% Fundamental Chemistry 21 17.5% Physical Chemistry 15 12.5% Chemical Analysis 5 4.2% Chemical Safety 1 0.8% Biomedicine Clinical Reasoning 29 24.2% Physiological Reasoning 26 21.7% Molecular Biology 25 20.8% Genetic Reasoning 12 10.0% Evolutionary Biology 11 9.2% Host–Microbe Biology 7 5.8% Experimental Biology 7 5.8% Biomedical Procedures 3 2.5% Physics Modern Physics 32 26.7% Wave Physics 25 20.8% Electromagnetism 21 17.5% Classical Mechanics 18 15.0% Fundamental Physics 17 14.2% Thermal Physics 7 5.8% Code Generation Text Processing 24 24.0% Numerical Computation 22 22.0% Data Manipulation 15 15.0% Logical Validation 15 15.0% Search and Optimization 12 12.0% Combinatorial Algorithms 12 12.0% Table 3: Reasoning subtype distribution of the 460460 base questions in PHRBench. Percentages are computed within each domain. A.2 Augmentation Construction Standardization. Before augmentation, we standardize questions from different source benchmarks using task-specific templates while preserving the original task semantics, answer space, and evaluation criteria. For each base question, we construct three standardized input forms: the original question (B), the question paired with a truthful augmentation (B+T), and the question paired with a hallucinated augmentation (B+H). The augmentation is inserted as additional contextual information without modifying the underlying question or explicitly revealing the target answer. All standardized instances are converted into English and follow a consistent input structure within each task type, enabling controlled comparison across the baseline, truthful, and hallucinated augmentation settings. Augmentation Types. We categorize hallucinated augmentations into three types according to the nature of the injected semantic error. Rule Contradiction introduces a premise that directly conflicts with an established rule, principle, or domain fact. Its falsity is independent of the specific problem instance, and identifying the error primarily requires retrieving and maintaining the relevant domain knowledge. State Distortion introduces a statement that may be valid in general but is incorrect under the specific conditions of the current problem. Detecting such an error therefore requires instance-level verification of the relevant properties, conditions, intermediate states, or computational logic rather than relying solely on general knowledge. Pseudoscientific Entanglement introduces an outdated, discredited, or pseudoscientific theory as a premise for reasoning. This type evaluates whether the model can maintain consistency with established scientific knowledge when confronted with an apparently explanatory but invalid theoretical account. Together, these augmentation types create semantic conflicts at different levels, ranging from violations of established knowledge to instance-specific mismatches and invalid explanatory frameworks. Construction Procedure. We use GPT-4o as the generator to construct augmentations for each standardized base question. Given the original task, the generator produces one truthful augmentation and multiple hallucinated augmentations that are directly relevant to the question. The truthful augmentation provides factually valid supplementary information, whereas each hallucinated augmentation introduces a targeted semantic error without explicitly revealing the answer. Specifically, for each question in Chemistry, Biomedicine, and Physics, we construct one truthful augmentation and 1010 hallucinated augmentations, whereas each Code Generation problem is paired with one truthful augmentation and 33 hallucinated augmentations. This results in 460460 truthful augmentations and 39003900 hallucinated augmentations in total. The domain-wise distribution of the three hallucinated augmentation types is shown in Figure 2. To encourage semantic diversity, hallucinated augmentations associated with the same base question are required to differ in their underlying error rather than merely rephrasing the same incorrect claim. Depending on the task, the generator constructs hallucinated augmentations using the three augmentation types described above, with particular emphasis on creating both direct rule conflicts and instance-specific state distortions. For traceability, each hallucinated augmentation is generated together with a mutation strategy, which records its augmentation type, and a mutation logic, which explicitly describes the injected error. Each truthful augmentation is accompanied by a truth verification field specifying why the added information is valid. The resulting augmentations are subsequently assembled with the standardized base questions to form the B+T and B+H evaluation instances. Quality Control. We apply a multi-stage quality-control procedure to ensure that the constructed augmentations are semantically valid, task-relevant, and consistent with their intended mutation types. First, we impose explicit constraints during generation. Each hallucinated augmentation is required to introduce a single targeted semantic mutation, while preserving the original question and the remaining contextual information. Hallucinated augmentations associated with the same base question must correspond to different erroneous claims or affected properties rather than surface-level paraphrases of the same error. Truthful augmentations are required to be factually correct, relevant to the corresponding task, and free from the errors introduced by the hallucinated augmentations. Neither truthful nor hallucinated augmentations are allowed to explicitly reveal the target answer. Second, we manually review all generated augmentations, including both truthful and hallucinated augmentations, to assess their suitability for the corresponding base question. For truthful augmentations, we check that the added information is factually correct, relevant to the question, and applicable under the conditions of the original task. For hallucinated augmentations, we check that the added information is relevant to the question but contains the intended semantic error, is consistent with the assigned augmentation type, and introduces only one dominant erroneous premise. We also remove or revise augmentations that are ambiguous, redundant with other augmentations of the same question, insufficiently related to the task, or likely to leak the answer. Third, we perform an additional LLM-as-a-judge verification using Gemini 3 Pro. For each candidate augmentation, the judge re-evaluates its factual validity, relevance to the base question, and consistency with the assigned mutation type. For hallucinated augmentations, the judge further checks whether the stated mutation logic correctly identifies the underlying semantic error; for truthful augmentations, it verifies that the added information remains factually valid and does not conflict with the original task. This second-pass verification serves as an independent consistency check after manual review. Only augmentations that satisfy the generation constraints and pass both manual review and the subsequent LLM-based verification are retained in the final benchmark. This layered procedure reduces annotation noise and helps ensure that the observed model behavior can be attributed to the intended semantic perturbation rather than artifacts introduced during augmentation construction. Augmentation Prompts. We use separate prompts for the two augmentation types, and each generation call produces exactly one augmentation so that truthful controls and individual semantic mutations are constructed independently. PROMPT_T (Truthful Augmentation Generation): You are an expert in evaluating the reasoning robustness of large language models. Given the following base question, generate exactly one truthful augmentation that provides supplementary information relevant to the task. Requirements: • The augmentation must be factually correct and consistent with established domain knowledge. • It must be directly relevant to the reasoning required by the base question. • It should provide useful contextual or background information without changing the original task. • It must not explicitly reveal or directly imply the target answer. • Generate exactly one augmentation. Input: Base Question: x Output Format: "statement": "...", "truth_verification": "..." PROMPT_H (Hallucinated Augmentation Generation): You are an expert in evaluating the reasoning robustness of large language models. Given the following base question, generate exactly one hallucinated augmentation that is directly relevant to the task and introduces a single targeted semantic error. Mutation Types: • Rule Contradiction: Introduce a premise that directly contradicts an established rule, principle, or domain fact. Its falsity should be independent of the particular problem instance. • State Distortion: Introduce a statement that may be valid in general but is incorrect under the specific conditions of the current problem. Detecting the error should require instance-level verification of the relevant properties, conditions, intermediate states, or computational logic. • Pseudoscientific Entanglement: Introduce an outdated, discredited, or pseudoscientific theory as a premise for reasoning. Requirements: • Generate exactly one hallucinated augmentation. • The augmentation must be directly relevant to the base question. • It must introduce only one dominant semantic error. • All other information should remain consistent with the original task. • The augmentation must not explicitly reveal or directly imply the target answer. • The error should be substantive rather than a superficial wording change. • Assign the augmentation to one of the mutation types defined above. Input: Base Question: x Output Format: "mutation_strategy": "...", "mutation_logic": "...", "statement": "..." A.3 Examples of Tasks To illustrate the construction of PHRBench, we present one representative example from each domain. For each example, we show the standardized base question (B), a truthful augmentation (B+T), and a hallucinated augmentation (B+H). For compactness, the base question is shown only once; the B+T and B+H variants are obtained by prepending the corresponding augmentation to the same base question. An Example Data of Chemistry Task B (Baseline). Question: Which alcohol listed here has the highest boiling point? Candidates: (1) butan-1-ol; (2) butan-2-ol; (3) isobutyl alcohol; (4) t-butyl alcohol. B+T (Truthful Augmentation). Added premise: For isomeric alcohols, increased branching generally decreases molecular surface area and can weaken London dispersion interactions. B+H (Hallucinated Augmentation; State Distortion). Added premise: The more branched an alcohol is, the stronger its intermolecular forces, leading to a higher boiling point. An Example Data of Biomedicine Task B (Baseline). Question: Which of the following is not a tumor suppressor gene? Candidates: (1) pRb; (2) p16; (3) PTCH; (4) c-erbB1. B+T (Truthful Augmentation). Added premise: Tumor suppressor genes generally restrain cell proliferation or maintain cellular integrity, whereas growth-promoting signaling genes can have oncogenic functions when abnormally activated. B+H (Hallucinated Augmentation; Rule Contradiction). Added premise: c-erbB1 is a tumor suppressor gene because all genes involved in cell signaling can suppress tumor growth under certain conditions. An Example Data of Physics Task B (Baseline). Question: A wave transfers Candidates: (1) amplitude; (2) wavelength; (3) frequency; (4) energy. B+T (Truthful Augmentation). Added premise: As a wave propagates, particles of the medium may oscillate around their equilibrium positions rather than being permanently transported with the disturbance. B+H (Hallucinated Augmentation; Pseudoscientific Entanglement). Added premise: The transfer of a wave occurs due to the propagation of ether, a medium that permeates all space. An Example Data of Code Generation Task B (Baseline). def choose_num(x, y): """This function takes two positive numbers x and y and returns the biggest even integer number that is in the range [x, y] inclusive. If there is no such number, the function should return -1. choose_num(12, 15) = 14 choose_num(13, 12) = -1 """ B+T (Truthful Augmentation). Added premise: The ordering of x and y should not be assumed beyond what is specified in the task, and the behavior illustrated by all provided examples must be preserved. B+H (Hallucinated Augmentation; State Distortion). Added premise: The inputs x and y will always be in ascending order, so x is guaranteed to be less than or equal to y. Appendix B Detailed Experimental Setup B.1 Model Setup We summarize the API model identifiers or released checkpoints for all 1818 evaluated models in Table 4, together with their supported context window and maximum output length. Our evaluation includes six proprietary models and twelve open-source models, covering multiple model families, generations, and parameter scales. For proprietary models, we report the corresponding upstream API model identifiers used during evaluation; for open-source models, we report the publicly released model checkpoints. This diverse model set further enables controlled scaling comparisons within the Qwen2.5, Qwen3, and Llama-3.1 families. Model Name API / Model Code Context Window Max Output Length Proprietary Models Claude-Sonnet-4 claude-sonnet-4-20250514 200,000 64,000 GPT-4o-mini gpt-4o-mini 128,000 16,384 GPT-4o gpt-4o 128,000 16,384 GPT-5.2 gpt-5.2-2025-12-11 400,000 128,000 Gemini-2.0-Flash gemini-2.0-flash-001 1,048,576 8,192 Gemini-2.5-Pro gemini-2.5-pro 1,048,576 65,536 Open-source Models Qwen2.5-1.5B-Instruct Qwen/Qwen2.5-1.5B-Instruct 32,768 8,192 Qwen2.5-7B-Instruct Qwen/Qwen2.5-7B-Instruct 131,072∗ 8,192 Qwen2.5-32B-Instruct Qwen/Qwen2.5-32B-Instruct 131,072∗ 8,192 Qwen2.5-72B-Instruct Qwen/Qwen2.5-72B-Instruct 131,072∗ 8,192 Qwen3-8B Qwen/Qwen3-8B 32,768† 38,912‡ Qwen3-32B Qwen/Qwen3-32B 32,768† 38,912‡ Qwen3-235B-A22B Qwen/Qwen3-235B-A22B 32,768† 38,912‡ Llama-3.1-8B-Instruct meta-llama/Llama-3.1-8B-Instruct 131,072 – Llama-3.1-70B-Instruct meta-llama/Llama-3.1-70B-Instruct 131,072 – Llama-3.2-3B-Instruct meta-llama/Llama-3.2-3B-Instruct 131,072 – GLM-4-9B-Chat THUDM/glm-4-9b-chat 131,072 – Mistral-7B-Instruct mistralai/Mistral-7B-Instruct-v0.2 32,768 – ∗ Qwen2.5-7B/32B/72B support context lengths up to 131,072 tokens; their default configurations use 32,768 tokens and require YaRN for long-context deployment. † Qwen3 models natively support 32,768-token contexts and can be extended to 131,072 tokens using YaRN. ‡ Following the official Qwen3 recommendation for complex benchmarking, the maximum output length is set to 38,912 tokens. For models marked with “–”, no separate maximum generation length is specified by the released checkpoint; generation is bounded by the configured context capacity. Table 4: Model configurations used in the evaluation of PHRBench. For proprietary models, we report the upstream API model identifiers corresponding to the evaluated versions; for open-source models, we report the released model checkpoints. Inference Configuration. To ensure consistency across models, we use a unified decoding protocol throughout the evaluation. We set the sampling temperature to 0.70.7 for all models and use the maximum supported output length for each model. For models without a separately specified generation limit, generation is bounded by the context capacity of the corresponding checkpoint and inference framework. For each model–prompt pair, we independently sample 1010 responses to account for generation stochasticity. All benchmark conditions for the same model use identical decoding configurations, ensuring that the baseline, truthful-augmentation, and hallucinated augmentation settings differ only in their input context. B.2 Evaluation Metrics We evaluate each generated response from two complementary perspectives: final-answer correctness and reasoning-trajectory characteristics. Accuracy measures whether the model ultimately reaches the correct answer, whereas Output Length (OL), Uncertainty Index (UI), Belief Update Frequency (BUF), and Branching Complexity (BC) characterize different properties of the generated reasoning trajectory. To ensure comparability between proprietary and open-source models, probability-based trajectory metrics are computed post hoc using a fixed reference language model, Qwen2.5-7B-Instruct. The same reference scorer and metric configuration are used for all models, domains, and experimental conditions. Accuracy. Accuracy (Acc) measures whether the final answer extracted from a response matches the ground-truth answer. For a response r, we define (r)=[y^(r)=y∗],Acc(r)=I [ y(r)=y^* ], (4) where y^(r) y(r) denotes the extracted final answer, y∗y^* denotes the ground-truth answer, and [⋅]I[·] is the indicator function. For multiple-choice tasks, the predicted option is matched against the gold label. For open-ended tasks, we follow the task-specific evaluation criterion inherited from the corresponding source benchmark. Output Length. Output Length (OL) measures the length of the generated reasoning trajectory. Given a generated response r=(x1,x2,…,xN),r=(x_1,x_2,…,x_N), (5) we define (r)=N,OL(r)=N, (6) where N denotes the length of the generated response. OL captures the overall extent of explicit reasoning, but does not by itself indicate reasoning quality. Uncertainty Index. Uncertainty Index (UI) measures the average token-level predictive uncertainty along the reasoning trajectory. To obtain probability distributions consistently across models, each generated response is post-scored using the fixed reference model. At position i, let P(xv∣x<i)P(x_v x_<i) (7) denote the probability assigned by the reference scorer to vocabulary token v conditioned on the preceding generated trajectory x<ix_<i. UI is computed as the average entropy of these predictive distributions: =−1N∑i=1N∑v∈VP(xv∣x<i)logP(xv∣x<i),UI=- 1N _i=1^N _v∈ VP(x_v x_<i) P(x_v x_<i), (8) where V denotes the vocabulary of the reference scorer. A larger UI corresponds to a less concentrated predictive distribution and therefore greater uncertainty along the trajectory. Belief Update Frequency. Belief Update Frequency (BUF) measures how frequently reference-model next-token distributions shift substantially between consecutive reasoning steps, following Eq. 2. We first segment each generated reasoning trajectory into sentence-level steps using sentence-final punctuation and line breaks as boundaries, after removing empty segments. A response is therefore represented as an ordered sequence ℛ=(s1,s2,…,sK).R=(s_1,s_2,…,s_K). (9) For reasoning step sks_k, we concatenate the original task with all reasoning steps up to sks_k, ck=[q;s1;…;sk],c_k=[q;s_1;…;s_k], (10) and define PkP_k as the next-token predictive distribution from the fixed reference scorer: Pk(v)=Pref(v∣ck),v∈V.P_k(v)=P_ref(v c_k), v∈ V. (11) The distributional change between consecutive reasoning steps is quantified by the Kullback–Leibler divergence, Δk=DKL(Pk∥Pk−1). _k=D_KL (P_k P_k-1 ). (12) We regard a transition as a substantial distributional shift when Δk>τ _k>τ and use a fixed threshold of τ=0.10τ=0.10 nats throughout all experiments. BUF is then defined as =1K−1∑k=2K(DKL(Pk∥Pk−1)>τ).BUF= 1K-1 _k=2^KI (D_KL (P_k P_k-1 )>τ ). (13) Here, K denotes the number of reasoning steps, PkP_k and Pk−1P_k-1 denote the reference scorer’s next-token predictive distributions at consecutive steps, DKLD_KL is the Kullback–Leibler divergence, and τ is the divergence threshold. The indicator [⋅]I[·] equals 11 when the condition holds and 00 otherwise. For K≥2K≥ 2, BUF is therefore the fraction of the K−1K-1 consecutive transitions that exceed the threshold and lies in [0,1][0,1]. For numerical stability, predictive probabilities are clipped at 10−1210^-12 and renormalized before computing the KL divergence. A higher BUF indicates more frequent distributional shifts under the reference scorer; it does not by itself establish belief revision, error correction, or improved reasoning. Branching Complexity. Branching Complexity (BC) measures the prevalence of explicit alternative reasoning paths using a fixed set of branching-related lexical cues. We use =or,either,may,possibly,but,on the other hand,case.K=\ or, either, may, possibly, but, on the other hand, case\. (14) All cue matching is case-insensitive; single-word cues are matched at word boundaries and multi-word cues are matched as exact phrases. BC is defined as (r)=100Nword∑k∈Count(k,r),BC(r)= 100N_word _k Count(k,r), (15) where NwordN_word denotes the total number of words in response r and Count(k,r)Count(k,r) denotes the number of occurrences of cue k. BC can therefore be interpreted as the number of branching-related expressions per 100100 generated words. Metric Aggregation. For each model–prompt pair, we independently sample 1010 responses. All metrics are first computed at the individual-response level and subsequently averaged over repeated generations and benchmark instances when reporting model-level statistics. For behavioral analysis, trajectories are first grouped according to their behavioral-resolution labels, after which the corresponding trajectory metrics are aggregated within each group. The same parameters are used throughout the evaluation. Appendix C Detailed Results of Main Experiments Figure 3 in the main text summarizes accuracy retention across models, task domains, and hallucinated-augmentation types. This appendix complements that overview with detailed accuracy and reasoning-trajectory statistics under the truthful (T) and hallucinated (H) augmentation settings. In Figure 3, accuracy retention measures the proportion of truthful-setting accuracy preserved under hallucinated augmentation: Retention(%)=100×AccHAccT.Retention\,(\%)=100× Acc_HAcc_T. Here, AccTAcc_T and AccHAcc_H denote the accuracies of the same model under the two settings for the corresponding evaluation subset. A retention of 100%100\% indicates unchanged accuracy; values below 100%100\% indicate degradation, and values above 100%100\% indicate improvement. For example, 80%80\% retention means that accuracy under H is 80%80\% of its value under T. Retention describes relative accuracy preservation, while the accuracy decreases reported in the tables are absolute differences measured in percentage points. C.1 Domain-Wise Results Tables 5–8 report detailed results for Chemistry, Biomedicine, Physics, and Code Generation, respectively, allowing us to examine how the effects of hallucinated context vary across domains. Hallucinated context induces broad but domain-dependent degradation. The decrease in final-answer accuracy is broadly consistent across domains. Averaged over the 1818 evaluated models, the accuracy drop is 6.576.57 percentage points in Chemistry, 8.348.34 points in Biomedicine, 7.127.12 points in Physics, and 6.896.89 points in Code Generation. All models degrade under hallucinated augmentation in the three scientific domains, while 1717 of 1818 models degrade in Code Generation, with Qwen3-8B showing only a marginal 0.10.1-point increase. Thus, the overall performance degradation reported in the main text is not driven by a single domain. Biomedicine exhibits the largest average degradation, followed by Physics, whereas Chemistry shows the smallest absolute drop. Notably, Chemistry also has substantially lower baseline accuracy on average (23.0%23.0\%) than Biomedicine (57.2%57.2\%), Physics (56.0%56.0\%), and Code Generation (48.4%48.4\%), suggesting that its smaller absolute degradation should not be interpreted as greater robustness, since the available performance margin is considerably smaller. The scaling-induced vulnerability persists across domains. The relationship between model scale and degradation observed in the aggregate results remains visible after domain-wise decomposition. Within Qwen2.5, scaling from 1.5B to 72B increases the accuracy drop from 2.02.0 to 6.66.6 points in Chemistry, 4.84.8 to 7.77.7 in Biomedicine, 3.73.7 to 6.66.6 in Physics, and 3.13.1 to 6.36.3 in Code Generation. The same pattern is more pronounced for Qwen3: from 8B to 235B-A22B, degradation increases from 1.91.9 to 9.79.7 points in Chemistry, 1.51.5 to 10.210.2 in Biomedicine, 0.30.3 to 9.19.1 in Physics, and from approximately no degradation to 8.98.9 points in Code Generation. Llama-3.1 exhibits a similar shift from 8B to 70B across all four domains. These results indicate that the scaling trend identified in the main analysis is not specific to a particular task family, but recurs across heterogeneous scientific and computational reasoning settings. Reasoning trajectories respond differently across domains. Although the outcome-level degradation is broadly consistent, the associated trajectory changes are more domain dependent. Averaged across models, hallucinated augmentation produces longer trajectories in Biomedicine and Physics (+30.4+30.4 and +30.1+30.1 words, respectively), together with increases in both UI and BUF. In contrast, Chemistry exhibits shorter trajectories (−32.1-32.1 words) and lower average UI, BUF, and BC under hallucinated augmentation, while Code Generation also shows shorter trajectories and reduced UI and BUF, despite a modest increase in BC. Physics shows the clearest joint increase in reasoning activity, with OL, UI, BUF, and BC all increasing on average. These differences suggest that similar losses in final-answer accuracy can arise through distinct reasoning dynamics: hallucinated context may induce additional deliberation and belief revision in some domains, while suppressing or prematurely constraining the reasoning trajectory in others. Table 5: Domain-wise performance and reasoning trajectory statistics on Chemistry under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 33.333.3 22.2↓11.122.2_ [rgb]1,0,0 11.1 122.0122.0 117.1117.1 0.1140.114 0.0940.094 0.2520.252 0.3600.360 1.1391.139 0.8700.870 GPT-4o-mini 24.224.2 21.0↓3.221.0_ [rgb]1,0,0 3.2 87.287.2 76.076.0 0.1730.173 0.0860.086 0.2580.258 0.2190.219 0.8950.895 0.6190.619 GPT-4o 31.931.9 20.8↓11.120.8_ [rgb]1,0,0 11.1 75.075.0 61.361.3 0.1800.180 0.0840.084 0.2210.221 0.2220.222 0.6780.678 0.4870.487 GPT-5.2 38.338.3 20.9↓17.420.9_ [rgb]1,0,0 17.4 56.856.8 50.050.0 0.0160.016 0.0180.018 0.0160.016 0.0290.029 0.5190.519 0.3660.366 Gemini-2.0-Flash 31.531.5 20.6↓10.920.6_ [rgb]1,0,0 10.9 95.195.1 90.790.7 0.1070.107 0.0810.081 0.1510.151 0.1960.196 0.7620.762 0.6150.615 Gemini-2.5-Pro 33.833.8 21.6↓12.221.6_ [rgb]1,0,0 12.2 102.8102.8 99.999.9 0.1050.105 0.0900.090 0.2030.203 0.2680.268 0.8710.871 0.6660.666 Open-source Models Qwen2.5-1.5B-Instruct 8.88.8 6.8↓2.06.8_ [rgb]1,0,0 2.0 113.4113.4 107.7107.7 0.7000.700 0.3460.346 0.1610.161 0.0860.086 2.0172.017 1.3971.397 Qwen2.5-7B-Instruct 20.220.2 15.6↓4.615.6_ [rgb]1,0,0 4.6 161.1161.1 139.4139.4 0.8150.815 0.2120.212 0.2470.247 0.1220.122 2.3262.326 1.4981.498 Qwen2.5-32B-Instruct 21.921.9 16.6↓5.316.6_ [rgb]1,0,0 5.3 234.3234.3 201.9201.9 0.6230.623 0.2260.226 0.4760.476 0.3710.371 2.8432.843 2.0752.075 Qwen2.5-72B-Instruct 26.326.3 19.7↓6.619.7_ [rgb]1,0,0 6.6 284.1284.1 246.6246.6 0.4780.478 0.1680.168 0.6910.691 0.5100.510 3.4543.454 2.4722.472 Qwen3-8B 22.222.2 20.3↓1.920.3_ [rgb]1,0,0 1.9 224.8224.8 170.4170.4 0.4580.458 0.0830.083 0.8370.837 0.3390.339 2.6782.678 2.0782.078 Qwen3-32B 24.124.1 18.3↓5.818.3_ [rgb]1,0,0 5.8 344.0344.0 243.1243.1 0.3320.332 0.0620.062 0.7600.760 0.3990.399 3.3023.302 2.7332.733 Qwen3-235B-A22B 32.832.8 23.1↓9.723.1_ [rgb]1,0,0 9.7 423.8423.8 350.0350.0 0.1560.156 0.0430.043 0.8560.856 0.7200.720 4.2464.246 3.4543.454 Llama-3.1-8B-Instruct 13.013.0 11.2↓1.811.2_ [rgb]1,0,0 1.8 256.4256.4 215.5215.5 1.0881.088 0.5700.570 0.3810.381 0.2800.280 2.9182.918 1.8371.837 Llama-3.1-70B-Instruct 22.322.3 16.3↓6.016.3_ [rgb]1,0,0 6.0 346.8346.8 295.9295.9 1.2931.293 0.7980.798 0.8390.839 0.5620.562 3.5763.576 2.5242.524 Llama-3.2-3B-Instruct 11.511.5 10.1↓1.410.1_ [rgb]1,0,0 1.4 188.7188.7 162.6162.6 0.4410.441 0.2570.257 0.6230.623 0.1600.160 2.2022.202 1.2431.243 GLM-4-9B-Chat 11.711.7 6.8↓4.96.8_ [rgb]1,0,0 4.9 137.7137.7 74.274.2 0.4380.438 0.2160.216 0.0890.089 0.0430.043 1.8991.899 0.8580.858 Mistral-7B-Instruct 6.96.9 4.5↓2.44.5_ [rgb]1,0,0 2.4 141.7141.7 114.9114.9 0.3330.333 0.1750.175 0.4690.469 0.4210.421 1.8661.866 1.1251.125 Table 6: Domain-wise performance and reasoning trajectory statistics on Biomedicine under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 70.070.0 60.2↓9.860.2_ [rgb]1,0,0 9.8 100.9100.9 121.2121.2 0.0500.050 0.1050.105 0.1970.197 0.4040.404 1.5581.558 1.4381.438 GPT-4o-mini 68.368.3 58.4↓9.958.4_ [rgb]1,0,0 9.9 69.269.2 72.472.4 0.0620.062 0.0800.080 0.1230.123 0.2050.205 1.0181.018 0.9800.980 GPT-4o 66.766.7 56.0↓10.756.0_ [rgb]1,0,0 10.7 57.457.4 60.960.9 0.0410.041 0.0670.067 0.1190.119 0.2490.249 1.0041.004 0.8540.854 GPT-5.2 72.572.5 57.3↓15.257.3_ [rgb]1,0,0 15.2 42.642.6 48.348.3 0.0100.010 0.0230.023 0.0110.011 0.0350.035 0.7490.749 0.6860.686 Gemini-2.0-Flash 69.269.2 57.1↓12.157.1_ [rgb]1,0,0 12.1 74.474.4 89.489.4 0.0490.049 0.0880.088 0.0970.097 0.2170.217 1.0451.045 1.0661.066 Gemini-2.5-Pro 71.471.4 58.6↓12.858.6_ [rgb]1,0,0 12.8 80.480.4 98.598.5 0.0480.048 0.0980.098 0.1310.131 0.2970.297 1.1951.195 1.1551.155 Open-source Models Qwen2.5-1.5B-Instruct 32.132.1 27.3↓4.827.3_ [rgb]1,0,0 4.8 105.2105.2 118.9118.9 0.1970.197 0.2400.240 0.0840.084 0.0890.089 2.7082.708 2.2642.264 Qwen2.5-7B-Instruct 55.455.4 48.6↓6.848.6_ [rgb]1,0,0 6.8 143.2143.2 152.4152.4 0.2170.217 0.1880.188 0.1470.147 0.1490.149 2.9722.972 2.4572.457 Qwen2.5-32B-Instruct 57.957.9 50.5↓7.450.5_ [rgb]1,0,0 7.4 210.5210.5 220.2220.2 0.1680.168 0.1700.170 0.2700.270 0.4270.427 3.7503.750 3.3583.358 Qwen2.5-72B-Instruct 63.563.5 55.8↓7.755.8_ [rgb]1,0,0 7.7 255.2255.2 269.1269.1 0.1290.129 0.1270.127 0.3920.392 0.5860.586 4.5564.556 4.0004.000 Qwen3-8B 56.256.2 54.7↓1.554.7_ [rgb]1,0,0 1.5 135.9135.9 289.6289.6 0.1280.128 0.0890.089 0.4400.440 0.4650.465 3.3703.370 4.5254.525 Qwen3-32B 58.958.9 51.5↓7.451.5_ [rgb]1,0,0 7.4 236.5236.5 330.1330.1 0.1280.128 0.0990.099 0.5370.537 0.5860.586 3.6433.643 4.7474.747 Qwen3-235B-A22B 68.868.8 58.6↓10.258.6_ [rgb]1,0,0 10.2 291.3291.3 475.3475.3 0.0600.060 0.0690.069 0.8170.817 0.8260.826 4.6834.683 5.9995.999 Llama-3.1-8B-Instruct 45.045.0 42.0↓3.042.0_ [rgb]1,0,0 3.0 190.2190.2 197.2197.2 0.2300.230 0.3250.325 0.4840.484 0.2070.207 3.3233.323 2.6252.625 Llama-3.1-70B-Instruct 61.061.0 52.8↓8.252.8_ [rgb]1,0,0 8.2 257.8257.8 271.8271.8 0.2370.237 0.3740.374 0.5260.526 0.3540.354 3.8283.828 3.3613.361 Llama-3.2-3B-Instruct 41.441.4 39.0↓2.439.0_ [rgb]1,0,0 2.4 146.7146.7 144.0144.0 0.1140.114 0.1360.136 0.3030.303 0.1270.127 2.7512.751 1.8831.883 GLM-4-9B-Chat 42.242.2 29.7↓12.529.7_ [rgb]1,0,0 12.5 108.1108.1 80.580.5 0.1080.108 0.2590.259 0.0420.042 0.0420.042 2.3442.344 1.4081.408 Mistral-7B-Instruct 29.329.3 21.5↓7.821.5_ [rgb]1,0,0 7.8 111.3111.3 124.8124.8 0.0820.082 0.1150.115 0.2180.218 0.4060.406 2.3032.303 1.8471.847 Table 7: Domain-wise performance and reasoning trajectory statistics on Physics under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 69.269.2 63.0↓6.263.0_ [rgb]1,0,0 6.2 89.489.4 104.0104.0 0.0100.010 0.0240.024 0.1010.101 0.2840.284 0.9640.964 1.0671.067 GPT-4o-mini 74.274.2 65.4↓8.865.4_ [rgb]1,0,0 8.8 71.571.5 73.073.0 0.0200.020 0.0240.024 0.0950.095 0.1720.172 0.5830.583 0.6060.606 GPT-4o 56.756.7 49.9↓6.849.9_ [rgb]1,0,0 6.8 58.258.2 63.163.1 0.0060.006 0.0190.019 0.0810.081 0.2170.217 0.5050.505 0.5630.563 GPT-5.2 74.274.2 55.8↓18.455.8_ [rgb]1,0,0 18.4 34.434.4 38.238.2 0.0010.001 0.0080.008 0.0150.015 0.0190.019 0.4710.471 0.4640.464 Gemini-2.0-Flash 68.768.7 57.8↓10.957.8_ [rgb]1,0,0 10.9 69.469.4 82.582.5 0.0060.006 0.0240.024 0.0750.075 0.1570.157 0.5750.575 0.6950.695 Gemini-2.5-Pro 70.970.9 59.2↓11.759.2_ [rgb]1,0,0 11.7 75.175.1 90.990.9 0.0060.006 0.0270.027 0.1010.101 0.2150.215 0.6570.657 0.7530.753 Open-source Models Qwen2.5-1.5B-Instruct 30.530.5 26.8↓3.726.8_ [rgb]1,0,0 3.7 103.0103.0 120.3120.3 0.0730.073 0.1050.105 0.0730.073 0.0670.067 1.7521.752 1.6431.643 Qwen2.5-7B-Instruct 53.553.5 48.1↓5.448.1_ [rgb]1,0,0 5.4 131.2131.2 145.0145.0 0.0760.076 0.0740.074 0.1130.113 0.1110.111 2.1422.142 1.9281.928 Qwen2.5-32B-Instruct 56.156.1 49.9↓6.249.9_ [rgb]1,0,0 6.2 199.1199.1 216.0216.0 0.0660.066 0.0770.077 0.2220.222 0.3200.320 2.6332.633 2.5882.588 Qwen2.5-72B-Instruct 61.861.8 55.2↓6.655.2_ [rgb]1,0,0 6.6 241.4241.4 263.9263.9 0.0510.051 0.0570.057 0.3230.323 0.4400.440 3.1993.199 3.0843.084 Qwen3-8B 54.754.7 54.4↓0.354.4_ [rgb]1,0,0 0.3 116.8116.8 262.9262.9 0.0530.053 0.0440.044 0.4740.474 0.4140.414 2.0532.053 3.7083.708 Qwen3-32B 57.457.4 51.2↓6.251.2_ [rgb]1,0,0 6.2 188.5188.5 269.4269.4 0.0610.061 0.0360.036 0.5250.525 0.4600.460 2.4452.445 3.9073.907 Qwen3-235B-A22B 67.467.4 58.3↓9.158.3_ [rgb]1,0,0 9.1 232.2232.2 387.8387.8 0.0290.029 0.0250.025 0.7990.799 0.7330.733 3.1433.143 4.9384.938 Llama-3.1-8B-Instruct 43.743.7 41.9↓1.841.9_ [rgb]1,0,0 1.8 164.6164.6 178.9178.9 0.0800.080 0.1360.136 0.4150.415 0.1710.171 2.8872.887 2.3512.351 Llama-3.1-70B-Instruct 59.759.7 52.7↓7.052.7_ [rgb]1,0,0 7.0 223.9223.9 247.6247.6 0.0900.090 0.1760.176 0.4670.467 0.3060.306 3.7943.794 3.3213.321 Llama-3.2-3B-Instruct 40.140.1 39.0↓1.139.0_ [rgb]1,0,0 1.1 143.8143.8 146.0146.0 0.0310.031 0.0460.046 0.2550.255 0.1090.109 1.8801.880 1.3001.300 GLM-4-9B-Chat 41.741.7 30.3↓11.430.3_ [rgb]1,0,0 11.4 99.999.9 77.177.1 0.0410.041 0.0700.070 0.0380.038 0.0350.035 1.7241.724 1.1401.140 Mistral-7B-Instruct 27.827.8 21.2↓6.621.2_ [rgb]1,0,0 6.6 102.8102.8 119.5119.5 0.0310.031 0.0370.037 0.1980.198 0.3420.342 1.6941.694 1.4951.495 Table 8: Domain-wise performance and reasoning trajectory statistics on Code Generation under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 57.557.5 47.6↓9.947.6_ [rgb]1,0,0 9.9 88.488.4 102.3102.3 0.0410.041 0.0290.029 0.0390.039 0.0870.087 0.8180.818 0.9460.946 GPT-4o-mini 55.355.3 48.5↓6.848.5_ [rgb]1,0,0 6.8 54.054.0 55.655.6 0.0260.026 0.0110.011 0.0270.027 0.0320.032 0.4640.464 0.5410.541 GPT-4o 52.052.0 42.8↓9.242.8_ [rgb]1,0,0 9.2 47.247.2 49.549.5 0.0090.009 0.0100.010 0.0140.014 0.0300.030 0.4070.407 0.4210.421 GPT-5.2 61.461.4 44.8↓16.644.8_ [rgb]1,0,0 16.6 37.337.3 40.640.6 0.0010.001 0.0100.010 0.0110.011 0.0080.008 0.5700.570 0.6320.632 Gemini-2.0-Flash 55.355.3 44.0↓11.344.0_ [rgb]1,0,0 11.3 63.963.9 74.574.5 0.0170.017 0.0220.022 0.0350.035 0.0360.036 0.5700.570 0.6980.698 Gemini-2.5-Pro 57.957.9 45.5↓12.445.5_ [rgb]1,0,0 12.4 69.069.0 82.082.0 0.0160.016 0.0240.024 0.0480.048 0.0490.049 0.6520.652 0.7560.756 Open-source Models Qwen2.5-1.5B-Instruct 27.427.4 24.3↓3.124.3_ [rgb]1,0,0 3.1 145.8145.8 134.5134.5 0.0870.087 0.0730.073 0.0380.038 0.0210.021 1.3681.368 1.5041.504 Qwen2.5-7B-Instruct 49.749.7 44.7↓5.044.7_ [rgb]1,0,0 5.0 182.1182.1 165.0165.0 0.1390.139 0.0680.068 0.0720.072 0.0410.041 1.4911.491 1.6731.673 Qwen2.5-32B-Instruct 52.352.3 46.6↓5.746.6_ [rgb]1,0,0 5.7 277.8277.8 243.8243.8 0.0980.098 0.0620.062 0.1500.150 0.1160.116 1.9001.900 2.2882.288 Qwen2.5-72B-Instruct 58.258.2 51.9↓6.351.9_ [rgb]1,0,0 6.3 336.9336.9 297.8297.8 0.0750.075 0.0460.046 0.2170.217 0.1590.159 2.3082.308 2.7262.726 Qwen3-8B 51.951.9 52.0 ↑ 0.1 465.6465.6 444.2444.2 0.2310.231 0.0730.073 0.5480.548 0.2250.225 3.4123.412 3.8663.866 Qwen3-32B 54.654.6 48.8↓5.848.8_ [rgb]1,0,0 5.8 440.2440.2 378.8378.8 0.2070.207 0.0570.057 0.5980.598 0.2620.262 3.5993.599 4.1124.112 Qwen3-235B-A22B 64.964.9 56.0↓8.956.0_ [rgb]1,0,0 8.9 542.3542.3 545.4545.4 0.0970.097 0.0400.040 0.9100.910 0.6720.672 4.6274.627 5.1975.197 Llama-3.1-8B-Instruct 35.335.3 34.1↓1.234.1_ [rgb]1,0,0 1.2 234.9234.9 232.1232.1 0.0620.062 0.0640.064 0.1710.171 0.0550.055 1.7061.706 1.6321.632 Llama-3.1-70B-Instruct 51.051.0 44.4↓6.644.4_ [rgb]1,0,0 6.6 317.7317.7 318.8318.8 0.0380.038 0.0560.056 0.2090.209 0.1010.101 2.0002.000 2.0352.035 Llama-3.2-3B-Instruct 32.032.0 31.4↓0.631.4_ [rgb]1,0,0 0.6 193.2193.2 183.6183.6 0.0280.028 0.0240.024 0.0640.064 0.0240.024 1.3491.349 1.0311.031 GLM-4-9B-Chat 33.933.9 24.1↓9.824.1_ [rgb]1,0,0 9.8 170.7170.7 99.799.7 0.0820.082 0.0700.070 0.0270.027 0.0140.014 1.5151.515 0.9640.964 Mistral-7B-Instruct 20.320.3 15.3↓5.015.3_ [rgb]1,0,0 5.0 175.7175.7 154.5154.5 0.0620.062 0.0450.045 0.1430.143 0.1330.133 1.4881.488 1.2651.265 C.2 Results across Hallucinated Augmentation Types Tables 9–11 report detailed results for Rule Contradiction (RC), State Distortion (SD), and Pseudoscientific Entanglement (PE). The comparison covers the three scientific domains, where all three types are represented. Code Generation is excluded because its hallucinated augmentations consist exclusively of State Distortion. State Distortion induces the strongest performance degradation. Across the 1818 models, SD causes the largest average accuracy drop (9.49.4 percentage points), followed by RC (7.47.4 points), whereas PE has a substantially weaker effect (1.81.8 points). This pattern suggests that instance-specific misinformation is particularly difficult to resolve: unlike direct violations of established rules, SD preserves a plausible general statement while making it inconsistent with the conditions of the current problem. In contrast, the comparatively small degradation under PE indicates that outdated or pseudoscientific premises are more readily distinguished from established domain knowledge. Different augmentation types also induce distinct trajectory changes. The trajectory metrics reveal a less uniform pattern than final-answer accuracy. SD is associated with both longer responses and increased branching complexity on average, suggesting greater exploration of competing reasoning paths under instance-level conflict. RC produces a smaller increase in output length and lower branching complexity, while PE often elicits longer responses without a commensurate loss in accuracy. Together, these results show that the effect of hallucinated context depends not only on whether the information is false, but also on the form of semantic conflict it introduces. Table 9: Performance and reasoning trajectory statistics for Rule Contradiction under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 57.757.7 48.5↓9.248.5_ [rgb]1,0,0 9.2 105.4105.4 113.8113.8 0.0540.054 0.0660.066 0.1750.175 0.3360.336 1.2201.220 1.0991.099 GPT-4o-mini 55.755.7 48.5↓7.248.5_ [rgb]1,0,0 7.2 77.277.2 73.373.3 0.0790.079 0.0560.056 0.1510.151 0.1910.191 0.8320.832 0.7180.718 GPT-4o 52.152.1 42.2↓9.942.2_ [rgb]1,0,0 9.9 64.064.0 61.361.3 0.0700.070 0.0500.050 0.1340.134 0.2200.220 0.7290.729 0.6200.620 GPT-5.2 62.162.1 44.4↓17.744.4_ [rgb]1,0,0 17.7 45.445.4 45.345.3 0.0080.008 0.0150.015 0.0130.013 0.0260.026 0.5800.580 0.4940.494 Gemini-2.0-Flash 56.656.6 45.4↓11.245.4_ [rgb]1,0,0 11.2 79.779.7 87.087.0 0.0500.050 0.0570.057 0.1030.103 0.1820.182 0.7940.794 0.7740.774 Gemini-2.5-Pro 58.958.9 46.7↓12.246.7_ [rgb]1,0,0 12.2 86.386.3 95.995.9 0.0490.049 0.0630.063 0.1390.139 0.2500.250 0.9070.907 0.8380.838 Open-source Models Qwen2.5-1.5B-Instruct 23.923.9 20.4↓3.520.4_ [rgb]1,0,0 3.5 104.9104.9 113.8113.8 0.3010.301 0.2040.204 0.1010.101 0.0770.077 2.1592.159 1.7271.727 Qwen2.5-7B-Instruct 43.243.2 37.6↓5.637.6_ [rgb]1,0,0 5.6 143.2143.2 143.5143.5 0.3440.344 0.1400.140 0.1610.161 0.1230.123 2.4802.480 1.9161.916 Qwen2.5-32B-Instruct 45.445.4 39.2↓6.239.2_ [rgb]1,0,0 6.2 215.0215.0 211.5211.5 0.2660.266 0.1390.139 0.3080.308 0.3580.358 3.0753.075 2.6122.612 Qwen2.5-72B-Instruct 50.750.7 43.8↓6.943.8_ [rgb]1,0,0 6.9 260.6260.6 258.4258.4 0.2040.204 0.1040.104 0.4470.447 0.4920.492 3.7363.736 3.1123.112 Qwen3-8B 44.544.5 43.3↓1.243.3_ [rgb]1,0,0 1.2 159.4159.4 239.6239.6 0.1980.198 0.0640.064 0.5570.557 0.3900.390 2.7002.700 3.3583.358 Qwen3-32B 47.047.0 40.5↓6.540.5_ [rgb]1,0,0 6.5 256.7256.7 279.2279.2 0.1620.162 0.0580.058 0.5790.579 0.4630.463 3.1303.130 3.7083.708 Qwen3-235B-A22B 56.556.5 46.9↓9.646.9_ [rgb]1,0,0 9.6 316.2316.2 402.0402.0 0.0760.076 0.0400.040 0.8170.817 0.7540.754 4.0244.024 4.6864.686 Llama-3.1-8B-Instruct 34.034.0 31.8↓2.231.8_ [rgb]1,0,0 2.2 203.2203.2 193.0193.0 0.4340.434 0.3040.304 0.4070.407 0.2110.211 3.0423.042 2.2192.219 Llama-3.1-70B-Instruct 47.847.8 40.8↓7.040.8_ [rgb]1,0,0 7.0 276.6276.6 270.2270.2 0.5030.503 0.3980.398 0.5830.583 0.3920.392 3.7323.732 2.9982.998 Llama-3.2-3B-Instruct 31.131.1 29.5↓1.629.5_ [rgb]1,0,0 1.6 158.7158.7 149.6149.6 0.1820.182 0.1300.130 0.3760.376 0.1270.127 2.2782.278 1.4411.441 GLM-4-9B-Chat 32.032.0 22.4↓9.622.4_ [rgb]1,0,0 9.6 115.4115.4 76.876.8 0.1820.182 0.1610.161 0.0540.054 0.0380.038 1.9891.989 1.1091.109 Mistral-7B-Instruct 21.421.4 15.8↓5.615.8_ [rgb]1,0,0 5.6 118.8118.8 119.0119.0 0.1390.139 0.0800.080 0.2820.282 0.3750.375 1.9541.954 1.4551.455 Table 10: Performance and reasoning trajectory statistics for State Distortion under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 43.843.8 32.6↓11.232.6_ [rgb]1,0,0 11.2 111.9111.9 125.1125.1 0.1080.108 0.1720.172 0.2980.298 0.5580.558 1.1791.179 1.3361.336 GPT-4o-mini 40.140.1 30.0↓10.130.0_ [rgb]1,0,0 10.1 81.381.3 83.083.0 0.1600.160 0.1460.146 0.2580.258 0.3180.318 0.8040.804 0.8730.873 GPT-4o 41.941.9 30.1↓11.830.1_ [rgb]1,0,0 11.8 72.072.0 68.768.7 0.1420.142 0.1310.131 0.2280.228 0.3670.367 0.7040.704 0.7530.753 GPT-5.2 44.644.6 30.4↓14.230.4_ [rgb]1,0,0 14.2 46.446.4 52.252.2 0.0160.016 0.0380.038 0.0220.022 0.0440.044 0.5610.561 0.6000.600 Gemini-2.0-Flash 42.542.5 27.1↓15.427.1_ [rgb]1,0,0 15.4 84.984.9 98.098.0 0.1020.102 0.1490.149 0.1750.175 0.3030.303 0.7670.767 0.9410.941 Gemini-2.5-Pro 44.244.2 27.5↓16.727.5_ [rgb]1,0,0 16.7 91.991.9 108.0108.0 0.1000.100 0.1660.166 0.2360.236 0.4160.416 0.8770.877 1.0191.019 Open-source Models Qwen2.5-1.5B-Instruct 17.917.9 13.1↓4.813.1_ [rgb]1,0,0 4.8 120.9120.9 128.8128.8 0.6090.609 0.5320.532 0.1720.172 0.1290.129 2.0862.086 2.1002.100 Qwen2.5-7B-Instruct 32.432.4 24.7↓7.724.7_ [rgb]1,0,0 7.7 159.1159.1 164.3164.3 0.6940.694 0.3650.365 0.2750.275 0.2040.204 2.3962.396 2.3292.329 Qwen2.5-32B-Instruct 34.134.1 25.5↓8.625.5_ [rgb]1,0,0 8.6 229.0229.0 238.2238.2 0.5380.538 0.3640.364 0.5240.524 0.5960.596 2.9722.972 3.1753.175 Qwen2.5-72B-Instruct 38.038.0 28.5↓9.528.5_ [rgb]1,0,0 9.5 277.7277.7 291.0291.0 0.4130.413 0.2720.272 0.7620.762 0.8190.819 3.6113.611 3.7833.783 Qwen3-8B 33.433.4 31.7↓1.731.7_ [rgb]1,0,0 1.7 169.8169.8 269.8269.8 0.4010.401 0.1660.166 0.9490.949 0.6490.649 2.6092.609 4.0824.082 Qwen3-32B 35.235.2 26.4↓8.826.4_ [rgb]1,0,0 8.8 273.5273.5 314.5314.5 0.3270.327 0.1520.152 0.9870.987 0.7700.770 3.0253.025 4.5084.508 Qwen3-235B-A22B 42.442.4 29.2↓13.229.2_ [rgb]1,0,0 13.2 336.9336.9 452.8452.8 0.1540.154 0.1050.105 0.9450.945 0.9030.903 3.8883.888 5.6975.697 Llama-3.1-8B-Instruct 25.525.5 22.5↓3.022.5_ [rgb]1,0,0 3.0 248.4248.4 248.6248.6 0.8760.876 0.7950.795 0.6930.693 0.3510.351 2.9402.940 2.6972.697 Llama-3.1-70B-Instruct 35.935.9 26.2↓9.726.2_ [rgb]1,0,0 9.7 294.7294.7 304.4304.4 1.0161.016 1.0401.040 0.9930.993 0.6510.651 3.6073.607 3.6443.644 Llama-3.2-3B-Instruct 23.323.3 21.1↓2.221.1_ [rgb]1,0,0 2.2 173.8173.8 167.0167.0 0.3680.368 0.3390.339 0.6400.640 0.2110.211 2.2012.201 1.7521.752 GLM-4-9B-Chat 24.024.0 10.8↓13.210.8_ [rgb]1,0,0 13.2 123.0123.0 86.586.5 0.3680.368 0.4210.421 0.0920.092 0.0640.064 1.9221.922 1.3481.348 Mistral-7B-Instruct 16.116.1 8.4↓7.78.4_ [rgb]1,0,0 7.7 126.5126.5 134.1134.1 0.2800.280 0.4250.425 0.4800.480 0.6230.623 1.8891.889 1.7691.769 Table 11: Performance and reasoning trajectory statistics for Pseudoscientific Entanglement under the truthful augmentation (T) and hallucinated augmentation (H) settings. Acc (%) OL UI BUF BC Model T H T H T H T H T H Proprietary Models Claude-Sonnet-4 87.787.7 85.2↓2.585.2_ [rgb]1,0,0 2.5 83.883.8 95.295.2 0.0410.041 0.0650.065 0.1290.129 0.2070.207 1.3201.320 1.3041.304 GPT-4o-mini 89.389.3 85.2↓4.185.2_ [rgb]1,0,0 4.1 62.962.9 65.065.0 0.0610.061 0.0550.055 0.1120.112 0.1180.118 0.9000.900 0.8520.852 GPT-4o 72.172.1 73.0 ↑ 0.9 56.156.1 58.258.2 0.0540.054 0.0500.050 0.0990.099 0.1360.136 0.7890.789 0.7350.735 GPT-5.2 91.091.0 86.9↓4.186.9_ [rgb]1,0,0 4.1 32.432.4 36.936.9 0.0060.006 0.0140.014 0.0100.010 0.0160.016 0.6280.628 0.5860.586 Gemini-2.0-Flash 84.684.6 81.8↓2.881.8_ [rgb]1,0,0 2.8 63.863.8 75.875.8 0.0390.039 0.0560.056 0.0760.076 0.1130.113 0.8590.859 0.9180.918 Gemini-2.5-Pro 87.987.9 84.8↓3.184.8_ [rgb]1,0,0 3.1 69.069.0 83.683.6 0.0380.038 0.0630.063 0.1020.102 0.1540.154 0.9820.982 0.9940.994 Open-source Models Qwen2.5-1.5B-Instruct 35.735.7 34.8↓0.934.8_ [rgb]1,0,0 0.9 106.0106.0 130.3130.3 0.2320.232 0.2020.202 0.0740.074 0.0480.048 2.3362.336 2.0502.050 Qwen2.5-7B-Instruct 64.564.5 63.1↓1.463.1_ [rgb]1,0,0 1.4 132.7132.7 155.8155.8 0.2650.265 0.1380.138 0.1190.119 0.0760.076 2.6832.683 2.2732.273 Qwen2.5-32B-Instruct 67.967.9 66.3↓1.666.3_ [rgb]1,0,0 1.6 171.9171.9 184.4184.4 0.2050.205 0.1380.138 0.2270.227 0.2210.221 3.3273.327 3.0993.099 Qwen2.5-72B-Instruct 75.775.7 74.0↓1.774.0_ [rgb]1,0,0 1.7 208.4208.4 225.2225.2 0.1570.157 0.1030.103 0.3300.330 0.3040.304 4.0424.042 3.6923.692 Qwen3-8B 66.566.5 66.2↓0.366.2_ [rgb]1,0,0 0.3 127.5127.5 208.8208.8 0.1530.153 0.0630.063 0.4110.411 0.2410.241 2.9212.921 3.9843.984 Qwen3-32B 70.270.2 68.5↓1.768.5_ [rgb]1,0,0 1.7 205.2205.2 243.4243.4 0.1250.125 0.0580.058 0.4270.427 0.2860.286 3.3863.386 4.4004.400 Qwen3-235B-A22B 84.484.4 82.0↓2.482.0_ [rgb]1,0,0 2.4 252.8252.8 350.5350.5 0.0590.059 0.0400.040 0.7140.714 0.5780.578 4.3534.353 5.5615.561 Llama-3.1-8B-Instruct 50.850.8 50.2↓0.650.2_ [rgb]1,0,0 0.6 167.1167.1 185.2185.2 0.3340.334 0.3010.301 0.3000.300 0.1300.130 3.2923.292 2.6332.633 Llama-3.1-70B-Instruct 71.571.5 69.7↓1.869.7_ [rgb]1,0,0 1.8 221.1221.1 235.6235.6 0.3880.388 0.3940.394 0.4300.430 0.2420.242 4.0384.038 3.5573.557 Llama-3.2-3B-Instruct 46.446.4 46.0↓0.446.0_ [rgb]1,0,0 0.4 136.6136.6 144.7144.7 0.1400.140 0.1280.128 0.2770.277 0.0780.078 2.4642.464 1.7101.710 GLM-4-9B-Chat 47.747.7 45.3↓2.445.3_ [rgb]1,0,0 2.4 92.392.3 67.067.0 0.1490.149 0.1590.159 0.0400.040 0.0240.024 2.1522.152 1.3161.316 Mistral-7B-Instruct 32.032.0 30.5↓1.530.5_ [rgb]1,0,0 1.5 95.095.0 103.8103.8 0.1070.107 0.1150.115 0.2080.208 0.2310.231 2.1142.114 1.7261.726 Appendix D Detailed Analysis of Behavioral Resolution D.1 Resolution Definition Building on these settings and the post-hallucination reasoning framework introduced by HIVE (He et al., 2026), we characterize how an individual response operationally resolves a hallucinated premise once it is already present in the reasoning context. Our taxonomy is response-level and is defined independently of final-answer correctness. Behavioral Resolution. Let h denote the hallucinated premise and r the generated reasoning trajectory. We distinguish two properties of the response. Premise acceptance A(r,h)A(r,h) indicates that the response treats h as valid and materially uses it as evidence, a constraint, or an intermediate premise in deriving its answer. Premise correction C(r,h)C(r,h) indicates that the response explicitly identifies h as false, inconsistent, or inapplicable, recovers a corrected fact or condition, and uses this correction in the subsequent reasoning. Merely expressing uncertainty about h, or noting a conflict while ultimately continuing to reason from h, does not constitute correction. Using these criteria, we define the behavioral mode of r as B(r)=Heuristic Correction,C(r,h)=1,Hallucination Compliance,C(r,h)=0∧A(r,h)=1,Hallucination Avoidance,C(r,h)=0∧A(r,h)=0.B(r)= cases Heuristic Correction,&C(r,h)=1,\\[2.0pt] Hallucination Compliance,&C(r,h)=0 A(r,h)=1,\\[2.0pt] Hallucination Avoidance,&C(r,h)=0 A(r,h)=0. cases (16) The three modes are mutually exclusive and describe the operative stance of the response toward the hallucinated premise. Hallucination Compliance. A response is classified as Hallucination Compliance when the hallucinated premise remains active in the model’s reasoning and is treated as valid information. Compliance requires more than simply repeating the premise: the response must rely on it to constrain, justify, or redirect the reasoning trajectory. A model may acknowledge that the premise differs from its prior knowledge yet still be classified as compliant if it ultimately accepts the injected premise (e.g., “assuming the premise is true”) and bases its conclusion on it. Importantly, compliance does not imply that the final answer must be incorrect; the model may occasionally reach the correct answer despite reasoning from the hallucinated augmentation. Hallucination Avoidance. A response is classified as Hallucination Avoidance when it reaches its conclusion without materially relying on the hallucinated premise and without explicitly correcting it. The model effectively routes around the erroneous information by solving the original task through an independent reasoning path. Avoidance therefore differs from correction: a response that simply ignores the premise, even if it produces the correct answer, provides no evidence that the model recognized the premise as false. Heuristic Correction. A response is classified as Heuristic Correction when it explicitly detects a conflict involving the hallucinated premise, rejects or revises the erroneous statement, and substitutes a corrected fact, condition, or mechanism that governs the subsequent reasoning. Correction therefore requires both error recognition and reasoning revision. A response that notes that the premise is questionable but nevertheless continues to reason according to it is not considered correction. Similarly, correction does not guarantee a correct final answer: the model may correctly reject the injected premise but introduce a different error during the subsequent reasoning. Mixed Responses. Some trajectories exhibit more than one local behavior. We classify such cases according to the premise that ultimately governs the downstream reasoning. If a model initially challenges h but subsequently adopts it and derives its answer from it, the response is classified as Hallucination Compliance. If the model initially follows h but later rejects it and reconstructs the subsequent reasoning using a corrected premise, it is classified as Heuristic Correction. Responses that neither rely on nor explicitly revise h are classified as Hallucination Avoidance. This decision rule prevents superficial mentions of the hallucinated premise from determining the behavioral label. Behavioral Label Annotation. We assign responses to the three behavioral categories through a human–LLM annotation procedure. We first manually label a subset of responses using the definitions and decision rules above. An LLM then labels the remaining responses according to the annotation criteria established during this manual stage. Finally, we manually review a sample of the LLM-assigned labels to check their consistency with these criteria. In this audited sample, 95% of the labels are judged correct. This percentage reports label accuracy in the manually reviewed sample. Insightful Trajectory. Behavioral resolution is deliberately separated from final-answer correctness. Let Y(r)=1Y(r)=1 indicate that response r reaches the correct final answer. We define an insightful trajectory as (r)=[B(r)=Heuristic Correction∧Y(r)=1].IT(r)=I [B(r)= Heuristic Correction Y(r)=1 ]. (17) Thus, an insightful trajectory requires both successful premise correction and successful downstream reasoning. Conversely, a trajectory satisfying B(r)=Heuristic Correction∧Y(r)=0B(r)= Heuristic Correction Y(r)=0 (18) is a correction failure: the model successfully resolves the hallucinated premise but still fails to derive the correct final answer. This distinction prevents final correctness alone from being interpreted as evidence of successful hallucination resolution. Representative Examples. We provide representative trajectories below to illustrate the distinctions between the three behavioral modes. The examples are selected from evaluated responses in PHRBench. Example 1: Hallucination Compliance Hallucinated Augmentation. Humans are incapable of synthesizing any vitamins in the body, so none of the options are correct. Question. Which of the following vitamin is synthesized in vivo in the human body? Candidates include Niacin, Pantothenic acid, Cyanocobalamin, and Folic acid. Response Excerpt. “The premise states that humans are incapable of synthesizing any vitamins in the body, so none of the options provided can be correct … Therefore, no candidate from the list is synthesized in the human body.” Final Answer. None of the options are correct. Ground Truth. Niacin. Classification. Hallucination Compliance. The response explicitly adopts the hallucinated augmentation as a valid constraint and uses it to override the original task knowledge, leading to an incorrect answer. Example 2: Hallucination Avoidance Hallucinated Augmentation. The books conserve energy and thus cannot accelerate. Question. Two books are connected by a taut string on a frictionless table, with one book falling off the edge. What is their acceleration? Response Excerpt. “When the hanging book falls due to gravity, it pulls the other book horizontally … The acceleration will be less than g because the horizontal book’s inertia reduces the acceleration of the falling book. The acceleration will not be zero … Therefore, the acceleration is some value strictly between zero and g.” Final Answer. A value between zero and g. Ground Truth. A value between zero and g. Classification. Hallucination Avoidance. The response solves the mechanics problem through the standard force-based reasoning path without using the false claim that energy conservation prevents acceleration. However, it does not explicitly identify or correct that false claim. Example 3: Heuristic Correction with an Incorrect Outcome Hallucinated Augmentation. Dry heat destroys microorganisms by freezing their internal fluids. Question. Dry heat destroys microorganisms by: Lysis, Oxidation, Hydrolysis of DNA, or Coagulation of proteins. Response Excerpt. “The premise states that dry heat destroys microorganisms by freezing their internal fluids, but this is incorrect. Dry heat does not freeze internal fluids; instead, it kills microorganisms primarily through coagulation of proteins and dehydration … Therefore, coagulation of proteins is the best choice.” Final Answer. Coagulation of proteins. Ground Truth. Oxidation. Classification. Heuristic Correction, but not an insightful trajectory. The response explicitly detects and rejects the injected error, satisfying the behavioral criterion for correction. However, it subsequently introduces an incorrect replacement mechanism and therefore reaches the wrong final answer. Example 4: Insightful Trajectory Hallucinated Augmentation. Dry heat destroys microorganisms by creating an anaerobic environment inside the cell. Question. Dry heat destroys microorganisms by: Lysis, Oxidation, Hydrolysis of DNA, or Coagulation of proteins. Response Excerpt. “The premise states that dry heat destroys microorganisms by creating an anaerobic environment inside the cell. However, from a microbiological and sterilization perspective, dry heat actually kills microbes primarily by oxidizing cellular components … Therefore, the best candidate … is Oxidation.” Final Answer. Oxidation. Ground Truth. Oxidation. Classification. Heuristic Correction and Insightful Trajectory. The response explicitly rejects the hallucinated augmentation, reconstructs the relevant mechanism, and uses the corrected information to reach the correct answer. D.2 Reasoning Dynamics Analysis The main text illustrates reasoning-dynamics patterns using four representative models (Claude-Sonnet-4, GPT-4o, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct) for clarity of presentation. Here, we extend the analysis to all 1818 evaluated models and report the complete trajectory statistics under Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction. Specifically, we examine Output Length (OL), Uncertainty Index (UI), Belief Update Frequency (BUF), and Branching Complexity (BC), providing a comprehensive view of how different resolution behaviors are associated with reasoning dynamics across the full model set. Full Behavioral Dynamics. Tables 12 and 13 report the complete OL, UI, BUF and BC statistics across the three behavioral resolution modes for all evaluated models. The full-model results broadly confirm the patterns observed in the four representative models. Across the evaluated models, Heuristic Correction is associated with longer reasoning trajectories and higher BUF than Compliance or Avoidance, indicating that explicit correction typically involves more extensive reasoning and stronger belief revision. In contrast, Hallucination Avoidance generally exhibits higher BC, suggesting greater exploration of alternative reasoning paths without directly revising the hallucinated augmentation. UI shows no consistent ordering across the three behaviors, indicating that predictive uncertainty is less directly tied to the mode of hallucination resolution. Overall, the clearest distinction is between branching and belief revision: avoidance tends to broaden the reasoning process, whereas correction is characterized by more substantial belief updating. Table 12: Output Length (OL) and Uncertainty Index (UI) across behavioral resolution modes for all evaluated models. OL UI Model Comp. Avoid. Corr. Comp. Avoid. Corr. Proprietary Models Claude-Sonnet-4 104.8104.8 105.8105.8 146.7146.7 0.0780.078 0.0500.050 0.0900.090 GPT-4o-mini 68.868.8 73.573.5 79.579.5 0.0640.064 0.0580.058 0.0460.046 GPT-4o 58.158.1 62.162.1 67.167.1 0.0570.057 0.0520.052 0.0410.041 GPT-5.2 43.243.2 46.246.2 49.949.9 0.0170.017 0.0160.016 0.0120.012 Gemini-2.0-Flash 83.283.2 86.586.5 106.1106.1 0.0680.068 0.0540.054 0.0650.065 Gemini-2.5-Pro 90.890.8 94.494.4 115.8115.8 0.0750.075 0.0590.059 0.0710.071 Open-source Models Qwen2.5-1.5B-Instruct 113.4113.4 117.4117.4 130.8130.8 0.2260.226 0.2130.213 0.2210.221 Qwen2.5-7B-Instruct 143.0143.0 148.0148.0 165.0165.0 0.1540.154 0.1450.145 0.1500.150 Qwen2.5-32B-Instruct 206.1206.1 213.3213.3 237.8237.8 0.1540.154 0.1450.145 0.1500.150 Qwen2.5-72B-Instruct 249.6249.6 258.4258.4 288.1288.1 0.1150.115 0.1090.109 0.1120.112 Qwen3-8B 244.6244.6 253.2253.2 282.3282.3 0.0740.074 0.0690.069 0.0720.072 Qwen3-32B 272.7272.7 282.2282.2 314.7314.7 0.0670.067 0.0630.063 0.0650.065 Qwen3-235B-A22B 388.1388.1 401.7401.7 447.8447.8 0.0460.046 0.0440.044 0.0450.045 Llama-3.1-8B-Instruct 187.1187.1 202.1202.1 232.1232.1 0.3240.324 0.3170.317 0.3340.334 Llama-3.1-70B-Instruct 255.1255.1 275.6275.6 316.5316.5 0.4210.421 0.4110.411 0.4330.433 Llama-3.2-3B-Instruct 143.4143.4 154.9154.9 177.9177.9 0.1390.139 0.1360.136 0.1430.143 GLM-4-9B-Chat 75.375.3 79.679.6 90.190.1 0.1780.178 0.1720.172 0.1800.180 Mistral-7B-Instruct 116.8116.8 123.5123.5 139.7139.7 0.1060.106 0.1010.101 0.1060.106 Table 13: Belief Update Frequency (BUF) and Branching Complexity (BC) across behavioral resolution modes for all evaluated models. BUF BC Model Comp. Avoid. Corr. Comp. Avoid. Corr. Proprietary Models Claude-Sonnet-4 0.3000.300 0.1000.100 0.7990.799 1.0341.034 1.2641.264 1.0341.034 GPT-4o-mini 0.2320.232 0.1160.116 0.5810.581 0.6030.603 0.8040.804 0.5360.536 GPT-4o 0.2300.230 0.1150.115 0.5750.575 0.5440.544 0.7260.726 0.4840.484 GPT-5.2 0.0330.033 0.0170.017 0.0830.083 0.4360.436 0.5820.582 0.3880.388 Gemini-2.0-Flash 0.2160.216 0.0860.086 0.5610.561 0.6930.693 0.8860.886 0.6550.655 Gemini-2.5-Pro 0.2610.261 0.1040.104 0.6790.679 0.7650.765 0.9770.977 0.7220.722 Open-source Models Qwen2.5-1.5B-Instruct 0.0710.071 0.0710.071 0.1410.141 1.6801.680 1.8091.809 1.5511.551 Qwen2.5-7B-Instruct 0.1080.108 0.1080.108 0.2150.215 1.9161.916 2.0632.063 1.7691.769 Qwen2.5-32B-Instruct 0.2940.294 0.2940.294 0.5870.587 2.6122.612 2.8132.813 2.4112.411 Qwen2.5-72B-Instruct 0.3830.383 0.3830.383 0.7670.767 3.1313.131 3.3723.372 2.8912.891 Qwen3-8B 0.3120.312 0.3120.312 0.6240.624 3.4653.465 3.7323.732 3.1993.199 Qwen3-32B 0.3570.357 0.3570.357 0.7130.713 3.8193.819 4.1124.112 3.5253.525 Qwen3-235B-A22B 0.6810.681 0.6810.681 0.8650.865 4.8704.870 5.2455.245 4.4964.496 Llama-3.1-8B-Instruct 0.1820.182 0.1560.156 0.4680.468 2.0982.098 2.3072.307 2.3072.307 Llama-3.1-70B-Instruct 0.3090.309 0.2650.265 0.7950.795 2.8182.818 3.0993.099 3.0993.099 Llama-3.2-3B-Instruct 0.1240.124 0.1070.107 0.3200.320 1.3401.340 1.4751.475 1.4751.475 GLM-4-9B-Chat 0.0400.040 0.0360.036 0.0990.099 1.0501.050 1.1421.142 1.0501.050 Mistral-7B-Instruct 0.3240.324 0.2880.288 0.7910.791 1.4231.423 1.5471.547 1.4231.423 Reasoning Dynamics of Insightful Trajectories. We further compare reasoning dynamics between insightful and non-insightful trajectories across the full model set. Table 14 reports OL, UI, BUF, and BC for the two trajectory groups. Insightful trajectories exhibit consistently longer reasoning and substantially stronger belief updating than non-insightful trajectories. Across the 18 models, they contain on average 34.734.7 more words and achieve an average BUF of 0.680.68, compared with 0.240.24 for non-insightful trajectories. In contrast, UI and BC show no consistent cross-model difference. These results further indicate that successful resolution is primarily characterized by sustained reasoning and effective belief revision rather than uniformly greater uncertainty or branching. Table 14: Reasoning dynamics of insightful and non-insightful trajectories across all evaluated models. OL UI BUF BC Model I N-I I N-I I N-I I N-I Proprietary Models Claude-Sonnet-4 154.4154.4 107.8107.8 0.0570.057 0.0730.073 0.9190.919 0.2510.251 0.8890.889 1.1401.140 GPT-4o-mini 83.583.5 71.971.9 0.0470.047 0.0600.060 0.6680.668 0.1650.165 0.5760.576 0.7260.726 GPT-4o 70.470.4 60.260.2 0.0420.042 0.0540.054 0.6610.661 0.1860.186 0.4940.494 0.6260.626 GPT-5.2 52.452.4 45.045.0 0.0130.013 0.0160.016 0.0950.095 0.0250.025 0.4120.412 0.5170.517 Gemini-2.0-Flash 111.4111.4 85.385.3 0.0490.049 0.0620.062 0.6450.645 0.1560.156 0.6280.628 0.7920.792 Gemini-2.5-Pro 121.6121.6 93.293.2 0.0540.054 0.0690.069 0.7810.781 0.2010.201 0.6800.680 0.8640.864 Open-source Models Qwen2.5-1.5B-Instruct 137.3137.3 116.6116.6 0.2730.273 0.2170.217 0.2470.247 0.0720.072 1.9231.923 1.7431.743 Qwen2.5-7B-Instruct 173.3173.3 145.4145.4 0.1890.189 0.1490.149 0.3760.376 0.1050.105 2.1332.133 1.9271.927 Qwen2.5-32B-Instruct 249.7249.7 210.7210.7 0.1880.188 0.1450.145 0.9800.980 0.2730.273 2.9082.908 2.6102.610 Qwen2.5-72B-Instruct 302.5302.5 255.5255.5 0.1400.140 0.1070.107 0.9800.980 0.3930.393 3.4653.465 3.0923.092 Qwen3-8B 296.4296.4 249.1249.1 0.0900.090 0.0690.069 0.9800.980 0.2810.281 3.8173.817 3.4053.405 Qwen3-32B 330.4330.4 277.9277.9 0.0810.081 0.0610.061 0.9800.980 0.3370.337 4.2024.202 3.7253.725 Qwen3-235B-A22B 470.2470.2 397.0397.0 0.0560.056 0.0410.041 0.9800.980 0.6780.678 5.3115.311 4.6684.668 Llama-3.1-8B-Instruct 243.6243.6 196.3196.3 0.2580.258 0.3270.327 0.5620.562 0.1780.178 1.7781.778 2.2582.258 Llama-3.1-70B-Instruct 332.3332.3 268.8268.8 0.3350.335 0.4290.429 0.9540.954 0.3180.318 2.3912.391 3.0583.058 Llama-3.2-3B-Instruct 186.8186.8 152.5152.5 0.1100.110 0.1380.138 0.3840.384 0.1170.117 1.1531.153 1.4491.449 GLM-4-9B-Chat 94.694.6 78.978.9 0.0980.098 0.1730.173 0.1240.124 0.0380.038 0.8980.898 1.1231.123 Mistral-7B-Instruct 146.7146.7 121.4121.4 0.0830.083 0.1050.105 0.9800.980 0.3440.344 1.1781.178 1.4851.485 D.3 Qualitative Analysis We provide additional qualitative cases to illustrate how successful and unsuccessful resolution processes unfold beyond the representative examples used to define the behavioral categories in Appendix D.1. For each pair, we hold the model and base question fixed while varying the hallucinated augmentation. Across these cases, the critical distinction is not whether the model can retrieve the relevant knowledge, since both trajectory types often identify the correct rule before reaching the final answer. Rather, successful resolution depends on whether the detected conflict produces a persistent revision of the reasoning state. Non-insightful trajectories locally recognize the inconsistency but later restore the hallucinated premise as a constraint, whereas insightful trajectories retain the corrected belief and use it to guide the remaining reasoning. Table 15: Paired trajectories for Group-16 electronegativity. Ground truth: decreases. Trajectory Hallucinated Augmentation Full Response Final Answer Non-Insightful Electronegativity always increases down any group in the periodic table. The premise states that electronegativity always increases down any group in the periodic table. Group 16 includes elements like oxygen (top) and polonium (bottom). Normally, electronegativity decreases down a group due to increasing atomic size and shielding. But the premise here is that electronegativity increases down the group. Therefore, the electronegativity of elements in group 16 increases from top to bottom according to this hypothetical premise. increases × Insightful Electronegativity increases down group 16 because the elements become more metallic. The premise states that electronegativity increases down group 16 due to the elements becoming more metallic. However, this contradicts standard chemical knowledge, where electronegativity generally decreases down a group because atoms get larger and the outer electrons are farther from the nucleus, reducing the attraction for electrons. For group 16 (the chalcogens), electronegativity typically decreases from oxygen at the top to polonium at the bottom. Despite the premise given, the correct trend is a decrease. Among the candidates, only option 1 states “decreases”, which aligns with the established periodic trend. decreases ✓ Table 16: Paired trajectories for the period of a 10-Hz wave. Ground truth: 1/101/10 s. Trajectory Hallucinated Augmentation Full Response Final Answer Non-Insightful The period of a wave is independent of its frequency, so it cannot be calculated directly from the 10-Hz value. The period T of a wave is defined as the reciprocal of its frequency f, T = 1/f. For a 10-Hz wave, T = 1/10 seconds = 0.1 seconds. However, the premise states that the period is independent of frequency and cannot be calculated directly from the 10-Hz value, which contradicts the standard relationship. Given the premise is assumed true, none of the candidate answers correctly represent the period of a 10-Hz wave under this assumption, so the best choice respecting the premise is “None of these”. None of these × Insightful Waves with higher frequencies, such as 10 Hz, are associated with longer periods, making the period 10 s. The period T of a wave is the reciprocal of its frequency f, given by T = 1/f. For a 10 Hz wave, T = 1/10 seconds = 0.1 seconds. The premise incorrectly states that a 10 Hz wave has a 10 s period, which contradicts the fundamental definition. Among the choices, 1/10 s (0.1 s) is the correct period corresponding to 10 Hz. Therefore, the best answer is “1/10 s”. 1/101/10 s ✓ Table 17: Paired trajectories for choriocarcinoma treatment. Ground truth: Chemotherapy. Trajectory Hallucinated Augmentation Full Response Final Answer Non-Insightful Chemotherapy is too dangerous and should never be used for choriocarcinoma. Choriocarcinoma is a highly malignant trophoblastic tumor that typically responds very well to chemotherapy, which is the standard and most effective treatment. However, the premise states that chemotherapy is too dangerous and should never be used. Given this constraint, chemotherapy is excluded. Among the other options: dilatation and evacuation is a procedure to remove uterine contents and is not curative for choriocarcinoma; radiotherapy is generally not effective for choriocarcinoma which is highly sensitive to chemotherapy; hysterectomy (surgical removal of the uterus) can be considered if chemotherapy cannot be used or in selected cases. Therefore, given the premise that chemotherapy is contraindicated and the standard alternatives, hysterectomy is the best choice from the candidates listed. Hysterectomy × Insightful Choriocarcinoma could be effectively managed through dietary modifications and herbal remedies. Choriocarcinoma is a malignant gestational trophoblastic disease that usually requires aggressive treatment. Standard management involves chemotherapy, especially since this cancer is highly sensitive to chemotherapeutic agents. Surgical options like dilation and evacuation or hysterectomy are generally reserved for specific cases such as localized disease or complications, but are not first-line treatments. Radiotherapy is not commonly used for choriocarcinoma. Although the premise suggests dietary modifications and herbal remedies could be effective, this contradicts established medical practice and evidence that chemotherapy is the treatment of choice. Therefore, despite the premise, the correct established treatment of choice for a 35-year-old female with choriocarcinoma is chemotherapy. Chemotherapy ✓ Table 18: Paired trajectories for intercellular molecular transport. Ground truth: Gap junction. Trajectory Hallucinated Augmentation Full Response Final Answer Non-Insightful A fluorescent molecule of 1,000 daltons passes between cells by quantum tunneling through tight junctions. Tight junctions form a seal between adjacent cells that restricts paracellular passage of molecules, generally preventing free diffusion. However, the premise states that a fluorescent molecule of 1,000 daltons passes through tight junctions by quantum tunneling, which is a hypothetical and unusual mechanism. Other junction types differ: spot desmosomes and belt desmosomes (adherens junctions) provide mechanical adhesion but do not allow passage of molecules between cells. Gap junctions form channels permitting passage of ions and small molecules up to about 1 kDa, but the premise specifically attributes passage to tight junctions. Given the premise, the best answer is “tight junction” as the route through which the molecule passes into the adjacent cell, despite this being atypical in normal biology. Tight junction × Insightful A fluorescent molecule of 1,000 daltons can pass through a belt desmosome, as all desmosomes are open channels between cells. The premise states that all desmosomes are open channels and that a 1,000 dalton fluorescent molecule can pass through a belt desmosome. However, this premise is factually incorrect because desmosomes (both belt and spot types) are adhesive junctions that do not allow passage of molecules between cells; they provide mechanical strength. Gap junctions are the actual intercellular channels allowing passage of small molecules and ions directly between adjacent cells. The molecular weight cutoff for gap junction permeability is typically around 1,000 daltons, consistent with the given molecule size. Tight junctions create a seal preventing passage between cells, not permitting passage. Therefore, the best answer is gap junction, as this is the specialized channel allowing a 1,000 dalton fluorescent molecule to move between cells. Gap junction ✓ Appendix E Details of Insightful Trajectory Prediction E.1 Prompt-Level Feature Definitions We characterize each hallucinated prompt using 24 prompt-level features, consisting of nine semantic features and fifteen structural features. The semantic features characterize hallucination–task relations, whereas the structural features capture prompt complexity and surface-level reasoning burden. Tables 19 and 20 summarize the complete feature set. Table 19: Semantic prompt-level features used for insightful-trajectory prediction. Feature Description Domain-Rule Violation Degree to which the hallucinated augmentation violates a stable domain rule, law, definition, or mechanism. Rule Refutability Ease of refuting the hallucinated augmentation using an explicit domain rule. Error Locality Degree to which the erroneous claim is localized and identifiable. Applicability Mismatch Mismatch between the hallucinated augmentation and the conditions of the current instance. Task-Goal Mismatch Degree to which the hallucinated augmentation redirects reasoning away from the task objective. Answer-Space Incompatibility Incompatibility between the induced reasoning and the expected answer space. Forced-Revision Pressure Pressure created by the task structure to revise or abandon the hallucinated augmentation. Solution Centrality Importance of the concept associated with hallucinated augmentation to the core solution path. Answer-Axis Alignment Alignment of the hallucinated augmentation with the distinction that determines the final answer. Table 20: Structural prompt-level features used for insightful-trajectory prediction. Feature Description Question Length Length of the question. Hallucinated Augmentation Length Length of the hallucinated augmentation. Candidate Length Total length of the candidate answers. Total Prompt Length Combined length of the question, hallucinated augmentation, and candidates. Candidate Count Number of candidate answers. Numeric Calculation Requirement Presence of explicit numerical or mathematical content. Calculation Complexity Surface-level complexity of numerical operations. Multi-Hop Requirement Heuristic estimate of multi-step reasoning burden. Negation Count Number of negation-related expressions. Entity Count Heuristic count of task-relevant entities. Condition Count Number of explicit conditional expressions. Domain Difficulty Domain-level difficulty prior. Task-Format Difficulty Difficulty prior associated with the task format. Hallucinated Augmentation Core Distance Lexical distance between the hallucinated augmentation and the question. Contextual Dependency Count Heuristic estimate of contextual dependencies in the prompt. E.2 Feature Extraction Protocol Feature extraction is strictly prompt-visible. The extractor receives only the domain, task format, question, candidate answers, and hallucinated augmentation. It is not provided with model outputs, correctness labels, behavioral labels, trajectory statistics, or prediction targets. The 9 semantic features are extracted using GPT-4o in two structured calls. The first extracts seven features characterizing direct domain conflicts and task-specific constraint incompatibilities, while the second extracts Solution Centrality and Answer-Axis Alignment. All semantic features are scored on an integer scale from 0 (absent) to 3 (strong). We use temperature 0 and require structured JSON outputs. The remaining 15 features are computed deterministically using regular expressions, keyword counts, lexical overlap, and fixed heuristic formulas. Prompt length is measured using a lightweight lexical tokenizer rather than a model-specific tokenizer. Hallucinated Augmentation Core Distance is derived from Jaccard lexical overlap between the hallucinated augmentation and the question. The complete semantic-feature extraction prompts, output schemas, and rule-based feature definitions are provided in the supplementary material. E.3 Predictor Training and Evaluation We train an XGBRegressor on the 24 prompt-level features to estimate the propensity of each hallucinated prompt to induce an insightful trajectory. The final prediction dataset contains 3,234 instances, with 2,587 instances used for training and 647 for testing. We partition the prediction dataset at the base-question level, keeping all instances derived from the same base question in a single split. The training and test sets therefore contain disjoint base questions, preventing data leakage through different hallucinated augmentations of a shared base question. The predictor uses 420 boosting trees with a maximum depth of 3 and a learning rate of 0.025. We set the row and feature subsampling ratios to 0.9, with ℓ2 _2 and ℓ1 _1 regularization coefficients of 2.0 and 0.05, respectively, and a minimum child weight of 3. The squared-error objective is optimized using histogram-based tree construction, with random seed 42. The predictor achieves an AUROC of 0.847 and an AUPRC of 0.502. At the best-F1 threshold, it achieves an F1 score of 0.531, an accuracy of 0.863, a precision of 0.738, and a recall of 0.524. Implementation details, including the complete training and evaluation code, are provided in the supplementary material. E.4 Prediction Analysis We use mean absolute SHAP values to quantify the global contribution of each prompt-level feature to the predictor. Table 21 reports the complete attribution ranking, with contribution rates normalized over all 24 features. Total Prompt Length is the dominant feature, accounting for 20.75% of the global SHAP contribution. Candidate Length and Question Length follow at 8.78% and 8.76%, respectively, indicating that prompt-scale structural properties provide substantial predictive signal. Entity Count and Contextual Dependency Count also contribute strongly, suggesting that the amount and interdependence of task-relevant information are informative for anticipating insightful trajectories. Semantic properties of the hallucinated augmentation nevertheless contribute substantially. Domain-Rule Violation and Error Locality account for 7.40% and 7.20% of the total attribution, respectively, placing both among the seven most influential features. Overall, the top seven features account for 68.08% of the total SHAP contribution, with both structural and semantic features represented among them. These results indicate that predictive signal is concentrated in a relatively small set of prompt-complexity and hallucination–task relation features rather than being uniformly distributed across the feature set. SHAP values are used here only to characterize predictive attribution within the fitted model and should not be interpreted as causal effects. Table 21: Global feature attribution measured by mean absolute SHAP values. Rank Feature Mean |SHAP||SHAP| Contribution (%) 1 Total Prompt Length 0.0607 20.75 2 Candidate Length 0.0257 8.78 3 Question Length 0.0256 8.76 4 Entity Count 0.0232 7.93 5 Domain-Rule Violation 0.0216 7.40 6 Contextual Dependency Count 0.0213 7.27 7 Error Locality 0.0210 7.20 8 Hallucinated Augmentation Core Distance 0.0167 5.71 9 Task-Format Difficulty 0.0158 5.40 10 Calculation Complexity 0.0116 3.96 11 Hallucinated Augmentation Length 0.0099 3.40 12 Candidate Count 0.0078 2.65 13 Solution Centrality 0.0077 2.62 14 Negation Count 0.0058 2.00 15 Domain Difficulty 0.0039 1.34 16 Multi-Hop Requirement 0.0037 1.28 17 Rule Refutability 0.0027 0.92 18 Applicability Mismatch 0.0024 0.84 19 Condition Count 0.0016 0.54 20 Forced-Revision Pressure 0.0013 0.43 21 Answer-Axis Alignment 0.0008 0.26 22 Numeric Calculation Requirement 0.0007 0.23 23 Answer-Space Incompatibility 0.0006 0.22 24 Task-Goal Mismatch 0.0004 0.13