Paper deep dive
Amplified Does Not Mean Predictive: Reasoning Behaviors in Thinking Models
Jean de Dieu Nyandwi, Leena Mathur, Yonatan Bisk, Robert Hawkins, Graham Neubig
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/17/2026, 4:17:39 AM
Summary
This paper investigates the relationship between reasoning behaviors and model correctness in 'thinking' models (trained with extended reasoning traces) versus instruction-tuned models. The authors introduce 'Behavioral Lift,' a metric measuring how much correctness changes when a behavior is present versus absent. They identify an 'Amplification-Lift Gap': thinking models strongly amplify behaviors like self-correction, hypothesis testing, and uncertainty acknowledgment, but these behaviors have low or negative lift (association with correctness). Conversely, high-lift behaviors like confidence calibration, knowledge alignment, and self-awareness are barely amplified. The study concludes that reasoning-oriented training does not preferentially amplify the behaviors most tied to correctness, suggesting a need for process-level objectives that reward calibrated reasoning over surface form.
Entities (22)
Relation Signals (10)
Confidence Calibration → hashighlift → Correctness
confidence 96% · Confidence calibration is among the strongest positive signals of correctness in both modalities
Behavioral Lift → measuresassociationwith → Correctness
confidence 96% · Behavioral Lift, a metric that measures how much correctness changes when a behavior is present versus absent
Thinking Models → amplifies → Self-Correction
confidence 95% · Thinking models strongly amplify self-correction, hypothesis testing, and uncertainty acknowledgment
Thinking Models → amplifies → Uncertainty Acknowledgment
confidence 95% · Thinking models strongly amplify self-correction, hypothesis testing, and uncertainty acknowledgment
Thinking Models → amplifies → Hypothesis Testing
confidence 95% · Thinking models strongly amplify self-correction, hypothesis testing, and uncertainty acknowledgment
Self-Awareness → hashighlift → Correctness
confidence 95% · the highest-lift behaviors are confidence calibration, knowledge alignment, and self-awareness.
Knowledge Alignment → hashighlift → Correctness
confidence 95% · the highest-lift behaviors are confidence calibration, knowledge alignment, and self-awareness.
Uncertainty Acknowledgment → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Which reasoning behaviors are associated with correct answers in reasoning models, and does reasoning-oriented training amplify those behaviors? This distinction is important because reasoning-oriented training can make traces look more deliberative without amplifying the behaviors most tied to model correctness. We quantify this mismatch with Behavioral Lift, a metric that measures how much correctness changes when a behavior is present versus absent in a model's reasoning trace. Across 15 models and 6 benchmarks spanning text-only and vision-language reasoning, we annotate 15,282 traces with a taxonomy whose core behaviors are defined for both LLM and VLM traces. We find evidence for an Amplification-Lift Gap, in which thinking models strongly amplify self-correction, hypothesis testing, and uncertainty acknowledgment, while the highest-lift behaviors are confidence calibration, knowledge alignment, and self-awareness. Confidence calibration is among the strongest positive signals of correctness in both modalities, yet is barely amplified; uncertainty acknowledgment is amplified by 3--7$\times$, yet is weakly or negatively associated with correctness. We find that reasoning-oriented training does not preferentially amplify the highest-Lift behaviors, motivating process-level objectives that reward calibrated and grounded reasoning rather than surface form alone.
Tags
Links
- Source: https://arxiv.org/abs/2608.13760v1
- Canonical: https://arxiv.org/abs/2608.13760v1
Trouble viewing inline? Open PDF directly →
Full Text
135,234 characters extracted from source content.
Expand or collapse full text
Preprint Amplified Does Not Mean Predictive: Reasoning Behaviors in Thinking Models Jean de Dieu Nyandwi 1 , Leena Mathur 1 Yonatan Bisk 1 , Robert Hawkins 2 , Graham Neubig 1 1 Carnegie Mellon University 2 Stanford University jeandedi,gneubig@andrew.cmu.edu https://neulab.github.io/behavioral-lift/ https://github.com/neulab/behavioral-lift https://huggingface.co/datasets/neulab/behavioral-lift Abstract Which reasoning behaviors are associated with correct answers in reason- ing models, and does reasoning-oriented training amplify those behaviors? This distinction is important because reasoning-oriented training can make traces look more deliberative without amplifying the behaviors most tied to model correctness. We quantify this mismatch with Behavioral Lift, a metric that measures how much correctness changes when a behavior is present versus absent in a model’s reasoning trace. Across 15 models and 6 benchmarks spanning text-only and vision-language reasoning, we annotate 15,282 traces with a taxonomy whose core behaviors are defined for both LLM and VLM traces. We find evidence for an Amplification-Lift Gap, in which thinking models strongly amplify self-correction, hypothesis testing, and uncertainty acknowledgment, while the highest-lift behaviors are confidence calibration, knowledge alignment, and self-awareness. Con- fidence calibration is among the strongest positive signals of correctness in both modalities, yet is barely amplified; uncertainty acknowledgment is amplified by 3–7×, yet is weakly or negatively associated with correctness. We find that reasoning-oriented training does not preferentially amplify the highest-Lift behaviors, motivating process-level objectives that reward calibrated and grounded reasoning rather than surface form alone. 1 Introduction Recent progress in reasoning-oriented post-training has produced a wave of “thinking” models: language and vision-language models trained to generate extended reasoning traces before answering. OpenAI’s o1 (OpenAI, 2024), DeepSeek-R1 (Guo et al., 2025), Qwen3 (Yang et al., 2025), Kimi-k1.5 (Team et al., 2025), and others now routinely produce long chains of reasoning on difficult problems and often outperform their instruction-tuned counterparts. Longer traces, however, also make it easier to confuse deliberation with reliable reasoning. Accuracy indicates whether a model gets the answer correct; it does not indicate which behaviors the model used, which failures it encountered, or whether the behaviors more common in thinking traces are the ones most associated with correctness. In this paper, we study whether thinking models amplify the reasoning behaviors most associated with correct answers. The distinction is important because a behavior can become more frequent without becoming more diagnostic of successful reasoning. A model may exhibit more hedging behaviors because it is confused, self-correct more because it made an earlier mistake, or branch over hypotheses without testing the right ones. Conversely, less prevalent behaviors, such as calibrating confidence to the strength of the reasoning or applying the appropriate domain framework, may be more predictive of success. Recent work has begun to analyze reasoning traces through cognitively-inspired taxonomies, trainability signals, and strategy discovery from chain-of-thought traces (Gandhi et al., 2025; Kargupta et al., 2025; Lee et al., 2025). We go beyond describing which behaviors appear 1 arXiv:2608.13760v1 [cs.CL] 13 Aug 2026 Preprint Figure 1: The disconnect between what thinking-oriented training amplifies and what predicts success. Each point is one of the nine cross-modal higher-order behaviors, averaged across VLMs and LLMs (N=15,282). The top-right quadrant is empty: the behaviors most amplified by thinking training are not the ones most associated with correctness. in reasoning traces by comparing thinking and instruct models and separating behavioral prevalence from behavioral lift. To separate prevalence from lift, we annotate reasoning traces with a cross-modal behavioral taxonomy spanning reasoning behaviors, failure modes, reasoning-quality labels, reasoning-type labels, summary metrics, and, for multimodal VLMs, visual grounding. We formalize two metrics: (1) Behavioral Lift, which measures how much correctness differs when a behavior is present versus absent in a model’s trace; (2) Recovery Rate, which measures how often a model reaches the correct answer, despite exhibiting at least one reasoning failure. We use our taxonomy to evaluate 15 models from 7 families across 6 benchmarks spanning a task spectrum from pure visual puzzles (VisualPuzzles) (Song et al., 2025) and logical reasoning (LogiQA2) (Liu et al., 2020), through mathematical reasoning (MathVista, MATH-500) (Lu et al., 2024; Hendrycks et al., 2021), to knowledge-intensive QA (MMMU, MMLU-Pro) (Yue et al., 2024; Wang et al., 2024). Using LLM-as-judge annotation under our taxonomy, we produce 15,282 behavioral profiles. Our analysis reveals an Amplification-Lift Gap, in which the behaviors most amplified by thinking training are not the behaviors most associated with correctness. As shown in Figure 1, no behavior is both strongly amplified and strongly predictive of success. Thinking models selectively amplify self-correction, hypothesis testing, and uncertainty acknowledg- ment, while leaving six other behaviors, including confidence calibration and knowledge alignment, largely unchanged. Confidence calibration is one of the strongest positive signals of correctness in both modalities (+72–80%), while uncertainty acknowledgment is strongly amplified but weakly or negatively associated with correctness. Thinking models recover better on extended-reasoning tasks and make fewer failures on knowledge-heavy tasks, while instruct models can outperform on tasks where fast recognition of logical form is sufficient for success. This paper makes four contributions: •We introduce Behavioral Lift, a metric that separates how often a behavior appears from how strongly it is associated with reasoning correctness. •We introduce a cross-modal behavioral taxonomy with 9 behaviors and 15,282 annotated traces from 15 LLMs and VLMs across 6 benchmarks. •We contribute empirical evidence for an Amplification-Lift Gap, finding that thinking-oriented models do not exhibit the highest-Lift behaviors. •We establish recovery as a mechanism behind thinking-model gains; we find that thinking helps when tasks reward extended computation or recovery from failures. 2 Preprint We release all annotation prompts, metrics code, and behavioral annotations. 2 Taxonomy and Metrics We develop a behavioral taxonomy that characterizes reasoning traces along multiple dimensions, and we define two metrics that capture aspects of reasoning quality. The taxonomy includes cross-modal reasoning behaviors, failure modes, reasoning-quality labels, reasoning-type labels, summary metrics, and, for VLMs, visual grounding. 2.1 Design Principles The following three principles guide the taxonomy: (1) we annotate the reasoning process, in addition to the final answer, distinguishing valid logic from lucky guesses; (2) we define nine higher-order behaviors across modalities using modality-neutral language; (3) we distinguish behavioral presence from Behavioral Lift, since behaviors that appear frequently are not always the ones most associated with success. 2.2 Taxonomy Structure The taxonomy is organized into six groups, with full definitions in Appendix Tables 24 and 26. Two groups drive the main analyses in this paper: the nine higher-order behaviors, which support cross-modal comparison, and the failure modes, which support the Recovery Rate analysis. The remaining groups provide modality-specific grounding and descriptive context: reasoning quality distinguishes sound reasoning from lucky guesses, reasoning types characterize the form of reasoning used, summary metrics provide compact trace-level summaries, and visual grounding captures image use in VLM traces. We analyze these VLM-specific grounding labels separately in Appendix G. Higher-order behaviors (9, defined for both modalities). Following distinctions in the metacognition literature (Nelson, 1990; Brown, 1987), we organize the nine behaviors (Ta- ble 1) into three functional categories: control/regulation (planning, goal tracking, hypothesis testing, self-correction), monitoring/judgment (uncertainty acknowledgment, confidence cali- bration, self-awareness), and epistemic grounding (evidence citation, knowledge alignment). 1 These three categories are descriptive and situate cross-modal behaviors within the lit- erature, and our empirical analyses group behaviors by how strongly thinking-oriented training amplifies them. All nine behaviors are defined in modality-neutral language (e.g., self-correction refers to the same behavior, whether the model corrects a visual interpreta- tion or a mathematical derivation). These behaviors are the basis for the amplification and Behavioral Lift analyses. Failure Modes (7 per modality, 4 defined for both modalities). Each failure mode is annotated as present or absent (true means the failure occurred). Four failure modes use identical definitions in both LLM and VLM traces: logical failure (invalid inferences), post- hoc rationalization (reasoning reverse-engineered from the answer), shortcut (skipping necessary steps), and lucky guess (correct answer with wrong reasoning). The VLM variant adds failure modes in visual hallucination, visual neglect, and language bias. The LLM variant adds failure modes in factual error, context misread, and knowledge gap. Prior behavioral taxonomies (Gandhi et al., 2025; Kargupta et al., 2025) characterize what models do well, but do not emphasize failure cases. Explicit failure annotation enables our Recovery Rate analysis to measure whether models reach correct answers despite exhibiting failures. 1 Our grouping is motivated by the standard metacognitive distinction between monitoring and control/regulation (Nelson, 1990; Schraw & Dennison, 1994), while treating evidence citation and knowledge alignment as a separate epistemic-grounding family. We define these categories by their observable role in reasoning traces rather than claiming correspondence to specific internal cognitive mechanisms. 3 Preprint BehaviorShort definition PlanningBreaks the problem into explicit sub-steps. Goal trackingTracks progress toward intermediate or final goals. Hypothesis testingConsiders alternatives, cases, or interpretations. Self-correctionRevises or corrects a previous step or claim. Uncertainty ack.Expresses doubt, ambiguity, or confusion. Confidence calibrationCertainty tracks the strength of reasoning. Self-awareness Recognizes missing information or limits of support. Evidence citationGrounds claims in prompt evidence or constraints. Knowledge alignmentUses the appropriate domain facts or framework. Table 1: Nine behaviors used for cross-modal analysis. Full definitions appear in Appendix Tables 24 and 26. 2.3 Metrics We define two metrics to study the questions motivating this paper. Behavioral Lift. How much does correctness in model reasoning differ when a behavior is present? For each behavior b, we compute: Lift(b) = P(correct| b=true)− P(correct| b=false)(1) Positive Lift 2 means the behavior is associated with higher accuracy when present; negative lift means it is associated with lower accuracy. We note that Lift does not imply causation: a behavior may have high Lift as a consequence of the model being on the right track, rather than a cause of correctness. We therefore use Lift as a descriptive measure of reasoning behavior rather than as a causal estimate of a behavior ’s effect. We compute Lift on pooled samples across all models per modality for the main analyses and report per-modality values to assess cross-modal consistency. Supplementary analyses compute the same quantity within individual benchmarks and individual models to assess stability. Recovery Rate. Can a model reach correct answers despite exhibiting reasoning failures? For a set of responses from model m: Recovery(m) = P(correct|∃ f ∈F all : f =true)(2) whereF all includes all 7 failure modes for the relevant modality. A high recovery rate indicates that the model reaches correct answers despite detected failures. 3 Experimental Setup We evaluate 15 models across 6 benchmarks, selecting models to enable matched-family comparisons between thinking and instruct variants and benchmarks to span a spectrum from reasoning to knowledge-intensive tasks. 3.1 Models We select models where both thinking and instruction-tuned variants are publicly available from the same model family, enabling within-family comparison (Table 6 in Appendix). Our evaluation includes 15 open-weight models from 3B to 9B parameters: 4 thinking and 3 non-thinking VLMs, and 4 thinking and 4 non-thinking LLMs, with matched thinking/non- thinking pairs wherever available. 3.2 Benchmarks We select benchmarks to span three task types per modality. For VLMs: VisualPuzzles (visual reasoning), MathVista (visual mathematical reasoning), and MMMU (multimodal knowledge). For LLMs: LogiQA2 (logical reasoning), MATH-500 (mathematical reasoning), 2 We refer to the metric as Behavioral Lift, often shortened to Lift. 4 Preprint and MMLU-Pro (knowledge-intensive tasks across 14 domains). This design creates a task spectrum within each modality, from structure-based to knowledge-heavy reasoning: Structure-focusedMixedKnowledge-heavy VLMVisualPuzzles (350)MathVista (300)MMMU (350) LLMLogiQA2 (350)MATH-500 (350)MMLU-Pro (350) We target 350 responses per benchmark, except MathVista, where we use 300 from the testmini split. Final counts fall slightly below these targets because a small number of responses fail output parsing and are excluded. In total we annotate 15,282 responses. 3.3 Evaluation Protocol Inference. All models are evaluated under standardized conditions with full traces retained for annotation; details appear in Appendix A. Behavioral annotation. We annotate each response using LLM-as-judge (GPT-4o). The judge receives the question, ground-truth answer, and the model’s full output, and produces binary annotations for all behaviors in the relevant taxonomy. For confidence calibration, the judge markstrueonly when the model’s expressed certainty is consistent with the observed strength of its reasoning in the trace, andfalsewhen the trace is clearly overconfident or underconfident relative to that reasoning. The annotation prompt provides explicit definitions and criteria for each behavior (Appendix Pages 41 and 43). Judge validation. To assess annotation reliability, we compare GPT-4o 3 annotations with three independent judge models: DeepSeek-V3 (DeepSeek), Gemini-2.5-Flash (Google), and Gemini-3-Flash (Google) on 600 stratified MATH-500 samples across all 8 LLMs. For the behaviors central to the amplification and Behavioral Lift analyses, agreement is moderate to substantial: self-correction (κ =0.53–0.82), uncertainty acknowledgment (κ =0.71– 0.79), hypothesis testing (κ =0.51–0.59), and confidence calibration (κ =0.56–0.81). Mean agreement across all behaviors ranges from 83.7% to 88.8% (κ =0.46–0.56) across the three judge pairs. As an additional manual check, we verified 120 stratified LLM traces across six models and three benchmarks; agreement with GPT-4o is 95.1% over 720 binary decisions for the six focal behaviors (κ =0.902). More ambiguous behaviors, especially self-awareness and some organizational labels, show lower cross-judge agreement. 4 Results 4.1 Thinking-Oriented Models Amplify Correction, Search, and Hesitation Thinking training selectively amplifies correction, search, and hesitation, while leaving most other core behaviors largely unchanged. Three behaviors are amplified. Self-correction appears in roughly 21–55% of thinking-model responses versus 3–15% for comparison models, with consistently large thinking-minus- comparison gaps across benchmarks (Table 7). Hypothesis testing (22–52% vs. 4–18%) and uncertainty acknowledgment (25–85% vs. 4–28%) show comparable gaps. These gaps hold across all seven model families, both modalities, and all six benchmarks. The remaining behaviors are largely unchanged. Confidence calibration is roughly equal between thinking and instruct models, and on several LLM benchmarks instruct models score slightly higher (66.1% vs. 67.9% on MATH-500; 40.9% vs. 46.7% on MMLU-Pro). Planning and goal tracking are nearly identical on MATH-500 and MMLU-Pro (planning 87– 91% for thinking vs. 84–92% for instruct), with moderate differences on VLM benchmarks. 3 GPT-4o has also been shown to effectively monitor stronger reasoning models from CoT traces in a reward-hacking setting (Baker et al., 2025); our use is taxonomy annotation, which we validate with cross-judge agreement, manual checks, and robustness analyses. 5 Preprint Self-correction Hypothesis testing Uncertainty ack. Self-awareness Confidence calib. Planning Goal tracking 0 20 40 60 80 100 Prevalence (%) 51 39 69 43 39 67 56 8 11 16 26 34 49 42 VLM Thinking Instruct Self-correction Hypothesis testing Uncertainty ack. Self-awareness Confidence calib. Planning Goal tracking 0 20 40 60 80 100 Prevalence (%) 40 34 40 41 52 84 74 8 9 8 32 54 76 70 LLM Figure 2: Aggregate prevalence of a subset of the nine core higher-order behaviors across all benchmarks per modality. Self-correction, hypothesis testing, and uncertainty acknowledg- ment show the largest prevalence gaps between thinking and instruct models, while the remaining behaviors are more similar across training types. 4.2 The Most Amplified Behaviors Are Not the Highest-Lift Behaviors The prevalence analysis identifies three behaviors amplified by thinking training and six that remain largely unchanged. We then compute Behavioral Lift (Figure 3; full statistics for LLMs and VLMs appear in Table 12 and Table 13, respectively), which shows that the amplified behaviors are not the behaviors most associated with success. Confidence calibration is one of the strongest positive signals. Across VLM benchmarks, confidence calibration shows +72.2% Lift (98.8% accuracy when present vs. 26.7% when absent); across LLM benchmarks, +79.6% (99.6% vs. 20.0%). However, thinking models are no more likely to exhibit this behavior than instruct models. Confidence calibration is also largely absent when reasoning is weak: it appears in only 7.6% (VLM) and 3.4% (LLM) of lucky guesses and 3.4% (VLM) and 1.6% (LLM) of post-hoc rationalization traces, versus 94.7% and 95.9% on sound traces (Appendix Table 10 and Table 11). Confidence calibration is rare when the answer is correct but the reasoning is weak and common when the reasoning is sound, indicating that it tracks reasoning quality beyond final-answer correctness. The amplified behaviors rank lowest by Behavioral Lift. Uncertainty acknowledgment Lift is−16.1% (VLM) and−13.9% (LLM). Hypothesis testing Lift is +1.0% in both modalities. Self-correction shows modest positive Lift (+20.1% VLM, +12.4% LLM), stronger in thinking models than instruct. Appendix Table 10 and Table 11 show the complementary pattern for the amplified behaviors: uncertainty acknowledgment, and in LLMs also hypothesis testing and self-correction, appear more often on post-hoc traces than on sound traces. The ranking is stable across modalities, benchmarks, and controls. Confidence calibration, self-awareness, and knowledge alignment rank highest in both modalities, while hypothesis testing and uncertainty acknowledgment rank lowest. This pattern is not an artifact of pool- ing: the same broad ranking appears within individual benchmarks (Figure 8) and within each response-complexity stratum (Figure 4), and remains visible in per-model analyses (Appendix Figure 9, Figure 10). The same ordering appears under reverse conditioning: behaviors enriched in correct traces are the same ones with the highest Lift, while uncer- tainty acknowledgment is more common in incorrect traces (Appendix Table 9). As an additional validation, linear probes trained on Qwen3-4B hidden states recover several an- notated behaviors above chance, and in the thinking model the probe decodability ranking is directionally aligned with Behavioral Lift, especially on incorrect traces (Appendix E). 6 Preprint Table 2: Within-question analysis across language-only and vision-language models. Values are mean per-question∆accuracy between traces where a behavior is present versus absent, computed from 8 samples per question. Confidence calibration remains among the strongest positive signals, while uncertainty acknowledgment is null or negative. Asterisks indicate bootstrap 95% confidence intervals excluding zero; full results appear in Appendix C, Table 17 (LLMs), and Table 18 (VLMs). BehaviorLLM-ThinkLLM-InstVLM-ThinkVLM-Inst Confidence calibration+0.305 ∗ +0.518 ∗ +0.186 ∗ +0.342 ∗ Self-correction+0.267 ∗ +0.078 ∗ +0.153 ∗ +0.062 ∗ Hypothesis testing+0.048 ∗ +0.028-0.005+0.045 Uncertainty acknowledgment+0.006-0.094 ∗ -0.017-0.004 20020406080 Behavioral Lift (%) Uncertainty ack. Hypothesis testing Self-correction Planning Goal tracking Evidence citation Knowledge alignment Self-awareness Confidence calibration +72.2% +62.0% +53.7% +34.9% +30.6% +6.6% +20.1% +1.0% -16.1% VLM (N=7,000) 20020406080 Behavioral Lift (%) +79.6% +52.7% +80.3% +49.1% +52.5% +26.3% +12.4% +1.0% -13.9% LLM (N=8,282) Positive liftNegative lift Figure 3: Behavioral Lift for the nine cross-modal higher-order behaviors, ordered to highlight the relationship between amplified behaviors and high-Lift behaviors. Bars show Lift only: confidence calibration ranks highest for VLMs and among the highest for LLMs, while the three behaviors identified as amplified in the prevalence analysis cluster near the bottom. Uncertainty acknowledgment has negative Lift in both modalities. Within-question analysis confirms the pattern. To control for question difficulty, we gener- ate 8 traces per question from Qwen3-4B on 250 MATH-500 problems and compare accuracy between traces where a behavior is present versus absent on the same question. Confidence calibration shows a large within-question advantage (+0.31 thinking, +0.52 instruct; 95% CIs exclude zero). Uncertainty acknowledgment is non-predictive for thinking models (+0.01) and mildly negative for instruct (−0.09, CI excludes zero). Self-correction is stronger in thinking models (+0.27) than instruct (+0.08). This pattern persists under within-question comparisons (Table 2; extended discussion in Appendix C). The same-question and probing analyses suggest that the ranking is not a question-difficulty artifact or an arbitrary surface-labeling effect. 4.3 Thinking Models Often Succeed Through Recovery If the amplified behaviors are not the most predictive, how do thinking models achieve higher accuracy? On extended-reasoning tasks, we find that thinking models often recover after detected failures; on pattern-matching tasks, they gain little or are outperformed. Full failure rates appear in Appendix Table 16. Recovery is task-dependent. Recovery Rate, accuracy conditional on at least one failure being detected, varies sharply across benchmarks (Table 7, bottom row). On benchmarks 7 Preprint that reward step-by-step computation, thinking models recover at 2–3×the rate of in- struct models: VisualPuzzles (23.0% vs. 8.4%, 2.7×), MATH-500 (40.8% vs. 17.8%, 2.3×), and MMLU-Pro (16.8% vs. 6.3%, 2.7×). On mixed benchmarks, recovery rates are com- parable: MathVista (45.9% vs. 47.5%) and MMMU (32.9% vs. 33.8%). On LogiQA2, a pattern-matching task, the pattern reverses: instruct models recover better than thinking models (24.5% vs. 11.1%). Some behaviors matter mainly after failures occur. Self-correction has only modest overall Lift, but among traces with at least one detected failure it is strongly associated with recovery (+39p; Figure 7). This helps explain the Behavioral Lift and Recovery Rate results: some behaviors matter most after a trace has entered an error state. Thinking gains come from fewer failures or better recovery. On MathVista and MMMU, thinking models win through fewer failures: instruct models show logical failure rates of 53–55% versus 32–45% for thinking models, with comparable recovery rates. On LLM benchmarks and VisualPuzzles, thinking models sometimes exhibit more failures (51.1% vs. 41.4% logical failure on MMLU-Pro), but recover from these failures at much higher rates. Both mechanisms lead to higher accuracy for thinking models on 4 of 6 benchmarks. LogiQA2 functions as a counterexample to the “more thinking helps” assumption. Instruct models outperform thinking models on LogiQA2 (58.4% vs. 54.1%) by taking more shortcuts (34.2% vs. 20.3%) that work. This LogiQA2 task rewards recognizing argument structures quickly. These findings suggest that thinking helps when tasks reward computation, but can hurt model performance when the task primarily rewards pattern recognition. When no failures are detected, both model types reach 96–99% accuracy, suggesting that the performance difference between instruct models and thinking models in this case is due to model ability to recover from failures. 4.4 OLMo-3 SFT Show the Amplification–Lift Mismatch Before DPO/RLVR The main comparisons in our paper use released model variants, where each checkpoint reflects a full post-training recipe. To examine this phenomenon at an earlier point in model training, we evaluate OLMo-3-7B-Think-SFT and OLMo-3-7B-Instruct-SFT on MATH-500 and MMLU-Pro (N =1,443). These checkpoints share the OLMo-3-7B base and precede the later DPO and RLVR stages. The same amplification-Lift gap appears at the SFT checkpoint stage. The Think-SFT branch shows higher prevalence of the deliberative behaviors spanning self-correction (34.5% vs. 1.0%), hypothesis testing (22.7% vs. 1.0%), and uncertainty acknowledgment (33.1% vs. 1.0%). However, the highest within-checkpoint Lift comes from knowledge alignment and confidence calibration (+81.0%/+67.4% in Think-SFT; +81.4%/+81.3% in Instruct-SFT), and the amplified behaviors have lower Lift: self-correction (+16.4%/+20.6%) and uncertainty acknowledgment (-16.3%/-22.7%). Hypothesis testing is near zero in both checkpoints (-6.1%/+6.1%), consistent with its near-zero, least-stable Lift throughout other experiments described in this paper. The gap is therefore visible before the later DPO and RLVR stages, though these comparisons do not isolate the causal contribution of any single training stage. 4.5 The Amplification-Lift Gap Persists at Scale We test whether the amplification gap persists at scale by evaluating Qwen3-VL (Think/In- struct) from 2B to 32B on MathVista (Figure 14) and Qwen3 (Think) and Qwen2.5 (Instruct) on MATH-500 (Figure 15). The Amplification-Lift gap persists. At 32B, thinking models self-correct in 61.3% of responses versus 11.3% for instruct, a 50-point gap comparable to the pattern observed for 2B models. Hypothesis testing and uncertainty acknowledgment show the same pattern (Figure 14, left). We find that the gaps on the less-amplified behaviors narrow or reverse with scale: confidence calibration reaches 78.0% for the 32B instruct model versus 71.7% for thinking (Figure 14, right). Consistent patterns appear for Qwen3 LLMs on MATH-500 (Figure 15, Table 20). 8 Preprint Self-correction Lift diminishes at scale. At 2B, self-correction provides +30.0% Lift and an 18.3-point advantage. At 32B, self-correction Lift drops to +5.9% and the accuracy gap shrinks to 1.7 points. Confidence calibration Lift increases with scale (+57.7% at 2B to +68.7% at 32B), reinforcing the mismatch between amplified behaviors and high-lift behaviors. Larger thinking models express less uncertainty. Uncertainty acknowledgment drops from 66.7% at 2B to 43.7% at 32B in thinking models, while instruct models remain flat at 8–15%. This finding supports the interpretation that uncertainty acknowledgment tracks difficulty, rather than functioning as a measure of calibration. In a supplementary frontier-model validation on the full GPQA-Diamond benchmark (198 questions across 7 models;N=1,386), seven models from five providers show the same Lift ranking in visible responses (Appendix Table 14). Summary of Results Thinking-oriented models produce more deliberative traces, but this added deliberation is not concentrated in the behaviors most associated with reasoning correctness. The largest lifts are associated with confidence calibration, knowledge alignment, and self-awareness rather than visible search or hesitation. Thinking models help most on tasks that reward recovery from intermediate failures. The amplification–Lift gap persists across benchmarks, model families, and scale. 5 Discussion Amplified deliberation does not track correctness because accuracy cannot tell a frequent behavior from a useful one. Self-correction is one example: it can mark a successful recovery, but it can equally mark that the reasoning went wrong earlier. Uncertainty is another: it can reflect appropriate caution or simple confusion. Behavioral Lift separates these cases by measuring how much correctness differs when a behavior is present versus absent. Confidence calibration and uncertainty acknowledgment are easy to conflate, yet only one tracks correctness. Confidence calibration means expressed certainty tracks the strength of the reasoning: the model is cautious when evidence is weak and decisive when the reasoning is strong. Uncertainty acknowledgment is only the presence of explicit doubt or hesitation. Its weak or negative association with correctness suggests that explicit uncertainty is informative only when it tracks the available evidence (Kim et al., 2026). A model that says “maybe” at every step is not calibrated. This distinction has direct implications for training and evaluation. Current reasoning- oriented training often rewards final-answer correctness, which can produce long traces containing more correction, search, and hesitation; a line of recent work targets this inef- ficiency directly (Ma et al., 2025; Arora & Zanette, 2025). Visible deliberation, however, is an unreliable proxy for reasoning quality. Our recovery results sharpen this point: think- ing helps most when tasks reward extended computation or recovery from intermediate mistakes, and can hurt when fast recognition of logical form is sufficient. Future evalua- tions should therefore report both whether thinking helps and how: by preventing failures, recovering from them, improving calibration, or changing the shortcuts models take. For process supervision, reward behaviors associated with correctness; surface markers of deliberation are insufficient on their own. A process objective should reward evidence- grounded claims, appropriate domain framing, recognition of underspecified information, and confidence that tracks reasoning strength. Longer traces, backtracking, and expressions of uncertainty are not objectives in themselves. Behavioral Lift can audit whether a process objective rewards behaviors associated with success or only rewards deliberative surface form. Several limitations remain. Our analysis is limited to visible traces; written reasoning may be incomplete, post hoc (Boppana et al., 2026), or unfaithful to the model’s internal computation (Chen et al., 2025; Baker et al., 2025). Our labels are produced by automated 9 Preprint judges, so systematic judge bias remains possible despite multi-judge validation, robustness checks, same-question controls, and probing analyses. Finally, Behavioral Lift is descriptive: high-lift behaviors may cause better performance, reflect that the model is already on the right track, or co-occur with another useful property. Prompting results (Appendix F) are consistent with the ranking, but more controlled training studies are needed. 6 Related Work Reasoning-oriented training and long chain-of-thought. Recent reasoning models such as DeepSeek-R1 (Guo et al., 2025), Kimi-k1.5 (Team et al., 2025), and Qwen3 (Yang et al., 2025) show that RL or hybrid post-training can elicit long reasoning traces with behaviors such as backtracking and self-correction (Chu et al., 2025). Yeo et al. (2025); Yue et al. (2025) study how long CoT reasoning emerges during training, finding that some core abilities are already present in base models but require substantial RL compute to be elicited reliably. Our work studies the behavioral consequences of this elicitation: thinking-oriented models amplify self-correction, hypothesis testing, and uncertainty acknowledgment (Zhang et al., 2025), but these are not the behaviors most associated with correctness. Behavioral analysis of reasoning traces.Recent work analyzes reasoning traces through cognitive behaviors, taxonomies, strategy discovery, and trace dynamics. Gandhi et al. (2025) identify cognitive behaviors that predict RL trainability, Kargupta et al. (2025) annotate large-scale traces with a cognitively grounded taxonomy, Lee et al. (2025) cluster and steer reasoning strategies from CoT traces, Chang et al. (2026) analyze reasoning through semantic flow and latent computation, and Wang et al. (2025) study instability and underthinking in o1-like reasoning traces. We extend this line of work by comparing thinking and instruct variants across LLMs and VLMs, and by separating behaviors that are frequent from behaviors that have high Behavioral Lift. Process supervision and evaluation beyond accuracy. Chain-of-thought prompting showed that explicit reasoning can improve final-answer accuracy (Wei et al., 2023), while process supervision and process reward models evaluate reasoning at the step level (Light- man et al., 2023). Recent process-evaluation benchmarks and verifier-style methods further test whether models can identify reasoning errors or avoid shortcut solutions rather than only produce correct final answers (Zheng et al., 2025; Zhong et al., 2025). Our results give a concrete criterion for process supervision: before rewarding a reasoning behavior, one should ask whether it is associated with correctness. 7 Conclusion We quantified the behavioral effects of thinking training across 15,282 traces from 15 models and 6 benchmarks. Thinking training consistently amplifies self-correction, hypothesis testing, and uncertainty acknowledgment, while leaving several higher-lift behaviors largely unchanged. The central finding is that the amplified behaviors are not the ones most associated with correctness: confidence calibration is one of the strongest positive signals, whereas uncertainty acknowledgment is often non-predictive or negative. These results suggest that improving reasoning models requires more than making them think longer. Future training and evaluation should reward the behaviors that make reasoning reliable: calibrated confidence, grounded use of evidence, appropriate domain knowledge, and recovery when reasoning goes wrong. 10 Preprint Acknowledgments We thank Xiang Yue, Jacob Springer, and Seungone Kim for helpful discussions and feedback throughout the development of this work. Ethics Statement This paper studies reasoning behavior in open-weight language and vision-language models using automated annotation of model outputs. Our analysis is observational and focuses on benchmark responses rather than deployment in real-world decision settings. We will release prompts, metrics code, and annotations to support transparency and reproducibility. We do not claim that visible traces fully reflect internal reasoning, and we discuss this limitation explicitly in the paper. LLMs were used to assist with grammar and editing, and with implementation and verifica- tion of selected analyses. LLM-based annotation is described in the methodology. References Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025. URL https://arxiv.org/abs/2502.04463. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation, 2025. URLhttps://arxiv.org/abs/ 2503.11926. Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, and Jack Merullo. Reasoning theater: Disentangling model beliefs from chain-of-thought, 2026. URL https://arxiv.org/abs/2603.05488. A. Brown. Metacognition, executive control, self-regulation, and other more mysterious mechanisms. 1987. URL https://api.semanticscholar.org/CorpusID:147157394. Ruidi Chang, Jiawei Zhou, and Hanjie Chen. Prism: A dual view of llm reasoning through semantic flow and latent computation, 2026. URL https://arxiv.org/abs/2603.22754. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don’t always say what they think, 2025. URL https://arxiv.org/abs/2505.05410. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. URL https://arxiv.org/abs/2501.17161. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Good- man. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https://arxiv.org/abs/2503.01307. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, 11 Preprint Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. Priyanka Kargupta, Shuyue Stella Li, Haocheng Wang, Jinu Lee, Shan Chen, Orevaoghene Ahia, Dean Light, Thomas L. Griffiths, Max Kleiman-Weiner, Jiawei Han, Asli Celikyilmaz, and Yulia Tsvetkov. Cognitive foundations for reasoning and their manifestation in llms, 2025. URL https://arxiv.org/abs/2511.16660. Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?, 2026. URL https://arxiv.org/abs/2603.24472. Seongyun Lee, Seungone Kim, Minju Seo, Yongrae Jo, Dongyoung Go, Hyeonbin Hwang, Jinho Park, Xiang Yue, Sean Welleck, Graham Neubig, Moontae Lee, and Minjoon Seo. The cot encyclopedia: Analyzing, predicting, and controlling how a reasoning model will think, 2025. URL https://arxiv.org/abs/2505.10185. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020. URL https://arxiv.org/abs/2007.08124. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evalu- ating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255. Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking, 2025. URLhttps://arxiv.org/abs/ 2504.09858. Thomas O. Nelson. Metamemory: A theoretical framework and new findings. Psychology of Learning and Motivation, 26:125–173, 1990. URLhttps://api.semanticscholar.org/ CorpusID:39951989. 12 Preprint OpenAI.LearningtoreasonwithLLMs.https://openai.com/index/ learning-to-reason-with-llms/, 2024. Technical Blog Post. Gregory Schraw and Rayne Sperling Dennison. Assessing metacognitive awareness. Con- temporary Educational Psychology, 19(4):460–475, 1994. ISSN 0361-476X. doi: https://doi. org/10.1006/ceps.1994.1033. URLhttps://w.sciencedirect.com/science/article/ pii/S0361476X84710332. Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visu- alpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge, 2025. URL https://arxiv.org/abs/2504.10342. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang, and Zongyu Lin. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL https://arxiv.org/abs/2501.12599. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URLhttps://arxiv. org/abs/2406.01574. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of o1-like llms, 2025. URL https://arxiv.org/abs/2501.18585. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models, 2023. URL https://arxiv.org/abs/2201.11903. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms, 2025. URL https://arxiv.org/abs/2502.03373. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, 13 Preprint Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URLhttps://arxiv.org/abs/2311.16502. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. URL https://arxiv.org/abs/2504.13837. Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid- training, and rl on reasoning language models, 2025. URLhttps://arxiv.org/abs/2512. 07783. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayi- heng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning, 2025. URL https://arxiv.org/abs/2412.06559. Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. Impossiblebench: Measuring llms’ propensity of exploiting test cases, 2025. URL https://arxiv.org/abs/2510.20270. A Inference Details Inference and prompting. We evaluate LLMs withlm-evaluation-harnessand VLMs withlmms-eval, using the benchmark implementations provided by these evaluation suites. Unless otherwise noted, all runs are zero-shot (numfewshot=0), and we use benchmark- native chain-of-thought prompting when a CoT zero-shot variant is available. For instruct models, prompts therefore follow the benchmark or evaluation-suite templates rather than a paper-specific custom prompt. Full run scripts, exact subset files, prompt templates, and scoring code will be released with the codebase. LLM generation settings.We run all LLM evaluations with vLLM using family-specific de- coding settings chosen to match the recommended regime for each model family. Reasoning- enabled LLMs—Qwen3 thinking models, OLMo-3-Think, Nemotron-v2 reasoning models, and DeepSeek-R1-Distill—use temperature 0.6, top-p0.95, top-k20, and min-p0. Qwen2.5 instruct models use temperature 0.7, top-p0.8, top-k20, and min-p0. Nemotron-Base is evaluated with greedy decoding (temperature 0). Maximum generation length is 16,384 tokens for Qwen scaling runs, 32,768 tokens for OLMo-3 and DeepSeek-R1-Distill, and 8,192 tokens for Nemotron variants. VLM generation settings.We run all VLM evaluations with vLLM-backend using family- specific decoding settings. For Qwen3-VL-8B and GLM-4.1V-9B, thinking variants use temperature 1.0, top-p0.95, top-k20, repetition penalty 1.0, presence penalty 0.0, and a maximum generation length of 40,960 tokens; instruct variants use temperature 0.7, top-p 0.8, top-k20, repetition penalty 1.0, presence penalty 1.5, and a maximum generation length of 16,384 tokens. For Kimi-VL-A3B, the thinking variant uses temperature 0.8 and top-p0.8 with sampling enabled, while the instruct variant uses temperature 0.2 and top-p0.8 without sampling; both use a maximum generation length of 16,000 tokens. For InternVL3.5-8B, the thinking variant uses temperature 0.6, top-p0.8, top-k20, min-p0, and sampling enabled, while the instruct variant uses temperature 0.8, top-p0.8, top-k20, min-p0, and sampling disabled; both use a maximum generation length of 16,000 tokens. We otherwise retain the model-specific defaults and benchmark wrappers provided by the evaluation frameworks, and we do not add extra prompting intended to elicit particular behaviors. Table 3 lists the decoding settings used for each model family. B Qualitative samples and surface markers of annotated behaviors We include behavior-positive traces to make the annotation labels auditable in concrete model outputs. The examples show distinct surface forms, with overlap at label boundaries. 14 Preprint Table 3: Generation settings used for the main evaluations. We evaluate LLMs with lm-evaluation-harnessand VLMs withlmms-eval, using benchmark-native evaluation wrappers and zero-shot prompting. When a CoT zero-shot benchmark variant is available, we use that variant. We otherwise retain the model-specific defaults and benchmark wrap- pers provided by the evaluation frameworks, and we do not add extra prompting intended to elicit particular behaviors. Exact run scripts, subset files, prompt templates, and scoring code will be released with the codebase. (a) LLMs ModelTemp.Top-pTop-kMin-pMax gen toks Qwen30.60.9520016,384 Qwen2.5-Instruct0.70.820016,384 OLMo-3-7B-Think0.60.9520032,768 OLMo-3-7B-Instruct0.60.9520032,768 Nemotron-v20.60.952008,192 Nemotron-v2-Base0.0–8,192 DeepSeek-R1-Distill-Qwen-7B0.60.9520032,768 (b) VLMs ModelTemp.Top-pTop-kMin-pMax gen toks Qwen3-VL-8B-Thinking1.00.9520–40,960 Qwen3-VL-8B-Instruct0.70.820–16,384 GLM-4.1V-9B-Thinking1.00.9520–40,960 Kimi-VL-A3B-Thinking0.80.820016,384 Kimi-VL-A3B-Instruct0.20.820016,384 InternVL3.5-8B0.60.820016,384 InternVL3.5-8B-Instruct0.80.820016,384 In particular, uncertainty acknowledgment is often marked by explicit hedging and self- doubt language such as “I’m not sure” or “I’m confused,” while self-awareness is more often expressed as an information audit, for example when the model states that the passage does not mention or does not specify the information needed to justify a conclusion. Self- correction tends to appear as an explicit break in the reasoning path, with phrases like “wait,” “actually,” or “let’s start over,” and hypothesis testing is most clearly marked by branching language such as “alternatively,” “suppose,” or “case 1 / case 2.” Confidence calibration is less tied to any single phrase. Instead, it often appears as a confidence arc in which the model becomes more or less certain as the reasoning weakens or stabilizes. Knowledge alignment is often visible when the model converts the problem into the appropriate domain framework rather than relying on generic technical language, and goal tracking appears most clearly when the model explicitly states what has been established and what remains to be solved. Table 15 summarizes the most common recurring markers and qualitative patterns for the annotated behaviors. Pages 45–53 provide selected qualitative samples for the behaviors central to our findings. C Robustness to annotation noise and question difficulty Within-question analysis. To reduce confounding from question difficulty, we compare traces from the same model on the same question. For each MATH-500 problem, we generate 8 traces at temperature 0.6 and compute the per-question accuracy difference between traces where a behavior is present versus absent. The same broad ranking appears across Qwen3- 4B-Think, Qwen3-4B-Instruct, and Qwen3-4B-VL on MathVista: confidence calibration remains strongly positive, self-correction is positive but generally smaller, hypothesis testing is weak, and uncertainty acknowledgment is null or negative. The core ranking therefore does not appear to be driven only by easier questions eliciting different behaviors. These comparisons control for question difficulty. They do not control for the latent quality of the reasoning path: a trace already on a productive path may both express calibrated confidence 15 Preprint and reach the correct answer. We therefore read the within-question results as constraining a difficulty-based explanation, not as resolving causal direction. Sensitivity to random annotation noise.We also test whether the Behavioral Lift ranking is fragile to random annotation errors. For each behavior, we flip 5%, 10%, 15%, and 20% of labels at random and recompute Lift over 1000 trials. The main ranking is stable under this perturbation. For LLMs, confidence calibration and knowledge alignment remain the top behaviors, while uncertainty acknowledgment remains the lowest-ranked behavior in every trial. For VLMs, confidence calibration remains the top-ranked behavior and uncertainty acknowledgment remains the lowest-ranked behavior in every trial, with the rest of the ordering also largely unchanged. The only unstable behavior is hypothesis testing, whose clean Lift is already near zero. Lucky-guess analysis. To test whether confidence calibration is simply a downstream signal of correctness, we compare its prevalence on lucky guesses, where the final answer is correct despite flawed reasoning, versus non-lucky responses. In both LLMs and VLMs, and for both thinking and instruct models, calibration is far less common on lucky guesses than on non-lucky responses. Calibration therefore tracks sound reasoning more closely than final-answer correctness alone. Temporal position of behaviors. We analyze when each behavior first appears relative to answer commitment (Chang et al., 2026) across sampled thinking-model traces. All four behaviors typically appear before the answer (91–99% of traces), with mean behavior positions of 27–35% through the trace versus 79–94% for the answer. This pattern is shared across behaviors and reflects the general structure of reasoning traces rather than a property unique to calibration. D Length-Controlled Prevalence Analysis One concern is that thinking-oriented traces are longer, giving more opportunities for behaviors such as self-correction, hypothesis testing, and uncertainty acknowledgment to appear. To test whether length alone explains the amplification pattern, we bin responses into shared word-count quintiles within each modality and recompute thinking-minus- comparison prevalence gaps inside each bin. Length accounts for part of the pattern, especially among the shortest traces, but does not eliminate it: for LLMs, self-correction, hypothesis testing, and uncertainty acknowledgment remain more prevalent in thinking- oriented traces in each of the four non-shortest bins. For VLMs, self-correction remains amplified in 4/5 bins, uncertainty acknowledgment in 5/5, and hypothesis testing in 3/5. Under this trace-level definition of amplification, response length alone does not explain the main prevalence pattern. E Linear Probing of Behavioral Representations As a supporting validation, we train linear probes on hidden states from Qwen3-4B-Thinking and Qwen3-4B-Instruct to test whether the annotated behaviors correspond to internally decodable structure rather than arbitrary surface labels. For each behavior, we perform teacher-forced replay of the model’s saved traces, extract hidden states from the generated portion only (layer 35 of 36), and train logistic regression probes with group-aware cross- validation splitting by question to prevent leakage across repeated traces. Table 4 shows that several annotated behaviors are linearly decodable from Qwen hidden states. Two patterns are notable. First, in the thinking model, probe decodability is direc- tionally aligned with Behavioral Lift and is most strongly aligned on incorrect-only traces, where the rank correlation reachesρ = +0.86 (p =0.014). Among incorrect thinking traces, knowledge alignment and goal tracking are among the most linearly separable evaluated behaviors, while hypothesis testing and uncertainty acknowledgment are among the least separable. Second, in both models, confidence calibration remains strongly decodable on correct-only traces, indicating that it is not a trivial proxy for final correctness. 16 Preprint The instruct model shows a more mixed pattern. On incorrect traces, the rank correlation is negative (ρ =−0.71,p =0.071), with hypothesis testing and uncertainty acknowledgment among the most decodable behaviors. This is consistent with these behaviors being more distinctive in instruct traces, especially when they are relatively rare. By contrast, confidence calibration is nearly absent from incorrect traces in both models (1.7% thinking, 0.7% instruct), preventing reliable probe evaluation in that subset. The probing results are consistent with the annotation labels: several behaviors, including confidence calibration and knowledge alignment, are decodable from hidden states, and in the thinking model this decodability is directionally aligned with Behavioral Lift, especially on incorrect traces. Table 4: Linear probe AUC for higher-order behaviors in Qwen3-4B-Thinking and Qwen3- 4B-Instruct hidden states. Probes use layer 35 representations from the generated portion of teacher-forced replays, with the best AUC over five trace positions (25%, 50%, 75%, last token, mean pool) reported for each behavior. Dashes indicate insufficient minority-class samples (<30). Behaviors are sorted by Behavioral Lift rank from Figure 3. † marks behaviors amplified at least 3× by thinking training. ThinkingInstruct BehaviorLiftAllCorrectIncorrectAllCorrectIncorrect Confidence calibration1.775.771–.849.889– Self-awareness2.672.628–.609.599– Knowledge alignment3.798.795.704.807.886.624 Evidence citation4.798.802.651.821.832.652 Goal tracking5.793.744.690.800.774.645 Planning present6.684.664.608.704.695.605 Self-correction † 7.727.755.582.807.842.693 Hypothesis testing † 8.641.655.561.750.748.759 Uncertainty ack. † 9.709.723.598.873.844.731 Spearmanρ+.40+.32+.86 ∗ −.12+.15 −.71 N1989175123819991694305 ∗ p = 0.014. All otherρ values have p> 0.05. F Prompting Behavioral Lift at Inference Time To test whether the Behavioral Lift ranking points to behaviors worth trying at inference time, we prompt Qwen3-4B-Thinking on three benchmarks: MATH-500 (500 questions), MMLU-Pro Math (550 questions), and GPQA (448 questions). We compare three conditions under identical decoding settings: •High-Lift prompt: encourages confidence calibration, knowledge alignment, and self-awareness. These are the top-ranked Lift behaviors and are not strongly am- plified in thinking traces. The prompt instructs the model to let confidence track reasoning strength, identify the applicable domain framework, and recognize when information is insufficient. •Low-Lift prompt: encourages pervasive uncertainty acknowledgment, the most amplified behavior with consistently negative Lift. The prompt instructs the model to express doubt at every step, note confusion before continuing, and add caveats even when fairly confident. • Baseline: no behavioral prompt. High-Lift prompting improves accuracy on MATH-500 and MMLU-Pro and remains best on GPQA, while Low-Lift uncertainty prompting degrades accuracy on every benchmark (Table 5). The ordering High-Lift>Baseline>Low-Lift holds throughout. The effect scales with baseline model competence: on MATH-500 (baseline 84.8%), the high-Lift prompt 17 Preprint Table 5: Prompting results on Qwen3-4B-Thinking across three benchmarks. High-Lift encourages confidence calibration, knowledge alignment, and self-awareness. Low-Lift encourages pervasive uncertainty acknowledgment. Deltas are relative to baseline; signifi- cance from paired bootstrap (10,000 resamples, matched by question). MATH-500MMLU-Pro MathGPQA HiBaseLoHiBaseLoHiBaseLo Accuracy & Recovery (%) Accuracy90.684.880.262.058.446.349.849.145.6 ∆+5.8 ∗ – −4.6 ∗ +3.6 ∗ – −12.1 ∗ +0.7– −3.5 95% CI[+2.6,+9.0][−9.0,−0.2] [+0.5,+6.7][−16.4,−7.7] [−3.8,+5.1][−8.3,+0.9] Recovery68.957.146.220.218.513.434.836.128.2 Targeted behavior prevalence (%) Conf. calib.62.058.634.349.144.721.716.115.623.3 Knowl. align.88.880.878.675.371.571.239.533.735.3 Self-awareness45.649.055.546.240.536.823.221.428.9 Uncert. ack.33.855.097.662.260.294.455.673.799.3 Failure rates (%) Logical fail.25.231.029.343.546.250.675.277.273.6 Post-hoc rat.25.029.226.135.338.743.572.575.968.5 Context misr.11.818.020.019.624.524.259.666.361.7 Hi = High-Lift prompt; Lo = Low-Lift prompt; Base = no behavioral prompt. ∗ p<0.001, ∗ p<0.01, ∗ p<0.05 (paired bootstrap, 10,000 resamples). 95% CIs are for the accuracy delta vs. baseline. High- Lift vs. Low-Lift gap: MATH-500 +10.4p (p<0.001), MMLU-Pro +15.7p (p<0.001), GPQA +4.3p (p = 0.086). improves accuracy by 5.8p (p<0.001) and recovery from 57.1% to 68.9%, while reducing logical failure, context misread, and knowledge gap rates. On MMLU-Pro Math (baseline 58.4%), the gap between conditions reaches 15.7p (p<0.001), with the low-Lift prompt degrading accuracy by 12.1p (p<0.001). On GPQA, where the 4B model operates near chance (baseline 49.1%), the high-Lift prompt produces only a marginal gain (+0.7p, n.s.), but the low-Lift prompt still degrades accuracy (−3.5p,p =0.14), with the high-vs-low gap approaching significance (p = 0.086). The prompts shifted behaviors as intended. Across benchmarks, the high-Lift prompt increased confidence calibration (+0.4 to +4.4p), knowledge alignment (+3.8 to +8.0p), and reduced uncertainty acknowledgment (−18.1 to−21.2p). The low-Lift prompt drove uncertainty acknowledgment to near saturation (94.4–99.3%), while reducing confidence calibration by up to 24.3p. The prompting results are consistent with the Behavioral Lift ranking: encouraging high-lift behaviors improved accuracy in two of three benchmarks, whereas pervasive uncertainty prompting reduced accuracy on all three. The effect is clearest on tasks dominated by executable reasoning chains and fades as baseline competence drops, consistent with the task-dependent recovery mechanisms identified in our Recovery Rate analysis. This experiment is preliminary. The prompts change several properties of a trace at once, so the results do not isolate the effect of any single behavior, and the GPQA differences are not statistically significant. We report it as evidence that the Lift ranking carries usable signal at generation time. Testing whether these behaviors improve reasoning directly calls for controlled training interventions, which we see as the natural next step. 18 Preprint C2C3C4 Confidence calibration Knowledge alignment Self-awareness Goal tracking Evidence citation Planning Self-correction Hypothesis testing Uncertainty ack. +81.5+77.3+61.8 +65.6+58.7+63.2 +66.9+71.2+64.9 +34.5+43.0+47.2 +45.4+38.5+43.6 +11.2+17.3+19.6 +21.9+28.1+39.8 +3.5+3.8+5.7 -22.6-11.9-17.4 VLM C2C3C4 +88.5+79.5+70.9 +88.0+81.2+74.4 +50.3+54.4+49.5 +59.0+57.6+50.2 +62.5+54.6+44.3 +28.6+35.3+27.5 +10.5+10.2+21.7 +6.4-2.0+5.9 -23.9-20.0-17.7 LLM 20 0 20 40 60 80 Lift (%) Behaviors amplified 3× by thinking training. Figure 4: Behavioral Lift (%) within response complexity bins (C2–C4), pooled across all models and benchmarks per modality. The ranking from Figure 3 holds within each bin: confidence calibration remains among the strongest positive associations with correctness and uncertainty acknowledgment shows negative Lift in every bin in both modalities, confirming that the disconnect between amplification and Behavioral Lift is not an artifact of response length or problem difficulty. Word-count quintile analysis in Figure 5 yields consistent results. G Visual Claim Accuracy Matters More Than Image References For VLM traces, we additionally annotate two modality-specific grounding labels: visual references present, which records whether the trace explicitly mentions concrete image content, and visual claims accurate, which records whether those visual statements are factually correct. These labels separate image mention from correct image use. The distinction is sharp. Across all VLM traces, accurate visual claims have +60.9 Behavioral Lift: responses with accurate visual claims are correct 87.8% of the time, compared with 26.9% when their visual claims are inaccurate. By contrast, simply mentioning visual content has essentially no positive association with correctness (-3.3 Lift). The same pattern holds within each benchmark: visual claim accuracy is strongly positive on VisualPuzzles (+71.7), MathVista (+54.3), and MMMU (+45.1), whereas visual reference presence is near zero except for a modest positive effect on MathVista (+12.1). This supports the taxonomy design choice to distinguish behavioral presence from behavioral quality. Thinking models also shift the profile of visual failures. They show less visual neglect than instruct models overall (36.0% vs. 47.4%), with especially large reductions on MathVista and MMMU, but they exhibit more visual hallucination (24.7% vs. 18.0%). Longer reasoning traces appear to attend to the image more actively while also creating more opportunities to state incorrect visual claims. Recovery from visual failures is asymmetric: thinking models recover better from hallucination (20.8% vs. 12.1%), but worse from visual neglect (15.3% vs. 20.4%). In other words, neglect is often avoided upstream rather than repaired downstream. Confidence calibration remains strongly associated with correctness even after conditioning on visual grounding quality. Among traces with accurate visual claims, calibration has +43.9 Lift; among traces with inaccurate visual claims, its Lift rises to +73.5. Thus, visual grounding quality and metacognitive quality capture different aspects of successful VLM reasoning: accurate perception is highly predictive overall, while calibration is especially informative when the trace contains imperfect visual interpretation. 19 Preprint Q1Q2Q3Q4Q5 Confidence calibration Knowledge alignment Self-awareness Goal tracking Evidence citation Planning Self-correction Hypothesis testing Uncertainty ack. +49.4+90.8+81.9+66.2+69.2 +37.3+66.5+62.3+60.2+40.0 +36.3+73.6+71.2+66.1+47.9 +20.6+33.4+36.6+42.6+27.4 +25.8+40.5+30.7+41.3+26.7 +9.0+12.1+15.2+16.7+6.6 +20.9+33.2+33.2+52.0+30.2 +26.4+16.5+5.2+6.4 -29.2-2.3+4.0-5.4+9.2 VLM Q1Q2Q3Q4Q5 +85.4+95.5+88.1+72.4+72.8 +85.0+91.1+85.7+76.1+71.2 +60.7+54.5+57.6+53.4+44.9 +52.0+63.0+63.9+53.8+53.3 +52.5+64.8+55.1+54.5+48.9 +34.7+24.3+39.7+32.1+31.8 +21.8+16.9+26.3+28.7+17.9 +23.5+3.5+15.9+5.8+2.1 -18.5-24.6+0.6-18.2-15.1 LLM 20 0 20 40 60 80 100 Behavioral Lift (%) Behaviors amplified 3× by thinking training. = insufficient samples (N < 20 in one group). Quintile thresholds computed separately per modality. Figure 5: Behavioral Lift (%) within word-count quintiles, pooled across all models and benchmarks per modality (VLM:N=7,000; LLM:N=8,282). Quintile thresholds are com- puted separately per modality. Dashes indicate insufficient samples (N<20 in one group). The top-level pattern from Figure 3 holds: confidence calibration and knowledge alignment dominate, while the behaviors identified as amplified in the prevalence analysis (†) cluster lower. This complements Figure 4 using a purely mechanical binning strategy with no annotator involvement. Appendix D shows that the main prevalence-amplification pattern persists within shared word-count bins, so it is not explained by response length alone. VisPuzzMathVistaMMMULogiQA2MATH-500MMLU-Pro 0 10 20 30 40 50 Recovery Rate (%) 23.0 8.4 45.9 47.5 32.9 33.8 11.1 24.5 40.8 17.8 16.8 6.3 VLMLLM Pure reasoning MixedKnowledge- heavy Pure reasoning MixedKnowledge- heavy Thinking Instruct Figure 6: Recovery Rate across the task spectrum. Thinking models recover from detected failures at 2.3–2.7×the rate of instruct models on benchmarks that reward step-by-step com- putation (VisualPuzzles, MATH-500, MMLU-Pro). On LogiQA2, instruct models recover better. Recovery advantage tracks task type, not modality. 20 Preprint 020406080 Recovery Rate (%) Conf. calibration Knowledge alignment Self-correction Self-awareness Uncertainty ack. Hypothesis testing Goal tracking Evidence citation Planning +57p +47p +39p +32p +21p +17p +4p -1p -7p Which Behaviors Predict Recovery from Failures? Recovery with behavior Recovery without behavior Figure 7: Recovery rate conditional on behavior presence among traces with at least one de- tected failure (N=4,778). Confidence calibration, knowledge alignment, and self-correction are the strongest positive predictors of recovery. This links the Behavioral Lift and Recovery Rate analyses: self-correction has modest overall Lift but strongly predicts recovery from failures. VisPuzzMathVistaMMMULogiQA2MATH-500MMLU-Pro Confidence calibration Knowledge alignment Self-awareness Goal tracking Evidence citation Planning Self-correction Hypothesis testing Uncertainty ack. +82+54+72+82+67+84 +52+41+53+69+83+84 +71+44+61+56+37+62 +33+25+26+43+67+49 +37+32+25+28+61+53 +11+8+9+16+44+25 +32+15+15+6+11+9 +9-5+11-1+5-10 -4-15-8-20-8-17 Behavioral Lift (%) per Benchmark 20 0 20 40 60 80 Lift (%) Figure 8: Behavioral Lift (%) by benchmark for VLM and LLM tasks. Each cell computes Lift within a single benchmark. The broad ranking is stable: confidence calibration and knowledge alignment show the strongest positive associations with correctness, while the behaviors identified as amplified in the prevalence analysis cluster near the bottom. 21 Preprint Qwen3-4B Think Qwen3-4B Instruct OLMo3-7B Think OLMo3-7B Instruct DSR1-Distill 7B Qwen2.5-7B Instruct Nemotron-9B Think Nemotron-9B Base Conf. calibration Knowledge align. Self-awareness Goal tracking Evidence citation Planning Self-correction Hypothesis testing Uncertainty ack. +67+80+79+84+72+86+89+82 +79+80+76+80+77+87+85+81 +47+37+56+57+58+54+53+59 +58+48+54+48+54+51+56+47 +55+61+50+39+52+43+57+43 +29+18+34+19+26+12+33+29 +32+1+15+2+26+17+12+9 +4-11+1+4+2+12-1+12 -16-29-18-17-13-58-21-50 Behavioral Lift (%) per LLM Model 60 40 20 0 20 40 60 80 Lift (%) Figure 9: Behavioral Lift (%) by model for all LLMs. Each column computes Lift from a single model’s responses. The Behavioral Lift ordering holds across models: confidence calibration and knowledge alignment typically show the strongest positive associations with correctness, while the behaviors identified as amplified in the prevalence analysis tend to cluster lower. Qwen3-VL Think Qwen3-VL Instruct Kimi-VL Think Kimi-VL Instruct InternVL3.5 Think InternVL3.5 Instruct GLM-4.1V Think Conf. calibration Knowledge align. Self-awareness Goal tracking Evidence citation Planning Self-correction Hypothesis testing Uncertainty ack. +71+79+75+75+66+68+71 +61+52+62+38+61+38+60 +67+67+64+63+62+50+64 +47+38+39+7+40-2+35 +48+32+41+18+44+15+36 +22+12+10-10+13-16+5 +38+0+7-28+29-1+25 +0-4-12+24+0+13+1 -28-33-36-36-25-41-33 Behavioral Lift (%) per VLM Model 60 40 20 0 20 40 60 80 Lift (%) Figure 10: Behavioral Lift (%) by model for all VLMs. Each column computes Lift from a single model’s responses. The Behavioral Lift ordering holds across models: confidence calibration and knowledge alignment typically show the strongest positive associations with correctness, while the behaviors identified as amplified in the prevalence analysis tend to cluster lower. 22 Preprint Mathematical Logical Procedural Conceptual Factual recall Causal Analogical 0 20 40 60 80 100 Prevalence (%) Reasoning Type Composition per Benchmark LogiQA2MATH-500MMLU-Pro Figure 11: Reasoning type composition per benchmark. Each benchmark emphasizes different reasoning demands: MATH-500 is dominated by mathematical and procedural reasoning, LogiQA2 by logical and conceptual reasoning. This variation underlies the task-dependent recovery mechanisms observed across benchmarks. ModelVariant roleFamilyParamsCheckpoint Vision-Language Models Qwen3-VL-8B-ThinkingThink-orientedQwen8B Qwen/Qwen3-VL-8B-Thinking Qwen3-VL-8B-InstructNon-thinkingQwen8B Qwen/Qwen3-VL-8B-Instruct Kimi-VL-A3B-ThinkingThink-orientedKimi3B ∗ moonshotai/Kimi-VL-A3B-Thinking-2506 Kimi-VL-A3B-InstructNon-thinkingKimi3B ∗ moonshotai/Kimi-VL-A3B-Instruct InternVL3.5-8BThink-orientedInternVL8B OpenGVLab/InternVL35-8B InternVL3.5-8B-InstructNon-thinkingInternVL8B OpenGVLab/InternVL35-8B-Instruct GLM-4.1V-9B-Thinkingthink-orientedGLM9B zai-org/GLM-4.1V-9B-Thinking Text-Only Language Models Qwen3-4B-ThinkingThink-orientedQwen4B Qwen/Qwen3-4B-Thinking-2507 Qwen3-4B-InstructNon-thinkingQwen4B Qwen/Qwen3-4B-Instruct-2507 OLMo-3-7B-ThinkThink-orientedOLMo7B allenai/Olmo-3-7B-Think OLMo-3-7B-InstructNon-thinkingOLMo7B allenai/Olmo-3-7B-Instruct DeepSeek-R1-Distill-Qwen-7Bthink-orientedDeepSeek/Qwen7B deepseek-ai/DeepSeek-R1-Distill-Qwen-7B Qwen2.5-7B-InstructAdditional comparisonQwen7B Qwen/Qwen2.5-7B-Instruct Nemotron-Nano-9B-v2Reasoning variantNemotron9B nvidia/NVIDIA-Nemotron-Nano-9B-v2 Nemotron-Nano-9B-v2-BaseNon-thinkingNemotron9B nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base Table 6: Models evaluated. The checkpoint column lists the exact public model identifiers used for inference; the variant-role column is descriptive and does not assert that any pair dif- fers by a single isolated training stage. Five released same-family, same-scale thinking/non- thinking endpoint pairs are Qwen3-VL, Kimi-VL, InternVL3.5, Qwen3-4B, and OLMo-3-7B. GLM-4.1V and DeepSeek-R1-Distill-Qwen-7B are unmatched think-oriented auxiliary mod- els; Qwen2.5-7B-Instruct is an additional comparison baseline. Nemotron-Nano-9B-v2 and Nemotron-Nano-9B-v2-Base are included as a base-to-reasoning comparison rather than a clean instruct/thinking endpoint pair. Analyses restricted to same-family thinking/non- thinking endpoint pairs are consistent with the overall results. ∗ Kimi-VL-A3B has 3B active parameters (MoE architecture). 23 Preprint Conf. calib. Self-aware. Know. align. Evid. citation Goal tracking Planning Self-correction Hyp. testing Uncert. ack. Behavior present Conf. calib. Self-aware. Know. align. Evid. citation Goal tracking Planning Self-correction Hyp. testing Uncert. ack. Given behavior --61100999693251915 91--97989695392930 8052--989392302120 644279--8591272121 71478597--97292221 6142759387--302425 515377868093--5869 49517084789575--68 3547607667898061-- Conditional Co-occurrence of Metacognitive Behaviors 0 20 40 60 80 100 P(column | row) % Figure 12:Conditional co-occurrence of cross-modal higher-order behaviors: P(column behavior present | row behavior present). Behaviors in the top-left dashed block co-occur at high rates (60–100%), while the three behaviors most amplified by thinking training in the bottom-right dashed block also co-occur (58–80%) but show lower cross-block co-occurrence with the others (15–30%). This suggests the amplified behaviors form a relatively distinct cluster. 24 Preprint LogiQA2 (pure reasoning) MATH-500 (mixed) MMLU-Pro (knowledge) 0 10 20 30 40 50 Recovery Rate (%) 11.1 40.8 16.8 24.5 17.8 6.3 Recovery Rate by Benchmark Thinking Instruct Figure 13: Recovery Rate by benchmark for LLM models. Thinking models recover at 2–3× the rate of instruct models on MATH-500 and MMLU-Pro, but instruct models recover better on LogiQA2. The pattern tracks task structure: extended reasoning tasks favor thinking models, while pattern-matching tasks do not. 2B4B8B32B Model Size (Qwen3-VL) 0 10 20 30 40 50 60 70 80 90 Prevalence (%) Amplified Behaviors Self-correctionHypothesis testingUncertainty ack. ThinkingInstruct 2B4B8B32B Model Size (Qwen3-VL) 20 30 40 50 60 70 80 90 Prevalence (%) Largely Unchanged Behaviors Conf. calibrationPlanningGoal tracking ThinkingInstruct Figure 14: Behavioral prevalence across model scales for Qwen3-VL on MathVista. The gap in amplified behaviors (left) persists from 2B to 32B parameters. Largely unchanged behav- iors (right) converge as model size increases, with instruct models matching or exceeding thinking models at 32B. 25 Preprint 0.6B1.7B4B8B14B32B Model Size (Qwen3 / Qwen2.5) 0 20 40 60 80 100 Prevalence (%) Amplified Behaviors Self-correctionHypothesis testingUncertainty ack. ThinkingInstruct 0.6B1.7B4B8B14B32B Model Size (Qwen3 / Qwen2.5) 10 20 30 40 50 60 70 80 90 100 Prevalence (%) Largely Unchanged Behaviors Conf. calibrationPlanningGoal tracking ThinkingInstruct Figure 15: Behavioral prevalence across model scales for Qwen3 on MATH-500. The gap in the three behaviors most amplified by thinking training persists from 0.6B to 32B parameters, while the largely unchanged behaviors converge as model size increases, with instruct models matching or exceeding thinking models at 32B. Table 7: Behavioral profile of thinking (Th) and instruct (In) models across six benchmarks. Amplified behaviors (top) show large gaps between thinking and instruct models; non- amplified behaviors (middle) show small or reversed gaps. Accuracy and recovery rate (bottom) show where behavior prevalence translates into task performance. VLM BenchmarksLLM Benchmarks VisPuzzMathVistaMMMULogiQA2 MATH-500 MMLU-Pro ThInThInThInThInThInThIn Amplified behaviors (%) Self-correction50.6 14.9 55.24.348.75.221.53.354.612.544.57.1 Hypothesis testing 42.6 17.6 21.63.751.5 11.3 25.66.442.812.834.48.2 Uncertainty ack.85.1 27.5 51.67.268.9 10.6 25.03.943.58.152.810.2 Largely unchanged behaviors (%) Self-awareness27.1 21.6 57.3 30.6 47.0 27.0 33.1 26.0 52.636.837.834.1 Conf. calibration18.3 24.5 58.8 42.3 43.9 35.1 47.4 46.5 66.167.940.946.7 Planning63.0 67.0 63.7 38.6 74.5 39.7 74.3 50.2 91.292.386.983.8 Goal tracking40.4 51.6 63.4 38.1 65.1 35.8 68.3 50.0 80.481.772.376.0 Knowledge align.38.8 39.7 74.6 49.2 72.6 59.0 65.1 57.2 78.974.453.454.4 Evidence citation42.0 55.0 73.3 47.8 56.4 48.8 87.0 73.1 86.191.271.680.3 Accuracy (%)36.5 29.7 77.5 66.6 59.2 52.4 54.1 58.4 81.474.351.451.5 Recovery Rate (%)23.08.445.9 47.5 32.9 33.8 11.1 24.5 40.817.816.86.3 26 Preprint Table 8: Behavioral Lift for the nine cross-modal higher-order behaviors. Rows are grouped to show the amplified behaviors relative to the highest-Lift behaviors. Lift= P(correct | b=true)− P(correct | b=false). Arrows ( † ) mark behaviors identified as amplified in the prevalence analysis (≥3×thinking/instruct prevalence). These behaviors cluster near the bottom of the Lift ranking. VLM (N=7,000)LLM (N=8,282) BehaviorLiftPres.%LiftPres.% Conf. calibration+72.236.9 +79.652.7 Self-awareness+62.035.9 +52.736.9 Knowledge alignment +53.756.2 +80.364.0 Evidence citation+34.954.0 +49.181.7 Goal tracking+30.650.0 +52.571.7 Planning+6.659.4 +26.380.1 Self-correction † +20.132.9 +12.424.2 Hypothesis testing † +1.027.3 +1.021.9 Uncertainty ack. † −16.146.3 −13.924.2 Table 9: Conditional prevalence of cross-modal higher-order behaviors in correct versus incorrect LLM traces. Values reportP(b | ✓),P(b | ×), and∆ = P(b | ✓)− P(b | ×). Confidence calibration and knowledge alignment are strongly enriched in correct traces, while uncertainty acknowledgment is more common in incorrect traces. AllThinkingInstruct BehaviorP(✓)P(×)∆P(✓)P(×)∆P(✓)P(×)∆ confidencecali.84.7%0.6%+84.1 82.0%1.0%+81.0 87.5%0.2%+87.3 knowledgealig. 93.9% 15.4% +78.5 94.4% 18.5% +75.8 93.4% 12.3% +81.1 self awareness56.7%4.6%+52.1 62.2%6.5%+55.7 51.0%2.8%+48.2 goaltracking88.9% 43.7% +45.2 91.0% 45.1% +45.9 86.7% 42.3% +44.4 evidence cita.93.5% 62.3% +31.2 94.4% 60.4% +34.0 92.6% 64.3% +28.3 planningpres.86.9% 69.1% +17.8 90.7% 73.4% +17.3 82.9% 64.9% +18.1 selfcorrection27.9% 18.2%+9.6 46.6% 29.7% +16.9 8.4%6.7%+1.7 hypothesistest.22.2% 21.4%+0.7 34.5% 33.9%+0.59.4%8.9%+0.5 uncertaintyack. 20.1% 30.8%-10.8 34.9% 49.7%-14.94.7%11.9%-7.1 Table 10: Behavior prevalence on post-hoc rationalization traces, non-post-hoc traces, lucky guesses, and sound reasoning traces for VLM models. Confidence calibration is almost absent when reasoning is reverse-engineered or lucky, while uncertainty acknowledgment is more common on post-hoc traces than on sound reasoning traces. BehaviorPost-hocNo post-hocLuckySound confidencecalibration3.4%76.4%7.6%94.7% knowledgealignment29.6%87.2%41.9%95.5% selfawareness9.1%66.4%14.4%81.5% goaltracking26.1%76.7%19.0%81.5% evidencecitation28.7%82.8%24.1%88.2% planning present45.3%73.5%27.1%75.3% selfcorrection30.5%32.4%37.6%37.8% hypothesistesting26.4%26.0%21.7%28.7% uncertainty acknowledgment56.7%29.0%46.2%30.8% 27 Preprint Table 11: Behavior prevalence on post-hoc rationalization traces, non-post-hoc traces, lucky guesses, and sound reasoning traces for LLM models. Confidence calibration is almost absent when reasoning is reverse-engineered or lucky, while uncertainty acknowledgment, hypothesis testing, and self-correction are more common on post-hoc traces than on sound reasoning traces. BehaviorPost-hocNo post-hocLuckySound confidencecalibration1.6%72.5%3.4%95.9% knowledgealignment20.4%80.9%42.1%99.9% self awareness7.3%48.3%8.3%62.8% goaltracking34.7%86.0%30.2%95.9% evidencecitation50.5%93.8%40.1%99.8% planningpresent62.0%87.2%44.9%91.7% self correction34.2%20.3%46.2%24.0% hypothesis testing32.2%17.9%32.0%20.0% uncertaintyacknowledgment47.1%15.3%44.9%15.5% Table 12: Detailed Behavioral Lift statistics for LLM benchmarks (N=8,282). Lift= P(✓|b)− P(✓|¬b). Think-Lift and Inst-Lift show lift computed separately on thinking and instruct model pools. Accuracy given bSample count BehaviorLift Th-Lift In-Lift P(✓|b) P(✓|¬b)N b N ¬b Knowledge alignment +80.3 +79.1 +81.690.810.5 52992983 Confidence calibration +79.6 +76.2 +83.299.620.0 43623920 Context understanding +76.8 +75.1 +78.687.310.5 55422740 Logical steps valid+69.9 +68.7 +71.388.818.9 50943188 Self-awareness+52.7 +54.0 +52.195.242.5 30525230 Goal tracking+52.5 +55.6 +49.776.824.3 59372345 Evidence citation+49.1 +53.2 +44.970.921.8 67631519 Reasoning present+29.1 +42.1 +24.563.634.5 7787495 Planning+26.3 +30.5 +23.467.140.8 66361646 Self-correction+12.4 +16.5 +5.571.358.9 20036279 Hypothesis testing+1.0 +0.6 +1.462.761.7 18126470 Uncertainty ack. −13.9 −14.5 −24.451.465.3 20026280 Shortcut−49.1 −48.3 −50.023.172.2 17336549 Post-hoc rational. −51.5 −55.6 −47.524.876.3 23175965 Logical failure−71.1 −69.9 −72.318.689.6 32295053 Factual error−71.8 −68.4 −75.03.775.6 15746708 Knowledge gap −75.5 −74.3 −76.78.884.3 24525830 Context misread −76.8 −75.2 −78.410.487.2 27275555 28 Preprint Table 13: Detailed Behavioral Lift statistics for VLM benchmarks (N=7,000). Lift= P(✓|b)− P(✓|¬b) . Think-Lift and Inst-Lift show lift computed separately on thinking and instruct model pools. Accuracy given bSample count BehaviorLift Th-Lift In-Lift P(✓|b) P(✓|¬b)N b N ¬b Confidence calibration +72.2 +69.9 +74.898.826.7 25844416 Logical steps valid+66.5 +70.5 +60.690.523.9 30893911 Self-awareness+62.0 +63.6 +60.593.131.1 25114489 Visual claims accurate +60.9 +63.4 +56.887.826.9 30323968 Knowledge alignment +53.7 +61.7 +43.076.823.1 39333067 Evidence citation+34.9 +43.6 +22.769.434.5 37773223 Goal tracking+30.6 +40.8 +15.668.638.0 34983502 Self-correction+20.1 +24.5 +3.266.846.7 23024698 Planning+6.6 +12.7 −3.456.049.4 41562844 Hypothesis testing+1.0 −4.3 +5.354.053.0 19145086 Visual refs present −3.3 −3.1 −6.552.655.9 55791421 Reasoning present −8.8 +21.5 −16.752.561.3 6385615 Uncertainty ack. −16.1 −30.2 −24.844.760.7 32413759 Language bias−21.1 −33.2 −6.935.056.19406060 Shortcut−36.3 −44.3 −25.631.467.7 27844216 Visual hallucination −45.7 −47.7 −45.317.663.3 15315469 Post-hoc rational. −52.9 −61.3 −42.029.382.2 38243176 Visual neglect−60.7 −64.8 −55.117.177.8 28264174 Logical failure−67.3 −70.9 −61.823.690.9 39143086 29 Preprint Table 14: Behavioral Lift for seven frontier models on GPQA-Diamond, computed from visible responses only and pooled across models. The same broad ranking appears as in the main analysis: knowledge alignment and confidence calibration are the strongest positive signals, hypothesis testing remains near zero, and uncertainty acknowledgment remains negative. BehaviorLiftP(✓| b)P(✓|¬b)Present %N p N a Knowledge alignment+84.294.610.466.7925461 Confidence calibration+77.999.922.057.3794592 Goal tracking+67.779.211.681.31127259 Self-awareness+58.696.938.348.3670716 Evidence citation+51.471.319.890.91260126 Planning present+20.267.146.997.7135432 Self-correction+14.176.362.231.3434952 Hypothesis testing-2.565.167.638.8538848 Uncertainty acknowledgment-23.251.574.634.6480906 Table 15: Qualitative surface markers of annotated higher-order behaviors. For each be- havior, we manually reviewed behavior-positive traces and summarized the most common recurring surface markers and broader discourse pattern. These summaries are qualitative and intended to illustrate how the behaviors tend to appear in visible reasoning traces. BehaviorExample surface markersQualitative pattern Planning present “letmebreakthisdown,” “first. . . then. . . ,” “let me define. . . ,” “step 1 / step 2,” explicit headers Often marked by explicit upfront decom- position of the task into steps, variables, or constraints before the main reasoning begins. Hypothesis test- ing “alternatively,” “what if,” “suppose,” “case 1 / case 2,” “if X. . . if not X. . . ” Often marked by explicit branching over alternative interpretations or scenarios, rather than generic linear continuation. Self-correction“wait,” “actually,” “that’s not correct,” “let me rethink,” “let’s start over” Often marked by explicit revision and backtracking after a contradiction, de- tected error, or failed line of reasoning. Uncertainty ac- knowledgment “I’m not sure,” “I’m confused,” “maybe,” “am I missing something?” Often marked by explicit hedging and self-doubt language, including admis- sions of ambiguity or difficulty. Evidencecita- tion “the passage states,” “given that,” “from condition 1,” “based on the fact that. . . ” Often marked by explicit grounding of a claim in a specific fact, premise, or con- straint from the prompt. Confidence cali- bration “I’m not sure,” “maybe,” “let me double-check,” “must be,” “therefore, in all cases” Often marked not by isolated hedging (which signals uncertainty acknowledg- ment), but by a shift in expressed cer- tainty that tracks the strength of the rea- soning — increasing confidence when steps succeed, decreasing when they fal- ter. Self-awareness“the passage does not mention,” “the text does not specify,” “cannot be de- termined,” “insufficient information” Often marked by explicit recognition that the prompt lacks the information needed to justify a conclusion, rather than by generic hedging alone. Goal tracking“I need to. . . ,” “the goal is. . . ,” “so far we know. . . ,” “the next step is. . . ,” “going back to the question. . . ” Often marked by explicit monitoring of progress toward the main objec- tive, rather than generic step-by-step se- quencing alone. Knowledge alignment “necessary condition,” “sufficient con- dition,” “correlation vs. causation,” “translate into logical expressions,” domain-specific conceptual terms Often marked by converting the prob- lem into the appropriate domain frame- work and reasoning within that frame- work, rather than by generic technical- sounding language. 30 Preprint VLM BenchmarksLLM Benchmarks VisPuzzMathVistaMMMULogiQA2 MATH-500 MMLU-Pro ThInThInThInThInThInThIn Cross-modal failure modes (%) Logical failure77.6 72.7 31.9 55.3 44.6 53.1 39.7 44.4 28.629.151.141.4 Shortcut52.7 43.1 23.2 49.3 27.7 45.9 20.3 34.2 16.419.018.218.3 Post-hoc rational.79.1 61.2 35.1 48.7 47.9 51.7 33.1 34.8 22.914.240.023.2 Lucky guess15.15.111.9 27.0 11.5 18.34.411.87.93.66.62.7 VLM-specific (%) Visual hallucination 37.1 24.1 14.8 11.2 20.8 17.9 Visual neglect55.3 57.0 20.0 36.6 30.6 43.5 Language bias11.45.38.514.0 15.8 26.2 LLM-specific (%) Factual error7.19.816.724.026.129.7 Context misread 32.8 39.8 19.822.941.941.0 Knowledge gap25.0 29.6 18.223.141.640.3 Recovery Rate (%)23.08.445.9 47.5 32.9 33.8 11.1 24.5 40.817.816.86.3 Accuracy (%)36.5 29.7 77.5 66.6 59.2 52.4 54.1 58.4 81.474.351.451.5 Table 16: Failure mode rates, recovery rate, and accuracy across six benchmarks. cross- modal failures (top) are directly comparable across modalities. VLM-specific and LLM- specific failures (middle) apply only to their respective benchmarks; gray cells indicate non-applicable modality. Recovery Rate= P(correct| any failure detected). On reasoning- heavy benchmarks, thinking models recover at 2.3–2.7×the rate of instruct models. On LogiQA2, instruct models recover better. Table 17: Within-question paired analysis on MATH-500. For each of 250 questions, we generate 8 traces at temperature 0.6 and compute the per-question accuracy difference between traces where a behavior is present versus absent. Reported values are mean within- question∆accuracy, with bootstrap 95% confidence intervals and the number of questions for which both present and absent traces were observed. Qwen3-4B-ThinkQwen3-4B-Instruct BehaviorMean∆95% CINMean∆95% CIN Confidence calibration+0.305[+0.254, +0.363]189+0.518[+0.453, +0.582]145 Self-awareness+0.187[+0.150, +0.226]225+0.158[+0.131, +0.189]227 Knowledge alignment+0.596[+0.524, +0.673]110+0.728[+0.651, +0.794]91 Evidence citation+0.299[+0.201, +0.388]78+0.268[+0.143, +0.393]54 Goal tracking+0.405[+0.331, +0.481]112+0.434[+0.346, +0.516]99 Planning present+0.120[+0.033, +0.208]69+0.138[+0.035, +0.241]72 Self-correction+0.267[+0.203, +0.334]132+0.078[+0.035, +0.121]180 Hypothesis testing+0.048[+0.017, +0.082]214+0.028[ -0.031, +0.085]126 Uncertainty acknow.+0.006[-0.021, +0.034]222-0.094[ -0.168, -0.028 ]115 31 Preprint Table 18: Within-question paired analysis on MathVista for Qwen3-4B-VL-Think and Qwen3- 4B-VL-Instruct. For each question, we generate 8 traces and compare mean accuracy between traces where a behavior is present versus absent on the same question. Reported values are mean within-question∆accuracy, with bootstrap 95% confidence intervals and the number of questions for which both present and absent traces were observed. Qwen3-4B-VL-ThinkQwen3-4B-VL-Instruct BehaviorMean∆95% CINMean∆95% CIN Confidence calibration+0.186[+0.137, +0.240]115+0.342[+0.235, +0.452]62 Self-awareness+0.136[+0.094, +0.180]178+0.131[+0.082, +0.190]130 Knowledge alignment+0.166[+0.112, +0.216]134+0.109[+0.054, +0.164]96 Evidence citation+0.149[+0.103, +0.200]149+0.107[+0.051, +0.168]95 Goal tracking+0.080[+0.042, +0.121]172+0.057[+0.020, +0.101]108 Planning present+0.002[-0.030, +0.037]188+0.028[-0.006, +0.066]102 Self-correction+0.153[+0.103, +0.207]166+0.062[+0.015, +0.127]42 Hypothesis testing-0.005[-0.038, +0.025]198+0.045[-0.004, +0.113]43 Uncertainty acknow.-0.017[-0.054, +0.020]145-0.004[-0.061, +0.054]38 Table 19: Scaling analysis: Qwen3-VL (thinking) vs. Qwen3-VL (instruct) on MathVista, 2B to 32B parameters. Amplified behaviors show persistent gaps at all scales. Largely unchanged behaviors converge, with instruct matching or exceeding thinking at 32B. 2B4B8B32B ThInThInThInThIn Self-correction64.04.772.7 10.3 69.3 11.3 61.3 11.3 Hypothesis testing 17.32.031.08.728.09.717.7 11.0 Uncertainty ack.66.78.377.7 13.0 61.7 14.3 43.7 15.3 Self-awareness45.7 25.3 50.3 46.0 64.7 39.7 66.0 63.3 Conf. calibration40.7 32.0 43.3 56.3 61.0 51.3 71.7 78.0 Planning55.3 37.3 62.7 43.7 67.7 45.7 78.7 59.0 Goal tracking54.7 34.7 57.0 43.3 66.3 46.0 76.0 65.0 Summary Accuracy72.0 53.7 71.7 74.0 77.0 71.0 83.3 81.7 Recovery Rate48.1 34.4 48.5 45.8 38.2 44.5 43.7 31.6 Behavioral Lift (pooled per size) Self-correction+30.0+14.2+10.3+5.9 Conf. calibration+57.7+52.8+59.3+68.7 Uncertainty ack. −4.5 −12.2 −16.8 −18.5 32 Preprint Table 20: Scaling analysis: Qwen3 (thinking) vs. Qwen2.5 (instruct) on MATH-500, 0.6B to 32B parameters. The same patterns hold as in the VLM scaling analysis: amplified behavior gaps persist across scale, largely unchanged behaviors converge, self-correction Lift diminishes while confidence calibration Lift remains high. 0.6B1.7B4B8B14B32B ThInThInThInThInThInThIn Self-correction63.4 0.0 60.3 1.7 82.6 2.0 81.4 4.6 92.3 1.7 61.4 2.3 Hypothesis testing 46.6 0.6 42.3 1.1 55.7 3.1 64.3 4.6 74.3 3.7 46.3 4.3 Uncertainty ack.67.1 1.4 53.7 1.1 58.3 0.0 53.4 0.6 59.7 0.3 29.4 0.3 Self-awareness22.0 8.3 22.0 12.0 45.7 27.1 43.4 28.0 69.1 26.3 54.9 33.1 Conf. calibration32.3 19.1 39.7 30.9 60.9 58.3 54.9 69.1 76.0 74.0 77.1 78.9 Planning72.9 64.6 88.0 68.6 91.7 93.1 94.6 97.4 97.4 95.1 97.1 95.7 Goal tracking47.7 33.7 70.3 47.1 83.4 75.7 81.7 86.0 88.0 86.3 88.6 90.0 Summary Accuracy52.6 20.3 71.4 34.6 90.6 61.4 85.1 71.4 92.3 78.6 89.4 81.4 Recovery Rate28.4 1.8 47.0 5.8 71.8 8.2 59.1 3.9 60.9 15.7 48.6 12.5 Behavioral Lift (pooled per size) Self-correction+39.7+34.7+35.7+20.6+16.0+14.8 Conf. calibration+84.8+71.4+59.4+56.5+57.5+66.2 Uncertainty ack.+19.4+20.3+17.3+3.7+2.4 −1.0 33 Preprint Table 21: Sensitivity of LLM Behavioral Lift to simulated random annotation noise. The first subtable shows the clean baseline with no injected noise. At each nonzero noise level, we randomly flip a fixed fraction of behavior labels and recompute Lift over 1000 trials. Each noisy subtable reports the clean Lift, the mean noisy Lift, the 95% range across trials, the mean rank, and the fraction of trials that preserve the original sign. Behaviors are kept in the clean baseline order for easier comparison across noise levels. (a) Clean baseline (0% label flips) BehaviorLiftRankN Knowledge alignment+0.80318282 Confidence calibration+0.79628282 Self-awareness+0.52738282 Goal tracking+0.52548282 Evidence citation+0.49158282 Planning present+0.26368282 Self-correction+0.12478282 Hypothesis testing+0.01088282 Uncertainty acknowledgment-0.13998282 (b) 5% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Knowledge alignment+0.803+0.711[+0.705, +0.718]1.85100.0% Confidence calibration+0.796+0.716[+0.711, +0.721]1.15100.0% Self-awareness+0.527+0.468[+0.459, +0.477]3.02100.0% Goal tracking+0.525+0.453[+0.444, +0.463]3.98100.0% Evidence citation+0.491+0.392[+0.380, +0.404]5.00100.0% Planning present+0.263+0.213[+0.201, +0.225]6.00100.0% Self-correction+0.124+0.104[+0.092, +0.116]7.00100.0% Hypothesis testing+0.010+0.008[-0.005, +0.021]8.0089.7% Uncertainty acknowledgment-0.139-0.117[-0.128, -0.105]9.00100.0% (c) 10% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Knowledge alignment+0.803+0.623[+0.616, +0.631]1.99100.0% Confidence calibration+0.796+0.636[+0.629, +0.643]1.01100.0% Self-awareness+0.527+0.411[+0.400, +0.422]3.00100.0% Goal tracking+0.525+0.388[+0.375, +0.400]4.00100.0% Evidence citation+0.491+0.317[+0.301, +0.332]5.00100.0% Planning present+0.263+0.175[+0.159, +0.190]6.00100.0% Self-correction+0.124+0.088[+0.072, +0.103]7.00100.0% Hypothesis testing+0.010+0.008[-0.008, +0.023]8.0081.6% Uncertainty acknowledgment-0.139-0.098[-0.112, -0.082]9.00100.0% (d) 15% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Knowledge alignment+0.803+0.539[+0.529, +0.549]2.00100.0% Confidence calibration+0.796+0.556[+0.548, +0.565]1.00100.0% Self-awareness+0.527+0.356[+0.344, +0.369]3.00100.0% Goal tracking+0.525+0.329[+0.314, +0.343]4.00100.0% Evidence citation+0.491+0.256[+0.240, +0.273]5.00100.0% Planning present+0.263+0.143[+0.126, +0.160]6.00100.0% Self-correction+0.124+0.074[+0.056, +0.091]7.00100.0% Hypothesis testing+0.010+0.006[-0.013, +0.023]8.0072.6% Uncertainty acknowledgment-0.139-0.082[-0.098, -0.064]9.00100.0% (e) 20% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Knowledge alignment+0.803+0.457[+0.447, +0.468]2.00100.0% Confidence calibration+0.796+0.477[+0.467, +0.487]1.00100.0% Self-awareness+0.527+0.302[+0.288, +0.316]3.01100.0% Goal tracking+0.525+0.275[+0.259, +0.290]3.99100.0% Evidence citation+0.491+0.206[+0.189, +0.225]5.00100.0% Planning present+0.263+0.116[+0.098, +0.134]6.00100.0% Self-correction+0.124+0.060[+0.041, +0.079]7.00100.0% Hypothesis testing+0.010+0.005[-0.014, +0.023]8.0071.0% Uncertainty acknowledgment-0.139-0.068[-0.085, -0.050]9.00100.0% 34 Preprint Table 22: Sensitivity of VLM Behavioral Lift to simulated random annotation noise. The first subtable shows the clean baseline with no injected noise. At each nonzero noise level, we randomly flip a fixed fraction of behavior labels and recompute Lift over 1000 trials. Each noisy subtable reports the clean Lift, the mean noisy Lift, the 95% range across trials, the mean rank, and the fraction of trials that preserve the original sign. Behaviors are kept in the clean baseline order for easier comparison across noise levels. (a) Clean baseline (0% label flips) BehaviorLiftRankN Confidence calibration+0.72217000 Self-awareness+0.62027000 Knowledge alignment+0.53737000 Evidence citation+0.34947000 Goal tracking+0.30657000 Self-correction+0.20167000 Planning present+0.06677000 Hypothesis testing+0.01087000 Uncertainty acknowledgment-0.16197000 (b) 5% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Confidence calibration+0.722+0.641[+0.633, +0.649]1.00100.0% Self-awareness+0.620+0.549[+0.540, +0.558]2.00100.0% Knowledge alignment+0.537+0.482[+0.473, +0.491]3.00100.0% Evidence citation+0.349+0.314[+0.304, +0.323]4.00100.0% Goal tracking+0.306+0.275[+0.266, +0.285]5.00100.0% Self-correction+0.201+0.177[+0.166, +0.188]6.00100.0% Planning present+0.066+0.059[+0.048, +0.069]7.00100.0% Hypothesis testing+0.010+0.009[-0.003, +0.021]8.0091.9% Uncertainty acknowledgment-0.161-0.144[-0.155, -0.133]9.00100.0% (c) 10% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Confidence calibration+0.722+0.563[+0.553, +0.573]1.00100.0% Self-awareness+0.620+0.481[+0.470, +0.494]2.00100.0% Knowledge alignment+0.537+0.427[+0.416, +0.439]3.00100.0% Evidence citation+0.349+0.279[+0.266, +0.293]4.00100.0% Goal tracking+0.306+0.244[+0.232, +0.257]5.00100.0% Self-correction+0.201+0.154[+0.139, +0.168]6.00100.0% Planning present+0.066+0.053[+0.038, +0.066]7.00100.0% Hypothesis testing+0.010+0.007[-0.009, +0.023]8.0080.2% Uncertainty acknowledgment-0.161-0.128[-0.142, -0.115]9.00100.0% (d) 15% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Confidence calibration+0.722+0.487[+0.474, +0.499]1.00100.0% Self-awareness+0.620+0.416[+0.402, +0.431]2.00100.0% Knowledge alignment+0.537+0.372[+0.359, +0.386]3.00100.0% Evidence citation+0.349+0.244[+0.228, +0.260]4.01100.0% Goal tracking+0.306+0.214[+0.198, +0.230]4.99100.0% Self-correction+0.201+0.132[+0.114, +0.149]6.00100.0% Planning present+0.066+0.046[+0.029, +0.063]7.00100.0% Hypothesis testing+0.010+0.006[-0.013, +0.026]8.0074.9% Uncertainty acknowledgment-0.161-0.112[-0.128, -0.096]9.00100.0% (e) 20% label flips BehaviorCleanMean noisy95% rangeMean rankSign kept Confidence calibration+0.722+0.414[+0.399, +0.427]1.00100.0% Self-awareness+0.620+0.353[+0.337, +0.368]2.00100.0% Knowledge alignment+0.537+0.319[+0.303, +0.334]3.00100.0% Evidence citation+0.349+0.208[+0.191, +0.225]4.03100.0% Goal tracking+0.306+0.184[+0.166, +0.202]4.97100.0% Self-correction+0.201+0.111[+0.093, +0.131]6.00100.0% Planning present+0.066+0.039[+0.019, +0.057]7.01100.0% Hypothesis testing+0.010+0.005[-0.017, +0.026]7.9967.7% Uncertainty acknowledgment-0.161-0.096[-0.114, -0.077]9.00100.0% 35 Preprint Table 23: Temporal position of behaviors relative to answer commitment in thinking-model traces. All four behaviors typically appear before the answer, suggesting this pattern reflects the general structure of reasoning traces rather than a property unique to calibration. BehaviorNMean beh. pos.Before answerMean gap Hypothesis testing26626.7%97.0%65.0 p Self-correction30034.0%99.0%60.0 p Uncertainty acknowledgment25335.2%95.3%56.2 p Confidence calibration30034.8%91.0%47.8 p 36 Preprint Table 24: LLM annotation taxonomy: reasoning quality, metacognitive behaviors, and reasoning types. (a) Group 1: Reasoning quality FieldDefinitionTrue ifFalse if reasoningpresentIs there actual step-by-step rea- soning rather than a direct an- swer? Shows intermediate steps, expla- nations, or deductions Jumps directly to the answer with no reasoning shown logicalstepsvalidDo the reasoning steps logically follow from each other? Each step follows logically from previous steps Contains non-sequiturs, invalid inferences, circular reasoning, or logical gaps reachescorrect conclusionDoes the CoT reasoning lead to the correct answer? Final answer matches ground truth Final answer does not match ground truth context understandingDoes it correctly understand and use information from the ques- tion or context? Correctly interprets the ques- tion, extracts relevant informa- tion, and uses correct values Misreads the question, misinter- prets information, or uses wrong values (b) Group 2: Metacognitive behaviors FieldDefinitionTrue ifFalse if planningpresentDoes it break the problem into sub-steps or make a plan up- front? Explicit problem decomposition or plan before solving Solves directly without structure hypothesistestingDoes it propose multiple possibil- ities and test them? Considers alternatives or explic- itly checks different possibilities Commits to the first interpreta- tion without considering alterna- tives selfcorrectionDoes it revise or correct earlier claims? Explicitly revises or corrects an earlier statement Never revisits or corrects earlier statements uncertainty acknowledgmentDoes it explicitly note ambiguity or uncertainty? Explicitly acknowledges ambigu- ity or uncertainty States everything with full cer- tainty regardless of ambiguity evidence citationDoes it reference specific parts of the question or context to sup- port claims? Grounds claims in specific infor- mation from the prompt or con- text Makes claims without connecting them to specific evidence confidencecalibrationDoes the model’s confidence match the certainty of its reason- ing? Uncertain when reasoning is weak; confident when reasoning is strong Overconfident with weak rea- soning, or underconfident with strong reasoning selfawarenessDoes it recognize the limits of what can be determined from given information? Notes that some conclusion can- not be determined from the avail- able information Claims certainty about things that cannot be determined goal trackingDoes it maintain and reference sub-goals in multi-step prob- lems? Tracks progress toward interme- diate goals Loses track of the original goal or solves the wrong sub-problem knowledge alignmentDoes it invoke appropriate do- main knowledge to solve the problem? Applies relevant mathematical, scientific, or logical concepts Manipulates symbols without do- main understanding, or applies the wrong knowledge (c) Group 3: Reasoning types FieldDefinitionTrue ifFalse if mathematicalreasoningInvolves calculations, equations, formulas, or numerical opera- tions Uses arithmetic, algebra, geome- try, statistics, or proofs No mathematical operations are present logicalreasoningInvolves deductive or inductive logic, formal reasoning Uses if-then reasoning, deduc- tion, syllogisms, or truth-table style logic No formal logical operations are present causalreasoningAnalyzes cause-effect relation- ships Explains mechanisms, causes, or effects No causal analysis analogicalreasoningMakes comparisons or draws analogies Uses analogy or cross-domain comparison No analogies or comparisons proceduralreasoningFollows or describes step-by-step processes or algorithms Executes sequential operations or procedures No procedural steps factualrecallRelies primarily on retrieving fac- tual knowledge Uses definitions, historical facts, or established knowledge with- out derivation No factual retrieval is needed conceptualreasoningReasons about abstract concepts and their relationships Relates concepts or reasons about abstract ideas No abstract conceptual work 37 Preprint Table 25: LLM annotation taxonomy: failure modes and summary metrics. (a) Group 4: Failure modes FieldDefinitionTrue if (failure occurred)False if (no failure) factualerrorContains incorrect factual claims, knowledge errors, or calculation mistakes Includes wrong facts, wrong defi- nitions, wrong formulas, or math errors All facts and calculations are cor- rect logical failureReasoning steps do not logically connect or contain errors Contains invalid inferences, con- tradictions, or non-sequiturs All logical steps are valid context misreadMisunderstands or misinterprets the question or given information Answers the wrong question, misreads values, or misinterprets constraints Correctly understands the ques- tion and context knowledge gapLacks necessary domain knowl- edge to solve the problem Does not know required con- cepts, formulas, or facts Has the necessary domain knowl- edge posthocrationalizationAppears to give the answer first, then make up reasoning Reasoningseemsreverse- engineered and does not support the conclusion Reasoning genuinely leads to the conclusion shortcutSkips important logical or com- putational steps Jumps to the conclusion without necessary intermediate reasoning or calculation All necessary steps are shown luckyguessReaches the correct answer but the reasoning is wrong or irrele- vant Correct answer with invalid or irrelevant reasoning path Correct answer with valid sup- porting reasoning (b) Group 5: Summary metrics FieldValuesDefinition complexityscore1–51 = single-step inference; 2 = sequential 2–3 steps; 3 = multi-step with dependencies; 4 = hierarchical decomposition with sub-problems; 5 = complex reasoning with hypothesis testing and self-correction reasoningdirectionforward / backward / mixedforward = starts from given information and works toward the an- swer; backward = starts from answer options and works backward to justify; mixed = combines both approaches efficiencyconcise / verbose / insufficient concise = appropriate length with necessary steps; verbose = unnec- essarily long or repetitive; insufficient = too short and missing key steps overallqualitystrong / flawed / poorstrong = sound reasoning, accurate logic, and coherent chain to the correct conclusion; flawed = reasoning present with notable issues; poor = major failures, severe errors, or no real reasoning wouldmakegoodjudgetrue / falseWhether the response would be trustworthy for judging other mod- els’ reasoning 38 Preprint Table 26: VLM annotation taxonomy: visual grounding, reasoning quality, and ad- vanced/metacognitive behaviors. (a) Group 1: Visual grounding FieldDefinitionTrue ifFalse if visualreferences presentDoes the CoT explicitly mention specific visual elements? Mentions concrete visual details such as objects, colors, counts, or positions Generic reasoning with no visual specifics visual claimsaccurateAre all visual descriptions factu- ally correct? All mentioned visual details match the image Hallucinated objects, wrong col- ors, incorrect counts, or wrong spatial relations visualinputnecessaryCould this question be answered without seeing the image? Requires specific visual details to answer Could be answered from ques- tion text alone or general knowl- edge (b) Group 2: Reasoning quality FieldDefinitionTrue ifFalse if reasoningpresentIs there actual step-by-step rea- soning rather than a direct an- swer? Shows intermediate steps, expla- nations, or deductions Jumps directly to the answer with no reasoning shown logicalstepsvalidDo the reasoning steps logically follow from each other? Steps are coherent and validContains non-sequiturs, invalid inferences, circularity, or logical gaps reachescorrect conclusionDoes the CoT lead to the correct answer? Final answer matches ground truth Final answer is wrong (c) Group 3: Advanced and metacognitive behaviors FieldDefinitionTrue ifFalse if planningpresentDoes it break the problem into sub-steps or make a plan up- front? Explicit decomposition or plan before solving Solves directly without structure hypothesistestingDoes it propose multiple possibil- ities and test them? Considers alternatives or checks competing interpretations Commits to the first interpreta- tion selfcorrectionDoes it revise or correct earlier claims? Explicitly revises or retracts an earlier step Never revisits earlier claims uncertainty acknowledgmentDoes it explicitly note ambiguity or uncertainty? Notes uncertainty or ambiguity States everything with full cer- tainty evidencecitationDoes it reference specific visual evidence to support claims? Grounds claims in specific image details Makes claims without visual sup- port confidencecalibrationDoes confidence match reasoning strength? Uncertain when reasoning is weak; confident when reasoning is strong Overconfident with weak rea- soning, or underconfident with strong reasoning selfawarenessDoes it recognize the limits of what can be determined from the image? Notes that some conclusion can- not be determined from available visual information Claims certainty beyond what the image supports goaltrackingDoes it maintain and reference sub-goals in multi-step prob- lems? Tracks progress toward interme- diate goals Loses track of the original objec- tive knowledgealignmentDoes it invoke appropriate do- main schemas to interpret visual content? Applies relevant domain con- cepts to solve the problem Only describes surface features or applies the wrong domain knowledge 39 Preprint Table 27: VLM annotation taxonomy: reasoning types, failure modes, and summary metrics. (a) Group 4: Reasoning types FieldDefinitionTrue ifFalse if spatialreasoningAnalyzes position, layout, or rel- ative location Reasons about left/right, above/- below, containment, distance, or orientation No spatial analysis counting reasoningInvolves quantitative analysis or enumeration Counts objects, compares quanti- ties, or estimates numerically No counting or quantity compar- ison compositionalreasoningCombines multiple attributes or objects Integrates several visual at- tributes or elements Relies on a single attribute only causal reasoningAnalyzes cause-effect relation- ships Explains mechanisms or effectsNo causal analysis mathematicalreasoningInvolves calculations, equations, charts, or graphs Uses arithmetic, geometry, or quantitative interpretation No mathematical operations temporal reasoningInvolves sequence, time, or tem- poral relations Reasons about before/after, or- der, stages, or prediction over time No temporal analysis (b) Group 5: Failure modes FieldDefinitionTrue ifFalse if visualhallucinationClaims objects or attributes not present in the image Mentions nonexistent objects, col- ors, text, or features All visual claims are accurate visual neglectMisses critical visual details that are present Overlookskeyinformation needed for the answer Attends to relevant visual infor- mation logicalfailureReasoning steps do not connect or contain errors Invalid inferences,contradic- tions, or non-sequiturs All logical steps are valid language biasOver-relies on text priors instead of visual content Answer follows stereotypes or textual priors rather than image evidence Reasoning is grounded in the im- age posthocrationalizationAppears to give the answer first, then invent reasoning Reasoningseemsreverse- engineered and does not support the conclusion Reasoning genuinely leads to the conclusion shortcut Skips important reasoning steps or bypasses visual analysis Jumps to the conclusion without needed intermediate steps Complete reasoning chain is present luckyguessReaches the correct answer with wrong or irrelevant reasoning Correct answer but invalid sup- porting reasoning Correct answer with valid sup- porting reasoning (c) Group 6: Summary metrics FieldValuesDefinition complexityscore1–51 = single-step inference; 2 = sequential 2–3 steps; 3 = multi-step with dependencies; 4 = hierarchical with sub-problems; 5 = complex reasoning with hypothesis testing and corrections reasoningdirectionforward / backward / mixedforward = starts from given information and works toward the an- swer; backward = starts from answer options and works backward; mixed = combines both efficiencyconcise / verbose / insufficient concise = appropriate length with necessary steps; verbose = unnec- essarily long or repetitive; insufficient = too short and missing key steps overallqualitystrong / flawed / poorstrong = sound reasoning and accurate grounding; flawed = reason- ing present but with notable issues; poor = major failures or no real reasoning wouldmakegoodjudgetrue / falseWhether this model’s reasoning would be trustworthy for judging other models’ responses 40 Preprint LLM Behavioral Annotation Prompt You are evaluating the chain-of-thought reasoning quality of a text-only language model. Ana- lyze the model’s reasoning process using the taxonomy below. Be strict and objective. Group 1: Reasoning Quality reasoningpresent:Is there actual step-by-step reasoning rather than a direct an- swer?logicalstepsvalid: Do the reasoning steps logically follow from each other? reachescorrectconclusion:Does the CoT reasoning lead to the correct answer? contextunderstanding: Does it correctly understand and use information from the question or context? Group 2: Shared Higher-Order Behaviors planningpresent: Does it break the problem into sub-steps or make a plan upfront? hypothesistesting: Does it propose multiple possibilities and test them?selfcorrection: Does it revise or correct earlier claims? uncertainty acknowledgment: Does it explicitly note ambiguity or uncertainty?evidencecitation: Does it reference specific parts of the question or context to support claims?confidencecalibration: Does the model’s confidence match the certainty of its reasoning?selfawareness: Does it recognize the limits of what can be determined from given information?goaltracking: Does it maintain and reference sub-goals in multi-step problems?knowledgealignment: Does it invoke appropriate domain knowledge to solve the problem? Group 3: Reasoning Types mathematicalreasoning,logicalreasoning,causalreasoning,analogicalreasoning, proceduralreasoning, factualrecall, conceptualreasoning. Multiple can be true. Group 4: Failure Modes factualerror: Incorrect factual claims or calculation mistakes.logicalfailure: Reasoning steps don’t logically connect.contextmisread: Misunderstands the question or given in- formation.knowledgegap: Lacks necessary domain knowledge.posthocrationalization: Reasoning appears reverse-engineered from the answer.shortcut: Skips important logical or computational steps.luckyguess: Reaches correct answer but reasoning is wrong. For all failure modes, true means the failure occurred. Group 5: Summary Metrics complexityscore(1–5),reasoningdirection(forward/backward/mixed),efficiency(con- cise/verbose/insufficient),overallquality(strong/flawed/poor),wouldmakegoodjudge (true/false). Evaluate each metric independently. Be strict: only mark true if clearly demonstrated. For failure modes, false means the failure did not occur. Respond with valid JSON only. 41 Preprint LLM Annotation Output Format "evaluation": "reasoningquality": "reasoningpresent": true|false, "logical stepsvalid": true|false, "reachescorrectconclusion": true|false, "contextunderstanding": true|false , "advanced andmetacognitive": "planningpresent": true|false, "hypothesistesting": true|false, "selfcorrection": true|false, "uncertaintyacknowledgment": true|false, "evidence citation": true|false, "confidence calibration": true|false, "selfawareness": true|false, "goaltracking": true|false, "knowledge alignment": true|false , "reasoningtypes": "mathematicalreasoning": true|false, "logicalreasoning": true|false, "causalreasoning": true|false, "analogicalreasoning": true|false, "procedural reasoning": true|false, "factualrecall": true|false, "conceptualreasoning": true|false , "failuremodes": "factualerror": true|false, "logicalfailure": true|false, "context misread": true|false, "knowledgegap": true|false, "post hocrationalization": true|false, "shortcut": true|false, "luckyguess": true|false , "summarymetrics": "complexity score": 1--5, "reasoningdirection": "forward"|"backward"|"mixed", "efficiency": "concise"|"verbose"|"insufficient", "overallquality": "strong"|"flawed"|"poor", "would makegoodjudge": true|false , "evaluator notes": "Brief justification for key ratings" 42 Preprint VLM Behavioral Annotation Prompt You are evaluating the chain-of-thought reasoning quality of a vision-language model. Analyze the model’s reasoning process using the taxonomy below. Be strict and objective. Group 1: Visual Grounding visualreferencespresent: Does the CoT explicitly mention specific visual elements? visualclaimsaccurate: Are all visual descriptions factually correct?visualinputnecessary: Could this question be answered without seeing the image? Group 2: Reasoning Quality reasoningpresent:Is there actual step-by-step reasoning rather than a direct an- swer?logicalstepsvalid: Do the reasoning steps logically follow from each other? reachescorrectconclusion: Does the CoT reasoning lead to the correct answer? Group 3: Shared Higher-Order Behaviors planning present: Does it break the problem into sub-steps or make a plan upfront? hypothesistesting: Does it propose multiple possibilities and test them?selfcorrection: Does it revise or correct earlier claims?uncertaintyacknowledgment: Does it explicitly note ambiguity or uncertainty?evidencecitation: Does it reference specific visual evidence to support claims?confidencecalibration: Does the model’s confidence match the certainty of its reasoning?selfawareness: Does it recognize the limits of what can be determined from the image?goaltracking: Does it maintain and reference sub-goals in multi-step problems? knowledgealignment: Does it invoke appropriate domain schemas to interpret visual content? Group 4: Reasoning Types spatial reasoning,countingreasoning,compositionalreasoning,causalreasoning, mathematicalreasoning, temporalreasoning. Multiple can be true. Group 5: Failure Modes visualhallucination: Claims objects or attributes not present in the image.visualneglect: Misses critical visual details that are present.logicalfailure: Reasoning steps don’t log- ically connect.languagebias: Over-relies on text priors, ignores actual visual content. posthocrationalization: Reasoning appears reverse-engineered from the answer.shortcut: Skips important reasoning steps or bypasses visual analysis.luckyguess: Reaches correct answer but reasoning is wrong. For all failure modes, true means the failure occurred. Group 6: Summary Metrics complexity score(1–5),reasoningdirection(forward/backward/mixed),efficiency(con- cise/verbose/insufficient),overallquality(strong/flawed/poor),wouldmakegoodjudge (true/false). Evaluate each metric independently. Be strict: only mark true if clearly demonstrated. For failure modes, false means the failure did not occur. Respond with valid JSON only. 43 Preprint VLM Annotation Output Format "evaluation": "visualgrounding": "visualreferencespresent": true|false, "visual claimsaccurate": true|false, "visualinputnecessary": true|false , "reasoningquality": "reasoning present": true|false, "logicalstepsvalid": true|false, "reachescorrectconclusion": true|false , "advancedandmetacognitive": "planning present": true|false, "hypothesistesting": true|false, "selfcorrection": true|false, "uncertaintyacknowledgment": true|false, "evidence citation": true|false, "confidencecalibration": true|false, "selfawareness": true|false, "goaltracking": true|false, "knowledgealignment": true|false , "reasoningtypes": "spatialreasoning": true|false, "countingreasoning": true|false, "compositionalreasoning": true|false, "causalreasoning": true|false, "mathematicalreasoning": true|false, "temporalreasoning": true|false , "failuremodes": "visualhallucination": true|false, "visualneglect": true|false, "logicalfailure": true|false, "languagebias": true|false, "posthocrationalization": true|false, "shortcut": true|false, "luckyguess": true|false , "summarymetrics": "complexityscore": 1--5, "reasoningdirection": "forward"|"backward"|"mixed", "efficiency": "concise"|"verbose"|"insufficient", "overall quality": "strong"|"flawed"|"poor", "wouldmakegoodjudge": true|false , "evaluatornotes": "Brief justification for key ratings" 44 Preprint Confidence Calibration ✓ Correct Question: If 2 8 = 4 x , what is the value of x? Model trace: First, I remember that 4 is a power of 2. Specifically, 4 = 2 2 . So 4 x = (2 2 ) x = 2 2x . Setting 8 = 2x gives x = 4. Let me check that. If x = 4, then 4 4 = 256, and 2 8 = 256. Yep, that works. Wait, let me make sure I didn’t make a mistake. Let’s verify step by step: 4 4 = 4× 4× 4× 4 = 256. So both sides are 256 when x = 4. That checks out. Another way to think about it is to take logarithms: log 2 (2 8 ) = log 2 (4 x ). The left side is 8, and the right side is x log 2 (4) = 2x. So again 8 = 2x, hence x = 4. I think that’s solid.The key was recognizing that 4 is a power of 2, then confirming the same answer through direct substitution and logarithms.No steps seem to be missing here. Final answer: 4(Groundtruth: 4) Note:The model’s certainty grows with converging support: it derivesx =4, verifies by sub- stitution, and confirms through an equivalent logarithmic formulation before committing to the answer. Model: Qwen3-4B-ThinkBenchmark: MATH-500 45 Preprint Confidence Calibration ✓ Correct Question: If a snack-size tin of peaches has 40 calories and is 2% of a person’s daily caloric requirement, how many calories fulfill a person’s daily caloric requirement? Model trace: Let the daily caloric requirement be x. Since 2% of x equals 40, 0.02x = 40. Dividing by 0.02 gives x = 40 0.02 = 2000. The snack has 40 calories and that is supposed to be 2% of the daily need. To recover the full amount, divide the part by the percentage: 40÷ 0.02 = 2000. Maybe I can convert 0.02 into a fraction. Since 0.02 = 2 100 = 1 50 , dividing by 1 50 is the same as multiplying by 50: 40× 50 = 2000. That seems right. Wait, let me check another way. If 2% is 40, then 1% is 20. So 100% is 20× 100 = 2000. Yep, that matches.So both methods give me 2000. That makes me more confident. Final answer: 2000(Groundtruth: 2000) Note:An especially clear case: the model solves the percentage equation directly, rederives the answer via a fraction conversion, and confirms it with a 1%→100% argument, explicitly noting that the agreement increases its confidence. Model: OLMo3-7B-ThinkBenchmark: MATH-500 46 Preprint Confidence Calibration ✓ Correct Question: If n≡ 2 (mod 7), find the remainder when (n + 2)(n + 4)(n + 6) is divided by 7. Model trace: Let n = 7k + 2. Then n + 2≡ 4 (mod 7),n + 4≡ 6 (mod 7),n + 6≡ 1 (mod 7). So (n + 2)(n + 4)(n + 6)≡ 4· 6· 1 = 24≡ 3 (mod 7). The remainder is 3. But let me test with a specific value to make sure. Take n = 2: (2 + 2)(2 + 4)(2 + 6) = 4· 6· 8 = 192. Since 192 = 7· 27 + 3, the remainder is 3. That checks out. Another example: take n = 9, which also satisfies n≡ 2 (mod 7). Then (9 + 2)(9 + 4)(9 + 6) = 11· 13· 15. Modulo 7, these are again 4, 6, and 1, so the product has remainder 3. Computing directly also gives remainder 3.Correct again. So the solution is correct. Final answer: 3(Groundtruth: 3) Note:The model derives the remainder symbolically, then checks two concrete substitutions that independently recover the same result; only after both checks agree does it affirm the answer. Model: Qwen3-4B-ThinkBenchmark: MATH-500 47 Preprint Uncertainty Acknowledgment + Confidence Calibration ✓ Correct Question: Convert the point(0, 3)in rectangular coordinates to polar coordinates. Enter your answer in the form (r,θ), where r> 0 and 0≤θ< 2π. Model trace: We know thatr = p x 2 + y 2 andθ = tan −1 (y/x)forx>0. Butx =0 in this case, so we can’t use the formula forθ directly. Instead, we can look at the point(0, 3)on the coordinate plane. This point is on the positive y-axis. Therefore,θ =π/2. [. . . ] But I’m confusedwhy the solution says “we can’t use the formula forθdirectly.” Because the formulatan −1 (y/x)is undefined whenx =0, so we can’t use it directly. But they still got the right answer. Wait, maybe the solution is correct. Let me think again.Another way to think about polar co- ordinates:(r,θ)corresponds to the point(r cosθ,r sinθ). So we wantr cosθ =0 andr sinθ =3. [. . . ] Now,sinθ =3/r>0 becauser>0 and 3>0, soθ =π/2.Yes, that makes sense.Soθ =π/2 is the correct angle. So the polar coordinates are (3,π/2).I think the solution is correct. Note:The model voices confusion about the undefined arctangent formula, then resolves it by rederivingθfromr cosθ =0 andr sinθ =3; confidence rises only once the geometric and algebraic views agree. Model: Qwen3-4B-ThinkBenchmark: MATH-500 Uncertainty Acknowledgment + Confidence Calibration ✓ Correct Question: According to the above definition of the sucker effect, which of the following options belongs to the straw effect? Model trace: [. . . ] Wait, the answer given is A, but the question asks for “straw effect” while the passage talks about “sucker effect”.This is a bit confusing. Let me check the Chinese terms to see if there’s a translation mix-up.InChinese, “sucker effect” is often translated as “free rider effect”.“Straw effect” is not a standardterminthiscontext.It’s possible that the question meant to ask for the “sucker effect” but wrote “straw effect” by mistake. Given the passage defines the “sucker effect” [. . . ] Option A describes exactly this [. . . ] Given the context, it’s likely that the intended answer is A, as it fits the sucker effect de- scribed. Note:The model flags a likely translation error in the question, states that uncertainty explicitly, and commits to A only after checking that the option fits the definition the passage actually provides. Model: Qwen3-4B-ThinkBenchmark: LogiQA2Doc ID: 163 48 Preprint Knowledge Alignment ✓ Correct Question: Simplify sec x sin x − sin x cos x . Model trace: To simplify the expression, we first recall that sec x = 1 cos x . Rewriting each term: 1 cos x sin x − sin 2 x cos x sin x = 1− sin 2 x cos x sin x . Now, recall the Pythagorean identity: sin 2 x + cos 2 x = 1, which implies 1− sin 2 x = cos 2 x. Substituting: cos 2 x cos x sin x = cos x sin x = cot x. Final answer: cot x(Groundtruth: cot x) Model: Qwen3-4B-ThinkBenchmark: MATH-500 Knowledge Alignment ✓ Correct Question: All the water lilies bloom at night, and this lotus blossoms during the day, so this lotus is not a water lily. Which of the following is very similar to the formal structure of the above reasoning? Model trace: The reasoning in the passage is: All water lilies bloom at night, this lotus blossoms during the day, so this lotus is not a water lily. Let’s break down the structure:A= water lilies,B= bloom at night, C = this lotus. All A are BC is not B ∴ C is not A This is a valid syllogism known as “Denying the Consequent” or Modus Tollens in logic.[. . . ] Option B: All goats have beards, and this sheep doesn’t have beards, so this sheep is not a goat.This matches the structure of the passage.The other options commit the fallacy of affirming the consequent , which is a distinct and invalid form of reasoning. Final answer: B(Groundtruth: B) Note:The trace names Modus Tollens and Affirming the Consequent unprompted, applying an internalized logical framework instead of matching the surface wording of the options. Model: Qwen3-4B-ThinkBenchmark: LogiQA2 49 Preprint Uncertainty Acknowledgment× Incorrect Question: A figure skater is facing north when she begins to spin to her right. She spins 2250 degrees. Which direction (north, south, east or west) is she facing when she fin- ishes her spin? Model trace: She starts facing north. Spinning to her right means she turns clockwise. A full circle is 360 degrees. So, 2250 divided by 360 equals 6 with a re- mainder of 90. [. . . ] Starting from north, turning 90 degrees to the right leads to west. Wait, I think I might be confused.Let me recheck. If you start facing north and turn clock- wise 90 degrees, you face east.Wait, no! Wait, I think I messed up the direction of the turn. [. . . ]90 degrees right (clockwise) from north is east.But the solution says west.Hmm, that’s a problem. Wait, now I’m confused. Which is it?[. . . ]Final answer: west(Groundtruth: east) Model: Qwen3-4B-ThinkBenchmark: MATH-500 Uncertainty Acknowledgment× Incorrect Question: Givenf (x) = √ x−1 x−2 , what is the smallest possible integer value forxsuch thatf (x)has a real number value?Model trace: To ensuref (x)is real, we need x −1≥0 andx −2̸=0, givingx ≥1 andx ̸=2.The smallest integer satisfy- ing this isx =1.But the correct answer is x = 3. Why is x = 1 not acceptable?[. . . ] Whenx =1, the denominator is 1−2= −1, which is nonzero, sof (1)is defined. So why is the answer 3? Hmm, maybe there’s a mistake in the problem statement or the solution? [. . . ]Maybe the solution is wrong? Or maybe there’s a misunderstanding.Wait, maybe the original problem was different [. . . ] Final answer: 3(Groundtruth: 1) Model: Qwen3-4B-ThinkBenchmark: MATH-500 Self-Awareness ✓ Correct Question: People generally think that left-handed people are more likely to have operational accidents than right-handed people, which is a misunderstanding. In fact, most household accidents — from fire burns to cut fingers — involve right-handed people. Which of the following best summarizes the loophole in the above argument? Model trace: [. . . ]The argument doesn’t consider how many left-handed people there are compared to right-handed people.Theflawisthat it compares absolute numbers withoutconsidering the base rate of each group.For ex- ample, suppose 90% of the population is right-handed.If 50% of accidents involve right-handed people, that is actually a lower accident rate than for left-handed people. Even if the absolute count is higher,the proportional rate could still favor left-handers. Option D correctly identifies this missing baseline as the core loophole. Final answer: D(Groundtruth: D) Note:The trace recognizes the limits of what the passage can establish: it flags the missing base-rate data and constructs a concrete counterexample (90% right-handed) to expose the gap. Model: Qwen3-4B-ThinkBenchmark: LogiQA2 50 Preprint Evidence Citation ✓ Correct Question: Behavior refers to what people say and do, with more than one measurement scale, which can be observed, described and recorded [. . . ] It can be public or hidden. Accordingly, which of the following is not behavior? Model trace: Let’s analyze each option [. . . ]Option C: Xiao Jun is secretly angry with his mother in the house.This is a hidden emotion that cannot be directly observed or recorded. The passage states that behaviorcan be public or hidden, but the key point is that behavior must be something that can be observed,described and recorded.Secretly be- ing angry does not meet this criterion becauseit is an internal state, not an observable action. Therefore, it is not behavior. Final answer: C(Groundtruth: C) Note:The trace anchors its reasoning to specific clauses of the definition, applying the stricter observability criterion to defeat the trap set by the ’public or hidden’ clause. Model: Qwen3-4B-ThinkBenchmark: LogiQA2 Hypothesis Testing ✓ Correct Question: Letx,y, andzbe positive real numbers such thatxyz =2. Find the minimum value of x 4 + 4y 2 + 4z 4 . Model trace: Wecanwritetheexpressionasx 4 +4y 2 +4z 4 andapplyAM-GM[. . . ] Alternative approach: apply AM-GM to x 4 , 4, 4y 2 , 4z 4 . But this does not give the minimum because 4 is constant and we don’t know if the minimum occurs when x 4 = 4. Alternative approach: apply AM-GM to x 4 , x 4 , x 4 ,y 2 , y 2 , z 4 , z 4 , z 4 .Let’s checktheproduct[. . . ]This is not constant, so it doesn’t work. Alternative approach: apply AM-GM to x 4 , 4y 2 , 4z 4 .The equality condition would re- quirex 4 =4y 2 =4z 4 [. . . ]which is not satisfied. Therefore, this approach is incorrect. Alternative approach: fix y and z such that yz = 2/x,then minimize over x [. . . ] Final answer: 16(Groundtruth: 16) Note:The model cycles through four distinct AM-GM configurations, explicitly ruling each out before pivoting to the substitution that yields the minimum. Model: Qwen3-4B-ThinkBenchmark: MATH-500 51 Preprint Self-Correction ✓ Correct Question: Compute 1− 2 + 3− 4 + 5−· + 99− 100. Model trace: First, group the sequence as (1− 2) + (3− 4) +· + (99− 100). Each pair sums to−1, and there are 50 pairs, so the total is−50. Alternatively, write 1 + (−2 + 3) + (−4 + 5) +· + (−98 + 99)− 100. Each inner pair sums to 1, and there are 49 such pairs. Then 49− 100 =−51. Wait, now I’m confused. Which one is correct? But earlier, when I tried grouping differently, I got−51. Maybe I made a mistake there. Wait, let’s recount that alternative approach. There are 49 inner pairs, so the total is 1 + 49− 100 =−50. Earlier, when I thought I got−51, I must have miscounted the grouped terms. Final answer: −50(Groundtruth: −50) Note:After its two derivations disagree (−50 vs.−51), the model recounts the terms in the alternate grouping, locates the miscount, and repairs the calculation; the correction changes the final answer. Model: Qwen3-4B-ThinkBenchmark: MATH-500 52 Preprint Self-Correction ✓ Correct Question: The function f satisfies f (x) + f (y) = f (x + y)− xy− 1 for all real numbers x and y. If f (1) = 1, find all integers n such that f (n) = n. Model trace: The model first proposes f (x) = g(x) + 1 2 x 2 + x + c, then concludes that c = 1 2 and therefore that there are no integer solutions. But wait, let me check if I made any mistakes. Hmm, maybe I should try a different approach. It then assumes f (x) = ax 2 + bx + c and compares coefficients in the functional equation: 0 = 2a− 1⇒ a = 1 2 ,2c = c− 1⇒ c =−1. Wait, this is different from the previous solution. Where did I go wrong in the first approach? In the first approach, I incorrectly set c = 1 2 . Actually, c =−1. The corrected form is f (x) = 1 2 x 2 + 3 2 x− 1. Setting f (n) = n gives 1 2 n 2 + 3 2 n− 1 = n and hence (n + 2)(n− 1) = 0. Final answer: −2, 1(Groundtruth: −2, 1) Note:The model reaches a wrong solution set, rechecks the algebra, identifies the mistaken constant term (c = 1 2 vs. c =−1), and updates the final answer. Model: Qwen3-4B-ThinkBenchmark: MATH-500 53