Paper deep dive
Measuring Reasoning Trace Legibility: Can Those Who Understand Teach?
Dani Roytburg, Shreya Sridhar, Daphne Ippolito
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/24/2026, 3:23:26 AM
Summary
This paper introduces Legi-Val, a unified framework for measuring the legibility of reasoning traces produced by Reasoning Language Models (RLMs). The authors evaluate 90k+ traces from 12 RLMs across three datasets (MATH, GPQA, Connections) using efficiency-based metrics (length, redundancy, backtracking) and a novel transfer utility metric that measures how well a strong model's reasoning trace helps weaker student models arrive at correct answers. Key findings include: (1) highest-accuracy models rank lowest on legibility; (2) reward models do not intrinsically reward legibility; (3) efficiency and transfer utility form a Pareto frontier; (4) legibility is task- and audience-dependent.
Entities (40)
Relation Signals (31)
Dani Roytburg → affiliatedwith → Carnegie Mellon University
confidence 99% · Machine Learning Department, Carnegie Mellon University, Pittsburgh, United States
Shreya Sridhar → affiliatedwith → Carnegie Mellon University
confidence 99% · Machine Learning Department, Carnegie Mellon University, Pittsburgh, United States
Daphne Ippolito → affiliatedwith → Carnegie Mellon University
confidence 99% · Language Technologies Institute, Carnegie Mellon University, Pittsburgh, United States
Legi-Val → evaluatedon → Connections Dataset
confidence 99% · Measured on 99,528 reasoning traces from 12 models across three diverse datasets (MATH, GPQA, Connections).
Legi-Val → evaluatedon → GPQA Dataset
confidence 99% · Measured on 99,528 reasoning traces from 12 models across three diverse datasets (MATH, GPQA, Connections).
Legi-Val → evaluatedon → MATH Dataset
confidence 99% · Measured on 99,528 reasoning traces from 12 models across three diverse datasets (MATH, GPQA, Connections).
Legi-Val → includesmetric → Regression Rate
confidence 99% · Regression Rate measures how often additional steps confuse rather than help.
Legi-Val → includesmetric → Transfer Utility
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language models are increasingly being trained to "reason" before answering users' queries, outputting hundreds or even thousands of tokens worth of deliberation before their final answer. While the main intention of reasoning is to improve models' ability to arrive at a correct answer, we argue that these models should be assessed for the legibility of their reasoning traces in addition to the correctness of their final answers. In this paper, we evaluate 90k traces from 12 Reasoning Language Models (RLMs) for the quality of their reasoning traces. We introduce the concept of transfer utility, which assesses how useful an RLM's reasoning traces are for guiding a weaker, non-reasoning model toward arriving at the correct answer. We find that the reasoning traces of the highest-performing models rank among the lowest for legibility. Furthermore, we uncover tensions between efficiency-based measurements of legibility (such as trace length) and transfer utility. These tensions establish a legibility Pareto frontier, and we demonstrate that an RLM's ability to output highly legible traces can be a task- and audience-dependent goal. Crucially, we find that reward models used to train RLMs do not intrinsically reward legibility. Together, these metrics and the findings they surface chart a path towards scaffolding reasoning traces for a multi-agent future.
Tags
Links
- Source: https://arxiv.org/abs/2603.20508v1
- Canonical: https://arxiv.org/abs/2603.20508v1
Trouble viewing inline? Open PDF directly →
Full Text
76,858 characters extracted from source content.
Expand or collapse full text
Measuring Reasoning Trace Legibility: Can Those Who Understand Teach? Dani Roytburg * 1 Shreya Sridhar * 1 Daphne Ippolito 2 Abstract Language models are increasingly being trained to “reason” before answering users’ queries, out- putting hundreds or even thousands of tokens worth of deliberation before their final answer. While the main intention of reasoning is to im- prove models’ ability to arrive at a correct answer, we argue that these models should be assessed for the legibility of their reasoning traces in addition to the correctness of their final answers. In this paper, we evaluate 90k traces from 12 Reason- ing Language Models (RLMs) for the quality of their reasoning traces. We introduce the concept of transfer utility, which assesses how useful an RLM’s reasoning traces are for guiding a weaker, non-reasoning model toward arriving at the cor- rect answer. We find that the reasoning traces of the highest-performing models rank among the lowest for legibility. Furthermore, we uncover ten- sions between efficiency-based measurements of legibility (such as trace length) and transfer utility. These tensions establish a legibility Pareto fron- tier, and we demonstrate that an RLM’s ability to output highly legible traces can be a task- and audience-dependent goal. Crucially, we find that reward models used to train RLMs do not intrin- sically reward legibility. Together, these metrics and the findings they surface chart a path towards scaffolding reasoning traces for a multi-agent fu- ture. 1. Introduction “Test-time compute”—scaling the number of intermediate tokens language models generate before outputting answers to hard questions—has emerged as the dominant paradigm for approximating human System-I thinking (Kahneman, * Equal contribution 1 Machine Learning Department, Carnegie Mellon University, Pittsburgh, United States 2 Language Tech- nologies Institute, Carnegie Mellon University, Pittsburgh, United States.Correspondence to: Dani Roytburg <droyt- bur@andrew.cmu.edu>. Preprint. March 24, 2026. Figure 1. Ranking of 12 reasoning models byLegi-Val, broken down into key metrics. Each model exhibits distinct strengths and weaknesses across dimensions. A full, metric-wise breakdown can be found in Appendix B.3 2011). These intermediate tokens, often referred to as “rea- soning traces” are trained primarily through reinforcement learning on verifiable rewards (RLVR), which means they are optimized largely for one objective: final answer cor- rectness. This optimization strategy treats reasoning traces as ancil- lary outputs to be minimized or discarded, a framing that underestimates the potential of reasoning traces as primary artifacts in their own right. In this paper, we argue for the need for high-quality, legible reasoning traces that are de- signed not just as a means to arrive at correct answers but as a mechanism for auditing and verifying model behavior. For highly capable AI systems requiring human oversight or automated verification, the intrinsic properties of reason- ing traces become invaluable. Indeed, interpretable natural language reasoning traces offer a path to scalable oversight (Korbak et al., 2025; Emmons et al., 2025)–the ability for hu- mans to supervise the actions of superhuman AI—but only if weak verifiers (humans or smaller models) can identify logical gaps and errors. Likewise, distillation of reason- ing capabilities into smaller, deployable models remains essential, with recent work showing that trace quality dra- matically impacts learning outcomes (Li et al., 2025; Zhang et al., 2025a). A similar need for legibility arises in caching 1 arXiv:2603.20508v1 [cs.MA] 20 Mar 2026 Measuring Reasoning Trace Legibility Figure 2.Legi-Val(1) elicits reasoning traces from RLMs on a common problem set; (2) analyzes the length of traces, the number of semantically redundant steps, and the frequency of backtracking; (3) passes progressively longer prefixes of traces to weaker student models, making measurements on student accuracy as a function of prefix length. In the given example,GPT-OSS-120Bproduces a trace which scores well on efficiency metrics at the cost of weak student transfer, whileDeepSeek-R1produces a more “teachable” trace at the cost of efficiency. and re-use of steps in a trace (Didolkar et al., 2025; Ahmed et al., 2025), which assumes demarcated atomic reasoning units for retrieval (Wei et al., 2021). Crucial to these applications is a notion of legibility: the ability to easily disentangle, comprehend and verify indi- vidual steps in a reasoning trace. However, research on measuring legibility for weak verifiers and human beings remains under-specified. Length-based heuristics risk ob- fuscation via compression (Sun et al., 2025; Hassid et al., 2025), coherence measures (Samineni et al., 2025) capture local consistency without guaranteeing global validity, and prior definitions of “legibility” (Kirchner et al., 2024; Em- mons et al., 2025) focus narrowly on verification, neglecting knowledge transfer and compositional reuse. A coherent, unified metric measuring legibility presupposes the neces- sary reward signals for downstream use. Contributions We proposeLegi-Val, a unified frame- work decomposing legibility along two axes: Efficiency- based dimensions measured by step- and token-length, re- dundancy in embedding space, and backtracking on strategy oscillations; and transfer-based dimensions measured by the effectiveness and smoothness of transferring reason- ing to weaker models through progressive rollouts. For both axes, we implement a practical set of metrics which can be computed at scale on large corpora of reasoning traces. While prior work examines individual properties, Legi-Valuniquely relates legibility to the multi-agent task of passing traces to weaker verifiers. Measured on 99,528 reasoning traces from 12 models across three diverse datasets (MATH, GPQA, Connections), we find that: (i) state-of-the-art models perform relatively weakly on either efficiency- or transfer-based metrics. (i) re- ward models only reward legibility as a function of accuracy; on correct-only traces, no correlation persists between scalar rewards and transfer success (ρ = .05). (i) legibility axes have clear trade-offs, producing a Pareto frontier between efficiency-based measures and weak-to-strong behavior. Legi-Val provides the first suite of targets for reward signals on multi-agent legibility, establishing the intrinsic— not instrumental—value of reasoning. We release code 1 , data, and a dashboard for reproducibility. 2. Background and Related Work 2.1. The Reasoning Paradigm Reasoning is an emergent behavior that can be elicited from general-purpose LLMs via prompting (Wei et al., 2023; Ko- jima et al., 2022). Subsequent work showed that supervised fine-tuning on reasoning trace could accentuate this behavior (Cobbe et al., 2021). More recently, reinforcement learning methods that reward reasoning have created breakthroughs in model reasoning capabilities using reward models (Light- man et al., 2023; Kumar et al., 2024; Zhang et al., 2025b) and verifiable rewards (DeepSeek-AI et al., 2025) to guide internal reasoning states toward current, state-of-the-art per- formance. We’l refer to models that have been explicitly trained to reason as reasoning language models (RLMs). All these methods are designed to take advantage of test- time compute—scaling the number of intermediate tokens generated before a final answer is produced (Muennighoff et al., 2025). Use of test-time compute is increasingly rival- ing model size as a scaling dimension to achieve improved model performance (Snell et al., 2024; Ke et al., 2025). These techniques produce natural language traces optimized almost exclusively for correctness. While work on “over- thinking” (Hassid et al., 2025; Hong et al., 2025) proposes efficiency objectives (Lu et al., 2025; Ning et al., 2025), such objectives focus on computational cost rather than multi-agent utility. At the extreme, efficiency converges to latent reasoning architectures (Hao et al., 2025) that entirely 1 code:github.com/ippolito-cmu/legible_cot_ preprint and dashboard: 104.131.95.244 2 Measuring Reasoning Trace Legibility sacrifice legibility for performance. 2.2. Defining and Measuring Legibility Prior work defines legibility inconsistently. Kirchner et al. (2024) defines it by how well weak verifiers can invalidate misaligned responses. Emmons et al. (2025) define it as whether a human can read and follow the reasoning trace, which they approximate with an AI judge. Jose (2025) like- wise use an AI judge that looks for semantically meaningless text. Bhambri et al. (2025) ask human to verify whether rea- soning traces are interpretable. Through reliance on strong LLM or human annotations, these definitions may limit rel- evance to the capability-asymmetric and cost-constrained use cases that require legibility, such as monitoring founda- tion models or distilling their traces for re-use, training, or routing. Reasoning trace efficiency has also been measured in prior work. Token and step counts directly capture efficiency, though findings are contradictory—some work shows neg- ative correlation between length and accuracy (Sun et al., 2025; Nayab et al., 2025), while others pursue compres- sion (Xia et al., 2025; Ning et al., 2025). Works looking at redundancy have used attention heads to prune redundant content(Choi et al., 2025; Hwang et al., 2025) and have applied semantic similarity with moving windows to pe- nalize generation of redundant steps. (Hong et al., 2025). Finally, metrics measuring backtracking identify strategy exploration through lexical cues (Lu et al., 2025) or evaluate linearized versus parallel search (Qin et al., 2025). Prior work has also explored content-based metrics that assess textual quality. Lee & Hockenmaier (2025) catalog dimensions of factuality, validity, and coherence. Samineni et al. (2025) distinguish local step-to-step coherence from global validity, finding that RLM optimize for coherence over validity; coherence, however, captures logical flow between adjacent steps, not whether weak verifiers can spot reasoning gaps; a locally coherent trace might skip critical reasoning only strong models can fill. Our work combines efficiency metrics introduced in prior work with a novel transfer utility measurement that records the pedagogical effectiveness of strong RLMs teaching weaker LLMs how to do tasks. Our transfer utility met- ric complements efficiency-based metrics that cannot detect reasoning gaps. 2.3. Weak-to-Strong Generalization Transfer utility connects to weak-to-strong generalization, where large, strong models train with respect to signals produced by weaker models (Burns et al., 2023). Zhang et al. (2025a) study teacher-to-student transfer for accelerating post-training, while Lang et al. (2025); Wu et al. (2025) propose active learning curricula based on traces. Pal et al. (2026) follow a similar framework, measuring the cross- model response consistency conditioned on a reasoning trace. We index these behaviors over teachers rather than students, measuring how teacher trace properties scaffold student learning. Legibility of strong model outputs has been tied to weak-to- strong generalization. Kirchner et al. (2024) defines legibil- ity with a prover-verifier game, where a weak verifier must learn to invalidate responses from a stronger, misaligned prover. We extend this setup to intermediate reasoning traces and reframe it as a non-adversarial, pedagogical interaction, enabling connections between legibility for safety and for distillation, caching and re-use. 3. Legi-Val Framework TheLegi-Valframework for assessing reasoning trace legibility decomposes legibility into an efficiency dimen- sion, which measures the length, semantic redundancy, and backtracking tendencies; and an interaction-based dimen- sion, which measures the success of a weaker LLM tasked with continuing a partial, prefixed trace. 3.1. Efficiency Dimension Motivation. Efficiency captures the verbosity of reason- ing traces. Empirical findings in applied psychology suggest that passage length (Forrin et al., 2019), redundant semantics (Kalyuga et al., 1999), and non-linear reasoning structures (De Bock et al., 2003) increase cognitive load for human beings, thereby reducing readability and task performance. We operationalize each of these to capture how “excess” in reasoning traces impedes legibility. Our three efficiency metrics measure length, redundancy, and backtracking. Length. The most intuitive legibility metric is conciseness— how economically traces convey necessary reasoning. For a reasoning traceRcontainingntokens split intomsteps, we count the efficiency of a trace as the reciprocal of its token length ( 1 n ) or step length ( 1 m ). We use NLTK’s sentence tokenizer to segment reasoning traces into steps. Token efficiency captures direct token counts, while step efficiency captures the number of steps in a trace For cross-problem comparison, we normalize by median lengths for correct solutions. Redundancy. Length metrics do not measure information density. A critical failure mode in test-time compute is se- mantic redundancy—repeating similar reasoning steps with- out adding new information. Following Hong et al. (2025), we compute sentence embeddings and measure maximum cosine similarity between each steps i and all preceding steps: 3 Measuring Reasoning Trace Legibility Redundancy(s i ) = max j<i sim(s i ,s j )(1) Using MiniLM-L6-v2 to capture embeddings, we set a thresholdτ = 0.8(validated manually; see Appendix A.2) and compute the proportion of steps exceeding it as the trace-level redundancy score. Backtracking. Strategy switches and reversals indicate exploratory reasoning but also create discontinuities that impede comprehension. We use an LLM judgeJto classify whether each steps i represents backtracking given prior context.Jreturns a list of backtracking steps found in the input trace, We report the proportion of steps that are labeled as backtracking. See Appendix A.3 for details. 3.2. Transfer Utility: Multi-Agent Dynamics Figure 3. Transfer utility measures how effectively a teacher model’s reasoning trace scaffolds a weaker student model toward correct answers. We evaluate student accuracy as they progres- sively receive more steps from the teacher’s trace, isolating peda- gogical quality from correctness. Motivation. Our efficiency metrics measure excess reason- ing, but they do not indicate the quality of the reasoning trace. For example, a single sentence reasoning trace saying “the answer is 5” is minimally short, with no backtracking or redundancy. However, most would agree this isn’t really a reasoning trace at all since it simply gives the answer. Thus, we need a way to measure that a reasoning trace actually contains the steps that would help someone else, either a human or a weaker model, to arrive at the correct answer. We make the following assumption–that a good reasoning trace is one where additional steps of reasoning are helpful. A reasoning trace that skips steps which are essential for task completion, is less legible than one that has all steps. We measure this property through “transfer utility,” whether a weaker LLM, when shown the reasoning trace from an RLM can use it to arrive at the correct answer. Moreover, for a highly legible RLM, we would expect each additional step of the RLM’s reasoning trace to increase the chance that the weak LLM arrives at the correct answer. Transfer utility curve. Consider a reasoning trace as a sequence of stepsR p = s 1 ,s 2 ,...,s m produced by a teacher RLMTwhich successfully answered problemp. LetSbe the correctness of a weak student prompted to continue a reasoning trace prefixR (k) p = s 1 ,s 2 ,...,s k fork = 1,...,m. Then, we can define a transfer utility curve f (k) measuring student accuracy after k steps: f (k) = 1 |p| X p∈P S(R (k) p )(2) We quantify three properties of the transfer utility curve. First-order transfer utility. This measures the overall effectiveness of all possible prefixes in a reasoning trace by measuring the area under the curve: FOTU(T,S,P ) = 1 m m X k=1 f (k)(3) A high FOTUS score indicates that the student needed fewer of the teacher’s reasoning steps to arrive at the correct an- swer, and more reliably stayed at that answer when provided additional reasoning steps. Intuitively, if an RLM tends to skip critical steps, student accuracy in expectation will lag until the answer is released in the final steps of the trace. Second-order transfer utility. This measures informa- tion density—whether correctness gains are meted out uni- formly across the trace or concentrated into a “bursts” of relevant information. Second-order transfer utility acts as a “cognitive regularizer” against RLMs that may maximize first-order utility by putting the answer in the first step. For each problemp, letτ p be the shortest prefix ofR p that the student succeeds with:τ p = k m ,k = minj : S(R (j) p ) = 1. Aggregated over many rollouts,τ p forms an empirical probability distribution over k: q T,S,P (x) = Pr(τ p = x),x∈ [0, 1](4) Now, we define second-order TU as the normalized entropy of this distribution: SOTU(T,S,P ) =−E x∼q T,S,P [logq T,S,P (x)](5) By taking the entropy of the distribution over first correct steps, we measure how evenly correctness gains are dis- tributed. If a reasoning trace front-loads a correct answer or else sandbags by including many irrelevant steps before placing crucial logic at the end, the distribution will be peaked and SOTU will be low. Conversely, if correctness gains are spread evenly, SOTU will be high. This opera- tionalizes the Uniform Information Density hypothesis (Wei et al., 2021), an inductive bias for language modeling which posits that cognitive information channels should distribute information uniformly. 4 Measuring Reasoning Trace Legibility Regression Rate.We measure how often additional steps confuse rather than help by counting the number of times student accuracy decreases with additional reasoning steps: R(T,S,P ) = m X k=2 ⊮[f (k) < f (k− 1)](6) Lower R indicates each step generally adds value, while high R suggests inconsistent or confusing reasoning that is difficult for the student to follow. Unlike FOTU and SOTU, R is a single, per-trace count rather than a continuous measure over the transfer utility curve; this allows us to measure local inconsistencies rather than global trends. 4. Experimental Setup 4.1. Models and Datasets Reasoning models. We evaluate 12 open-source mod- els that were post-trained to output chain-of-thought rea- soning traces: GPT-OSS-20B, GPT-OSS-120B, Qwen3- [4B, 8B, 32B], QwQ-32B, DeepSeek-R1, DeepSeek-R1- Distill-Qwen-32B NVIDIA-Llama-3.1-Nemotron-Nano-8B, NVIDIA-OpenReasoning-Nemotron-32B, Gemma-3-[12B, 27B], and Magistral-Small-2509. We do not include re- cent proprietary, closed-weight reasoning models (GPT-5, Claude-4.5-Sonnet, Gemini-2.5-Flash) in our analysis, as their APIs only provide abstractive summaries of the actual reasoning traces; comparing against these would confound legibility by forcing an apples-to-oranges comparison be- tween actual reasoning traces and the effects of different summarization techniques. Student models. For transfer utility, we use Microsoft Phi-3.5-Mini (3.8B) and Meta LLaMA-3.2-1B. Both were not explicitly trained to output reasoning, and of the two Phi is generally a more capable model. Datasets. We select three diverse reasoning domains. MATH (Hendrycks et al., 2021): 5,000 competition mathe- matics problems with terse, step-by-step solutions. GPQA (Rein et al., 2024): 448 graduate-level science questions (biology, chemistry, physics). Connections (Lopez et al., 2025): 652 word puzzles from the New York Times Con- nections puzzle. The game task is to cluster 16 words into 4 groups. 4.2. Implementation Details We briefly enumerate key details below; full details appear in Appendix A as well as in our code release. Trace generation. We generate traces using model- specific recommended hyperparameters. System prompts appear in Appendix A.1. We extract reasoning by parsing Table 1. Rank-wise model comparison by metric. Model (Acc) uses mean accuracy integers. Metrics show ordinal rank (1=best) with rank change relative to Acc. in tiny parentheses. Len.: To- ken Length; Red.: Redundancy; Back.: Backtracking; FOTU: first-order TU; SOTU: hazard-entropy TU. Bold denotes #1 rank; underline denotes #2 rank. See Table 7 in the appendix for raw values. MODEL (ACC, %)LEN. RED. BACK. FOTU SOTU GPT-OSS-120B (82)5 (-4)5 (-4)4 (-3) 11 (-10)4 (-3) DEEP-R1-0528 (77)11 (-9) 10 (-8)6 (-4)3 (-1)7 (-5) R1-DISTILL (76)6 (-3)9 (-6)9 (-6)6 (-3)5 (-2) MAGISTRAL-S (73)3 (+1)4 (0)5 (-1)10 (-6)3 (+1) QWEN3-8B (72)9 (-4) 11 (-6)12 (-7)5 (0)10 (-5) GPT-OSS-20B (70)7 (-1)6 (0)8 (-2)12 (-6)6 (0) GEMMA-3-27B-IT (68)1 (+6)3 (+4)2 (+5)7 (0)1 (+6) QWEN3-4B (68)8 (0) 12 (-4)11 (-3)8 (0)9 (-1) GEMMA-3-12B-IT (63)2(+7)2(+7)1 (+8)9 (0)2(+7) QWQ-32B (60)10 (0)7 (+3)10 (0)4 (+6)8 (+2) OPENREAS-32B (51)12 (-1)8 (+3)7 (+4)2 (+9)11 (0) LLAMA-NANO (49)4 (+8) 1 (+11)3 (+9)1 (+11)12 (0) dedicated think tokens, and grade correctness using parsers from established evaluation harnesses. Efficiency metrics.For all models, we counted tokens us- ing OpenAI’s tiktoken library, and we separated reasoning traces into steps using the NLTK sentence tokenizer. To compute embeddings for each step, we used the sentence- transformers library’sall-MiniLM-L6-v2model. To assess backtracking, we used Gemini 2.5-Flash as the AI judge. The rubric passed to the judge can be found in Ap- pendix A.3. We conducted manual validations of redun- dancy and backtracking metrics (Appendix A.3.4) as well as offer sensitivity analyses (Appendix A.2). Transfer utility.We prompted student models to continue reasoning from a truncated teacher trace, allowing students 1,024 tokens to continue reasoning. If no extractable answer came from the generated output, we appended “The final answer is: ” to the response and continue the generation for another 100 tokens. We only evaluate traces for prob- lems where the teacher answered correctly. Since traces vary by length, we use binned percentage-based progression (0-100%) rather than absolute step counts, then compute transfer-utility metrics per Equations 3–6. More details in Appendix A.4. 5. Results 5.1. Overall Rankings Evaluating our 12 teacher models across all metrics, we found that no one RLM dominates all axes of legibility, and that the relative ability of different RLMs to arrive at correct answers is decoupled from the legibility of their reasoning 5 Measuring Reasoning Trace Legibility traces. The accuracy-legibility paradox. GPT-OSS-120B and DeepSeek-R1 had the highest task accuracy among the 12 models. However, Table 1 shows that GPT-OSS-120B ranks 11th (second to last) on first-order transfer utility (FOTU) and slightly above the median on efficiency metrics; sim- ilarly, DeepSeek-R1 ranks third on FOTU at the cost of middling second-order transfer utility (SOTU) and poor efficiency and redundancy. Meanwhile, LLaMA-Nemotron- Nano, the lowest-capability model, ranks first on FOTU, redundancy, and regression rates. This pattern persists across the table: high-accuracy models underperform on either efficiency-based or transfer-based metrics, while low- capability models rank highly on FOTU. No one “legible reasoner.” The metric-based decompo- sition shows that no one model manages to rank in the top three across all six metrics. This implies a Pareto frontier, where post-training artifacts leave different models with dif- ferent legibility profiles, such that the traces of no model can both optimally scaffold the understanding of weaker models and maintain efficiency. However, we do observe two distinct profiles (see Figure 4). Efficiency at the cost of transfer utility (Gemma-27B, Gemma-12B, gpt-oss-120b): These models produce concise traces with low redundancy and minimal backtracking. For example,Gemma-12B generates median 602 tokens versus 4,931 for OpenReasoning-32B (8.2×shorter). However, their transfer utility ranks moderately (7th, 9th, 11th), sug- gesting conciseness comes at pedagogical cost. Transfer utility at the cost of efficiency (DeepSeek-R1, OpenReasoning-32B, QwQ-32B): These models generate verbose traces (3,645–4,931 tokens) with high redundancy (23–26%) and frequent backtracking, yet achieve top-3 transfer utility rankings. These profiles persist along model families. For instance, the Qwen family models exhibit moderate performance across all dimensions. DeepSeek models (R1, R1-Distill) con- sistently rank highly on transfer utility despite verbosity. Gemma models uniformly optimize for efficiency. Interpretation. Optimizing only for efficiency-based measurements of legibility is unlikely to yield RLMs whose reasoning traces have strong transferability to weak mod- els. In addition, highly capable models like GPT-OSS-120B likely have good reasoning efficiency because they don’t actually need that many reasoning tokens to arrive at the correct answer. When this good efficiency doesn’t lead to good transfer utility to weak students, we enter precisely the scenario which scalable oversight efforts seek to avoid. Figure 4. Trade-off between teacher accuracy and transfer utility, average across datasets. Colors denote models, clustered by fami- lies. 5.2. Cross-Metric Correlations To better understand the tradeoffs in Pareto optimality for our metrics, we directly consider the correlations between teacher-level statistics. Figure 10 shows model-level Spear- man correlations of rankings on our legibility metrics and task accuracy. Tradeoffs between accuracy and legibility. At the teacher level, accuracy strikes a strange relationship with transfer- based metrics. There exists an equal and opposite effect be- tween first-order (ρ =−0.47) and second-order (ρ = 0.46) transfer utility. This suggests that reasoning traces from highly capable RLMs tend not to invite student follow-on, causing higher-entropy correctness distributions. Mean- while, no correlation exists between teacher accuracy and student regression. suggests that models which scaffold students effectively do so either by front-loading correct- ness (high FOTU, low SOTU) or by distributing information evenly (low FOTU, high SOTU). This tension reflects dif- ferent pedagogical strategies: some models prioritize early wins to build student confidence, while others focus on steady knowledge building. With respect to efficiency met- rics, we find no significant correlations between accuracy and length or backtracking. Surprisingly, we find a mod- est, positive correlation between redundancy and accuracy (ρ = 0.30), suggesting that more capable models may also produce more semantically repetitive reasoning. Transfer utility metrics show internal tension. First-order transfer utility (FOTU) exhibits a strong, negative correla- tion with second-order utility (ρ =−0.68), demonstrating the necessity of “cognitive regularization” to measure leg- ibility; a high area under a student’s transfer utility curve could be driven by front-loaded correctness, which SOTU penalizes. SOTU correlates positively with a student’s re- 6 Measuring Reasoning Trace Legibility Figure 5. Scatter plots of first-order (x-axis) transfer against second-order transfer utility (uniform information), RLM accuracy, and reward model scores (y axes). Each point is a (model, dataset) pair. Colors denote models, clustered by families gression rate (ρ =−0.22); the more often a teacher induces reversions in student logic, the more uniform the informa- tion density. Optimizing for all three—high overall transfer, and uniform information density with minimal regressions— forces a balance between making reasoning accessible to weaker models without leakage. 5.3. Comparison with Reward Models To assess whether reward models capture legibil- ity dimensions, we gather scalar rewards from two models:(allenai/Llama-3.1-8B-Instruct-RM-RB2, Skywork/Skywork-Reward-V2-Llama-3.1-8B)and Legi-Valmetrics for all traces. Because reward models are trained on pairwise preferences, we analyze a subset of 902 problems which all 12 teacher models answered correctly for an even comparison with transfer metrics. This allows us to assign each trace a scalar reward and each teacher an average reward over scalars. At both the model- and trace-level, we discover no evidence of a relationship between scores and legibility metrics. At the trace level, first-order transfer utility and reward model scores bear near-zero correlation (Spearman’sρ =−0.041, Appendix B.1). At the model level, both reward models show some positive correlation with transfer metrics; how- ever this appears entirely driven by accuracy: when isolating correct traces, all correlations decrease, with only SOTU correlating with Skywork (Table 2). Table 2. Spearman correlations between reward-model scores and transfer-utility metrics. Values show “All→ Correct only”. RMFOTU AFOTU CSOTU ASOTU CRR ARR C Skywork0.108-0.0730.1870.1470.008-0.084 Allenai0.1970.0410.015-0.0520.1400.067 Mean0.153-0.016 0.1010.0470.074-0.009 5.4. Ablations and Sensitivity Analyses Rank stability across tasks. If an RLM tends to pro- duce legible reasoning traces for one task, does it also pro- duce legible traces for very different task? We computed Spearman’s rank correlation (ρ) between model rankings across task pairs (see Appendix Table 6), and found that token length and redundancy show high-to-moderate stabil- ity across datasets (avgρ∈ [0.46, 0.65]), with particularly strong agreement between GPQA and MATH (ρ≥ 0.79). Transfer utility also reveals domain-specific pedagogical challenges around reasoning trace transferability. DeepSeek- R1-0528 and OpenReasoning-Nemotron-32B are the top-2 most pedagogically effective models for both GPQA and MATH. In fact, these two domains show strong ranking agreement across for first order TU (ρ = 0.66respectively), exceeding accuracy stability (ρ = 0.47for GPQA–MATH). However, Connections exhibits low overlap with the two STEM domains among top-3 models. This suggests that a reasoning trace style that is pedagogically useful for STEM questions may be less useful for word puzzles. Similarly, QwQ-32B ranks 3rd on Connections but 10th on MATH, possibly because its exploratory reasoning style involving high backtracking benefits clustering words (the model back- tracks to revise its tentative clustering) but impedes system- atic algebraic manipulation. The low overall average corre- lation across pairs (ρ = 0.17) reflects this intentional task diversity rather than measurement noise. Unlike first-order TU’s domain-specificity, rankings for second-order TU (TU Uniformity) maintain high stability across all domain pairs (ρ ∈ [0.58, 0.91]). Information density—whether each reasoning step adds consistent incre- mental value—appears universally important regardless of whether the task involves word puzzles, physics problems, or mathematical proofs. Sensitivity to Student Model Choice We find that choice of student model does not substantially impact the stability of rankings. Choice of student model shows more stability than choice of dataset. Table 4 reports Spearman’sρbetween rankings derived from Phi-3.5-Mini and LLaMA-3.2-1B students, with rankwise stability of0.55across datasets for first-order TU and 0.50 for second-order TU. 7 Measuring Reasoning Trace Legibility 6. Discussion We presentLegi-Valas a paradigm for moving past to- ken length and answer correctness in evaluating reason- ing. While our findings show an accuracy-legibility anti- correlation and poor conditioning from reward models, these findings should not be taken as intractable. They are a call to action for future benchmarking in multi-agent systems. RL training for efficiency. Today, legibility metrics largely show artifacts of post-training strategies rather than explicit optimization. We see the results of efficiency-minded con- densed reasoning in models such as GPT-OSS-20B, as well as the ensuing impact on scaffolding for weaker students. We consider whether such a tradeoff is inevitable or a conse- quence of under-optimization. Future work may consider in- corporating legibility as an explicit objective, for instance by using multi-objective reward signals which balance correct- ness with legibility dimensions. How to weight efficiency- and scaffold-based reward depends on application area; for instance, oversight- or distillation-oriented reasoning might emphasize transfer utility, while human-facing applications require efficiency metrics in accordance with principles of cognition. Furthermore, transfer utility metrics may be directly inte- grated into GRPO algorithms by swapping a weaker stu- dent to sample rollouts, instead of the model under training. Alternatively,Legi-Valmay also be posed as a comple- ment to prover-verifier adversarial games (Kirchner et al., 2024), where our second-order transfer metrics provide use- ful checks against “cheating” pressures from post-hoc rea- soning. Domain-adaptive legibility. Transfer utility’s low stabil- ity between GPQA/MATH and Connections suggests task- dependent pedagogical requirements. RL training may re- quire domain-specific legibility signals rather than universal targets. For instance, mathematical reasoning may benefit from explicit algebraic manipulation, while commonsense reasoning may require verbose narrative explanation. Future work should investigate whether fine-grained rewards can generalize transfer utility across diverse tasks. Limitations (1) Student models. We evaluate transfer utility as in-context scaffolding, rather than after fine-tuning. To fulfill our goal of producing unified metrics for many trace-focused appli- cations, inference-time behaviors hold the most relevance. For distillation in particular, however, there may be a gap between transfer utility and training. Future work could tieLegi-ValtoAlice(Wu et al., 2025) to articulate optimal teacher policies for distillation. (2) Human evaluation. The limited scale of our human eval- uation limits our ability to determine alignment between our metrics and embodied cognitive load. While interdisci- plinary work has investigated cognitive aspects of our effi- ciency metrics, future work should compare directly with transfer-based definitions. (3) LLM Judge for Backtracking Detection. Our backtrack- ing metric relies on LLM judge classification. While we validate against human annotation (Appendix A.3), judge reliability varies. Future work should explore lightweight rule-based alternatives or train specialized classifiers. (4) Redundancy threshold. Ourτ = 0.8threshold for re- dundancy was selected through manual inspection. Optimal thresholds may vary by domain or model family. 7. Conclusion Legi-Val is a unified framework which decomposes rea- soning trace legibility into efficiency-based and transfer- based dimensions, enabling future research on bench- marking and optimizing reasoning traces as primary arti- facts. Evaluating 99,528 traces across 12 models and three datasets, we establish three core findings that challenge current reasoning model development: Accuracy paradoxically anti-correlates with teachability. Models achieving the highest accuracy (GPT-OSS-120B: 82%) rank among the lowest for transfer utility (11th of 12), while lower-accuracy models (LLaMA-Nano: 49%) rank first. Spearmanρ =−0.43demonstrates that current optimization strategies produce traces effective for strong models but pedagogically opaque—precisely the scenario scalable oversight seeks to avoid. No universal “legible reasoner” exists. Model rankings vary dramatically across tasks and metrics. Among top-3 models for STEM reasoning, GPQA and MATH agree on 2 models; Connections exhibits zero overlap with STEM domains. This task-dependent legibility (ρ = 0.75within STEM vs.ρ = 0.19overall) reveals fundamentally different pedagogical requirements across reasoning domains. Reward models ignore legibility beyond correctness. While reward model scores correlate moderately with trans- fer utility overall (ρ ≈ 0.23), this relationship vanishes when controlling for accuracy (ρ ≈ 0.05). Current RLVR paradigms optimize an incomplete objective, treating reason- ing traces as instrumental rather than intrinsically valuable. These findings establish that reasoning traces serve as pri- mary artifacts for multi-agent applications requiring over- sight, knowledge transfer, and compositional reuse. Opti- mizing solely for correctness leaves critical capabilities on the table.Legi-Valprovides concrete targets for multi- objective reward modeling that balances correctness with legibility. As reasoning-based AI systems scale, the flow of information between models—and between models and 8 Measuring Reasoning Trace Legibility humans—becomes the bottleneck. We offer metrics to mea- sure and optimize this flow. Future work should investigate: (1) RL training with Legi-Valmetrics as explicit rewards, (2) domain- adaptive legibility optimization for diverse reasoning tasks, (3) human evaluation validating automated metrics, and (4) transfer utility measured through fine-tuning rather than in-context learning. Impact Statement This paper presents work measuring the legibility of reason- ing traces for weak verifiers and human beings. We acknowl- edge important societal implications, particularly in service of monitoring and reliability for rapidly deployed, reasoning- based AI systems. Legibility offers: (1) Improved safety monitoring through more legible reasoning traces, enabling better human oversight of AI systems and (2) Democratized access to reasoning capabilities in smaller, deployable mod- els via distillation. The most proximate social impact of this work is its position on weak oversight as a desirable reward signal. However, we recognize social limitations to our understand- ing of legibility. For human beings, legibility is not a mono- lith; lived experiences, personal circumstances and ability play a large role in shaping legibility as a broad, cognitive phenomenon. We do not intend forLegi-Valto supplant human-oriented definitions of legibility used in communica- tion studies and pedagogy; rather, our focus on multi-agent dynamics produces a general framework for future optimiza- tion trajectories. References Ahmed, A., Khan, A. A., Ahmad, A., Di, S., Liu, Z., and Anwar, A. Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts, September 2025. URLhttp: //arxiv.org/abs/2509.21743. Bhambri, S., Biswas, U., and Kambhampati, S. Do Cog- nitively Interpretable Reasoning Traces Improve LLM Performance?, August 2025. URLhttp://arxiv. org/abs/2508.16695. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, December 2023. URLhttp://arxiv. org/abs/2312.09390. Choi, D., Lee, J., Tack, J., Song, W., Dingliwal, S., Jayan- thi, S. M., Ganesh, B., Shin, J., Galstyan, A., and Bo- dapati, S. B. Think Clearly: Improving Reasoning via Redundant Token Pruning. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, p. 21437–21451, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979- 8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp. 1169. URLhttps://aclanthology.org/2025. findings-emnlp.1169/. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems, November 2021. URL http://arxiv.org/abs/2110.14168. De Bock, D., Verschaffel, L., Janssens, D., Van Dooren, W., and Claes, K. Do realistic contexts and graphical representations always have a beneficial impact on students’ performance? Negative evidence from a study on modelling non-linear geometry problems. Learning and Instruction, 13(4):441–463, August 2003. ISSN 0959-4752.doi: 10.1016/S0959-4752(02)00040-3. URLhttps://w.sciencedirect.com/ science/article/pii/S0959475202000403. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., et al. DeepSeek-R1: In- centivizing Reasoning Capability in LLMs via Reinforce- ment Learning. Nature, 645(8081):633–638, Septem- ber 2025. ISSN 0028-0836, 1476-4687. doi: 10.1038/ s41586-025-09422-z. URLhttp://arxiv.org/ abs/2501.12948. Didolkar, A., Ballas, N., Arora, S., and Goyal, A. Metacog- nitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors, September 2025. URLhttp:// arxiv.org/abs/2509.13237. Emmons, S., Zimmermann, R. S., Elson, D. K., and Shah, R. A Pragmatic Way to Measure Chain-of-Thought Mon- itorability, October 2025. URLhttp://arxiv.org/ abs/2510.23966. Forrin, N. D., Risko, E. F., and Smilek, D. On the rela- tion between reading difficulty and mind-wandering: A section-length account. Psychological Research, 83(3): 485–497, April 2019. ISSN 1430-2772. doi: 10.1007/ s00426-017-0936-9. URLhttps://doi.org/10. 1007/s00426-017-0936-9. Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. Training Large Language Models to Reason in a Continuous Latent Space, November 2025. URL http://arxiv.org/abs/2412.06769. Hassid, M., Synnaeve, G., Adi, Y., and Schwartz, R. Don’t Overthink it. Preferring Shorter Thinking Chains for 9 Measuring Reasoning Trace Legibility Improved LLM Reasoning, May 2025. URLhttp: //arxiv.org/abs/2505.17813. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. Hong, J., Zhen, T., Chen, K., Liu, J., Zhu, W., Huo, J., Gao, Y., Wang, D., Wan, H., Yang, X., Wang, B., and Meng, F. Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning, August 2025. URL http://arxiv.org/abs/2508.02178. Hwang, J., Yun, S., Heo, B., and Han, D. Less is Not Worse: Effective Reasoning Without Complete Rea- soning Chains. In NeurIPS 2025 Workshop on Effi- cient Reasoning, November 2025. URLhttps:// openreview.net/forum?id=Hpizo3Fa8Y. Jose, A. Reasoning Models Sometimes Output Illegible Chains of Thought. In The Thirty-ninth Annual Confer- ence on Neural Information Processing Systems, October 2025. URLhttps://openreview.net/forum? id=w1TjXJk846. Kahneman, D. Thinking, Fast and Slow. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, NY, US, 2011. ISBN 978-0-374-27563-1 978-1-4299-6935-2. Kalyuga, S., Chandler, P., and Sweller, J. Managing split- attention and redundancy in multimedia instruction. Ap- plied Cognitive Psychology, 13(4):351–371, 1999. ISSN 1099-0720. doi: 10.1002/(SICI)1099-0720(199908)13: 4<351::AID-ACP589>3.0.CO;2-6.URLhttps: //onlinelibrary.wiley.com/doi/abs/10. 1002/%28SICI%291099-0720%28199908% 2913%3A4%3C351%3A%3AAID-ACP589%3E3.0. CO%3B2-6. Ke, Z., Jiao, F., Ming, Y., Nguyen, X.-P., Xu, A., Long, D. X., Li, M., Qin, C., Wang, P., Savarese, S., Xiong, C., and Joty, S. A Survey of Frontiers in LLM Reason- ing: Inference Scaling, Learning to Reason, and Agen- tic Systems. Transactions on Machine Learning Re- search, March 2025. ISSN 2835-8856. URLhttps: //openreview.net/forum?id=SlsZZ25InC. Kirchner, J. H., Chen, Y., Edwards, H., Leike, J., McAleese, N., and Burda, Y. Prover-Verifier Games improve leg- ibility of LLM outputs, August 2024. URLhttp: //arxiv.org/abs/2407.13692. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems, October 2022. URLhttps://openreview.net/ forum?id=e2TBb5y0yFf. Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., Chen, M., Cooney, A., Dafoe, A., Dra- gan, A., Emmons, S., Evans, O., Farhi, D., Greenblatt, R., Hendrycks, D., Hobbhahn, M., Hubinger, E., Irving, G., Jenner, E., Kokotajlo, D., Krakovna, V., Legg, S., Lindner, D., Luan, D., M ̨adry, A., Michael, J., Nanda, N., Orr, D., Pachocki, J., Perez, E., Phuong, M., Roger, F., Saxe, J., Shlegeris, B., Soto, M., Steinberger, E., Wang, J., Zaremba, W., Baker, B., Shah, R., and Miku- lik, V. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, July 2025. URL http://arxiv.org/abs/2507.11473. Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., Zhang, L. M., McKinney, K., Shrivastava, D., Paduraru, C., Tucker, G., Precup, D., Behbahani, F., and Faust, A. Training Language Models to Self- Correct via Reinforcement Learning, October 2024. URL http://arxiv.org/abs/2409.12917. Lang, H., Huang, F., and Li, Y. Debate helps weak-to- strong generalization. In Proceedings of the Thirty- Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, vol- ume 39 of AAAI’25/IAAI’25/EAAI’25, p. 27410–27418. AAAI Press, February 2025. ISBN 978-1-57735-897- 8.doi: 10.1609/aaai.v39i26.34952.URLhttps: //doi.org/10.1609/aaai.v39i26.34952. Lee, J. and Hockenmaier, J. Evaluating Step-by-step Rea- soning Traces: A Survey, May 2025.URLhttp: //arxiv.org/abs/2502.12289. Li, Y., Yue, X., Xu, Z., Jiang, F., Niu, L., Lin, B. Y., Ra- masubramanian, B., and Poovendran, R. Small Models Struggle to Learn from Strong Reasoners, February 2025. URL http://arxiv.org/abs/2502.12143. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s Verify Step by Step, May 2023. URL http://arxiv.org/abs/2305.20050. Lopez, A. Y. L., McDonald, T., and Emami, A. NYT- Connections: A Deceptively Simple Text Classifica- tion Task that Stumps System-1 Thinkers. In Ram- bow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.), Proceed- ings of the 31st International Conference on Compu- tational Linguistics, p. 1952–1963, Abu Dhabi, UAE, January 2025. Association for Computational Linguis- tics. URLhttps://aclanthology.org/2025. coling-main.134/. 10 Measuring Reasoning Trace Legibility Lu, X., Han, S., Acuna, D., Kim, H., Jung, J., Prabhumoye, S., Muennighoff, N., Patwary, M., Shoeybi, M., Catan- zaro, B., and Choi, Y. Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning, April 2025. URL http://arxiv.org/abs/2504.04383. Meeker, W. Q., Escobar, L. A., and Pascual, F. G. Statis- tical Methods for Reliability Data. John Wiley & Sons, December 2021. ISBN 978-1-118-11545-9. Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., and Hashimoto, T. S1: Simple test-time scaling, March 2025. URL http://arxiv.org/abs/2501.19393. Nayab, S., Rossolini, G., Simoni, M., Saracino, A., But- tazzo, G., Manes, N., and Giacomelli, F.Concise Thoughts: Impact of Output Length on LLM Reason- ing and Cost, January 2025. URLhttp://arxiv. org/abs/2407.19825. Ning, Y., Li, W., Fang, J., Tan, N., and Liu, H. Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learning, May 2025. URL http://arxiv.org/abs/2505.11827. Pal, K., Bau, D., and Singh, C. Do explanations generalize across large reasoning models?, January 2026. URL http://arxiv.org/abs/2601.11517. Qin, T., Alvarez-Melis, D., Jelassi, S., and Malach, E. To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning, October 2025. URLhttp: //arxiv.org/abs/2504.07052. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A Graduate-Level Google-Proof Q&A Bench- mark.In First Conference on Language Modeling, August 2024. URLhttps://openreview.net/ forum?id=Ti67584b98. Samineni, S. R., Kalwar, D., Gangal, V., Bhambri, S., and Kambhampati, S. Local Coherence or Global Valid- ity? Investigating RLVR Traces in Math Domains, Oc- tober 2025. URLhttp://arxiv.org/abs/2510. 18176. Snell, C. V., Lee, J., Xu, K., and Kumar, A. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. In The Thirteenth International Conference on Learning Representations, October 2024. URLhttps://openreview.net/ forum?id=4FWAwZtd2n. Sun, Y., Wang, H., Li, J., Liu, J., Li, X., Wen, H., Yuan, Y., Zheng, H., Liang, Y., Li, Y., and Liu, Y. An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint, May 2025. URLhttp://arxiv. org/abs/2504.14350. Wei, J., Meister, C., and Cotterell, R. A Cognitive Regular- izer for Language Modeling. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), p. 5191–5202, Online, August 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2021.acl-long. 404. URLhttps://aclanthology.org/2021. acl-long.404/. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models, January 2023. URLhttp://arxiv.org/ abs/2201.11903. Wu, S., Qian, C., Fung, Y. R., Liang, P. P., and Ji, H. Alice: Proactive Learning with Teacher’s Demonstrations for Weak-to-Strong Generalization, April 2025. URLhttp: //arxiv.org/abs/2504.07316. Xia, H., Leong, C. T., Wang, W., Li, Y., and Li, W. To- kenSkip: Controllable Chain-of-Thought Compression in LLMs, September 2025. URLhttp://arxiv.org/ abs/2502.12067. Zhang, X., Huang, Z., Li, Y., Ni, C., Chen, J., and Oymak, S. BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning, June 2025a. URL http://arxiv.org/abs/2506.17211. Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, J. The Lessons of Developing Pro- cess Reward Models in Mathematical Reasoning. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Findings of the Association for Computational Linguis- tics: ACL 2025, p. 10495–10516, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl. 547. URLhttps://aclanthology.org/2025. findings-acl.547/. 11 Measuring Reasoning Trace Legibility A. Implementation Details A.1. System Prompts A.1.1. MATH DATASET “You are an expert on answering and explaining math questions. Read the question and formulate a response. Please reason step by step. Formatting instructions: 1. When you have a final answer, you will wrap the final answer around a function so that it can be evaluated. If no such answer is found, you will receive no credit regardless of how much reasoning you did. 2. Your answer should exclusively use standard LaTeX math formatting (e.g., ab for fractions, x for square roots, ∞ for infinity etc.). Do not use any math symbols or characters other than LaTeX formats – they are ugly and indecipherable. 3. Do not include any decorators like dollar signs ($) in your final answer. 4. Do not surround the answer in a LaTeX code block. Return it as-is. 5. Do not include any units, suffixes, or variable definitions unless they are absolutely necessary. 6. Do not include any pre-phrasing like "the answer is..." or "x=...". Your goal is to create the easiest-to-parse string for a latex2sympy evaluator. 7. Your response MUST be ONLY the LaTeX-formatted final answer, with no additional text, explanations, or JSON structure. Do not terminate until you have come up with a final answer.” A.1.2. GPQA DATASET “You are an expert on answering and explaining graduate-level scientific reasoning questions. To achieve success on this task, you will carefully consider the question and reason step by step. You will choose a single option from a set of possible choices. Exactly one of the options is correct – the rest are false. What is the single, most likely answer choice among the options? When you are ready to respond, format your final answer as follows: "The correct answer is boxedX", where X is whichever letter corresponds to your choice. The response inside of boxed MUST be ONLY the corresponding letter, with no additional text, explanations, or JSON structure. If you do not put your answer in boxed, your answer will not be evaluated and you will gain no reward. Do not terminate until you have come up with a final answer.” A.1.3. CONNECTIONS DATASET “You are an assistant configured to solve the New York Times Connections Word game. Make four groups of four words that share something in common. Categories will always be more specific than ‘5-LETTER-WORDS‘, ‘NAMES‘ or ‘VERBS.‘ Example 1: Words: [’DART’, ’HEM’, ’PLEAT’, ’SEAM’, ’CAN’, ’CURE’, ’DRY’, ’FREEZE’, ’BITE’, ’EDGE’, ’PUNCH’, ’SPICE’, ’CONDO’, ’HAW’, ’HERO’, ’LOO’] Answer: “‘ "Groupings": "1": "Category": "Things to sew", "Words": [ "DART", "HEM", "PLEAT", "SEAM" ] , "2": "Category": "Ways to preserve food", "Words": [ "CAN", "CURE", "DRY", "FREEZE" ] , "3": "Category": "Sharp quality", "Words": [ "BITE", "EDGE", "PUNCH", "SPICE" ] , "4": "Category": "Birds minus last letter", "Words": [ "CONDO", "HAW", "HERO", "LOO" ] “‘ Example 2: Words: [’COLLECTIVE’, ’COMMON’, ’JOINT’, ’MUTUAL’, ’CLEAR’, ’DRAIN’, ’EMPTY’, ’FLUSH’, ’CIGARETTE’, ’PENCIL’, ’TICKET’, ’TOE’, ’AMERICAN’, ’FEVER’, ’LUCID’, ’PIPE’] Answer: “‘ "Groupings": "1": "Category": "Shared", "Words": [ "COLLECTIVE", "COMMON", "JOINT", "MUTUAL" ] , "2": "Category": "Rid of contents", "Words": [ "CLEAR", "DRAIN", "EMPTY", "FLUSH" ] , "3": "Category": "Associated with ’stub’", "Words": [ "CIGARETTE", "PENCIL", "TICKET", "TOE" ] , "4": "Category": "__ Dream", "Words": [ "AMERICAN", "FEVER", "LUCID", "PIPE" ] “‘ Categories share commonalities: - There will never be a miscellaneous category - No word will ever appear in two categories - There will always be four words in a category - As the category number increases, the connections between the words and their category becomes more obscure. The category 1 is the most easy and intuitive, category 4 is the hardest - There may be a red herring category - Category 4 often contains words with a common preposition or postposition, like the category 4 in the example Please reason step by step. Please respond in a JSON format.” Follow-up prompts. For cases where student models fail to produce extractable answers, we append forcing prompts: • MATH/GPQA: “ The answer is \ ” • Connections: “ My final groupings are: “‘” 12 Measuring Reasoning Trace Legibility A.2. Redundancy Threshold Validation Embedding model. We use the sentence-transformers library with modelall-MiniLM-L6-v2, which produces 384- dimensional embeddings. All embeddings are L2-normalized before computing cosine similarity. Similarity computation. For a trace with m steps (sentences tokenized via NLTK), we: 1. Encode all steps: embeddings = model.encode(steps, normalize_embeddings=True) 2. Compute pairwise cosine similarity matrix:sim_matrix = util.cos_sim(embeddings, embeddings) 3. For each step i≥ 1, compute maximum similarity to all previous steps: max_sim i = max j<i sim_matrix[i,j] 4. Count fraction of steps where max_sim i > τ Threshold selection. We conducted elbow curve analysis across 21 evenly-spaced thresholds from 0.0 to 1.0. Manual inspection of 100 random traces per dataset confirmed thatτ = 0.8reliably identify semantic redundancy—step pairs scoring above these thresholds exhibit near-identical meaning with only minor paraphrasing. Figure 6 shows both an absolute and rankwise comparison of different redundancy comparisons. We show that 0.8 is the strictest threshold which preserves maximum rank stability over the 12 models analyzed. Furthermore, this threshold also shows the least partitioning of models from different families. 0.00.20.40.60.81.0 Redundancy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 Mean Redundant Fraction QwQ-32B Qwen3-4B Qwen3-8B DeepSeek-R1-Distill-Qwen-32B deepseek-r1-0528 gemma-3-12b-it gemma-3-27b-it Magistral-Small-2509 Llama-3.1-Nemotron-Nano-8B-v1 OpenReasoning-Nemotron-32B gpt-oss-120b gpt-oss-20b Redundancy Elbow by Model (Mean Redundant Fraction vs Threshold) (a) By Model 0.00.20.40.60.81.0 Redundancy Threshold 2 4 6 8 10 12 Rank by Mean Redundant Fraction (1=best) Redundancy Elbow (Rankwise) (b) Rankwise Figure 6. Distribution of maximum step-to-previous-step cosine similarities. A.3. Backtracking Judge Validation A.3.1. RUBRIC DESIGN MATH Backtracking Rubric. Backtracking is detected when the model: • Uses explicit revision keywords: “Wait”, “But”, “However”, “Hmm”, “Going back”, “Backtrack”, “Alternatively” • Changes solution method or direction abruptly • Revisits or corrects earlier calculations/assumptions • Explores alternative approaches after a failed attempt Minor arithmetic corrections or detail additions are not counted as backtracking. GPQA Backtracking Rubric. Domain-specific indicators for graduate-level science: 13 Measuring Reasoning Trace Legibility • Revision phrases: “Wait, that’s not right”, “Actually”, “Re-evaluating”, “On second thought” • Correcting domain-specific constants, formulas, or sign errors • Realizing overlooked constraints in the problem statement • Switching theoretical frameworks when initial approach leads to contradiction Connections Backtracking Rubric. Word-grouping specific indicators: • Abandoning proposed groups: “No, that word belongs elsewhere” • Recognizing red herrings: “Actually, these could be...” • Recalculating remaining words after failed attempt • Changing category themes mid-reasoning A.3.2. OUTPUT SCHEMA The judge returns structured JSON for each trace: ‘backtracking_detected": bool, ‘final_answer": str, ‘backtracking_steps": [ "step_number": int, ‘reason": str, ... ], ‘confidence": float (0.0-1.0), ‘overall_reasoning": str A.3.3. PROMPT “You are an expert in analyzing mathematical reasoning traces generated by large language models. Your task is to identify instances of ’backtracking’ within these traces. A reasoning trace is a step-by-step derivation of a solution to a mathematical problem. Backtracking occurs when the model revises or abandons a previous step, approach, or calculation in favor of a new one. This often indicates a correction, a change in strategy, or exploration of an alternative path. Your goal is to determine if the provided reasoning trace exhibits backtracking and, if so, pinpoint where it occurs and explain why. Look for indicators such as: - Explicit keywords or phrases (e.g., ’But’, ’Wait’, ’Alternatively’, ’However’, ’Hmm’, ’Hmmm’, ’Not sure’, ’Going back’, ’Backtrack’, ’Trace back’, ’Another’). - A sudden change in the method or direction of the solution. - Revisiting or correcting a calculation or assumption made earlier in the trace. - Exploring an alternative approach after a previous one failed or seemed incorrect. Do not confuse minor arithmetic errors, rephrasing, or adding detail with backtracking. Backtracking implies a more significant deviation from a previously taken path or a clear change in strategy. You will be given a reasoning trace as input. You must return a JSON object with the following structure: "backtrack- ing_detected": boolean, "final_answer": "The final answer extracted from the reasoning trace.", "backtracking_steps": [ "step_number": int, "reason": "A brief explanation of why this step indicates backtracking." ], "confidence": float (0.0 to 1.0), "overall_reasoning": "A brief explanation for your overall decision regarding backtracking." **Core Task:** - Read the reasoning trace carefully, following the steps logically. - Identify any points where the model seems to change its mind, correct a previous error, or switch to a different method after initially pursuing another, using the indicators mentioned above. **Definitions:** 1. **Reasoning Trace:** A sequence of steps leading to a mathematical solution. 2. **Backtracking:** A deviation in the reasoning trace where a previous idea, calculation, or approach is abandoned or corrected, and a new one is adopted. This often appears as a restart, a sudden change in direction, or an explicit correction of a prior statement or calculation. 14 Measuring Reasoning Trace Legibility **Filtering Rules (IMPORTANT):** - **‘backtracking_detected‘**: ‘true‘ if the trace shows evidence of backtracking, ‘false‘ otherwise. - **‘backtracking_steps‘**: If ‘backtracking_detected‘ is ‘true‘, list the step numbers (starting from 1 for the first logical step) where backtracking is evident, along with a brief ‘reason‘ for each. If no backtracking is detected, this array should be empty. - **‘confidence‘**: How sure are you of your decision? 1.0 for very sure, 0.5 for uncertain. - **‘overall_reasoning‘**: A concise explanation for your overall decision regarding backtracking. If backtracking was detected, summarize why. If not, state that the trace appeared linear or consistent. **Your Response:** - Your response MUST be a single, valid JSON object. Do not include any other text or formatting before or after the JSON. ” A.3.4. VALIDATION RESULTS To further understand cross-ranking stability with backtracking, we conduct a human validation on 60 traces sampled from our dataset (30 MATH, 15 GPQA, 15 Connections). Two human annotators received 35 traces each (20 MATH, 10 GPQA, 10 Connections), sharing 10/5/5 traces to do cross-comparison. Decisions on backtracking labels carry some subjectivity—for instance, do a chain of four consecutive approach suggestions constitute four backtracks, one backtrack into uncertainty, or none? Because of the difficulty of working with exact numerical labels, we instead calculate Spearman rankwise correlation between the two annotators on a per-trace basis. If the relative extent of backtracking agrees between humans as well as between models, then the metric operates according to our rankwise expectations as reported in findings. We observe a high inter-annotator agreement on the 20 examples set aside (ρ=0.91), allowing us to continue with the analysis relative to our LLM judge. To reconcile absolute differences on common examples, we take the average of the two scores. We also include a categorical Cohen’s Kappa on both the absolute count of steps as well as the binary variable (backtracking detected) to represent fundamental agreement on observations. Table 3 shows overall positive results in rankwise correlation. All rankwise correlations exceed 0.7, with MATH and GPQA atρ = 0.857, 0.916. Our Cohen’s Kappa results demonstrate low agreement on absolute counts—especially for Connections—which we analyze qualitatively below. The binary Kappa, however, suggests that agreement with the LLM annotator is moderately high on establishing whether or not backtracking took place (overall ρ = 0.712). Table 3. Annotator agreement on backtracking counts: Spearman’s rho (rankwise), Kendall’sτcross-dataset, and Cohen’s Kappa (categorical). Dataset Spearman’s Rho (annotator-judge) Kendall’s τ (cross-dataset stability) Kappa (Exact) Kappa (Binary >0) Overall0.717–0.2340.712 Connections0.7170.6410.1450.623 GPQA0.9160.4670.5520.800 Math0.8570.7790.2670.745 These correlations confirm that backtracking is a valid signal—models exhibiting more backtracking rank differently on correctness metrics, with strongest effect in mathematical reasoning. Consistently, Connections appears to cause divergence with our metrics. In the observed examples, we noted many instances which annotators marked as backtracks while the LLM judge did not. More often than not, human annotators claimed higher rates of backtracking than LM judges. In the most extreme case, one annotator reported 55 backtracks while the LM judge reported 4. For Connections, which requires high variability, exploration and context switching (reasoning is more horizontal than vertical), many models produced a significant number of strategy-switching cues such as “wait”, “alternatively”, “on the other hand”. While human annotators marked each one as a unique backtrack, it appears that the LM judge only considered cases where that strategy was not aborted by a rejection thought. Below, we include an abridged example where bold shows the 4 LM-judge backtracks, while italics show some of the 55 human backtracks. 15 Measuring Reasoning Trace Legibility Abridged Connections Trace (QwQ-32B) Okay, let’s tackle this problem step by step. I have the list of words: BLAST, KNOCK, SLAM, TRASH, APOLOGIES, OOPS, PARDON, SORRY, DEFENSE, EXCUSE, OUT, REASON, DATE, DEAD, DIRTY, DOH. First, I need to group them into four groups of four words each, with each group having a common theme that’s more specific than just categories like verbs or names. Also, the categories should get progressively harder from group 1 (easiest) to group 4 (most obscure). ... Next, looking at APOLOGIES, OOPS, PARDON, SORRY. These all relate to apologizing or expressing regret. Words like “Oops” is an apology for a mistake, “Sorry” is an apology, “Pardon” can mean forgiveness, and “Apologies” is plural but still fits. So Group 2 might be “Expressions of apology or regret.” That makes sense. Now, moving on to the remaining words: DEFENSE, EXCUSE, OUT, REASON; then DATE, DEAD, DIRTY, DOH. Hmm. Let’s see. DEFENSE, EXCUSE, OUT, REASON. Maybe these are related to making excuses or justifications. “Defense” can be a justification, “Excuse” is direct, “Reason” supports an argument, but what about “OUT"? Maybe not exactly. Alternatively, “Out” could fit in another context. Let me think again. Alternatively, looking at DATE, DEAD, DIRTY, DOH. “Date” can mean a fruit, a time, or a romantic meeting. “Dead” is obviously death-related. “Dirty” is unclean. “Doh” is the Simpsons’ Homer sound. Not seeing it here. Wait, maybe they have something in common with prefixes or suffixes? Like “do” + something? Doh starts with Do, Dead has Dea..., not sure. Alternatively, maybe they can all be preceded by “THE” in famous titles? “The Date", “The Dead", “The Dirty", “The D’Oh” (unlikely). Hmm. Wait, “DOH” is an exclamation, so maybe all are interjections? “Oops", “pardon", “sorry” were in the second group. So “doh” is an interjection, but the rest aren’t. So probably not. Wait, looking back at the words again: Leftover words: DEFENSE, EXCUSE, OUT, REASON; DATE, DEAD, DIRTY, DOH. Maybe“DEFENSE", “EXCUSE", “REASON", “OUT” can be categorized as “Ways to justify or avoid blame". That feels plausible. Then the fourth group needs to be more obscure. Let’s try DATE, DEAD, DIRTY, DOH. Maybe they are all words that end with T? No: DATE ends with E, DEAD ends with D, DIRTY ends with Y, DOH ends with H. No. Starting letters are D, so maybe “four-letter words starting with D". But DIRTY has five letters (without apostrophe) and DOH is three. Unless written as “D’OH” which is four characters but counts as four letters? Not sure. Alternatively, maybe all can be preceded by “THE” in famous titles? “The Date", “The Dead", “The Dirty", “The D’Oh” (unlikely). Hmm. ... Maybe I need to re-examine the initial groupings. Could there be a different way to categorize? Alternative approach for group 1: Maybe “BLAST, KNOCK, SLAM, TRASH” as verbs meaning to hit something physically? But blast is more explosion, knock is a tap. Alternatively, they can all be nouns too: blast (explosion), knock (a strike), slam (forceful closing), trash (debris). Maybe “Types of noises"? Blast, knock, slam, trash (rubbish noise?) Not great. Alternatively, “BLAST” can mean criticize, as before. Let’s stick with the first idea of harsh criticism verbs. Second group is clear with the apologies. For the remaining words, maybe DEFENSE, EXCUSE, REASON, and OUT can form “Methods of justification or avoidance", while DATE, DEAD, DIRTY, DOH are left. Alternatively, maybe “DEFENSE", “EXCUSE", “REASON", “OUT” can be parts of a baseball game? Out is a term in baseball, but defense, excuse don’t fit. Hmm. Alternatively, “DEFENSE", “EXCUSE", “OUT", “REASON” could all be answers to “What’s needed in court?” Defense, evidence (not listed), excuse, reason? Not precise. Wait, “OUT” in the context of “out of options” or “out of excuses"—maybe “Reasons for quitting"? Not sure. Alternatively, maybe “DEFENSE", “EXCUSE", “OUT", “REASON” are all nouns that can be preceded by “LAST” – last defense, last excuse, last out, last reason. “Last out” in baseball is a term, so that could be a stretch but possible. So category: “Terms that can be preceded by ’LAST’". That might work for group 3. Then group 4 would be the leftover four words: DATE, DEAD, DIRTY, DOH. ... So putting it all together: Group 1: Harsh criticism verbs (BLAST, KNOCK, SLAM, TRASH) Group 2: Apology expressions (APOLOGIES, OOPS, PARDON, SORRY) Group 3: Words following “GOOD” (DEFENSE, EXCUSE, REASON, DATE) Group 4: Phrases with “down and ___” (DEAD, DIRTY, OUT, DOH) ... A.4. Transfer Utility Implementation A.4.1. STUDENT MODEL PROMPTING For each problem where the teacher answered correctly, we create progressive prompts at multiple rollout depths. Given a teacher trace with m steps (sentences tokenized via NLTK): 1. Sample rollout points: 0, 3, 6, 9,...,m− 1 (every 3 steps, always including first and last) 2. For each rollout depth k, construct chat messages: messages = [ 16 Measuring Reasoning Trace Legibility "role": ‘system", ‘content": system_prompt, "role": ‘user", ‘content": problem, "role": ‘assistant", ‘content": reasoning[:k] ] 3. Apply chat template with continue_final_message=True to prompt student continuation 4. Student generates completion (max 8,192 tokens) A.4.2. ROLLOUT STRATEGY Step interval: 3 sentences per increment (configurable via -step_interval) Two-pass grading: 1. First pass: Extract and grade answer using standard extraction logic 2. If extraction fails: Append forcing prompt (“The answer is ” for MATH/GPQA, “My final groupings are:” for Connections) and re-generate for up to 128 additional tokens A.4.3. METRIC COMPUTATION All three metrics are computed per trace by aggregating student performance across rollout steps. First, we create bins at 2% granularity.A student completion of a prefix that is k m will fall into a bin under (0,.02], (.02,.04],..., (.98, 1]. First-order TU (Equation 3): For a given trace, we calculate first-order TU by (1) averaging all steps in a bin and (2) taking the unweighted average across all steps. As our x axis moves from 0 to 1, this is equivalent to taking the area under the transfer utility curve. We aggregate statistics on the model-level or (model, dataset)-level by averaging the trace-level statistics together. Second-Order TU (Equation 5): To calculate second-order transfer utility, we identify the first point where a student correctly completes a problem from teacher rollouts (in Meeker et al. (2021) this is referred to as the hazard). This point falls into one of the percentage bins above. From this bin, we construct a histogram of hazard frequencies by bin, and collect the Shannon entropy along this bin. A.4.4. IMPLEMENTATION DETAILS Student models: Microsoft Phi-3.5-Mini (3.8B) and Meta LLaMA-3.2-1B, representing different capability tiers signifi- cantly weaker than teacher models. Computational cost: For each teacher model on datasetDwithNproblems and average trace length ̄m, we perform approximatelyN × ( ̄m/3)student inferences per student model. Total across all experiments:>2M student model inferences. Each student, due to their small parameter size, would fit on a single A100 GPU, and our package includes support for producing data-sharded jobs to parallelize if possible. 17 Measuring Reasoning Trace Legibility Figure 7. Trace-level relationship between reward model scores and first-order transfer utility. PhiLlama 2 4 6 8 10 12 Global PU Rank (rank-points, lower is better) QwQ-32B Qwen3-4B Qwen3-8B DeepSeek-R1-Distill-Qwen-32B deepseek-r1-0528 gemma-3-12b-it gemma-3-27b-it Magistral-Small-2509 Llama-3.1-Nemotron-Nano-8B-v1 OpenReasoning-Nemotron-32B gpt-oss-120b gpt-oss-20b PU Rank Progression by Student (All Models) Figure 8. Rank progression of transfer utility metrics across student models for all evaluated models. 18 Measuring Reasoning Trace Legibility Table 4. Cross-student rank stability. Spearman’sρbetween TU rankings when measured with Phi-3.5-Mini vs LLaMA-3.2-1B as student models. Higher values indicate rankings are consistent regardless of student model choice. METRICCONNECTIONSGPQAMATHAVG ρ TU CORRECTNESS (1ST ORDER)0.4200.6480.5820.550 TU UNIFORMITY (2ND ORDER)0.3850.5910.5230.500 Table 5. Model-level variation in transfer utility between student models. Models ordered by average absolute difference (|∆TU|) between Phi-3.5-Mini and LLaMA-3.2-1B scores across three datasets. Higher values indicate greater sensitivity to student model choice. MODELAVG|∆| MAX|∆| ∆ MSE PHI AVG LLAMA AVG DEEPSEEK-R1-DISTILL0.2720.5670.0210.4170.558 GPT-OSS-20B0.2430.3590.0190.2910.107 QWEN3-0.6B0.0930.0830.1000.3080.204 QWEN3-32B0.0920.1090.1920.4640.549 LLAMA-NANO0.0910.3330.0470.3860.278 MAGISTRAL-S0.0820.1190.0120.3990.478 GEMMA-3-27B-IT0.0730.2080.1180.4100.473 QWEN3-4B0.0570.0520.0260.3540.377 QWEN3-8B0.0490.0860.1180.4200.440 QWQ-32B0.0440.1190.0270.4450.472 OPENREAS-32B0.0430.0840.0340.5010.503 DEEP-R1-05280.0370.0640.1190.4770.592 GEMMA-3-12B-IT0.0250.0490.0200.4110.433 B. Additional Figures and Results B.1. Reward Model Correlations with Transfer Utility B.2. Rank Stability Analyses B.2.1. STUDENT B.2.2. DATASET B.3. Full Results B.4. Correlation Matrix 19 Measuring Reasoning Trace Legibility Table 6. Cross-dataset stability for all Legi-Val metrics measured by Spearman’sρ. Bold indicates high stability (ρ > 0.7); underline indicates moderate (0.4≤ ρ≤ 0.7). METRICCONN.–GPQA CONN.–MATH GPQA–MATH AVG ρ Efficiency Metrics ACCURACY0.61-0.120.470.32 TOKEN LENGTH0.760.380.790.64 REDUNDANCY0.360.250.860.49 BACKTRACKING0.620.390.540.52 Transfer Utility Metrics FOTU-0.37-0.130.660.06 SOTU0.910.56 0.580.68 REGRESSION RATE0.290.130.070.17 TU COMBO-0.150.080.240.06 RM MEAN RAW0.630.370.730.58 Figure 9. Metric-wise decomposition of results in Fig. 1, unweighted averages across datasets. 20 Measuring Reasoning Trace Legibility Table 7. Model comparison by axis. Model (Acc) uses mean accuracy integers. Len.: Token Length; Red.: Redundancy; Back.: Backtracking; FOTU: first-order TU; SOTU: second-order TU; Regression Rate: TU regression rate. Bold denotes best performance; underline denotes second-best. MODEL (ACC)LEN.RED.BACK.FOTUSOTUREGRESSION RATE Connections DEEPSEEK-R1-DISTILL (91)180814.94.553.2%73.4%20.3% DEEP-R1-0528 (90)377913.03.844.2%62.1%20.7% QWQ-32B (90)386312.85.743.4%56.8%22.2% MAGISTRAL-S (86)10755.26.049.4%84.3%24.5% GPT-OSS-120B (81)311717.14.428.4%67.5%19.7% QWEN3-8B (76)301022.66.843.0%50.8%22.5% GEMMA-3-27B-IT (76)5244.2 1.052.1%89.1%21.8% QWEN3-4B (71)294729.76.641.7%51.9%21.2% GEMMA-3-12B-IT (69)5594.71.145.1%89.0%21.7% GPT-OSS-20B (55)561415.66.122.1%64.8%16.8% OPENREAS-32B (42)715914.36.537.5%56.8%18.6% LLAMA-NANO (28)267 0.70.847.5%9.1%5.6% GPQA GPT-OSS-120B (70)12624.32.450.3%80.1%10.6% GPT-OSS-20B (62)18474.93.950.9%75.0%11.7% R1-DISTILL (59)353515.86.047.3%71.4%11.7% DEEP-R1-0528 (55)555215.82.263.3%66.3%9.1% QWEN3-8B (50)435427.05.356.0%64.6%11.1% QWQ-32B (50)461112.15.654.4%64.8%11.4% MAGISTRAL-S (48)7359.71.050.5%89.9%10.6% GEMMA-3-27B-IT (42)7666.10.847.4%89.5% 11.9% QWEN3-4B (42)421133.25.052.7%63.2%12.3% GEMMA-3-12B-IT (37)7575.40.850.1%89.0%12.1% LLAMA-NANO (37)8547.50.884.4%59.5%1.6% OPENREAS-32B (29)681611.42.167.4% 61.1%8.0% MATH GPT-OSS-120B (96)5814.70.934.7%84.7%7.5% GPT-OSS-20B (94)7485.6 1.635.9%87.5%8.9% QWEN3-8B (90)237716.46.343.0%83.3%12.3% QWEN3-4B (89)209217.75.641.6%84.5%12.5% MAGISTRAL-S (85)6849.50.834.2%85.3%8.3% DEEP-R1-0528 (85)345615.53.054.9%78.0%10.9% GEMMA-3-27B-IT (84)61011.51.240.2%89.5%9.9% LLAMA-NANO (84)197711.52.840.7%85.8%12.2% GEMMA-3-12B-IT (83)593 11.30.839.1%88.9%10.1% OPENREAS-32B (81)447113.21.857.7%73.7%10.5% R1-DISTILL (79)131814.52.540.6%85.0%11.1% QWQ-32B (40)13119.54.550.1%82.4%14.2% 21 Measuring Reasoning Trace Legibility Figure 10. Correlation matrix of all metrics across all datasets. Correlation values refer to Spearman’s rankwiseρ, calculated at the model-level by taking points across models, averaged over datasets. 22