Paper deep dive
Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation
Eric Bigelow, Amir Zur, Satchel Grant, Tal Haklay, Can Rager, Owen Lewis, Thomas McGrath, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/21/2026, 3:22:03 AM
Summary
This paper introduces a computationally efficient method for estimating uncertainty dynamics in Large Language Model (LLM) text generation, addressing the high cost of resampling-based analyses like Forking Paths Analysis (FPA). The authors demonstrate that uncertainty dynamics converge to stable patterns and that sampling noise follows a multinomial distribution, decreasing in proportion to the square root of the sample size. They develop a statistical model utilizing change point detection (PELT) and kernel-weighted Dirichlet pooling to smooth noisy low-sample data, effectively approximating high-sample ground truth. This approach significantly reduces sampling costs (up to 15x efficiency gains) while maintaining accuracy in identifying critical 'forking points' in reasoning chains.
Entities (8)
Relation Signals (7)
Forking Fast → evaluatedon → DeepSeek-R1-Distill-Llama-8B
confidence 98% · and the native reasoning model DeepSeek-R1-Distill-Llama-8B... as they solve problems in tinyMMLU
Forking Fast → evaluatedon → Llama-3-8B-Instruct
confidence 98% · We evaluate our approach by studying Chain-of-Thought reasoning... in Llama-3-8B-Instruct
Forking Paths Analysis → suffersfrom → High Computational Cost
confidence 96% · However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly.
Forking Fast → uses → PELT
confidence 95% · First, we apply pruned exact linear time (PELT) change point detection... These change points divide the base path into segments
Forking Fast → reduces → Sampling Cost
confidence 94% · allowing us to significantly cut sampling costs... cut the total budget by1/8
Forking Fast → uses → Multinomial Distribution
confidence 93% · Since we drawScontinuations independently... the answer counts form a multinomial distribution... we expect sampling noise to decrease in proportion to1/ √ S.
Uncertainty Dynamics → exhibits → Forking Points
confidence 92% · uncertainty dynamics converge to stable patterns... except at key forking points which have sharp changes.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncertainty. Resampling-based analyses characterize this distribution, revealing which steps of a rollout determine how the model arrives at its answer. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly. Our work strives to make resampling analysis more computationally efficient, while also shedding light on an important scientific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resampling many reasoning chains, uncertainty dynamics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM's sensitivity to each individual token or reasoning step. We develop a statistical model for smoothing noisy low-sample rollout data to better approximate high-sample data, allowing us to significantly cut sampling costs.
Tags
Links
- Source: https://arxiv.org/abs/2608.19611v1
- Canonical: https://arxiv.org/abs/2608.19611v1
Trouble viewing inline? Open PDF directly →
Full Text
34,573 characters extracted from source content.
Expand or collapse full text
Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation Eric Bigelow, Amir Zur, Satchel Grant, Tal Haklay, Can Rager, Owen Lewis, Thomas McGrath, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger Abstract LLM reasoning is stochastic, and so under- standing a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncer- tainty. Resampling-based analyses character- ize this distribution, revealing which steps of a rollout determine how the model arrives at its answer. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly. Our work strives to make resam- pling analysis more computationally efficient, while also shedding light on an important sci- entific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resam- pling many reasoning chains, uncertainty dy- namics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM’s sensitivity to each individual token or reasoning step. We develop a statistical model for smoothing noisy low-sample rollout data to better approximate high-sample data, allowing us to significantly cut sampling costs. 1 Introduction When Large Language Models (LLMs) engage in long-form text generation such as reasoning, there are many possible paths they might take at each step or token that they generate. Prior work has shown that specific steps during reasoning can have a significant impact on the model’s overall behav- ior, while many other steps are not as important (Bigelow et al., 2024a; Bogdan et al., 2025; Zhang et al., 2025; Macar et al., 2026; Boppana et al., 2026). One method for understanding which steps of reasoning (and text generation more generally) are most important is resampling: given a single reasoning rollout, resample alternate rollouts at Code and interactive dashboard:https://github.com/ ericb-goodfire/forking-fast each step of reasoning, then collect the final an- swers of all rollouts and aggregate them into a dis- tribution. Resampling methods such as Forking Paths Analysis (FPA) (Bigelow et al., 2024a) can pro- vide critical insight into uncertainty dynamics in text generation, where uncertainty measures how likely an LLM is to pick one answer over another. However, this approach is very costly and often requires millions of tokens to analyze a single rea- soning chain. In this work, we propose a more efficient way to approximate these uncertainty dy- namics by developing a statistical model of their distribution, and using this model to smooth noisy low-sample estimates. We evaluate our approach by studying Chain- of-Thought reasoning (Kojima et al., 2022) in Llama-3-8B-Instruct (Grattafiori et al., 2024) and the native reasoning model DeepSeek-R1-Distill- Llama-8B (DeepSeek-AI et al., 2025) as they solve problems in tinyMMLU (Maia Polo et al., 2024; Hendrycks et al., 2021). After collecting and ana- lyzing reasoning data with nearly two billion tokens in total, we find three main results: 1.When resampling a small number of times, the uncertainty dynamics of reasoning rollouts are quite noisy (Bigelow et al., 2024a). However, when resampling hundreds of times, the uncer- tainty dynamics become increasingly smooth, except at key forking points which have sharp changes. We find that variation across reason- ing rollouts is modeled well as multinomial sampling noise (§3.3), and reconstruction er- ror decays in proportion to the square root of the amount of samples collected. 2.We develop a statistical model which allows us to approximate high-sample uncertainty dy- namics by smoothing lower-sample data. This model uses change point detection to identify forking points, and kernel-pooling to smooth 1 arXiv:2608.19611v1 [cs.CL] 20 Aug 2026 0 0.5 1 (a) FPA reference o t : S=1000, every token ABCDOther 0 0.5 1 (b) Smoothed Data o t : S=20, every 2 tokens 050100150200250300 position t in the base path (tokens) 0 0.5 1 (c) Raw Data o t : S=20, every 2 tokens Figure 1: Uncertainty dynamicso t for one question (tinyMMLU question 39; Llama-3-8B-Instruct). Col- ors represent the fraction of rollouts attthat end with different final answers. (a) When collecting a massive number of resampled rollouts (S=1000) at every to- ken (N=1), the outcome distribution becomes increas- ingly smooth, except at forking points. (c) Uncertainty dynamics with a lower-sample analysis (S=20,N=2) using1%of the reference’s sampling cost. (b) Smooth- ing the lower-sample datao t closely approximates the high-sample data and recovers the same forking points. More examples in Appendix B and our interactive dashboard (https://github.com/ericb-goodfire/ forking-fast). estimates between these points. By smoothing low-sample data, we can effectively multiply the effective sample size by3.3×(S = 30) to 5× (S = 5) (§3.1). 3.We find that resampling everyNtokens or steps, instead of every step, can improve data efficiency without smoothing. However, this approach loses precision in estimating fork- ing points, and it benefits less from smoothing compared toN = 1. By combining this ap- proach with our smoothing model, we can cut the total budget by1/8with only a small increase in error (§3.2). 2 Methods Forking Paths AnalysisFollowing Bigelow et al. (2024a), we analyze uncertainty dynamics relative to fixed reasoning chainx, a base path, gener- ated by greedy decoding. At each token or sen- tence positiontper intervalN, for example ev- ery token or every sentence (N=1), we resam- ple continuations from the prefixx <t . Specifi- cally, for every alternative token with probabil- ityp(x t | x <t ) ≥ 0.05, we sampleScontinua- tionsx >t with temperatureτ = 1.0and extract each continuation’s outcome, in our case a multiple choice answer A/B/C/D/Other. We then aggregate these final answers into a weighted distribution over outcomes, such that each timesteptindexes a distributiono t over final answers weighted by the probability of the tokenp(x t | x <t )and continu- ationp(x >t | x ≤t ), which can be visualized as a timeseries (Fig. 1). In this work, we consider two methods for re- ducing the sampling cost of FPA: decreasing the number of samplesSat each token position, and increasing the sampling interval to resample every Ntokens or sentences, instead of at every step (i.e. N=1). Our goal is to use this lower-sample data to approximate uncertainty dynamics ino t with high S and N=1. Estimating uncertainty dynamics When do sharp changes in the outcome distributiono t corre- spond to decision points in the reasoning path, and when do they correspond to sampling noise? Fig- ure 1a shows a referenceo t with a high resample rateS=1000andN=1. We find that with large amounts of resampling data, the outcome distribu- tiono t becomes increasingly smooth at almost all positionst, except for key forking points whereo t suddenly and dramatically changes (Bigelow et al., 2024a). Compare this to Figure 1c, which shows a lower-sample outcome distributiono t withS=20 andN=2. It is difficult to visually discern from the lower-sample data whether sharp changes ino t are genuine uncertainty dynamics, or whether these fluctuations are an artifact of sampling noise. Since we drawScontinuations independently for each possible next-tokenx t = w, the answer counts form a multinomial distributionc (w) t ∼ Multinomial(S,o (w) t ) , and the branch weights ̃p w are known exactly from the next-token distribution p(x t = w | x <t ), so the weighted estimate ofo t has variance P w ̃p 2 w o (w) t,k (1−o (w) t,k )/Sfor each an- 2 swer categoryk. Sampling a branch weighted by ̃p w and one of its continuations yields i.i.d. draws fromo t itself, so counts constructed this way sat- isfyc t ∼ Multinomial(S,o t )exactly. Therefore, we expect sampling noise to decrease in proportion to1/ √ S. That is, we expect a negative linear re- lation between the log distance between the true outcome distribution (which we approximate with highSandN=1) and √ Sas we increase the num- ber of resampled outputs. We empirically test this in Section 3.3. Statistical modelIf we simply increase the sam- ple countS, we can expect to get closer to the model’s true uncertainty dynamicso t at a rate of √ S. Next, we propose a method for more effi- ciently approximating the trueo t using reduced sampling rates by varying S and N . Our statistical model builds on a few qualitative observations about the shape ofo t . We observe, as do Bigelow et al. (2024a), that uncertainty dynam- ics ino t follow a pattern where large segments ofo t are relatively stable, remaining flat or very slowly drifting overt. These stable segments are sepa- rated by sharp changes at forking points, where the outcome distribution suddenly changes, sometimes after only a single token. Our model follows a three-stage process: change point segmentation, kernel-weighted Dirichlet pool- ing, and cross-validation tuning. First, we apply pruned exact linear time (PELT) change point detec- tion (Killick et al., 2012; Truong et al., 2020) with an exact multinomial cost. These change points divide the base path into segments in which we expect the outcome distribution to either remain flat or slowly drift. Next, within these segments we pool answer counts from neighboring data points, weighting them with a Gaussian kernel, and use the result to parameterize a Dirichlet distribution. Finally, we use cross-validation to tune three hyper- parameters of the model: the cost function (multi- nomial likelihood vs.L 2 loss) and penalty value for PELT, and the kernel bandwidth used for Gaussian smoothing within segments. This cross-validation operates over the same data being smoothed, by splitting e.g.S=20into 5 folds each withS=4 data points and finding the hyperparameters for each S=16 that best explains the held out fold. We evaluate the overall success of our statistical model by measuring total variation distance (TVD) of smoothed low-sampleo t relative to ground truth high-sampleo t data withS=200,N=1. TVD mea- 5101520305075100 samples per position S 0.02 0.03 0.05 0.1 TVD to S=200 reference Llama-3-8B-Instruct observation spacing (tokens) Smoothed Data Raw Data 1 token 2 tokens 4 tokens 8 tokens 16 tokens 32 tokens reference noise floor Figure 2: Accuracy versus number of resampled roll- outs. We find that with raw data observationso t (dot- ted lines), greater resample spacingNleads to slightly lower accuracy with equalS, with considerably more difference at higherS. This effect is significantly more pronounced with the smoothedo t (solid lines), which also shows that increasinglogShas a consistent linear effect on log-TVD. LargerNhaving decreasing accu- racy for a givenShas a relatively simple explanation, since this means fewer samples collected overall. Accu- racy on the y-axis is measured by total variation distance (TVD) relative to the S = 200 reference o t . sures the distance between two distributionspand q:TVD(p,q) = 1 2 P K k=1 | p k − q k | . In Ap- pendix E we report ablations to our model, which confirm that PELT can accurately identify forking points and that kernel pooling produces accurate flat regions. 3 Experimental Results Data. We analyze forking paths on the tinyMMLU dataset (Maia Polo et al., 2024) (n=100), collecting a high-sample reference FPA dataset withS=200,N=1.For LLama-3-8B- Instruct, we resample at every token, whereas for DeepSeekR1-Distill-Llama-8B we resample at every sentence, since reasoning models generate much longer reasoning chains.In total, we collected 1.77B tokens for this FPA dataset. We evaluate forking path analyses with decreas- ing sampling ratesS ∈ [5, 100]and intervals N ∈1, 2, 4, 8, 16, 32, against the reference anal- ysis withS=200andN = 1. We report total variation distance (TVD) between this reference FPA against each lower-sample analysis to charac- 3 terize the tradeoff between precision and sampling efficiency. For each of these conditions, we then use our statistical model to produced smoothedo t estimates, and compare these to the S=200 data. 3.1 Estimating o t from fewer samples Across all questions, the statistical model’s excess TVD above the reference noise floor falls monoton- ically from0.0265(S=5) to0.0056(S=100). For Llama, the sample efficiency multiplier at 4-token spacing is5.0×atS=5,3.3×atS=30, and1.0× atS=100(Figure 2, dark blue lines). At 1-token spacing, the sample efficiency multiplier grows to 22.1×atS=5,7.3×atS=30, and1.6×atS=100 (Figure 2, red lines). We find similar results with the reasoning model’s sentence resampling with multipliers4.9×atS=5,3.1×atS=30, and1.1× at S=100 (Appendix C). We find that with raw data, increasing the spac- ing between observation has relatively little effect on accuracy overall (Fig. 2, dotted lines), except with highS. This may be because sampling noise dominates fluctuations ino t unlessSis sufficiently high (e.g., Fig. 1, Bottom). An exception to this trend is regions around forking points, where us- ing a wider stride gives less accurateo t estimates (Fig. 7). However, the smoothed data shows a very different pattern, where higher sampling gran- ularity reliably leads to more accurateo t estimates (Fig. 2, solid lines), which also holds for forking regions (Fig. 7). 3.2 Tradeoff between accuracy and cost Figure 3 maps accuracy against token cost, com- paring the raw data against model-smoothed data. Here, we see a clear pattern with the raw data: wider token strides lead to more accurate approxi- mations of high-sample data, given the same token budget. Intuitively, averaging data over a fixed win- dow leads to theo t estimates having lower variance, since sampling noise is averaged out. When data is smoothed, however, we instead see that chang- ing sampling window has relatively little effect on token efficiency, but instead represents a different section of the same Pareto frontier. In other words, with smoothed data, changingSand the spacing be- tween observation samplesthas roughly the same effect (Fig. 3, solid lines). 3.3Sampling noise decreases with sample size To measure how sampling noise changes with sam- ple size, we divided theS=1000samples for each 10 4 10 5 10 6 tokens generated per question 0.02 0.03 0.05 0.1 TVD to S=200 reference Llama-3-8B-Instruct observation spacing (tokens) Smoothed Data Raw Data 1 token 2 tokens 4 tokens 8 tokens 16 tokens 32 tokens status quo (S=30, every token) reference noise floor Figure 3: Accuracy versus token cost. We find that overall, with rawo t data, wider spacing between re- sampled pointsNleads to better accuracy with lower overall token cost (dotted lines). Our statistical model significantly increases the accuracy for mostSandN, but also negates the improvement of largerNincreasing accuracy with equal cost (solid lines). TVD between theS=200reference and each low-sample estimator is shown on the y-axis, with colors representing different N. The black star represents the sampling parameters used in (Bigelow et al., 2024a). of two questions into disjoint, equally sized subsets. At each position, we then computed the pairwise to- tal variation distance (TVD) between the outcome distributions estimated from these subsets. As Fig- ure 11 shows, the pooled TVD between estimates from these independent sample subsets decreases steadily asSincreases. On log-log axes, the fitted slope is−0.4903throughS=200, closely match- ing the−1/2slope expected under independent multinomial sampling, suggesting that noise de- creases in proportion to 1/ √ S. The fullS=200dataset shows similar scaling, with slopes of−0.4757for Llama when resampling every token and−0.4838for DeepSeek when re- sampling every sentence. AtS=200, the variation across runs (TVD = 0.0451) is nearly identical to the sampling variation within a run (TVD = 0.0442). These results support modeling the ob- served counts at each position as multinomial sam- ples: c t ∼ Mult(S,o t ). 4 4 Discussion Here we have shown that heavy cost of resampling steps in reasoning and text generation is in fact a modeling problem. The noise ofo t is exactly multinomial and the curves are smooth with sparse forking points, so a large resampling budget mostly re-measures structure that neighboring positions already contain. An estimator that pools within segments and breaks at change points converts that structure into∼ 15×efficiency gains within seg- ments, and∼ 4×efficiency gains without losing accuracy in identifying forking points. This work opens a number of exciting avenues for future work. It suggests that there could be a mechanistic theory of in-context learning that can explain these kinds of learning dynamics (Nanda et al., 2023; Bigelow et al., 2024b). Finally, while our method enables us to post-hoc smooth ap- proximate outcome distributionso t , an alternate approach which integrates modeling would be to more efficiently choose particular token indices to sample from, similar to approaches in optimal experiment design (Chaloner and Verdinelli, 1995). References Eric Bigelow, Ari Holtzman, Hidenori Tanaka, and Tomer Ullman. 2024a. Forking paths in neural text generation. Preprint, arXiv:2412.07961. Eric Bigelow, Ekdeep Singh Lubana, Robert Dick, Hide- nori Tanaka, and Tomer Ullman. 2024b. In-context learning dynamics with random binary sequences. In International Conference on Learning Representa- tions, volume 2024, pages 56330–56373. Paul C Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. 2025. Thought anchors: Which llm reason- ing steps matter? arXiv preprint arXiv:2506.19143. Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, and Jack Merullo. 2026. Reasoning theater: Disentangling model beliefs from chain-of-thought. arXiv preprint arXiv:2603.05488. Kathryn Chaloner and Isabella Verdinelli. 1995. Bayesian experimental design: A review. Statisti- cal science, pages 273–304. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, et al. 2025. DeepSeek- R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Preprint, arXiv:2501.12948. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations. Rebecca Killick, Paul Fearnhead, and Idris A. Eckley. 2012. Optimal detection of changepoints with a lin- ear computational cost. Journal of the American Statistical Association, 107(500):1590–1598. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. Uzay Macar, Paul Bogdan, Senthooran Rajamanoharan, and Neel Nanda. 2026. Thought branches: Inter- preting llm reasoning requires resampling. In Inter- national Conference on Learning Representations, volume 2026, pages 92567–92600. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinyBenchmarks: Evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress mea- sures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217. Charles Truong, Laurent Oudre, and Nicolas Vayatis. 2020. Selective review of offline change point detec- tion methods. Signal Processing, 167:107299. Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Au- rojit Panda, Jinyang Li, and He He. 2025. Rea- soning models know when they’re right: Probing hidden states for self-verification. arXiv preprint arXiv:2504.05419. 5 A Limitations Scope. Our experiments are limited to tinyMMLU multiple-choice questions and two 8B-parameter models: one instruction-tuned model and one reasoning-distilled model. We analyze five-class outcome distributions over relatively short rollouts (continuations of at most 400 tokens for Llama and 1536 tokens for DeepSeek). Whether the re- sults generalize to longer horizons, open-ended outcomes, or larger models remains to be tested. Hyperparameter tuning. The Full Model and its hyperparameter tuning procedure were devel- oped using two questions from one model. The tinyMMLUS=200dataset and all DeepSeek data were used only for evaluation; however, they come from the same task family and use models at the same parameter scale as the development data. Estimator effectiveness depends on how big the forks are. The estimator’s advantage in fork regions is established at a forking threshold of0.10, but it degrades at0.15(n=35questions), and at the largest forks of0.20(n=21), smoothing is a marginally worse estimator than raw data. Point estimates only. The estimators’ 90% cred- ible intervals do not achieve their nominal coverage on the observed data: the Full Model’s empirical coverage ranges from0.48to0.64across the devel- opment questions. We therefore report and recom- mend the point estimates rather than the credible intervals. Reference curves are estimates. The refer- ence curves are themselves finite-sample estimates (S=200orS=1000) and thus have a nonzero noise floor. Figure-level TVDs computed against the full reference include this floor. For hypothesis-bearing comparisons, we use leave-replicate-out references to prevent overlap between an estimate and its refer- ence from artificially reducing the measured error. B Additional Example Reconstructions Figure 4 extends the comparison in Figure 1 to three questions from the tinyMMLUS=200eval- uation set. For each question, the reference is theS=200every-token curve. The reduced run usesS=15at 4-token spacing, with samples nested within the same set of draws. C DeepSeek-R1-Distill Results Figures 5 and 6 reproduce the analyses in main-text Figures 2 and 3 for DeepSeek-R1-Distill-Llama- 8B. They show accuracy and sampling cost on the reasoning model with sentence-level resampling, with observation spacings fromN=1toN=8sen- tences. The main trends carry over: TVD decreases asSincreases at every spacing, the densest obser- vation spacing performs best, and the Full Model yields its largest improvement over Raw Data at smallS. Section 3.1 reports the corresponding effective-sample multipliers. D Forking Region Accuracy and Cost Figures 7 and 8 reproduce the accuracy and budget analyses from the main text after restricting evalua- tion to forking regions, where a forking region is defined as positions within±10tokens of a forking point. Forking points are defined as pointstsuch thatTV D(o t ,o t+N ) > ε, i.e. points where there is a significant difference in the outcome distribution between two adjacent steps. We report results at forking thresholds ε∈0.10, 0.15, 0.20. We observer that performance in forking regions deteriorates earlier and more rapidly than overall performance as the observation spacing becomes coarser andNincreases. AtS=30, fork-region TVD increases from0.0669to0.0936to0.1587as spacing increases fromN=4toN=8andN=16 tokens. It then saturates near0.16at spacings of 32– 64 tokens, indicating that the fork has effectively been missed. Flat-region performance degrades more gradually, and the same ordering holds at scale for both models. Second, the Full Model’s advantage over Raw Data near forks is concentrated at moderate forks (threshold0.10), disappears for larger forks, and even reverses with the largest forks (Section 3.2). In the threshold-0.15panels, the solid and dotted curves at dense spacings nearly coincide. This agreement reflects the powered null result of the preregistered test rather than a plotting artifact. E Component Ablation of the Full Model Ablating the Full Model’s components one at a time on theS=200tinyMMLU dataset reveals which components contribute to accuracy in flat and fork- ing regions. We report paired per-question TVD differences relative to the Full Model, computed against leave-replicate-out references, with 95% t-intervals. E.1 Estimators We compare four estimators of increasing complex- ity: 6 0 0.5 1 S=200 reference (every token) question 46 ABCDOther 0 0.5 1 question 59 0 0.5 1 question 4 0 0.5 1 Smoothed Data (S=15, every 4 tok) 0 0.5 1 0 0.5 1 050100150200250 position t (tokens) 0 0.5 1 Raw Data (S=15, every 4 tok) 0100200300 position t (tokens) 0 0.5 1 0100200300 position t (tokens) 0 0.5 1 Figure 4: Reconstructions for three questions from the full tinyMMLU evaluation set using Llama-3-8B-Instruct. From top to bottom, the rows show theS=200every-token reference, the Full Model’s reconstruction from a nested S=15 run at 4-token spacing, and Raw Data from the same reduced run. 5101520305075100 samples per position S 0.03 0.05 0.1 0.15 TVD to S=200 reference DeepSeek-R1-Distill-Llama-8B observation spacing (sentences) Smoothed Data Raw Data 1 sentence 2 sentences 4 sentences 8 sentences reference noise floor Figure 5: DeepSeek counterpart to Figure 2. Pooled TVD to the fullS=200reference is plotted against the number of samples per position,S, with one line per ob- servation spacing in sentences. Dotted lines are results for raw data, solid lines are smoothed data. Raw Data:Empirical frequencies with linear in- terpolation between observed positions. Kernel Pooling: Gaussian-kernel-weighted Dirichlet pooling of neighboring counts. This estimator is designed for regions between forks, where increments are small and nearby positions have similar values of o t . 10 5 10 6 tokens generated per question 0.03 0.05 0.1 0.15 TVD to S=200 reference DeepSeek-R1-Distill-Llama-8B observation spacing (sentences) Smoothed Data Raw Data 1 sentence 2 sentences 4 sentences 8 sentences status quo (S=30, every sentence) reference noise floor Figure 6: DeepSeek counterpart to Figure 3. Pooled TVD is plotted against the number of tokens generated per question; colors match Figure 5. Dotted lines are results for raw data, solid lines are smoothed data. The star marks the baseline setting ofS=30at 1-sentence spacing. Segment+Pool:PELT change point detection (Killick et al., 2012; Truong et al., 2020) with an exact multinomial cost, followed by per-segment Dirichlet pooling. The resulting piecewise-constant fit preserves boundaries at 7 0.03 0.05 0.1 0.15 0.2 jump-region TVD to S=200 reference Llama-3-8B-Instruct — Forking Threshold 0.10 0.03 0.05 0.1 0.15 0.2 Llama-3-8B-Instruct — Forking Threshold 0.15 spacing (solid Smoothed Data, dotted Raw Data) 1 token 2 tokens 4 tokens 8 tokens 16 tokens 32 tokens 5101520305075100 samples per position S 0.03 0.05 0.1 0.15 0.2 0.3 jump-region TVD to S=200 reference DeepSeek-R1-Distill-8B — Forking Threshold 0.10 5101520305075100 samples per position S 0.03 0.05 0.1 0.15 0.2 0.3 DeepSeek-R1-Distill-8B — Forking Threshold 0.15 Figure 7: Forking region counterpart to Figure 2. TVD to theS=200reference is evaluated only at forking regions, using forking thresholds of0.10(left) and0.15(right). Solid lines show Smoothed Data, and dotted lines show Raw Data; colors match those in the main text. detected forks. Full Model: PELT segmentation followed by Ker- nel Pooling truncated at change point bound- aries. This estimator accommodates grad- ual drift within segments while preserving sharp changes at forks. The penalty and band- width hyperparameters are selected by cross- validation on low-sample runs. E.2 Further Model Ablations Segmentation preserves forks. Removing seg- mentation and using Kernel Pooling alone re- tains 93–96% of the pooled improvement but de- grades accuracy in forking regions. The TVD difference relative to the Full Model is+0.0113 [+0.0039, +0.0187]for Llama at forking thresh- old0.15and+0.0091 [+0.0036, +0.0145]for DeepSeek at threshold0.10. Figure 10 shows two examples in which Kernel Pooling blurs forks that the Full Model preserves.On the recorded DeepSeek example, the respective TVDs are 0.2720 and 0.0088. Cross-validated tuning matters at large forks. Replacing cross-validated hyperparame- ters with fixed values increases TVD by0.0126 [+0.0051, +0.0202] for Llama at threshold 0.15. Kernel pooling provides a small but consis- tent pooled gain. Replacing kernel pooling with flat per-segment pooling (Segment+Pool) increases pooled TVD by0.0021 [+0.0015, +0.0026]for Llama at S=30. The detection cost has little end-to-end effect. Fixing PELT to the L2 cost increases pooled TVD by≤ 0.0010throughout the ablation. Its perfor- mance is statistically indistinguishable from that of the Full Model in both headline fork conditions; for example, the difference for Llama atS=30and threshold0.10is+0.0003 [−0.0004, +0.0009]. This variant retains 97–100% of the Full Model’s improvement over Raw Data. Figure 9 summarizes these paired differences. For pooled accuracy, every smoothing variant is within a few thousandths of the Full Model, whereas Raw Data has substantially higher TVD. In forking regions, the no-segmentation variant differs reliably from zero, while the L2-detection variant does not. F The 1/ √ S Noise Law Figure 11 presents the replicate-noise measure- ments underlying Section 3.3. For theS=1000 8 10 4 10 5 10 6 tokens generated per question 0.03 0.05 0.1 0.15 0.2 jump-region TVD to S=200 reference Llama-3-8B-Instruct — Forking Threshold 0.10 10 4 10 5 10 6 tokens generated per question 0.03 0.05 0.1 0.15 0.2 Llama-3-8B-Instruct — Forking Threshold 0.15 spacing (solid Smoothed Data, dotted Raw Data) 1 token 2 tokens 4 tokens 8 tokens 16 tokens 32 tokens 10 5 10 6 tokens generated per question 0.03 0.05 0.1 0.15 0.2 0.3 jump-region TVD to S=200 reference DeepSeek-R1-Distill-8B — Forking Threshold 0.10 10 5 10 6 tokens generated per question 0.03 0.05 0.1 0.15 0.2 0.3 DeepSeek-R1-Distill-8B — Forking Threshold 0.15 Figure 8: Fork-region counterpart to Figure 3. TVD in forking regions is plotted against the number of tokens generated per question at forking thresholds of0.10and0.15. Solid lines show Smoothed Data, and dotted lines show Raw Data. development set, pooled TVD between disjoint replicates is plotted againstSon log–log axes. The fitted slope is−0.4903overall and−0.5047 in the tail. The figure also shows the exact i.i.d. multinomial null, computed by resampling the per- branch histograms through the same statistic; the ratios of measured to null TVD range from0.98 to1.01. Finally, the per-model means from the S=200tinyMMLU dataset have slopes of−0.4767 and−0.4577. Their absolute TVDs are lower be- cause those questions are less variable on average, but the error decreases at a similar rate. G Segmentation Cost in PELT Implementation During this work, we uncovered a bug in the im- plementation of PELT in theruptureslibrary (Truong et al., 2020). The segment-based estimators use PELT change point detection with an exact multinomial cost. If a custom cost object does not inherit fromBaseCost, the library silently replaces it with the default least- squares cost,CostL2. The fit still completes and returns change points without warning, but results are significantly affected (Figure 12). H LLM Use Statement The research reported in this paper and the initial manuscript draft were produced by the autonomous LLM agent Silico (https://w.goodfire.com/ silico) under human direction. 9 0.04 0.05 0.07 0.1 0.15 0.2 TVD to S=200 reference Full Dataset 0.04 0.05 0.07 0.1 0.15 0.2 Forking Regions (τ = 0.10) 51530100 samples per position S 0.04 0.05 0.07 0.1 0.15 0.2 TVD to S=200 reference 51530100 samples per position S 0.04 0.05 0.07 0.1 0.15 0.2 Raw DataKernel PoolingSegment+PoolFull Model Full ModelFixed hyperparameters (no CV tuning)L2 detection costTrend-aware detection cost Figure 9: Paired per-question TVD differences between each ablation variant and the Full Model for DeepSeek-R1- Distill-Llama-8B at 1-sentence spacing. Error bars show 95%t-intervals over tinyMMLU, and the dashed zero line denotes the Full Model. Results are shown for pooled accuracy (left) and forking region accuracy at threshold0.10 (right). Removing segmentation harms accuracy near forks but not pooled accuracy; fixing the detection cost to L2 is indistinguishable from the Full Model in both panels. 10 0 0.5 1 o t S=200 reference — Llama-3-8B-Instruct, question 46 (computed) ABCDOther 0 0.5 1 S=200 reference — DeepSeek-R1-Distill-8B, question 3 (recorded) 0 0.5 1 o t Full Model 0 0.5 1 0 0.5 1 o t L2 detection cost 0 0.5 1 0 0.5 1 o t no kernel pooling (Segment+Pool) 0 0.5 1 0 0.5 1 o t no segmentation (Kernel Pooling) 0 0.5 1 050100150200250 position t in the base path (tokens) 0 0.5 1 o t Raw Data 02004006008001000 position t in the base path (tokens) 0 0.5 1 Figure 10: Ablation overlays for two example questions. Reconstructions useS=30on the densest spacingNand are evaluated against leave-replicate-outS=200references; dashed lines mark reference forks. The left column shows Llama-3-8B-Instruct question 46 using the ablation variants, cross-validation, and operating point from the recordedS=200store. The right column shows the recorded DeepSeek example, where the fork occurs when continuations stop resolving to an answer. Removing segmentation (Kernel Pooling) blurs the forks, whereas removing kernel pooling (Segment+Pool) reduces gradual drift to discrete steps. The L2 detection-cost variant is visually indistinguishable from the Full Model. 11 2050100200 samples per position S 10 −1 replicate TVD measured (2 questions, pooled) exact i.i.d. multinomial null 1/√ S reference Llama (100 questions) DeepSeek (100 questions) Figure 11: Replicate TVD versus the number of samples per position,S, on log–log axes. The measured curve remains within2%of the exact i.i.d. multinomial null throughS=200. Triangles show the full dataset means for both models. 12 2 0 2 2 2 4 2 6 2 8 2 10 PELT penalty 0 10 20 30 40 segments detected question 12 2 0 2 2 2 4 2 6 2 8 2 10 PELT penalty question 39 multinomial cost trend-aware cost L2 cost (silent fallback) ideal segment count Figure 12: Number of detected segments as a function of the PELT penalty forS=30dense-grid counts on the two development questions. Curves show the exact multinomial cost, a trend-aware cost, and the silently substituted L2 cost. The dashed line marks the ideal segment count. 13