Paper deep dive
Does an Agent's History Tell You When Compaction Will Hurt? A Modest, Bounded Effect on the TRACE Paired-Replay Corpus
Egor Pakhomov, Erik Nijkamp
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many long-horizon agents compact their context on a global rule, usually a token budget, blind to what the agent was doing. We ask whether the agent's recent behaviour predicts when a compaction will hurt. TRACE's public corpus of 590 harness-triggered AppWorld compaction boundaries replays each boundary from a re-executed prefix state under the pre-compaction context and under the summary, and records the burden of the next actions: calls that error or repeat a call already made. We find that pre-boundary history predicts post-compaction harm only weakly. An internally prespecified contrast by prefix placement is a wide null, and the naive "has-written" label behind it turns out to measure trajectory phase. The best extension-protocol trigger reaches held-out AUROC 0.66 (0.64 on the replicate's own label) against a same-boundary replicate of 0.72; the best frozen, interpretable trigger avoids 21% of harmful (positive-burden) boundaries while keeping 84% of compaction opportunities, and exceeds the random-rule expectation on count but not on burden mass (a post hoc comparison). Whether the best trigger beats a token-budget rule at matched retention cannot be evaluated on the release. We state what corpora should ship to answer it.
Tags
Links
- Source: https://arxiv.org/abs/2610.08722v1
- Canonical: https://arxiv.org/abs/2610.08722v1
Trouble viewing inline? Open PDF directly →
Full Text
80,211 characters extracted from source content.
Expand or collapse full text
Interpreting Agent Behavior (IAB) Does an Agent’s History Tell You When Compaction Will Hurt? A Modest, Bounded Effect on the TRACE Paired-Replay Corpus Egor Pakhomov Affiliation: Salesforce AI Research Email: epakhomov@salesforce.com Erik Nijkamp Affiliation: Salesforce AI Research Email: erik.nijkamp@salesforce.com Abstract Many long-horizon agents compact their context on a global rule, usually a token budget, blind to what the agent was doing. We ask whether the agent’s recent behaviour predicts when a compaction will hurt. TRACE’s public corpus of 590 harness-triggered AppWorld compaction boundaries replays each boundary from a re-executed prefix state under the pre-compaction context and under the summary, and records the burden of the next actions: calls that error or repeat a call already made. We find that pre-boundary history predicts post-compaction harm only weakly. An internally prespecified contrast by prefix placement is a wide null, and the naive “has-written” label behind it turns out to measure trajectory phase. The best extension-protocol trigger reaches held-out AUROC 0.66 (0.64 on the replicate’s own label) against a same-boundary replicate of 0.72; the best frozen, interpretable trigger avoids 21% of harmful (positive-burden) boundaries while keeping 84% of compaction opportunities, and exceeds the random-rule expectation on count but not on burden mass (a post hoc comparison). Whether the best trigger beats a token-budget rule at matched retention cannot be evaluated on the release. We state what corpora should ship to answer it. 1 Introduction Many long-horizon agent harnesses compact [1, 2, 3, 4, 5]: past a budget, the raw history is replaced by a generated summary [6], and the decision of when is a global rule, blind to what the agent was doing (in the coding-agent scaffolds surveyed by Rombaut [7], typically a token threshold or a structural cap)—the natural next step is to defer compaction where the recent history says it is about to hurt. Agent-invoked compression [8], RL-learned context editing [9] and learned folding [10] condition the timing on the history but are judged by task success, which cannot say whether a given compaction hurt; that needs a counterfactual outcome per boundary. TRACE [11, 12] supplies it (§2). The paired design, the burden channels, the first/later stratification and the thesis that compaction should be judged at the boundary by the agent’s next actions, not only by task success (process-level evaluation [13]; failure attribution [14]; summaries can lengthen trajectories by masking failure signals [15]) are the source’s [11, 12]. Behavioural interpretation of agents mostly labels completed transcripts [16]; a paired replay is the intervention such a science calls for [17]. We use the released cohort as is and ask: does what the agent did before a boundary predict whether compaction there will hurt? Contributions. (i) An internally prespecified placement test with a post hoc diagnostic of what a naive write label measures (§3). (i) A task-held-out risk ranking of boundaries and an opportunity-retention frontier against global ON/OFF and the random-rule expectation at matched retention (retention: the share of opportunities a rule still allows; the random-rule expectation: what a uniformly random rule achieves), not reported by the source [11] (§4). (i) A benchmark of two fixed learners against a split-half replicate yardstick under a frozen extension protocol (post-freeze additions dated in App. A). The placement contrast is a wide null. The seven Table 2 policies sit two to seven points above the random-rule expectation on count (only the stump’s interval crosses zero; the two-feature logistic does so too when refit under stem-grouped folds, App. F, Table 16) and zero to eleven points on burden mass, clear of zero under replay resampling for three of the seven (App. E, F). Held-out AUROC [18] is 0.404–0.635 across Table 11’s original-label cells for the interpretable policies (0.55–0.62 on the primary target) and 0.66 for the boosted ones, against 0.72 for a replicate of the same boundary. Interpretability lessons: the phase proxy (§3) and first-compaction fragility (§5). 2 Corpus and measurement Cohort. TRACE’s frozen cohort holds 590 compaction boundaries from 275 of 294 trajectories on 147 AppWorld tasks [19] (140 with a boundary; App. A). Agent and compressor are one model (minimax-m3; 4,096-token window) [11, §3]; boundaries are harness-triggered, never agent-chosen; 275 are first compactions, 315 later. At each boundary the recorded code prefix is re-executed into a fresh environment to rebuild the state (the replay is identified by a hash of task, boundary step and prefix code, not of the reached state; deltas inherit any restoration variance) and the agent runs at most five actions from the PRE rendering (full raw prefix at first compactions; previous summary plus raw suffix later) or the POST rendering (recorded summary plus retained last turn); five samples per arm at first compactions, three at later ones [11, §3.3]. Channels. Blocked, refetch and union channels are defined in App. A. One boundary. The boundary comes from task 6ea6792_2 (easy), a first compaction after seven steps, chosen for clarity (dU@3 1.8, five times the cohort mean of 0.35). Its prefix function set is api_docs.show_api_descriptions, api_docs.show_api_doc, supervisor.show_account_passwords, venmo.login and venmo.show_received_payment_requests; its last observation is a page of pending payment requests. Under PRE (the full raw prefix) all five rollouts request page 2 (empty), then print the requests already held in Python variables or keep browsing; none of the first three actions is blocked or a refetch. Under POST (the summary plus the retained turn) all five request page 2; four then hit a NameError because the variables the summary describes are no longer bound (blocked) and re-issue the prefix call show_received_payment_requests(page_index=0) (refetch); the fifth re-reads the Venmo API list it had already fetched (refetch). The boundary’s dU@3 is 1.8 (PRE 0.0, POST 1.8; blocked 0.8, refetch 1.0). For each boundary and horizon k∈1,3,5k∈\1,3,5\, each rollout contributes its count over the first min(k,L) (k,L) actions, L its executed actions; the paired delta is mean(POST) −- mean(PRE) of that count, giving dE (blocked), dR (refetch) and dU (union); exposure and completion by arm are in App. C. dU at k=3k=3 is the preregistered primary outcome; 344 of 590 boundaries have dU@3 >0>0. Statistics. Unless stated otherwise, every interval is a 95% percentile bootstrap over tasks as clusters [20, 21] with fixed seeds, 10,000 draws for the preregistered analyses and 2,000 for the extensions (App. F). What the release exposes before a boundary. The release exposes sorted call and function sets, prefix step count, compaction index, task identifier and difficulty, and the last three observations—not the ordered action sequence, token counts, compression ratio or summary text (App. A). 3 Is burden localised by pre-boundary placement? Label. Each AppWorld endpoint carries an HTTP method in its decorator (GET = read; POST/PUT/PATCH/DELETE = write); the map covers all 119 functions observed (App. B), so every boundary is labelled. The map was checked by an LLM-agent-authored table encoded in the analysis (43/43 non-GET functions) and, in a second audit, by two labelers of different model families given the READ/WRITE and SESSION/SUBSTANTIVE definitions but not the map (119/119; relabelling moves no boundary; App. B). Cells: 494 boundaries with a write in the prefix, 96 without (read-only). Prespecified contrast. dU@3, post-write minus read-only: −0.062-0.062 [−0.209-0.209, +0.079+0.079] (any prior write +0.335+0.335 [+0.269+0.269, +0.405+0.405] over 494 boundaries / 135 tasks); stem-clustered, post hoc sensitivity: [−0.238-0.238, +0.100+0.100] (App. E). The frozen criterion for an informative answer (interval excluding zero, or half-width at most 0.10) was not met (half-width 0.144; criterion (b) was out of reach at the realized cell sizes, computed after the fact from the observed SDs, App. A). Table 1: Boundaries by pre-boundary function set; the no-write row is the frozen two-cell label’s read-only cell and the substantive-write row is the post hoc label’s cell, both with frozen-seed intervals (App. C); the login-only interval is derived (seed 20260950); login-only = writes only to session endpoints (App. B). Frozen two-cell label and prespecified contrast: §3. Pre-boundary function set Boundaries Tasks Prefix steps Compaction index dU@3 [95% CI] No write 96 60 7.9 1.22 +0.398+0.398 [+0.266+0.266, +0.533+0.533] Login-only write 368 123 15.1 1.89 +0.391+0.391 [+0.313+0.313, +0.472+0.472] Substantive write 126 64 22.1 3.14 +0.172+0.172 [+0.048+0.048, +0.291+0.291] What the label measures. A post hoc diagnostic (App. A): of the 494 “write-prefixed” boundaries, 368 have no write beyond login or session calls; 126 contain a state change to task-relevant application data. Here “has-written” is mostly “has been running for a while”: a trajectory-phase proxy. Post hoc relabelling. Excluding the login/session endpoints from the write set is outcome-free but was defined after the original-label results had been seen, so it is post hoc and carries no gate. It yields an uncontrolled contrast of −0.22-0.22 that falls to −0.004-0.004 [−0.19-0.19, +0.18+0.18] once task fixed effects, log prefix steps and a first-compaction indicator are added (App. C): the apparent placement effect is consistent with confounding by the measured phase variables. 4 Can a history-based trigger beat random selection at matched retention? Policies. A trigger policy scores each boundary from pre-boundary observables and blocks compaction at or above a threshold. Three interpretable classes: a stump and a depth-2 tree over six observables (App. E), and a logistic regression on two: has-written and recent errors. Target: 1[dU@3>0]1[dU@3>0]; evaluation task-held-out five-fold, all scores out-of-fold. Offline value. An allowed boundary incurs the POST arm; a blocked one keeps the pre-compaction rendering and incurs the PRE arm; both arms are observed for every boundary, so a policy’s value is a plug-in sum over logged arms with no new rollouts, not an off-policy estimate [22]. The x-axis is the fraction of compaction opportunities retained (no token counts are released, so opportunities are equal-weight; App. A); the y-axis is the fraction of positive-burden boundaries blocked; the random-rule expectation is the diagonal y=1−xy=1-x (analogous to a risk–coverage curve [23, 24]). Figure 1: Out-of-fold frontier (task-held-out folds for the three frozen policies; stem-grouped folds averaged over 20 repetitions for the two boosted ones; Table 2’s two lower logistic rows are not drawn): positive-burden boundaries (dU@3 >0>0; 344 of 590) blocked vs. equal-weight compaction opportunities retained; diagonal = random-rule expectation; hollow circles = global ON (1,0)(1,0) / OFF (0,1)(0,1); markers with 95% intervals at ≥80%≥ 80\% retained (Table 2). Hollow diamond: the same-boundary replicate used as a trigger score (a yardstick, not a policy; App. F). (Full width: App. E.) Table 2: Operating points at ≥80%≥ 80\% retained. Δ = paired advantage over the random-rule expectation 1−1-Ret. (a fixed expectation, not a realized draw); replicate row scored against half-sample labels. Rows are read at their own operating points (Ret.) under different fold schemes (task-held-out for the frozen policies; stem-grouped × 20 repetitions for the extension models), so rows are not directly comparable. Policy Folds Ret. Avoided [95% CI] Δ vs random [95% CI] Stump task 0.908 0.108 [0.083, 0.238] +0.016+0.016 [−0.001-0.001, +0.048+0.048] Depth-2 tree task 0.842 0.195 [0.096, 0.246] +0.037+0.037 [+0.012+0.012, +0.064+0.064] Logistic (2 features) task 0.837 0.206 [0.156, 0.256] +0.044+0.044 [+0.017+0.017, +0.070+0.070] 17 engineered summaries (boosted) stem 0.800 0.247 [0.217, 0.277] +0.047+0.047 [+0.019+0.019, +0.079+0.079] Richer representation (144 features, boosted) stem 0.800 0.267 [0.238, 0.290] +0.067+0.067 [+0.040+0.040, +0.092+0.092] Logistic, 144 features stem 0.800 0.273 [0.243, 0.296] +0.073+0.073 [+0.045+0.045, +0.098+0.098] Logistic, summaries + difficulty stem 0.800 0.273 [0.243, 0.299] +0.073+0.073 [+0.045+0.045, +0.100+0.100] Replicate (yardstick) half-sample 0.866 0.243 [0.198, 0.286] +0.109+0.109 [+0.082+0.082, +0.134+0.134] Result (Fig. 2, Table 2). At ≥80%≥ 80\% retained, the logistic policy avoids 0.206 [0.156, 0.256] of positive-burden boundaries at 0.837 retained (strict threshold; conditional on fitted out-of-fold scores; App. E); the advantage over the random-rule expectation of 0.163 is +0.044+0.044 [+0.017+0.017, +0.070+0.070] (a post hoc interval; App. A); by burden mass (share of all positive dU@3) the logistic policy has no advantage (−0.001-0.001 [−0.034-0.034, +0.035+0.035]), while the depth-2 tree and the boosted trigger clear zero on the task-only bootstrap with the threshold reselected per draw (Table 16 for the boosted trigger; App. E for the depth-2 tree). The frozen gate G4 (App. A) criterion (half-width ≤0.15≤ 0.15 at the 80% point) was met, with half-width 0.050. Engineered summaries of the released observables. Under a second protocol (frozen before any extension output; App. F), an L2 logistic regression and a shallow gradient-boosted classifier were given seventeen engineered summaries of the released observables (App. F) under five-fold cross-validation grouped by task stem, the scenario identifier a task shares with its variants (48 stems, 20 repetitions); the logistic cell with the largest operating-point advantage (summaries plus task difficulty, Table 16) is also in Table 2. The best cell, the boosted classifier on the seventeen history features, reaches held-out AUROC 0.66 [0.61, 0.70] (App. F); at 80% retained it blocks 0.247 [0.217, 0.277] of positive-burden boundaries and 0.238 [0.201, 0.284] of positive burden mass, against a random-rule expectation of 0.200 (+0.047+0.047 [+0.019+0.019, +0.079+0.079] by count; Table 2, Fig. 2); these extension intervals are unadjusted for selection and multiplicity (App. E, F). Richer representations. Adding one indicator per prior function and app (144 features) gives held-out AUROC 0.66 [0.61, 0.71] and +0.067+0.067 [+0.040+0.040, +0.092+0.092] over the random-rule expectation at 80% retained; hashed observation text or exact calls did not improve held-out AUROC point estimates (0.61–0.65) (full statement in App. F.1; both 144-feature models are in Table 2). Replicate yardstick. Splitting each boundary’s PRE and POST rollouts into disjoint halves (200 random splits), one half’s paired delta predicts the sign of the other’s with AUROC 0.69–0.72 at k=3k=3 depending on which half supplies the label (0.72 [0.69, 0.75] under the frozen extension protocol’s assignment; first 0.74, later 0.68); the half–half correlation is 0.40 and the task-level share of variance is near zero (App. F). Burden is thus moderately reproducible in ranking terms (AUROC 0.72, r 0.40) and only weakly in sign (three-class agreement 0.55, κ 0.28); little of its variation is attributable to task identity in this cohort. Against this yardstick the three frozen policies (0.55–0.62 at k=3k=3) fall short of what a replicate of the same boundary predicts; the extension protocol’s overlap rule would read the boosted trigger (0.66 [0.61, 0.70]) as within the replicate’s range, but in the post hoc same-label comparison (amendment D) the replicate leads the seventeen-summary trigger by +0.07+0.07 [+0.04+0.04, +0.11+0.11] and the 144-feature one by +0.06+0.06 [+0.02+0.02, +0.10+0.10] (App. F), so on that comparison the best trigger falls short of what a replicate of the same boundary predicts. 5 Discussion and limitations For harness design. At a boundary the released observables barely separate harmful compactions from harmless ones, and the evidence does not show a per-boundary trigger a harness designer would adopt on this corpus. For the representations and learners tested, the count advantage over the random rule is two to seven points. Against the same half-sample label, the boosted seventeen-summary trigger’s 0.64 recovers about two-thirds of the replicate’s 0.72 discrimination above chance (0.14 of 0.22 AUROC points; App. F); with three-class sign reliability of only 0.55 (κ 0.28) the target itself is noisy, so the same numbers also support an attenuation reading. When this agent is fragile: the source’s shipped curves already show first compactions carry more burden than later ones; under task fixed effects the gap is +0.274+0.274 [+0.084+0.084, +0.474+0.474], the stump’s shorter-history leaf carries the higher positive-burden rate, and the tree’s leaf for a short history ending in a near-empty observation reaches 0.89–0.93 in four of five folds on its fitted rows (0.71 in the fifth; 28–36 boundaries per fold, mostly first compactions; App. E). The interpretability lesson: check a “has-written” label against prefix length and compaction index before reading it as placement. The outcome is a meaningful prediction target—the replicate yardstick shows the burden is reproducible enough—but whether its signal is available in the pre-boundary history is not settled by the models and representations tested here; corpora should ship ordered actions, token counts, summary text and more replays. Limitations. One agent model, one compressor and one window (4,096 tokens, deliberately tight); harness-triggered boundaries on AppWorld only, so agent-chosen compaction [8] is out of scope; the offline value prices only the next k actions, not a later compaction a blocked one may only postpone; coarse released observables; 140 task clusters, 55 with both placement cells; per-boundary deltas from three to five replays per arm (noisy); frontiers pooled over fold models (5 or 100), not a deployable fitted policy (App. E); no data redistribution (the source carries no licence file)—we cite the release; the analysis record is available on request. LLM-use statement: App. G. Acknowledgments and Disclosure of Funding This work was funded by Salesforce. References [1] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems, 2023. URL https://arxiv.org/abs/2310.08560. [2] Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. ACON: Optimizing context compression for long-horizon LLM agents. In Forty-third International Conference on Machine Learning (ICML), 2026. URL https://arxiv.org/abs/2510.00615. [3] Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou. ReSum: Unlocking long-horizon search intelligence via context summarization, 2025. arXiv:2509.13313. [4] Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Prannay Hebbar, Jiankai Sun, Yiyuan Li, Pramod Srinivasan, Ishan Gupta, Christopher Settles, Daniel Wang, Derek Chen, Pranav Raja, Albert Liu, Marek Šuppa, Nevasini Sasikumar, Luyang Kong, Erik Quintanilla, Xiangyi Li, Ivan Bercovich, and Steven Dillmann. SWE-Marathon: Can agents autonomously complete ultra-long-horizon software work?, June 2026. URL https://arxiv.org/abs/2606.07682. arXiv preprint, v1, 5 June 2026. [5] Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, and Yuxiao Dong. CompactionRL: Reinforcement learning with context compaction for long-horizon agents, July 2026. URL https://arxiv.org/abs/2607.05378. arXiv preprint, v1, 6 July 2026. [6] Qingyue Wang, Yanhe Fu, Yanan Cao, Shuai Wang, Zhiliang Tian, and Liang Ding. Recursively summarizing enables long-term dialogue memory in large language models, 2023. URL https://arxiv.org/abs/2308.15022. Accepted at Neurocomputing (per arXiv listing). [7] Benjamin Rombaut. Inside the scaffold: A source-code taxonomy of coding agent architectures, April 2026. URL https://arxiv.org/abs/2604.03515. arXiv preprint, v2, 10 April 2026. [8] Shukai Liu, Bo Jiang, Jian Yang, Yizhi Li, Jinyang Guo, Xianglong Liu, and Bryan Dai. Context as a tool: Context management for long-horizon SWE-agents. In Findings of the Association for Computational Linguistics: ACL 2026, pages 20604–20617, 2026. doi: 10.18653/v1/2026.findings-acl.1032. arXiv:2512.22087; author order as in the ACL Anthology record. [9] Yuxiang Zhang, Jiangming Shu, Ye Ma, Xueyuan Lin, Shangxi Wu, and Jitao Sang. Memory as action: Autonomous context curation for long-horizon agentic tasks, October 2025a. URL https://arxiv.org/abs/2510.12635. arXiv preprint, v3, 7 May 2026. [10] Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. Scaling long-horizon LLM agent via context-folding, October 2025. URL https://arxiv.org/abs/2510.11967. arXiv preprint, v1, 13 October 2025. [11] Guanghui Min, Liang Wu, Mayank Darbari, Chen Chen, and Liangjie Hong. Toward reliable context compression for long-horizon agents: An empirical study of execution instability, August 2026. URL https://arxiv.org/abs/2608.06503. arXiv preprint, v1, 6 August 2026. [12] Nokia Applied Research. TRACE: a verifier for agent context-compression policies. Software repository, GitHub, 2026. URL https://github.com/nokia-applied-research/Trace. Git revision 9be7d66911b7600fdec66e64f4825865397e944 (committed 2026-08-06T01:06:32Z); accessed 2026-09-02. No LICENSE or CITATION file present at that revision. [13] Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An analytical evaluation board of multi-turn LLM agents. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. URL https://arxiv.org/abs/2401.13178. [14] Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? On automated failure attribution of LLM multi-agent systems, 2025b. arXiv:2505.00212. [15] Tobias Lindenbauer, Igor Slinko, Ludwig Felder, Egor Bogomolov, and Yaroslav Zharov. The complexity trap: Simple observation masking is as efficient as LLM summarization for agent context management. In 4th Workshop on Deep Learning for Code (DL4C), NeurIPS 2025, 2025. arXiv:2508.21433, v3 (camera-ready). [16] Jie Gao, Kaiser Sun, Jen-tse Huang, Katherine Van Koevering, Sijie Ji, Heyuan Huang, Weiyan Shi, Zhuoran Lu, Ziang Xiao, Daniel Khashabi, and Mark Dredze. How to interpret agent behavior, May 2026. URL https://arxiv.org/abs/2605.13625. arXiv preprint, v1, 13 May 2026. [17] Lin Chen, Yunke Zhang, Jie Feng, Haoye Chai, Honglin Zhang, Bingbing Fan, Yibo Ma, Shiyuan Zhang, Nian Li, Tianhui Liu, Nicholas Sukiennik, Keyu Zhao, Yu Li, Ziyi Liu, Fengli Xu, and Yong Li. AI agent behavioral science, June 2025. URL https://arxiv.org/abs/2506.06366. arXiv preprint, v3, 12 June 2025. [18] James A. Hanley and Barbara J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1):29–36, 1982. doi: 10.1148/radiology.143.1.7063747. [19] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16022–16076, 2024. doi: 10.18653/v1/2024.acl-long.850. URL https://aclanthology.org/2024.acl-long.850/. [20] A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3):414–427, 2008. doi: 10.1162/rest.90.3.414. [21] Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall, New York, 1993. [22] Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on Machine Learning (ICML), 2011. URL https://arxiv.org/abs/1103.4601. [23] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), 2017. arXiv:1705.08500. [24] Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53):1605–1641, 2010. [25] Charles Spearman. Correlation calculated from faulty data. British Journal of Psychology, 3(3):271–295, 1910. doi: 10.1111/j.2044-8295.1910.tb00206.x. [26] William Brown. Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3(3):296–322, 1910. doi: 10.1111/j.2044-8295.1910.tb00207.x. [27] Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non-determinism of “deterministic” LLM settings, August 2024. URL https://arxiv.org/abs/2408.04667. arXiv preprint, v5, 2 April 2025. [28] Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and mitigating numerical sources of nondeterminism in LLM inference. In Advances in Neural Information Processing Systems 38, pages 188045–188077, 2025. doi: 10.52202/085713-5653. arXiv:2506.09501. Appendix A Reproduction and preregistration record Source. TRACE repository at revision 9be7d669, cloned once; no re-download and no copy of the data outside the clone. The corpus documentation states 147 tasks, 294 trajectories, 590 boundaries, 4,641 stored rollouts (4,640 analysed: one rollout is kept with a recorded collection error and skipped by scoring) and 20,133 executed steps; 140 of the 147 tasks and 275 of the 294 trajectories carry at least one boundary; the release’s run configuration names the agent model minimax-m3; the source preprint states the same model served as compressor [11]; window 4,096, matching the source preprint’s main experiments (its §5.1). The source also releases its own annotation of compaction mechanisms on 46 compactions in 14 trajectories per the source’s consolidated annotation summary (its data glossary counts the 12 trajectories of the first annotation pass), which we do not use (46 annotated compactions against 590 boundaries). Renderings. At a trajectory’s first compaction the PRE rendering is the full raw prefix; at later compactions it is the previous real summary plus the raw suffix since it, and no previous summary is ever fabricated. The POST rendering is always the recorded summary plus the single retained raw turn. Released observables. Sorted sets of prior calls and functions, prefix step count, compaction index, task identifier and difficulty, and the last three observations, but not the ordered action sequence, token counts, compression ratio or summary text; three preregistered features are therefore unmeasurable and the two-cell collapse is a dated amendment (below). Channels. Per executed action, blocked means the action returned an error under AppWorld’s native error contract (the source’s per-step error flag); refetch means its exact action signature (function and arguments) had already been observed before the boundary or earlier in the same rollout; union means blocked or refetch (this channel sense of blocked is distinct from a policy blocking a compaction, §4). These are the source’s definitions [11, Fig. 4]; per rollout each channel is counted over the first min(k,L) (k,L) actions (§2). Reproduction (gate G1). The source ships six scan tables and a test that regenerates them. Under Python 3.10.8 four of the six regenerated tables were byte-identical to the shipped ones and the two bootstrap-bearing tables differed at the 10−1510^-15 level (Table 3); the cause is float summation order across interpreter versions, not randomness (the source bootstrap is seeded), and under Python 3.12.13 all six were byte-identical and the test passed 6/6. Downstream analysis used the Python-3.10 fresh scan as its frozen basis; its deterministic tables are byte-identical to the shipped ones, so no number in this paper depends on the difference; the row-level comparison is in the frozen reproduction record. Table 3: Shipped scan tables versus a fresh scan (SHA-256, first 16 hex digits). Table Shipped SHA-256 Python 3.10 fresh Python 3.12 fresh boundary_wide.csv 8534faccb3a97dec identical identical boundary_long.csv 34a61f99a9d7af7 identical identical step_level.csv eba9b5d8075c90ea identical identical curves.csv 0cd6cee8d4f5dfda differs (6fba4773d74415b7) identical marginal.csv 2d144932f1bd365 differs (8b2adbb1326b5d98) identical error_subtypes.csv 6a2969efd8f15b91 identical identical Gates as frozen, and their status. The criteria were written before any stratified analysis ran and reference execution quality and informativeness only, never the direction of an estimate. • G1 Reproduction. The source’s reproduction test passes and a fresh scan matches the shipped tables by SHA-256. Met on Python 3.12 (above). • G2 Taxonomy coverage. At least 90% of the 590 boundaries receive a deterministic read/write label; an audit clause; each analysed cell has at least 30 boundaries, else a prespecified collapse to two cells. Met on coverage (590/590) and cell size (494/96); the audit clause as written (“hand-audit agreement ≥ 45/50”, frozen text, verbatim) was not met—no person audited—and was replaced by the dated G2 audit-clause amendment below (a map-blind audit by two LLM labelers, 119/119, App. B.1, reported descriptively with no pass threshold); a separate encoded semantic-map check (43/43, App. B) preceded it; G2 is met only as amended: the amended audit clause is descriptive (agreement reported, no threshold), so G2 as amended reduces to its coverage and cell-size clauses, both met. • G3 Informative heterogeneity answer (either direction). The post-write minus read-only contrast on dU@3 has a task-clustered 95% interval that (a) excludes zero or (b) has half-width ≤0.10≤ 0.10; wide and crossing zero means the placement half is not the headline and the paper falls back to the policy half if G4 passes. Not met: −0.062-0.062 [−0.209-0.209, +0.079+0.079], half-width 0.144. Criterion (b) was out of reach at the realized cell sizes by a post hoc unclustered normal approximation: the unclustered half-width at the realized cell sizes is already 0.129 (sample SDs 0.557 and 0.728 at n=96n=96 and 494; computed after the fact from the observed SDs), so G3 could only have passed by excluding zero; the null’s width is a design consequence, not a finding. • G4 Informative policy answer (either direction). Task-held-out frontier estimates for every policy class with bootstrap intervals; the best policy’s avoided fraction at the 80% point has interval half-width ≤0.15≤ 0.15; beating or losing to global rules is reportable either way. Met: the frozen logistic’s half-width is 0.050 task-only and 0.052 nested (App. E); the criterion (≤0.15≤ 0.15) was met. • G5 Layer distinctness from the source. The source has not already published the taxonomy-heterogeneity layer or the offline-trigger layer; the paper claims only clear layers. Met: the source preprint [11] reports the paired PRE/POST design, aggregate post-compaction action-position effects (its §3.3, Fig. 4), a compression-prompt optimizer driven by summary preferences, and a comparison of fixed context budgets, 16K/8K/4K/2K, under FIFO truncation versus summarisation (its §3.1–3.2, Figs. 1–2); it reports no boundary-risk trigger, selective-trigger frontier or task-held-out policy evaluation. • Write-up proceeds iff G1, G2 and G5 pass and at least one of G3/G4 passes. Satisfied via G4, with G2 as amended; this paper is the policy half, and §3 reports the G3 null as a null. Amendments (dated; the originals were never edited). All freezes were internal: dated files with SHA-256 hashes, available from the authors on request; no third-party registry was used. • 2026-09-01, G1′. After the Python-3.10 run, the byte-identity requirement was relaxed for the two bootstrap-bearing tables to numerically equivalent point estimates (differences at the 10−1510^-15 level) and statistically consistent intervals, with the fresh scan as the frozen downstream basis and the discrepancy reported here verbatim. 2026-09-02, correction: the source bootstrap is seeded; the differences are summation order across interpreter versions; Python 3.12 gives byte identity for all six tables. G1 is met on its original letter; the amendment stands as a provenance record. • 2026-09-02, G2′ (label and observables). Logged before any amended-label output was generated in the full run. (a) A second, post hoc label excludes session endpoints (name suffix in login, logout, set_access_token, signup, send_verification_code, reset_password, send_password_reset_code) from the write set; the original label and all its outputs are kept unchanged, and every downstream stage is run under both labels with the same seeds. The two-cell collapse had already been made in the earlier script, and the original-label results had been computed and seen on 2026-09-01 before the second label was defined on 2026-09-02; that is why it is post hoc and carries no gate. (b) The four-cell taxonomy collapsed to two cells because last_action_type, unresolved_commitment and compression_ratio are unmeasurable in the release, not because of cell size; “any write anywhere in the pre-boundary function set” is the substitute observable and the task-id stem the app-domain proxy. (c) position_fraction is dropped from every feature set because it is 1.0 in all 590 rows by construction; the two-feature logistic keeps its artifact name. (d) A first-compaction indicator is added to the controlled regression and a first-compaction-only contrast is added with its own seed. • 2026-09-02, G4′ (frontier axis). Logged before any frontier output was read. The release carries no per-boundary token counts, so the frontier x-axis is the fraction of equal-weight compaction opportunities retained; the G4 criterion is unchanged and direction-blind. • 2026-09-02, extension protocol. The replicate yardstick and the engineered-summary trigger (App. F) were frozen, with their interpretation rules, before any extension output was computed. • 2026-09-02, G2 audit clause. The frozen clause named a hand audit that no person performed; a map-blind audit by two LLM labelers of different model families was frozen as a separate mini-protocol at 12:20, before any audit call and with no threshold deciding anything, and run at 12:38 (App. B.1); the G2 status above records the substitution. • 2026-09-02, advantage over the random rule. Paired task bootstrap of avoided −- (1−retained)(1-retained) at the operating point, for the frozen policies and then for the extension models: a post-freeze addition made during internal pre-submission review, computed 13:00; not preregistered. • 2026-09-02, amendments A–F, all direction-blind post-freeze additions made during internal pre-submission review. A (12:40, nested task × replay bootstrap) and B (12:40, mixed-label paired replicate −- trigger difference): computed 13:02. C (13:52, EXT-3 richer representations): computed from 13:57. D (same-label paired difference) and E (stem-clustered intervals): direction-blind sensitivity analyses specified after their motivating result was known, computed 13:48 and logged 13:52—not preregistered (dated correction in the frozen prereg record). F: signed plug-in value (Table 10); logged 16:59, computed 17:03; post-freeze, not preregistered. Analysis environment. Seeded analysis (base seed 20260901; per-stage seeds in the frozen analysis record), 10,000 task-clustered bootstrap draws in every preregistered interval (2,000 in the extension analyses, App. F), Python 3.10.8 with NumPy 1.26.4 for the least-squares solve. The statistical analysis pipeline made no model API calls and collected no new rollouts; the endpoint-map audit of App. B.1 used $1.15 of hosted-API calls to two LLM labelers. Total downstream wall time was 786 s. Appendix B Endpoint method map and audit protocol Map. The map records the HTTP method attached to each AppWorld v0.1.3.post1 endpoint decorator; GET is read and POST/PUT/PATCH/DELETE write. The map was extracted once from the AppWorld v0.1.3.post1 endpoint decorators and is recorded with the analysis (aliases in Table 4); of the 119 functions observed in the cohort’s pre-boundary function sets, 18 are explicit aliases. Method counts: 76 GET, 35 POST, 5 DELETE, 3 PATCH; 0 unmapped. A boundary is post-write under the original label if any function in its pre-boundary function set is non-GET, and substantive-write under the post hoc label if any non-GET function’s name suffix is outside the session list of App. A. Table 4: The 18 explicit aliases in the endpoint method map. Function Method Function Method file_system.list_directory GET supervisor.show_account GET file_system.read GET supervisor.show_account_credentials GET file_system.read_file GET supervisor.show_account_profile GET file_system.show_file_metadata GET supervisor.show_account_usernames GET phone.get_contact GET supervisor.show_phone_contacts GET phone.set_access_token POST venmo.show_api_descriptions GET simple_note.show_api_descriptions GET venmo.show_balance GET spotify.get_music_queue GET venmo.show_friends GET spotify.show_music_player GET venmo.show_transaction_aggregate_stats GET Check of the map. A separately encoded semantic map—an agent-authored table encoded in the analysis—gives a per-endpoint judgement for the 43 non-GET functions observed in the cohort (function name plus AppWorld API semantics, one reason each) and a read-verb rule for the remaining 76 (name begins with show_, search_, get_, read or list_); the table agrees with the decorator method on 43/43. A 50-boundary comparison (25 per original cell, seed 20260908) follows mechanically from that table and the pre-boundary function sets and is not a second audit; an earlier comparison that derived its labels from the map itself was discarded as circular. A further audit by two labelers of different model families, blind to the map, is reported in App. B.1. B.1 Blind two-labeler audit In a second, map-blind audit under a protocol frozen beforehand, two LLM labelers of different model families were given the READ/WRITE and SESSION/SUBSTANTIVE definitions but not the map. They were called through a hosted API; the frozen protocol specified temperature 0 for both, but one model’s endpoint rejects that setting, so it ran at its default temperature (a logged deviation) and the other at temperature 0. Each labeler saw each of the 119 endpoint names with the one-line AppWorld description where one could be extracted from the corpus’s documentation calls (101 of 119; the 18 alias names have none) and never the method map, and answered READ or WRITE and, for writes, SESSION or SUBSTANTIVE. Both labelers agree with the decorator map on 119 of 119 endpoints (43 of 43 non-GET, 76 of 76 GET), with each other on 119 of 119, and with the session-suffix rule on 43 of 43 writes (11 session, 32 substantive); there were no parse failures. Replacing the map by either labeler’s calls moves 0 of 590 boundaries under the original split (494/96) and 0 under the post hoc split (126/464). API cost $1.15; per-call cost records are kept in the analysis record. Table 5: Session endpoints among the 494 boundaries labelled post-write under the original label: boundaries whose pre-boundary function set contains the function. Function Boundaries spotify.login 200 phone.login 146 venmo.login 143 file_system.login 99 simple_note.login 73 gmail.login 4 spotify.send_password_reset_code 4 venmo.logout 2 venmo.send_verification_code 2 phone.set_access_token 2 spotify.reset_password 1 Appendix C Cell estimates, exploratory contrasts and regressions All intervals are 95% task-clustered percentile bootstrap intervals with 10,000 draws. Under the original label the cells are read-only (96 boundaries, 60 tasks) and any prior write (494, 135); under the post hoc label, no substantive write (464, 132) and substantive write (126, 64). Table 6: Per-cell paired deltas by horizon and channel, original label. Cell k dU dE dR Read-only 1 ++0.135 [++0.071, ++0.205] ++0.053 [++0.010, ++0.101] ++0.086 [++0.026, ++0.148] Read-only 3 ++0.398 [++0.266, ++0.533] ++0.130 [++0.061, ++0.212] ++0.272 [++0.160, ++0.384] Read-only 5 ++0.681 [++0.504, ++0.859] ++0.179 [++0.107, ++0.266] ++0.516 [++0.370, ++0.665] Any prior write 1 ++0.130 [++0.097, ++0.163] ++0.119 [++0.096, ++0.144] ++0.021 [−-0.008, ++0.048] Any prior write 3 ++0.335 [++0.269, ++0.405] ++0.229 [++0.188, ++0.270] ++0.140 [++0.088, ++0.194] Any prior write 5 ++0.476 [++0.384, ++0.570] ++0.299 [++0.246, ++0.353] ++0.231 [++0.157, ++0.305] Table 7: Per-cell paired deltas by horizon and channel, post hoc label. Cell k dU dE dR No substantive write 1 ++0.153 [++0.119, ++0.188] ++0.110 [++0.087, ++0.134] ++0.049 [++0.023, ++0.075] No substantive write 3 ++0.393 [++0.326, ++0.464] ++0.224 [++0.186, ++0.264] ++0.197 [++0.145, ++0.250] No substantive write 5 ++0.566 [++0.478, ++0.658] ++0.291 [++0.243, ++0.343] ++0.316 [++0.248, ++0.385] Substantive write 1 ++0.048 [−-0.004, ++0.099] ++0.104 [++0.053, ++0.160] −-0.034 [−-0.096, ++0.030] Substantive write 3 ++0.172 [++0.048, ++0.291] ++0.171 [++0.077, ++0.266] ++0.031 [−-0.065, ++0.123] Substantive write 5 ++0.301 [++0.122, ++0.472] ++0.241 [++0.116, ++0.366] ++0.136 [−-0.013, ++0.274] Contrasts. (Only the original-label all-boundaries contrast was prespecified; the first-compaction restriction is amendment G2′(d), and the substantive-write contrasts use the post hoc label.) Original label, dU@3, post-write minus read-only: all boundaries −0.062-0.062 [−0.209-0.209, +0.079+0.079], half-width 0.144 (590 boundaries, 140 tasks, 55 tasks with both cells); first compactions only +0.069+0.069 [−0.086-0.086, +0.225+0.225], half-width 0.156 (275 boundaries, 42 tasks with both cells). Post hoc label, substantive-write minus no-substantive-write: all boundaries −0.221-0.221 [−0.359-0.359, −0.093-0.093], half-width 0.133 (56 tasks with both cells); first compactions only −0.389-0.389 [−0.546-0.546, −0.234-0.234], half-width 0.156, with only 5 tasks containing both cells (18 substantive-write first compactions out of 275). Table 8: Exploratory horizon × channel contrasts (not prespecified; 590 boundaries, 140 tasks). Left: original label, post-write minus read-only. Right: post hoc label, substantive-write minus no-substantive-write. k Channel Original label Post hoc label 1 dU −-0.005 [−-0.078, ++0.066] −-0.105 [−-0.167, −-0.044] 1 dE ++0.067 [++0.013, ++0.117] −-0.006 [−-0.062, ++0.053] 1 dR −-0.066 [−-0.135, ++0.002] −-0.083 [−-0.150, −-0.012] 3 dU −-0.062 [−-0.207, ++0.079] −-0.221 [−-0.359, −-0.092] 3 dE ++0.099 [++0.003, ++0.182] −-0.053 [−-0.153, ++0.047] 3 dR −-0.132 [−-0.254, −-0.010] −-0.166 [−-0.274, −-0.066] 5 dU −-0.205 [−-0.405, −-0.009] −-0.265 [−-0.461, −-0.081] 5 dE ++0.120 [++0.018, ++0.209] −-0.050 [−-0.183, ++0.083] 5 dR −-0.285 [−-0.457, −-0.118] −-0.180 [−-0.341, −-0.033] Regressions. Outcome: per-boundary dU@3. Uncontrolled: intercept and label. Controlled: intercept, label, log prefix steps, a first-compaction indicator and 139 task indicators; ordinary least squares; task-clustered bootstrap intervals. Original label, post-write coefficient: uncontrolled −0.062-0.062 [−0.207-0.207, +0.078+0.078]; controlled −0.027-0.027 [−0.216-0.216, +0.174+0.174]. Post hoc label, substantive-write coefficient: uncontrolled −0.221-0.221 [−0.358-0.358, −0.091-0.091]; controlled −0.004-0.004 [−0.193-0.193, +0.176+0.176]. The uncontrolled OLS coefficient on a binary regressor is the difference of cell means, so the three printed −0.221-0.221 figures are one estimate (−0.2209-0.2209); their intervals differ only because each table draws its own 10,000 task resamples (seeds 20260924, 20260930, 20260932); the same holds for the three −0.062-0.062 intervals of the original-label contrast. Exposure by arm. A rollout’s L is its number of recorded steps (equal to its valid primary actions; a rollout stops at five actions, when the environment reports the task complete, or at a step with no valid call). Boundary-weighted over the 590 boundaries (2,320 rollouts per arm; five zero-step rollouts enter with L=0L=0): valid actions through k=3k=3, PRE 2.768 / POST 2.761; completed by k=3k=3 (terminated with task complete at a step ≤3≤ 3), PRE 0.221 / POST 0.199; through k=5k=5, 4.249 / 4.299 and 0.368 / 0.328 (the release’s own whole-rollout columns). The exposure imbalance is small; burden is reported as a total effect that includes failure to continue; a reach-k sensitivity is future work. Appendix D The first-compaction confound Figure 2: Paired deltas at k=3k=3 (dU, dE, dR; mean per boundary, 95% task-clustered intervals) for the three pre-boundary constructs of Table 1: no write (96 boundaries; mean prefix 7.9 steps), login-only write (368; 15.1), substantive write (126; 22.1). The no-write and substantive-write intervals are those of Table 1; the login-only interval is derived (seed 20260950). The login-only group is not a cell of either frozen label; the figure is descriptive. The cohort has 275 first and 315 later compactions. Under the original label the read-only cell is 78/96 first compactions (81%) and the write cell 197/494; under the post hoc label 257 of the 464 no-substantive-write boundaries and 18 of the 126 substantive-write boundaries are first compactions. Mean compaction index is 1.22 for no-write, 1.89 for login-only and 3.14 for substantive-write boundaries (Table 1). Any label that correlates with prefix length therefore also correlates with whether the boundary is a first compaction, and the repository’s shipped curves already separate first from later boundaries and stratify by compaction index [12]; first and later boundaries are also sampled differently (five versus three per arm) [11]. This is why the controlled regressions of App. C include a first-compaction indicator and why the first-compaction-only contrasts are reported: under the original label the first-compaction-only contrast is +0.069+0.069 [−0.086-0.086, +0.225+0.225], and under the post hoc label the corresponding contrast rests on 5 tasks that contain both cells. In the controlled regressions the first-compaction indicator itself carries +0.274+0.274 [+0.084+0.084, +0.474+0.474] under the original label and +0.273+0.273 [+0.091+0.091, +0.466+0.466] under the post hoc label (log prefix steps −0.103-0.103 [−0.368-0.368, +0.178+0.178] and −0.113-0.113 [−0.366-0.366, +0.158+0.158]). Appendix E Trigger policies: definitions, folds, frontiers and sensitivity Observables. • post_write_prefix: the original label (the post hoc runs use the post hoc substantive-write label substantive_write_prefix); • log_prefix_steps =log(1+prefix steps)= (1+ prefix steps); • recent_error_count: the number of the last three prefix observations that begin with the environment’s failure contract; • last_observation_chars: the length of the last prefix observation; • compression_index: 1 for a trajectory’s first compaction, 2 for its second, and so on; • n_historical_exact: the number of distinct exact call signatures before the boundary. The stump and the depth-2 tree choose among all six; the logistic uses the label and recent_error_count. Fitting. Trees: Gini impurity, at most 25 candidate thresholds per feature, no split of fewer than 20 rows. Logistic: features standardised on the training fold, full-batch gradient descent (1,500 iterations, step 0.05, L2 penalty 0.01 on non-intercept weights). Folds: the 140 task identifiers are shuffled once with a fixed seed and assigned round-robin to five folds; a boundary is scored by the model fitted on the other four folds. Frontier: boundaries are sorted by out-of-fold risk score; a threshold allows compaction for every boundary with score strictly below it; the operating point is the threshold that maximises the avoided fraction among all thresholds retaining at least 80% of boundaries; intervals are task-clustered, conditional on the fitted out-of-fold scores and the realized labels (10,000 draws; seeds 20261112, 20261124, 20261128 for stump, depth-2, logistic); a nested bootstrap that also resamples the 3–5 replays within each boundary (Table 9; 2,000 draws, seed 20260960, scores fixed, no refit) leaves the widths essentially unchanged but shifts the intervals: the count advantages over the random-rule expectation become +0.044+0.044 [+0.006+0.006, +0.065+0.065] (logistic), +0.037+0.037 [+0.010+0.010, +0.069+0.069] (depth-2), +0.016+0.016 [−0.005-0.005, +0.051+0.051] (stump) and +0.047+0.047 [+0.025+0.025, +0.089+0.089] (boosted), and no burden-mass advantage then excludes zero among the four cells of Table 9 (the 144-feature cell’s does; App. F.1); the count-advantage half-widths grow by less than 0.004 (full widths by up to 0.007; the avoided-fraction half-widths change by at most 0.006, Table 9), and we read the nested family as a sensitivity analysis, not a second confidence family. Table 9: Nested (replays within boundary, then tasks) versus task-only bootstrap intervals for the avoided fraction at ≥80%≥ 80\% retained; half-widths in parentheses; last column: burden-mass advantage over the random-rule expectation under the nested bootstrap. Under replay resampling 11.5% of boundaries change sign label and the resampled positive rate is 0.533 against the realized 0.583. Policy Avoided Nested 95% CI Task-only 95% CI Mass advantage (nested) Stump 0.108 [0.084, 0.239] (0.077) [0.083, 0.238] (0.078) +0.001+0.001 [−0.032-0.032, +0.033+0.033] Depth-2 tree 0.195 [0.112, 0.251] (0.069) [0.096, 0.246] (0.075) +0.042+0.042 [−0.003-0.003, +0.074+0.074] Logistic (2 features) 0.206 [0.146, 0.250] (0.052) [0.156, 0.256] (0.050) −0.001-0.001 [−0.041-0.041, +0.031+0.031] 17 summaries (boosted) 0.247 [0.223, 0.288] (0.032) [0.217, 0.277] (0.030) +0.038+0.038 [−0.005-0.005, +0.083+0.083] Table 10: Signed plug-in value at the ≥80%≥ 80\% operating point (amendment F, specified after review). V = burden avoided by keeping PRE on the blocked set B, with boundaries where POST had less burden counted against the policy (Σ+ ^+, Σ− ^-, number negative); Erand=(|B|/590)ΣallE_rand=(|B|/590)\, _all with ΣalldU@3=203.9 _all\,dU@3=203.9 over 590 boundaries (mean 0.346). Intervals: task-clustered, 2,000 draws (seed 20260964), and nested task × replay (seed 20260965), scores and thresholds fixed. The positive-part version (max(0,dU) (0,dU)) reproduces the burden-mass column of Table 12 and the mass advantages of App. F in point estimate; these intervals hold the operating threshold fixed, whereas Table 16’s reselect it in every draw, so a borderline cell’s zero crossing can differ between the two. Policy |B||B| (share) V (Σ+ ^+, Σ− ^-; neg.) ErandE_rand V−ErandV-E_rand [task 95% CI] Stump 54 (0.092) 22.00 (25.00, −3.00-3.00; 8) 18.67 +3.33+3.33 [−4.46-4.46, +11.67+11.67] Depth-2 tree 93 (0.158) 49.67 (54.27, −4.60-4.60; 12) 32.15 +17.52+17.52 [+4.62+4.62, +29.69+29.69] Logistic (frozen) 96 (0.163) 38.20 (44.00, −5.80-5.80; 11) 33.18 +5.02+5.02 [−6.25-6.25, +16.82+16.82] Boosted, 17 summaries 118 (0.200) 59.27 (64.73, −5.47-5.47; 12) 40.79 +18.48+18.48 [+5.52+5.52, +31.11+31.11] Boosted, 144 features 118 (0.200) 74.07 (76.80, −2.73-2.73; 8) 40.79 +33.28+33.28 [+19.14+19.14, +47.78+47.78] Logistic, 144 features 118 (0.200) 78.53 (82.87, −4.33-4.33; 9) 40.79 +37.75+37.75 [+23.72+23.72, +52.78+52.78] Logistic, summaries + difficulty 118 (0.200) 66.00 (72.13, −6.13-6.13; 11) 40.79 +25.21+25.21 [+11.52+11.52, +38.34+38.34] Policy V−ErandV-E_rand [nested 95% CI] V/|B|V/|B| [task 95% CI] [nested] Stump [−5.59-5.59, +12.51+12.51] 0.407 [0.257, 0.550] [0.237, 0.575] Depth-2 tree [+3.80+3.80, +31.46+31.46] 0.534 [0.378, 0.676] [0.370, 0.696] Logistic (frozen) [−6.90-6.90, +18.16+18.16] 0.398 [0.266, 0.531] [0.262, 0.550] Boosted, 17 summaries [+5.09+5.09, +32.46+32.46] 0.502 [0.376, 0.624] [0.369, 0.639] Boosted, 144 features [+18.90+18.90, +48.81+48.81] 0.628 [0.502, 0.753] [0.500, 0.765] Logistic, 144 features [+22.77+22.77, +55.28+55.28] 0.666 [0.535, 0.804] [0.531, 0.820] Logistic, summaries + difficulty [+10.61+10.61, +40.94+40.94] 0.559 [0.431, 0.680] [0.426, 0.702] Signed plug-in value. The signed plug-in value exceeds its random-blocking expectation by +5.0+5.0 [−6.3-6.3, +16.8+16.8] burden-steps for the frozen logistic (96 blocked; not clear of zero), +18.5+18.5 [+5.5+5.5, +31.1+31.1] for the boosted seventeen-summary trigger and +33.3+33.3 [+19.1+19.1, +47.8+47.8] for the 144-feature one (118 blocked each; 0.40–0.63 per blocked boundary against a mean of 0.35; post hoc; Table 10). Amendment F extended the nested analysis from five to seven policies: the nested positive-part interval excludes zero for three of the seven (boosted-144 [+8.3+8.3, +35.2+35.2], logistic-144 [+13.4+13.4, +41.7+41.7], logistic on summaries + difficulty [+4.5+4.5, +28.5+28.5] burden-steps); the depth-2 tree’s touches zero ([−0.04-0.04, +21.3+21.3]); the task-only positive-part interval clears zero in four of seven (the boosted seventeen-summary model’s, [−0.6-0.6, +21.0+21.0], does not), the signed value in five of seven under both bootstraps. Δ compares against the fixed expectation 1−Ret.1-Ret.; a realized uniformly random rule blocking 54 / 93 / 96 / 118 of 590 boundaries has a hypergeometric standard deviation of 0.010 / 0.013 / 0.013 / 0.014 in the avoided fraction, so the stump’s +0.016+0.016 is about 1.6 such deviations and the frozen logistic’s +0.044+0.044 about 3.4. In Fig. 2 the depth-2, logistic and 144-feature markers are offset horizontally by 0.008 and the markers at 0.80 overprint, their intervals (±0.03± 0.03) lying inside the marker. The frontier pools out-of-fold scores from the five fold models and is therefore a mixture of five fitted policies (a deployable stump has a single operating point; the stump’s interior frontier points exist only because the fold models have different leaf rates); per-fold AUROC, mean (range), beside the pooled value: stump 0.618 (0.570–0.669) against 0.592, depth-2 0.633 (0.578–0.701) against 0.619, logistic 0.582 (0.525–0.642) against 0.547; the pooled scores take 10 / 20 / 24 distinct values and the fold positive rates range 0.514–0.641. Advantage over the random rule by burden mass at the operating points of Table 2: logistic −0.001-0.001 [−0.034-0.034, +0.035+0.035], stump +0.001+0.001 [−0.028-0.028, +0.034+0.034], depth-2 +0.042+0.042 [+0.007+0.007, +0.079+0.079], boosted +0.038+0.038 [+0.003+0.003, +0.086+0.086]. Fitted models (descriptive). The stump splits on n_historical_exact ≤15.5≤ 15.5 in four folds and on compression_index ≤1.5≤ 1.5 in one, the shorter-history leaf having the higher positive-burden rate in all five (0.68–0.71 against 0.40–0.47). The depth-2 tree’s short-history × short-last-observation leaf (distinct prior calls ≤15.5≤ 15.5, or compaction index ≤1.5≤ 1.5 in one fold, and last observation ≤ 50–60 characters) has a positive-burden rate of 0.89–0.93 in four of five folds on the rows the fold model was fitted on (n = 28–36 per fold; 0.71 in the fifth); these are in-sample leaf purities of outcome-selected splits, reported descriptively. The release caps observations at 200 characters and 464 of 590 boundaries sit at the cap; the 74 boundaries at ≤60≤ 60 characters are ones whose last call returned almost nothing—an empty result list or a one-line status or lookup reply—mostly first compactions after short histories; their positive-burden share is 0.73 against 0.58 overall. The depth-2 tree refines that split mostly with last_observation_chars on the shorter-history side and log_prefix_steps on the longer-history side, in one fold each with compression_index and n_historical_exact. The logistic coefficients on the standardised features are negative for both features in every fold under both labellings. This records what the policies used, not a mechanism. Table 11: Task-held-out AUROC [18] by horizon and channel for the target 1[Δ>0]1[ >0] of that channel. Left block: original label; right block: post hoc label. Each policy is refit out-of-fold to that horizon × channel target under the same task-held-out folds; only the k=3k=3 union cells coincide with the frozen dU@3 policies. Original label Post hoc label k Channel Stump Depth-2 Logistic Stump Depth-2 Logistic 1 dU 0.546 0.577 0.467 0.546 0.572 0.539 1 dE 0.490 0.567 0.503 0.490 0.567 0.475 1 dR 0.572 0.602 0.507 0.572 0.602 0.515 3 dU 0.592 0.619 0.547 0.592 0.628 0.560 3 dE 0.468 0.466 0.489 0.468 0.466 0.438 3 dR 0.600 0.635 0.532 0.600 0.635 0.533 5 dU 0.580 0.597 0.540 0.585 0.608 0.551 5 dE 0.474 0.491 0.404 0.474 0.491 0.450 5 dR 0.610 0.604 0.547 0.610 0.625 0.570 The horizon × channel grid of Table 11 and the exploratory contrasts of App. C are descriptive and not adjusted for multiplicity. Stem-clustered sensitivity (48 task-id stems as clusters, 2,000 draws; task-clustered value in parentheses; these intervals re-estimate the policies’ uncertainty, not their fitting, which stays task-held-out). Placement contrast [−0.238-0.238, +0.100+0.100] ([−0.209-0.209, +0.079+0.079]); avoided at ≥80%≥ 80\%—stump [0.085, 0.239], depth-2 [0.099, 0.252], logistic [0.146, 0.260], boosted [0.217, 0.278]; count advantage—stump [−0.003-0.003, +0.050+0.050], depth-2 [+0.013+0.013, +0.063+0.063], logistic [+0.013+0.013, +0.071+0.071], boosted [+0.018+0.018, +0.079+0.079]; mass advantage—stump [−0.032-0.032, +0.038+0.038], depth-2 [+0.008+0.008, +0.080+0.080], logistic [−0.041-0.041, +0.039+0.039], boosted [−0.002-0.002, +0.088+0.088]; replicate AUROC [0.689, 0.747]; same-label difference boosted [+0.031+0.031, +0.117+0.117], depth-2 [+0.052+0.052, +0.153+0.153], logistic [+0.146+0.146, +0.262+0.262]; boosted AUROC [0.607, 0.709]. Stem clustering widens the intervals modestly (placement half-width 0.144 → 0.169, the rest by at most 0.01) and changes no sign conclusion except the boosted mass advantage’s lower bound (+0.003→−0.002+0.003→-0.002), consistent with the nested result. Sensitivity. Leave-one-task-stem-out values are means of within-stem AUROCs (48 stems, 1–35 boundaries per stem; 47 finite values, one stem single-class) and are a different estimator from the pooled AUROC of Table 11, not comparable to it: 0.630 (stump), 0.670 (depth-2), 0.578 (logistic) under the original label and 0.630 / 0.649 / 0.567 under the post hoc label. Restricting to boundaries with |dU@3|≥0.2|dU@3|≥ 0.2 (n=462n=462; |dU@3|≥0.2|dU@3|≥ 0.2 within 10−910^-9, 453 under exact comparison): 0.606 / 0.607 / 0.535 (original) and 0.606 / 0.632 / 0.546 (post hoc). Post hoc label operating points at ≥80%≥ 80\% retained: stump unchanged; depth-2 0.235 [0.108, 0.253] at 0.805 retained; logistic 0.116 [0.066, 0.181] at 0.886 retained. Table 12: Full out-of-fold frontiers, original label: every threshold at which the allowed set changes. Retained = fraction of the 590 boundaries allowed to compact; avoided = fraction of the 344 positive-burden boundaries blocked; mass = fraction of positive burden mass blocked. Policy Allowed Retained Avoided Mass Stump 0 0.000 1.000 1.000 59 0.100 0.904 0.885 104 0.176 0.837 0.818 144 0.244 0.802 0.788 212 0.359 0.698 0.682 258 0.437 0.657 0.639 330 0.559 0.506 0.498 401 0.680 0.363 0.339 471 0.798 0.230 0.206 536 0.908 0.108 0.092 590 1.000 0.000 0.000 Depth-2 0 0.000 1.000 1.000 15 0.025 0.977 0.972 33 0.056 0.948 0.938 34 0.058 0.945 0.935 57 0.097 0.927 0.919 72 0.122 0.904 0.894 90 0.153 0.898 0.888 120 0.203 0.855 0.849 173 0.293 0.773 0.769 214 0.363 0.706 0.688 236 0.400 0.677 0.664 259 0.439 0.654 0.637 327 0.554 0.515 0.505 392 0.664 0.387 0.369 449 0.761 0.285 0.275 497 0.842 0.195 0.200 561 0.951 0.076 0.088 565 0.958 0.064 0.079 571 0.968 0.047 0.062 577 0.978 0.032 0.038 590 1.000 0.000 0.000 Logistic 0 0.000 1.000 1.000 1 0.002 1.000 1.000 2 0.003 0.997 0.998 4 0.007 0.991 0.994 5 0.008 0.991 0.994 7 0.012 0.991 0.994 21 0.036 0.968 0.967 38 0.064 0.939 0.931 57 0.097 0.919 0.916 64 0.108 0.913 0.911 73 0.124 0.910 0.910 168 0.285 0.738 0.704 257 0.436 0.576 0.538 342 0.580 0.433 0.398 429 0.727 0.302 0.259 494 0.837 0.206 0.162 496 0.841 0.201 0.156 497 0.842 0.198 0.154 498 0.844 0.195 0.146 499 0.846 0.195 0.146 517 0.876 0.148 0.116 531 0.900 0.116 0.090 550 0.932 0.076 0.055 571 0.968 0.035 0.028 590 1.000 0.000 0.000 Figure 3: Figure 2 reproduced at full width (same data and legend). Appendix F Extension analyses: replicate yardstick and engineered-summary trigger EXT-1 and EXT-2 were frozen, with their interpretation rules, before any output was computed (App. A); no seed, feature list, hyperparameter, fold scheme or target was changed afterwards; amendments A–C (App. A) were frozen before their own computation; D, E and F are direction-blind post-freeze analyses specified after their motivating result was known and are not preregistered (dated correction in the frozen prereg record). EXT-1, replicate yardstick. Splitting each boundary’s PRE and POST rollouts into disjoint halves (200 random splits), one half’s paired delta predicts the sign of the other’s with AUROC 0.69–0.72 at k=3k=3 depending on which half supplies the label (0.72 [0.69, 0.75] under the frozen extension protocol’s assignment; at later boundaries the smaller half is a single rollout per arm; first 0.74, later 0.68); the half–half correlation is 0.40, the Spearman–Brown step-up gives a nominal full-sample reliability of 0.57 [0.50, 0.62] (the halves are unequal, so this is an extrapolation), and the estimated task-level share of variance is near zero (intraclass correlation 0.01 [−0.05-0.05, 0.08]; task stem 0.04, point only). Per rollout, union burden at k is the number of the first k steps that are blocked or refetch; per-boundary mean(POST) −- mean(PRE) is required to reproduce the released dU@k for all 590 boundaries before anything else runs. For 200 random splits, each boundary’s PRE samples and POST samples are partitioned into two disjoint halves (2/3 at first compactions, 1/2 at later ones) and a half-A and half-B delta are formed. Statistics, averaged over splits with 5th–95th percentiles and a task-clustered bootstrap interval on a fixed 20-split subset: Pearson and Spearman correlation between the halves (all / first / later); the replicate AUROC of the half-B delta as a score for 1[half-A delta>0]1[ half-A delta>0]; sign agreement and Cohen’s κ on the three-class sign; the Spearman–Brown step-up [25, 26], reported with the caveat that it assumes equal parallel halves and ours are unequal; and one-way intraclass correlations of per-rollout burden across boundaries for each arm, and of the per-boundary delta by task and by task stem. Fixed interpretation rule: the history-based held-out AUROC is reported beside the replicate AUROC; overlapping intervals are read as “within the range set by the boundary’s own replicate”, a replicate interval entirely above as “fall short of what a replicate of the same boundary predicts”. The replicate AUROC is a yardstick, not a strict ceiling (a half-sample score against a half-sample label); its own noise floor includes the sampling of three to five rollouts per arm and inference nondeterminism, which persists at temperature zero and across serving configurations [27, 28]. The split is asymmetric: the smaller half (2 of 5 rollouts per arm at first compactions, 1 of 3 at later ones) supplies the label (positive rate 0.431 against 0.583 realized) and the larger half the score; the mirrored assignment gives 0.69 [0.66, 0.72] at k=3k=3 (supplementary, not a preregistered quantity). On the frontier axes (half-B delta as risk score for the half-A label, operating point at ≥80%≥ 80\% retained under the rule of App. E), the yardstick avoids 0.243 [0.198, 0.286] at 0.866 retained at k=3k=3, +0.109+0.109 [+0.082+0.082, +0.134+0.134] above the random rule (its operating point cannot be placed nearer 0.80 because a 2–3-rollout delta takes few distinct values, so comparisons should be made on the advantage over the random rule, not on the raw avoided fraction); at k=1k=1 it avoids 0.355 (+0.223+0.223 [+0.180+0.180, +0.267+0.267]) and at k=5k=5 0.311 (+0.124+0.124 [+0.099+0.099, +0.144+0.144]); at k=3k=3 within first compactions 0.241 (+0.091+0.091 [+0.062+0.062, +0.112+0.112]) and within later ones 0.247 (+0.126+0.126 [+0.083+0.083, +0.183+0.183]). Same-label paired differences (replicate −- trigger, both against the half-A label; 2,000 task draws): boosted +0.073+0.073 [+0.037+0.037, +0.111+0.111] (replicate 0.72 against the half-sample label, trigger 0.64 against the same label); depth-2 +0.101+0.101 [+0.057+0.057, +0.146+0.146]; logistic +0.204+0.204 [+0.155+0.155, +0.253+0.253] (the logistic’s scores discriminate the half-sample label at 0.513 against 0.547 for the realized label). The mixed-label comparison (App. A, amendment B; replicate against the half-sample label, triggers against the realized label: +0.061+0.061 [+0.008+0.008, +0.113+0.113]; +0.098+0.098 [+0.048+0.048, +0.152+0.152]; +0.170+0.170 [+0.108+0.108, +0.230+0.230]) is retained as context only. Results are in Table 13; the interpretation sentence fixed by the rule above is the one in §4. One-way intraclass correlations at k=3k=3: per-rollout union burden across boundaries, PRE arm 0.60 [0.54, 0.66], POST arm 0.49 [0.44, 0.53]; per-boundary delta by task 0.01 [−-0.05, 0.08], by task stem 0.04 (point only: the task-resampling interval is not valid for stem grouping). Table 13: EXT-1 split-half replicate statistics by horizon and subset (all 590 boundaries; 275 first compactions; 315 later). Means over 200 random splits; intervals are task-clustered bootstrap intervals (2,000 draws) of the mean over a fixed subset of 20 splits. Replicate AUROC = half-B delta as a score for 1[half-A delta>0]1[ half-A delta>0]; rfullr_full = Spearman–Brown step-up of the half–half Pearson correlation; sign agreement treats zero as its own class. k Subset n Replicate AUROC [95% CI] Pearson rfullr_full (Spearman–Brown) [95% CI] Sign agr. κ 1 all 590 0.726 [0.691, 0.760] 0.407 0.578 [0.514, 0.631] 0.630 0.331 1 first 275 0.765 [0.725, 0.807] 0.550 0.710 [0.656, 0.763] 0.661 0.393 1 later 315 0.673 [0.608, 0.726] 0.320 0.484 [0.372, 0.568] 0.602 0.263 3 all 590 0.718 [0.686, 0.746] 0.397 0.568 [0.505, 0.624] 0.551 0.282 3 first 275 0.742 [0.699, 0.783] 0.509 0.674 [0.615, 0.737] 0.609 0.296 3 later 315 0.683 [0.634, 0.725] 0.331 0.496 [0.384, 0.580] 0.500 0.237 5 all 590 0.723 [0.688, 0.745] 0.416 0.587 [0.523, 0.638] 0.573 0.303 5 first 275 0.760 [0.711, 0.795] 0.512 0.676 [0.610, 0.735] 0.654 0.346 5 later 315 0.677 [0.623, 0.713] 0.364 0.532 [0.426, 0.605] 0.502 0.238 EXT-2, engineered-summary trigger. Seventeen pre-boundary features fixed in advance (log prefix steps, compaction index, first-compaction flag, distinct exact and function calls, prefix steps minus distinct exact calls (a repeat proxy that is negative when a step issues several calls), distinct apps, documentation-lookup calls and their share, session-call count, the original write label and the post hoc substantive-write label, count of substantive-write functions, recent errors, log last-observation length, any failure in the three-observation tail, presence of the task-completion call), plus task difficulty as a separately reported group (a three-level one-hot, so the H+T cells carry 20 columns). Models with fixed hyperparameters: L2 logistic regression (C=1C=1; lbfgs solver, at most 5,000 iterations) and a gradient-boosted classifier (depth 3, learning rate 0.05, 200 iterations, minimum 20 rows per leaf, L2 regularisation 1.0, no early stopping). Five-fold cross-validation grouped by task stem (48 stems, stricter than the task-held-out folds of §4), 20 repetitions with shuffled group assignment; task-grouped folds as a sensitivity. Primary target 1[dU@3>0]1[dU@3>0]; sensitivity targets at k=1k=1 and k=5k=5 and on the blocked and refetch channels; a regression variant scored by held-out Spearman correlation. Metrics: held-out AUROC (mean and range over repetitions, task-clustered bootstrap interval), average precision, and the operating point at ≥80%≥ 80\% retained under exactly the rule of App. E with the random-rule expectation beside it; out-of-fold permutation importance reported descriptively. “Best” is the model × feature set with the highest mean held-out AUROC on the primary target, named before any sensitivity target is examined; it is one of four eligible cells and its interval is not selection-adjusted. At the operating point the largest advantage is the logistic model on the summaries plus difficulty (+0.073+0.073 [+0.045+0.045, +0.100+0.100]), not the cell with the highest AUROC; the two post hoc substantive-write comparators sit below the random-rule expectation (Table 16). Results for the primary target under stem-grouped folds are in Tables 14–16. Two estimators are reported: the AUROC of the repetition-averaged out-of-fold scores, whose task-clustered bootstrap interval (2,000 draws) is the protocol’s and which the body uses everywhere (best cell 0.66 [0.61, 0.70]; with task difficulty 0.65 [0.61, 0.70]; frozen feature pairs 0.48–0.51 [0.43, 0.56]); and the mean over the 20 repetitions of the per-repetition pooled AUROC (best cell 0.65, range 0.62–0.66; frozen pairs 0.54–0.56, supplementary interval [0.50, 0.60]). For the seventeen-feature models the two agree within 0.012; for the two-feature models (10–24 distinct score values) the per-repetition pooled value differs from the repetition-averaged value, and since the per-repetition scores are not released the difference cannot be decomposed further; the frozen G4 numbers of §4, from a single task-held-out split, are unaffected. For the best model (boosted classifier, seventeen history features), on the repetition-averaged-score estimator, the sensitivity targets give held-out AUROC 0.61 at k=1k=1, 0.61 at k=5k=5, 0.56 on the blocked channel and 0.65 on the refetch channel; task-grouped folds give 0.66; the regression variant reaches held-out Spearman correlation 0.30 [0.21, 0.38] with dU@3. Out-of-fold permutation importance for the boosted model with task difficulty, reported descriptively as mean AUROC drop: number of distinct prior exact calls 0.045, compaction index 0.039, post hoc substantive-write label 0.018, number of distinct prior functions 0.017, log last-observation length 0.012. Two implementation notes: H17 (task-completion call present in the exact set) is constant zero over the 590 boundaries and drops out; and for one boundary the POST sample carries a different observation tail, so the first error-free PRE rollout’s tail was used. Table 14: EXT-2 held-out discrimination, primary target 1[dU@3>0]1[dU@3>0], five-fold cross-validation grouped by task stem, 20 repetitions. First AUROC column: AUROC of the repetition-averaged out-of-fold scores with its task-clustered bootstrap 95% CI (the protocol’s interval); second: mean over repetitions of the per-repetition pooled AUROC (min–max); AP = average precision (mean over repetitions). H = the seventeen history features; T = task difficulty; the last four rows are the two frozen feature pairs of §4 under these folds (“errors” = recent errors). Model Features AUROC of rep.-avg. scores [95% CI] Per-rep. mean (min–max) AP Boosted H (17) 0.657 [0.608, 0.703] 0.645 (0.623–0.659) 0.699 L2 logistic H (17) 0.629 [0.575, 0.680] 0.624 (0.606–0.641) 0.698 Boosted H+T (20) 0.654 [0.605, 0.700] 0.642 (0.621–0.656) 0.698 L2 logistic H+T (20) 0.626 [0.574, 0.676] 0.621 (0.599–0.638) 0.701 L2 logistic has-written + errors 0.482 [0.429, 0.532] 0.544 (0.526–0.565) 0.617 Boosted has-written + errors 0.483 [0.429, 0.535] 0.543 (0.524–0.563) 0.616 L2 logistic post hoc subst.-write + errors 0.512 [0.458, 0.561] 0.563 (0.536–0.584) 0.615 Boosted post hoc subst.-write + errors 0.502 [0.449, 0.554] 0.558 (0.542–0.581) 0.613 Table 15: EXT-2 operating point at ≥80%≥ 80\% retained under the rule of App. E, same models and folds as Table 14; the random-rule expectation is 1−1-retained. Model Features Ret. Avoided [95% CI] Mass [95% CI] Boosted H (17) 0.800 0.247 [0.217, 0.277] 0.238 [0.201, 0.284] L2 logistic H (17) 0.800 0.262 [0.233, 0.286] 0.248 [0.211, 0.286] Boosted H+T (20) 0.800 0.250 [0.219, 0.273] 0.249 [0.208, 0.287] L2 logistic H+T (20) 0.800 0.273 [0.243, 0.299] 0.266 [0.227, 0.303] L2 logistic has-written + errors 0.829 0.209 [0.180, 0.255] 0.164 [0.134, 0.214] Boosted has-written + errors 0.805 0.227 [0.183, 0.256] 0.179 [0.134, 0.213] L2 logistic post hoc subst.-write + errors 0.802 0.177 [0.104, 0.205] 0.166 [0.093, 0.202] Boosted post hoc subst.-write + errors 0.814 0.142 [0.111, 0.183] 0.130 [0.091, 0.179] Table 16: EXT-2 paired advantage over the random-rule expectation at the operating points of Table 15, by count and by burden mass (task-clustered, 2,000 draws). Model Features Δ count [95% CI] Δ mass [95% CI] Boosted H (17) ++0.047 [++0.019, ++0.079] ++0.038 [++0.003, ++0.086] L2 logistic H (17) ++0.062 [++0.035, ++0.088] ++0.048 [++0.012, ++0.088] Boosted H+T (20) ++0.050 [++0.021, ++0.075] ++0.049 [++0.010, ++0.088] L2 logistic H+T (20) ++0.073 [++0.045, ++0.100] ++0.066 [++0.029, ++0.105] L2 logistic has-written + errors ++0.038 [−-0.002, ++0.065] −-0.007 [−-0.051, ++0.026] Boosted has-written + errors ++0.032 [−-0.004, ++0.067] −-0.016 [−-0.053, ++0.028] L2 logistic post hoc subst.-write + errors −-0.021 [−-0.052, ++0.010] −-0.032 [−-0.069, ++0.008] Boosted post hoc subst.-write + errors −-0.044 [−-0.072, −-0.011] −-0.057 [−-0.095, −-0.015] F.1 Richer representations (144–8,336 features; EXT-3) Adding one indicator per prior function and app to the summaries (144 features) gives held-out AUROC 0.66 [0.61, 0.71] and +0.067+0.067 [+0.040+0.040, +0.092+0.092] over the random-rule expectation at 80% retained; hashed observation text or exact calls did not improve held-out AUROC point estimates (0.61–0.65); the 144-feature model’s burden-mass advantage stays clear of zero under replay resampling (+0.083+0.083, nested [+0.026+0.026, +0.123+0.123]; task-only [+0.040+0.040, +0.125+0.125]), and the same-label gap to the replicate is +0.06+0.06 [+0.02+0.02, +0.10+0.10]. Table 17: EXT-3 held-out discrimination, primary target 1[dU@3>0]1[dU@3>0], stem-grouped five-fold cross-validation, 20 repetitions; same learners, folds, repetitions, target, metrics and bootstrap draws as EXT-2. AUROC: repetition-averaged-score estimator [95% CI] and per-repetition mean (min–max); AP = average precision. Learner Repr. Features AUROC [95% CI] Per-rep. mean (min–max) AP L2 logistic R1 144 0.654 [0.606, 0.698] 0.644 (0.622–0.661) 0.712 L2 logistic R2 4,240 0.606 [0.555, 0.655] 0.594 (0.569–0.621) 0.672 L2 logistic R3 8,336 0.612 [0.561, 0.660] 0.601 (0.569–0.628) 0.680 Boosted R1 144 0.663 [0.612, 0.708] 0.651 (0.635–0.667) 0.709 Boosted R2 4,240 0.642 [0.592, 0.686] 0.628 (0.607–0.648) 0.688 Boosted R3 8,336 0.651 [0.603, 0.695] 0.636 (0.621–0.655) 0.695 Table 18: EXT-3 operating point at 0.800 retained, same cells as Table 17; Δ = paired advantage over the random-rule expectation, by count and by burden mass (intervals paired with EXT-2’s). Learner Repr. Features Avoided [95% CI] Δ count [95% CI] Δ mass [95% CI] L2 logistic R1 144 0.273 [0.243, 0.296] +0.073+0.073 [+0.045+0.045, +0.098+0.098] +0.105+0.105 [+0.053+0.053, +0.144+0.144] L2 logistic R2 4,240 0.253 [0.221, 0.276] +0.053+0.053 [+0.023+0.023, +0.077+0.077] +0.083+0.083 [+0.035+0.035, +0.123+0.123] L2 logistic R3 8,336 0.253 [0.225, 0.279] +0.053+0.053 [+0.026+0.026, +0.080+0.080] +0.081+0.081 [+0.042+0.042, +0.124+0.124] Boosted R1 144 0.267 [0.238, 0.290] +0.067+0.067 [+0.040+0.040, +0.092+0.092] +0.083+0.083 [+0.040+0.040, +0.125+0.125] Boosted R2 4,240 0.262 [0.231, 0.284] +0.062+0.062 [+0.033+0.033, +0.085+0.085] +0.096+0.096 [+0.050+0.050, +0.131+0.131] Boosted R3 8,336 0.262 [0.234, 0.288] +0.062+0.062 [+0.036+0.036, +0.089+0.089] +0.093+0.093 [+0.054+0.054, +0.135+0.135] R1 = the seventeen summaries plus one binary indicator per function observed in the cohort (119) and per app (8); R2 = R1 plus a hashed binary bag of word tokens from the three tail observations (4,096 buckets); R3 = R2 plus a hashed bag of exact call signatures (4,096 buckets); same learners, folds, repetitions, target, metrics and bootstrap draws as EXT-2 (intervals paired with EXT-2’s). Best cell by the fixed rule: boosted trees on R1; it is one of six cells and its interval is not selection-adjusted. Its sensitivity targets (repetition-averaged AUROC): k=1k=1 0.59 [0.55, 0.64]; k=5k=5 0.62 [0.58, 0.66]; blocked 0.56 [0.51, 0.61]; refetch 0.66 [0.62, 0.70]; task-grouped folds 0.66 [0.62, 0.71]. Permutation importance (AUROC drop): distinct prior calls 0.048, distinct prior functions 0.022, post hoc substantive-write label 0.016, compaction index 0.013, last-observation length 0.013; the largest identity feature is the spotify-app indicator at 0.007; 101 of 144 features never change the AUROC when permuted. Robustness of the best cell: the nested task × replay bootstrap gives avoided [0.245, 0.304], Δ count [+0.048+0.048, +0.105+0.105] and Δ mass [+0.026+0.026, +0.123+0.123] (task-only [0.238, 0.290], [+0.040+0.040, +0.092+0.092], [+0.040+0.040, +0.125+0.125]); unlike the summary-only cell, its mass advantage stays clear of zero under replay resampling—17 trigger cells examined on burden mass; task-only intervals exclude zero upward in 11 (and downward in one, the boosted post hoc substantive-write model); the nested design was run for 7 (five under amendment A, all seven under amendment F) and excludes zero in 3 (the two 144-feature models and the logistic on summaries + difficulty). Same-label replicate −- trigger difference +0.061+0.061 [+0.025+0.025, +0.097+0.097] (stem-clustered [+0.021+0.021, +0.104+0.104]); stem-clustered AUROC [0.615, 0.714], avoided [0.240, 0.289], Δ count [+0.042+0.042, +0.090+0.090], Δ mass [+0.042+0.042, +0.125+0.125]. By AUROC the boosted cell is best; the logistic on the same 144 features has the larger operating-point advantage (+0.073+0.073 count, +0.105+0.105 mass). Appendix G LLM-use disclosure The outcomes in this paper are the source corpus’s recorded blocked and refetch flags under its environment’s error contract. We made no model call that judged an outcome or generated a rollout; the subject model is the corpus’s own. Two instruments were LLMs: the encoded semantic-map check of App. B and the two map-blind endpoint labelers of App. B.1 (hosted-API calls costing $1.15); the statistical pipeline itself made no model calls. Beyond those instruments, models assisted with the analysis code and with drafting the manuscript, directed and reviewed by the authors, who are responsible for every claim, number and measurement. Reported statistics were checked against the frozen analysis artifacts, and cited works against their sources.