Paper deep dive
How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language Models
Andres Algaba, Francesca Carlon, Lynn Delcon, Marthe Ballon, Bert Verbruggen, Vincent Ginis
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models often show users a final response and a short reasoning summary while the full reasoning trace stays hidden. We introduce an observability ladder that holds each completed run fixed and varies only what a reader inspects to judge whether the answer is correct: the response, a self-summary the model writes from the trace, the trace itself, and internal signals, each with and without the prompt. Across three benchmarks and five open-weight Qwen3 and gpt-oss models, we train matched linear correctness predictors on each access level. Without the prompt, summaries carry most of the trace's ranking signal (mean AUROC 0.774 versus 0.813) and add +0.156 over the response alone. With the prompt visible, the summary's gain collapses to +0.019, while the trace still adds +0.041. Even at equal length, the trace's last words predict correctness as well as summaries, or slightly better, and carry denser and more discriminative uncertainty and self-correction cues. On MMLU-Pro questions with both correct and incorrect runs, linear summary readers are near chance and trace readers retain only modest signal, both with and without the prompt (prompt-withheld AUROC 0.503-0.545 versus 0.544-0.590). With the prompt withheld, a GPT-5-mini reader recovers substantially more signal from both summaries and traces on gpt-oss-20b, and even then the trace keeps a small +0.034 advantage. Much of the linear readers' trace signal is associated with length. In the common case where users already hold the prompt, summaries are less helpful than the full trace for monitoring correctness. Monitorability is thus a joint property of the display and the reader, so any monitorability claim, including for faithfulness, should specify both.
Tags
Links
- Source: https://arxiv.org/abs/2608.02089v1
- Canonical: https://arxiv.org/abs/2608.02089v1
Trouble viewing inline? Open PDF directly →
Full Text
226,232 characters extracted from source content.
Expand or collapse full text
HOW MUCH DOES A REASONING SUMMARY REVEAL? AN OBSERVABILITY LADDER FOR LARGE LANGUAGE MODELS Andres Algaba 1,2,* 0000-0002-0532-3066 Francesca Carlon 1,2 0009-0004-2152-2745 Lynn Delcon 1,2 0009-0005-0815-4674 Marthe Ballon 1,2 0009-0000-4586-234X Bert Verbruggen 1,2 0000-0001-9776-2420 Vincent Ginis 1,2,3 0000-0003-0063-9608 1 Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium 2 imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussel, Belgium 3 School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA August 4, 2026 ABSTRACT Large language models often show users a final response and a short reasoning summary while the full reasoning trace stays hidden. We introduce an observability ladder that holds each completed run fixed and varies only what a reader inspects to judge whether the answer is correct: the response, a self-summary the model writes from the trace, the trace itself, and internal signals, each with and without the prompt. Across three benchmarks and five open-weight Qwen3 and gpt-oss models, we train matched linear correctness predictors on each access level. Without the prompt, summaries carry most of the trace’s ranking signal (mean AUROC0.774versus0.813) and add+0.156over the response alone. With the prompt visible, the summary’s gain collapses to+0.019, while the trace still adds+0.041. Even at equal length, the trace’s last words predict correctness as well as summaries, or slightly better, and carry denser and more discriminative uncertainty and self-correction cues. On MMLU-Pro questions with both correct and incorrect runs, linear summary readers are near chance and trace readers retain only modest signal, both with and without the prompt (prompt-withheld AUROC0.503–0.545versus0.544–0.590). With the prompt withheld, a GPT-5-mini reader recovers substantially more signal from both summaries and traces on gpt-oss-20b, and even then the trace keeps a small+0.034advantage. Much of the linear readers’ trace signal is associated with length. In the common case where users already hold the prompt, summaries are less helpful than the full trace for monitoring correctness. Monitorability is thus a joint property of the display and the reader, so any monitorability claim, including for faithfulness, should specify both. Keywords large language models· reasoning traces· chain-of-thought· monitorability· observability ladder 1 Introduction Reasoning traces can expose errors or misbehavior absent from a model’s final response [1,2,3]. Yet interfaces differ in what they reveal. In commercial deployments such as those from OpenAI, Google, and Anthropic, a user may receive only the final response or a short provider summary [4,5,6,7], while the full trace, answer probabilities, and hidden states require deeper access [8,9,10,11,12,13]. Evidence from a full trace therefore does not establish what a user can learn from a response or summary. What any display reveals also depends on its reader, so our claims concern the tested monitoring methods rather than the text alone. *Corresponding author: andres.algaba@vub.be 1 arXiv:2608.02089v1 [cs.LG] 3 Aug 2026 An Observability Ladder for Large Language Models Task q Target model completed run C reasoning trace R final response y correctness label a One completed run completed run C same target model, summary instruction S S= Summary(C): produced after the run from C alone, under an instruction requesting at most 90 words b A self-summary written after the run Prompt + response q R Add summary q R S Add trace q R SC Add internal features q R SC Z public prompt-visible levelsprovider-only features Controls, not additional ladder levels Subset control: all R, S, C subsets Prompt-withheld contrast Run control: repeat items c What each reader can see q C replay context target model (provider-side) Z prob Z hidden answer probabilities and hidden states, retrieved from a final-token replay d Internal features from a provider-side replay Figure 1: The observability ladder holds one completed run fixed and changes only the evidence used to predict whether its answer is correct. a, The run contains promptq, full reasoning traceC, final responseR, and correctness labely; the label is only the prediction target. b, Afterward, the same model receives the trace alone and writes self-summarySunder an instruction requesting at most 90 words. c, Public readers first see the prompt and response, then the summary, then the trace. Prompt-withheld and repeated-run comparisons are controls, not extra access levels. d, A separate analysis with direct model access adds answer probabilities and hidden states, denoted Z. We ask three plain questions. First, how well does a short summary predict correctness compared with the full trace? Second, how much does the summary add when the reader does or does not see the prompt? Third, can either text distinguish correct from incorrect runs of the same question? To answer them, after each run the target model writes a summary from its trace alone under an instruction requesting at most 90 words. This controlled summary is our proxy for deployed provider summaries, which may differ in context, instructions, models, or filters. We compare matched linear correctness readers on the response, summary, full trace, and a small set of signals requiring direct model access. We evaluate five open-weight models, Qwen3-4B, Qwen3-8B, Qwen3-14B, gpt-oss-20b, and gpt-oss-120b [14,15, 16,17], on GPQA-Diamond, MMLU-Pro, and Omni-MATH-2-Filtered [18,19,20]. The first two benchmarks are multiple choice. Omni-MATH-2-Filtered contains olympiad mathematics problems that require a written answer; GPT-5-mini grades them following the benchmark’s procedure. Open weights let us examine every access level on the same completed runs. Under the tested linear readers with the prompt withheld, summary-only AUROC is0.774, compared with0.813for the full trace, and adding the trace after the response and summary gives+0.035AUROC. At the same word count, extracts from the trace’s end match or slightly outperform the self-summaries and carry denser and more discriminative uncertainty and self-correction cues. Showing the prompt reduces the summary gain from +0.156to+0.019. A prompt-only reader already reaches AUROC0.734. In repeated-run comparisons on questions with both correct and incorrect runs, linear summary readers are near chance and linear trace readers retain only modest signal, whether the prompt is visible or withheld. For gpt-oss-20b, a GPT-5-mini reader recovers substantially more signal from both summaries and traces when the prompt is withheld, but the trace still performs slightly better than the summary. Ranking runs across a mixed set of questions and judging one particular run are therefore different tasks. Prior work uses intermediate reasoning to improve or check answers [21,22,23,24,25], although written reasoning need not faithfully describe the computation that produced the answer [26,27,28,29]. Other studies ask whether traces reveal unsafe intent, cheating, obfuscation, or failures hidden from final responses [1,2,3,30,31,8,9]. Recent results also show that conclusions can change with the reader [32,20], that structural patterns in a trace can help compare runs of the same question [33], and that shortening traces can remove useful clues [34]. Studies of reasoning summaries test whether they preserve information that another model can use [35], including a model less capable than the one that produced the trace [36]. Researchers have also used answer probabilities and hidden states to predict correctness or truthfulness [37, 38, 11, 10, 39, 40, 41, 42, 43, 12, 44, 13, 45, 46, 47, 48]. I An Observability Ladder for Large Language Models Our observability ladder changes only what a reader can inspect while keeping the completed run and correctness label fixed. It compares the final response, self-summary, full trace, and additional internal signals available only with direct model access, both with and without the prompt. Under the readers studied here, summaries help most when the prompt is withheld. In the common case where users already hold the prompt, the summary is less helpful than the full trace for monitoring correctness. For properties such as reward hacking, the decisive evidence can sit in the trace alone [2,1], and judging faithfulness requires more than the visible text itself [26,27]. Monitorability is thus a joint property of the display and the reader, and any monitorability claim, including for faithfulness, should state both. 2 The observability ladder For each benchmark itemiand target modelm, the model produces one completed reasoning run (Figure 1a). From that run we record the task promptq i , visible final responseR i,m , full reasoning traceC i,m , post-hoc self-summary S i,m , provider-side replay features Z i,m , and binary correctness label y i,m = 1[final answer is correct]. Correctness is the monitored property because the labels are defined externally to the monitor and held fixed across the displays being compared. For multiple-choice tasks,R i,m is the final response text, which is usually but not always a bare option letter. The extracted option is stored separately for grading. For open-response mathematics,R i,m is the visible final response and the extracted boxed expression, final expression, or concise answer is stored separately. The same target model generatesS i,m fromC i,m alone after the run, under an instruction requesting at most 90 words (Figure 1b). The grading procedure, the verbatim summary instruction, and an example summary are in the Methods. The provider-side replay feature setZ i,m contains answer-probability features and the last-layer hidden state at the final token, together with a small set of summaries of those quantities (Figure 1d). This prompt-conditioned replay captures only a narrow subset of what model internals can reveal [43, 12, 44, 45]. A display is the text made available to the monitor (Figure 1c). In the main prompt-visible setting, the three public access levels are O q 1 (i,m) = q i ⊕ R i,m , O q 2 (i,m) = q i ⊕ R i,m ⊕ S i,m , O q 3 (i,m) = q i ⊕ R i,m ⊕ S i,m ⊕ C i,m . The final, non-public level adds provider-side replay features: O q 4 (i,m) = q i ⊕ R i,m ⊕ S i,m ⊕ C i,m ⊕ Z i,m . The matched prompt-withheld contrast does not explicitly show q i : O 1 (i,m) = R i,m , O 2 (i,m) = R i,m ⊕ S i,m , O 3 (i,m) = R i,m ⊕ S i,m ⊕ C i,m . At the public levels,⊕joins the visible text. At the provider-side level, it appends numerical features extracted from a prompt-conditioned final-token replay. These replay features require direct model access and are not part of a public display. Appendix Table A1 summarizes the components and access levels. BecauseSis derived fromC, an ordered increment depends on whether the summary or trace is added first. We therefore fit public monitors to all seven subsets R, S, C, R + S, R + C, S + C, R + S + C. For each component, Shapley attribution averages its change in AUROC when added to every subset that does not already contain it. We use the Shapley values only to describe how predictive performance beyond chance is allocated among R, S, and C [49]; they are not an information decomposition. The prompt-visible ladder represents the most realistic setting when the monitor has the prompt. The prompt-withheld ladder remains important as an output-only contrast, especially when only responses and summaries are available. It does not includeqin the display, butSandCcan still paraphrase the prompt or expose item-difficulty cues. Additional interpretive controls remove long spans copied from the prompt, mask stated answers together with option text and prompt-derived numbers, or compare independent runs of the same MMLU-Pro item. The within-item metric in the repeated-run comparison tests how a monitor trained across items transfers to run-level discrimination after item-level cues such as topic, format, answer structure, and difficulty are removed. The prompt-conditioned and repeated-run controls are complementary but not equivalent. Addingqtests whether a display contributes beyond an explicitly visible prompt in across-item evaluation. Holdingqfixed removes between-item variation, but does not reproduce a jointly prompt-conditioned monitor because the encoder can represent prompt–response and prompt–trace interactions. We therefore run the prompt-visible repeated-run controls with the primary surface-plus-Cohere monitors under two objectives, transferred across-item fitting (Appendix Table A42) and direct item-conditional fitting (Appendix Table A48). We repeat both with the open surface-plus-TF-IDF monitors as encoder sensitivities (Appendix Tables A45 and A49). 3 An Observability Ladder for Large Language Models The experiment crosses five models with three benchmarks, giving 15 model–benchmark settings (Appendix Tables A2 and A4). Within a setting, every access level uses the same held-out items and splits, so differences between levels are paired on a fixed set of completed runs (Appendix Table A3). For a public display, the monitor computes surface features, including word and sentence counts, digit and mathematical-symbol density, and rates of uncertainty, self-correction, and answer-framing markers [50]. These features are concatenated with a 512-dimensional Cohere embed-v4.0 text embedding after train-only preprocessing [51,52]. The provider-side level appends replay-derived answer-probability and hidden-state features from the target model [10,11,12,13]. Anℓ 2 -regularized logistic probe maps the resulting vector to a correctness score (Appendix Table A8). AUROC is the probability that the monitor assigns a higher score to a randomly chosen correct run than to a randomly chosen incorrect run. A value of0.5is chance. The model–benchmark setting is the unit of inference, and the reported estimates average paired contrasts across the 15 settings. The full feature specification is in Appendix Table A6, and procedural details are in the Methods. Our hypothesis concerns the mean paired increments; individual settings may be flat or negative. Although the access sets are conceptually nested, each primary display is embedded and fitted separately. Finite-sample estimation and validation can therefore give a higher-access reader a lower AUROC. The ladder orders the evidence available to a reader but does not require fitted performance to increase at every step. For the main prompt-visible ladder, we report q∆ S = AUROC(q + R + S)− AUROC(q + R), q∆ C = AUROC(q + R + S + C)− AUROC(q + R + S), q∆ Z = AUROC(q + R + S + C + Z)− AUROC(q + R + S + C), withq∆ public = AUROC(q + R + S + C)− AUROC(q + R)andq∆ full = AUROC(q + R + S + C + Z)− AUROC(q + R). For the prompt-withheld contrast, we also report ∆ S = AUROC(R + S)− AUROC(R), ∆ C = AUROC(R + S + C)− AUROC(R + S), ∆ Z = AUROC(R + S + C + Z)− AUROC(R + S + C), and the public and full gains∆ public = AUROC(R + S + C)− AUROC(R)and∆ full = AUROC(R + S + C + Z)− AUROC(R) . The central summary-versus-trace comparison is the trace increment after the summary is visible. A small value means that the short-summary monitor performs close to the full-trace monitor in this ordered comparison. It does not quantify how much trace information the summary contains. Because Z is computed with both q and C in context, it sits outside the public ladders even when appended to prompt-withheld displays. Figure 1 summarizes these comparisons. 3 The summary adds little once the prompt is visible Figure 2a reports mean prompt-visible trajectories for each benchmark and overall, together with the matched prompt- withheld mean, on the same held-out items and splits, and Figure 2b reports the paired increments within the same runs. Prompt-visible setting values are in Appendix Table A35; the prompt-withheld setting values and increments are in Appendix Tables A9 and A10, and the pooled comparison is in Appendix Table A12. The prompt alone is already predictive, reaching mean AUROC0.734across the 15 settings (Figure 2a). Because each target model is evaluated separately, this score mainly reflects which questions tend to be easier or harder for that model. With the prompt visible, adding the summary to the response gives a small mean gain,q∆ S = +0.019AUROC (Figure 2b; 95% descriptive setting-bootstrap interval[+0.011, +0.027]over the 15 observed settings, Appendix Table A11); the mixed-model sensitivity interval for this increment includes zero (Appendix Table A16). Adding the full trace after the prompt, response, and summary givesq∆ C = +0.041(descriptive interval[+0.028, +0.052]). The total public gain fromq +Rtoq +R+S +Cis thereforeq∆ public = +0.060(descriptive interval[+0.040, +0.077]), and the replay features addq∆ Z = +0.029(descriptive interval[+0.023, +0.036]), forq∆ full = +0.089(descriptive interval [+0.066, +0.110]). TheZincrement measures the gain from this specific replay feature set (Appendix Figure A8 and Appendix Table A53). These gains improve across-item correctness ranking over the held-out mixture, but they are much smaller than the prompt-withheld ladder. The prompt-withheld contrast explains why response-only baselines can overstate the apparent value of summaries on multiple-choice tasks (Figure 2b, open points). Withoutq, the response is often a bare option letter and has mean AUROC0.623, while the prompt-only monitor reaches0.734. Moving fromRtoR +S +C +Zgains∆ full = +0.217 AUROC (95% descriptive setting-bootstrap interval[+0.177, +0.257]over the observed settings, Appendix Table A11), positive in every setting. The largest step is∆ S = +0.156. Prompt visibility shrinks that same summary increment to+0.019, a paired difference of−0.137(95% descriptive setting-bootstrap interval[−0.177,−0.095]; Appendix 4 An Observability Ladder for Large Language Models PromptPrompt + response Add summary Add trace Add internal features 0.4 0.5 0.6 0.7 0.8 0.9 Test AUROC a Prompt-visible trajectories GPQA-Diamond MMLU-Pro Omni-MATH-2-Filtered Prompt-visible mean Prompt-withheld mean Summary after response Trace after summary Trace after response Summary after trace −0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Paired AUROC gain filled: prompt shown open: prompt hidden b Effect of summary and trace access Qwen3-4BQwen3-8BQwen3-14Bgpt-oss-20bgpt-oss-120b Figure 2: The summary adds little once the prompt is visible. Results cover five models and three benchmarks. a, Correctness ranking as readers receive the prompt, response, summary, trace, and then internal features. Thin colored lines show the 15 settings and thick colored lines the benchmark means, all with the prompt visible. The black line is the prompt-visible mean and the dashed gray line the matched prompt-withheld mean. Shaded bands give 95% descriptive setting-bootstrap intervals for the black prompt-visible and dashed prompt-withheld mean lines. b, Effects of summary and trace access. Filled points show readers that see the prompt and open points show readers that do not. Black ticks show setting means with 95% descriptive setting-bootstrap intervals. Adding the summary after the response gains +0.019with the prompt versus+0.156without it. Adding the summary after the trace changes AUROC by about zero in both cases. Exact values and intervals are in Appendix Tables A11 and A12. Tables A11 and A12). By contrast, the trace increment is similar in both cases,∆ C = +0.035prompt-withheld andq∆ C = +0.041prompt-visible. The order controls in Figure 2b and Appendix Table A12 give the direct, non- cumulative contrasts. Adding the trace directly toq + Rgains+0.062, and adding the summary after the trace changes AUROC by−0.002. The directq+R+Cversusq+R+Scontrast is+0.043, positive in all 15 settings (95% descriptive setting-bootstrap interval[+0.033, +0.052]; Appendix Table A11). Thus, most of the large output-only summary increment overlaps with predictive information already available to a prompt-visible monitor; traces retain a small residual increment after prompt and summary are visible. The repeated-run comparison in Section 5 tests how well a reader distinguishes runs of the same question, which is a different quantity. Because the summary is generated from the trace, ordered increments do not uniquely attribute the public signal toSor C. The output-only all-subset analysis in Section 4 and the following prompt-conditioned factorials address that order dependence. The complete primary surface-plus-Cohere factorial gives mean full-display AUROC0.809. Withqas the baseline, its prompt-conditioned Shapley allocation is+0.005forR,+0.013forS, and+0.057forC(Appendix Table A37). A secondary open surface-plus-TF-IDF factorial reproduces this pattern, with+0.000for addingSafter q + R + Cand a Shapley allocation of+0.033forSversus+0.078forC(Appendix Tables A38 to A40). These paths concern set inclusion, not physical presentation order. The prompt-visible gains differ by benchmark (Figure 2a,b). Meanq∆ full is+0.036on GPQA-Diamond,+0.132 on MMLU-Pro, and+0.100on Omni-MATH-2-Filtered. The corresponding public gains before adding the replay features are+0.014,+0.094, and+0.071. GPQA-Diamond has the smallest prompt-visible gain and the widest setting-level scatter, consistent with its smaller sample (n = 198; Appendix Table A4). For comparison, the matched prompt-withheld full gains are much larger on GPQA-Diamond and MMLU-Pro (+0.230and+0.295), and closer on Omni-MATH-2-Filtered (+0.127), where the open-response final answer already carries substantial correctness- predictive information. Across setting means, the prompt-visible full gain ranges from+0.004to+0.143(Figure 2a), while the prompt-withheld full gain ranges from+0.075to+0.332and remains positive even where one intermediate increment is flat or negative (Appendix Figure A4). Mixed-model sensitivity estimates remain positive, with intervals excluding zero for every increment exceptq∆ S (Appendix Table A16; specification and fallback in the Methods). With only three benchmarks and two model families, these are sensitivity summaries for the observed design rather than population-level generalization intervals. Mean AUROC is0.812forq + Cand0.809forq + R + S + C(Appendix Table A35), and several setting-level steps are 5 An Observability Ladder for Large Language Models R SC R + S R + C S + C R + S + C 0.4 0.5 0.6 0.7 0.8 0.9 Test AUROC a AUROC by display combination GPQA-DiamondMMLU-ProOmni-MATH-2-Filtered S − R C − S ( R + C ) − ( R + S ) ( R + S + C ) − ( R + C ) −0.05 0.00 0.05 0.10 0.15 0.20 0.25 Paired ΔAUROC b Paired differences between displays Self summary Capped summary Trace start Trace end Last 90 words 0.65 0.70 0.75 0.80 0.85 Test AUROC c Summaries and simple trace extracts (open text reader) Figure 3: Across a mixed set of questions, short summaries carry most of the trace’s ranking signal for the tested linear readers. The prompt is withheld throughout. Black ticks show setting means with 95% descriptive setting-bootstrap intervals over the 15 observed settings. a, Mean AUROC for each available combination of response R, self-summaryS, and traceC. b, Paired differences between displays. c, Model-written summaries and simple trace extracts at matched word budgets under a reader based on surface and word-pattern features. Exact values are in Appendix Tables A22 and A27; instruction and masking controls are in Appendix Tables A24 and A34. negative (Appendix Table A36). A nested component-block control, which concatenates separately computed feature blocks instead of re-embedding each expanded display, changes AUROC by−0.004to+0.016and preserves every access increment (Appendix Table A13). Across all 15 settings, surface features, Cohere embeddings, and a fully open TF-IDF monitor reproduce the public access pattern, with full-public AUROC between0.777and0.815across the non-primary feature families (Appendix Figure A5 and Appendix Table A30). After train-only linear residualization of every non-length surface and Cohere feature on word and sentence counts, across-item AUROC at the full-public display remains0.717prompt-withheld and0.713prompt-visible (from0.814and0.809), while the corresponding public-ladder increments shrink to+0.101 and−0.014(Appendix Table A31). This analysis tests sensitivity to linear associations with length; causal attribution is outside its scope. For the prompt-withheld contrast, the mean full-ladder gain remains between+0.178and+0.262 under leave-one-family-out, leave-one-benchmark-out, and item-weighted aggregation (Appendix Tables A14 and A15). Item-clustered intervals give the same qualitative result (Appendix Table A17), and label permutation returns every level to chance (Appendix Table A18). Along the output-only ladder, higher-access monitors rank better but are less well calibrated before validation-fitted Platt scaling (Appendix Table A20). The appendix also reports secondary metrics (Appendix Table A19), answer and parser baselines (Appendix Table A21), selective risk (Appendix Table A57), failure strata (Appendix Table A60), replay quality (Appendix Table A55), and feature-selection checks (Appendix Tables A63 and A64). 4 Short summaries carry most of the trace’s ranking signal for linear readers In Figure 3, we compare summary- and trace-based displays under output-only access across a held-out mixture of questions, where monitors can use both item-level cues and evidence specific to the completed run. With the prompt withheld, the trace-only monitor reaches mean AUROC0.813across the 15 settings, while the summary-only monitor reaches0.774and the full public displayR + S + Creaches0.814(Figure 3a and Appendix Table A22). We define the normalized excess-AUROC ratio asr AUROC = (AUROC(S)− 0.5)/(AUROC(C)− 0.5). Using the unrounded setting means,r AUROC = 0.874(95% descriptive paired setting-bootstrap interval[0.855, 0.896]over the 15 observed settings; Appendix Table A14). This normalized performance comparison is not an information-retention measure because AUROC is nonlinear and non-additive. Adding the response to the trace changes little, with AUROC0.812for R +C. Paired contrasts place summary-containing displays0.033–0.039AUROC below their trace counterparts, while adding the summary toR +Cchanges AUROC by only+0.002(Figure 3b). The secondary Shapley allocation assigns mean contributions of+0.154toC,+0.118toS, and+0.042toR(Appendix Table A23). BecauseSandCoverlap, these values allocate shared predictive performance rather than identify independent information sources. The AUROC differences in Figure 3b also affect which runs a reader would retain. At 50% coverage, retaining the half of test runs ranked most likely correct, the mean error rate is0.243withR,0.139withR + S,0.114with 6 An Observability Ladder for Large Language Models R + S + C, and0.098after addingZ(Appendix Table A57). Benchmark format explains part of the variation. GPQA-Diamond and MMLU-Pro responses are usually bare option letters, whereas the open-response Omni-MATH-2- Filtered baseline already reaches AUROC0.755(Appendix Table A19). Response-only discrimination varies sharply by benchmark, model, and response format; neither selected-answer identity nor response length alone explains it (Appendix Table A61). In Figure 3c, we compare model-written summaries with deterministic trace extracts at the same word count. Under the primary surface-plus-Cohere family, hard-capping the summary at 90 words or replacing it with a conclusion-region trace extract changes mean AUROC little. Summary-only AUROC is0.774with the original summary,0.775with the 90-word hard cap (Cap90),0.780with the final-90-words extract (Last90), and0.782with masked Last90; the correspondingR+Svalues remain between0.774and0.780, compared with0.779originally (Appendix Table A26). In the open surface-plus-TF-IDF monitor, self-summary AUROC is0.782and becomes0.778after an exact 90-word cap. Trace extracts matched to each capped summary’s word count differ by position: first-word extracts are weaker (0.740), head–tail and random extracts are near the capped summary (0.781and0.777), and last-word extracts are stronger, at0.804versus0.778for the capped summary. The paired difference on the unrounded setting means is+0.025 (95% descriptive setting-bootstrap interval[+0.013, +0.038]over the 15 observed settings; Appendix Table A27). Under the same open surface-plus-TF-IDF specification, the fixed Last90 extract is similar (0.803), and masking stated answers, option text, and prompt-derived numbers before extraction leaves it at0.803. Within this open control, conclusion-region-biased extracts are especially predictive. When the requested extract is longer than the trace, it can include the whole trace. In Appendix Table A65, we compare the primary surface features of each self-summary with its same-run Last90 extract. Weighting the 15 settings equally, over the 80,763 retained pairs, Last90 contains higher rates of all three lexical marker families than the self-summary. The Last90-minus-summary differences are+0.009for uncertainty, +0.012for self-correction, and+0.051for answer markers, and all three 95% descriptive setting-bootstrap intervals exclude zero. The corresponding oriented single-feature AUROC differences are+0.090([+0.064, +0.118]),+0.068 ([+0.048, +0.089]), and+0.072([+0.049, +0.095]). Density results are mixed. Last90 has higher digit density but weaker digit discrimination (+0.018in level and−0.022in AUROC), whereas its math-character density is lower but more discriminative (−0.010and+0.044). The two density measures move in opposite directions, so the data do not indicate that summarization consistently removes numerical or symbolic content. A generic summary instruction changes mean AUROC for summary-containing displays by−0.011to+0.001(Ap- pendix Table A24). Summary decoding also has little effect. In a balanced 48-item-per-setting sensitivity with the monitor held fixed, the sampled-decoding SD of the summary increment is0.017and the greedy value sits+0.010 above the sampled mean (Appendix Figure A7 and Appendix Table A25; design in the Methods). Its absolute increment is not directly comparable with the full-data+0.156, but within this design the output-only gain is not attributable to one sampled summary draw. To test whether whole-display features miss localized trace evidence, we divide each trace into up to eight balanced contiguous chunks and pool chunk-level features or scores (design in the Methods). The best pooled versions match the whole-display trace within0.001AUROC in both feature families (Appendix Tables A28 and A29). These controls test whether the results depend on how the trace is divided and read; they do not locate the evidence causally, and a short trace can occupy a single chunk. A masking control replaces answer spans, choice letters, option text, and prompt-derived numbers before recomputing all features. The combined mask changes mean trace AUROC by+0.001, and no per-benchmark contrast moves by more than0.012(Appendix Figure A6a and Appendix Table A34). A companion control removes 8–14-word spans copied from the question and changes mean AUROC by between−0.002 and+0.001across the six summary- and trace-containing displays (Appendix Figure A6b and Appendix Table A33). These controls restrict near-verbatim reconstruction but cannot remove paraphrase or a higher-level indication that a question is difficult. 5 For repeated runs of the same question, linear summary readers are near chance The repeated-run control tests whether monitors separate correct from incorrect runs of the same item (Figure 4). We generate two additional MMLU-Pro runs per item and model, fit monitors on the primary run, and score all three runs. Both reported AUROCs use the same 939–1,389 outcome-discordant items per model that appeared in at least one held-out split. Across-item AUROC pools runs across selected questions, whereas within-item AUROC compares a correct and incorrect run of the same question and model, holding item-level cues fixed (Figure 4a and Appendix Table A41). The analysis is limited to MMLU-Pro because GPQA-Diamond has too few multiply sampled items and Omni-MATH-2-Filtered has one run per item. 7 An Observability Ladder for Large Language Models Summary Trace Response + summary Response + trace Prompt + response + summary Prompt + response + trace 0.46 0.48 0.50 0.52 0.54 0.56 0.58 0.60 0.62 AUROC a Across selected questions vs within the same question Across selected questions Within question Summary vs response Trace vs response Add summary to response Add trace to response Trace vs summary (prompt visible) −0.04 −0.02 0.00 0.02 0.04 0.06 0.08 Change in within-question AUROC b Summary and trace comparisons Summary minus response Trace minus summary Add summary to response Trace versus summary after response 0.00 0.05 0.10 0.15 0.20 Change in within-question AUROC c Comparing two readers Primary linear GPT-5-mini Primary Word count Words + sentences Length- adjusted Primary pairs within 20% Primary pairs within 10% 0.450 0.475 0.500 0.525 0.550 0.575 0.600 0.625 0.650 Trace-only within-question AUROC d Trace-only length sensitivity Qwen3-4BQwen3-8BQwen3-14Bgpt-oss-20bgpt-oss-120b Figure 4: For repeated runs of the same question, linear summary readers are near chance, while reader choice and trace length matter. All panels use the 939–1,389 outcome-discordant MMLU-Pro items per model that appeared in at least one held-out split. In panels a, b, and d, marker shapes identify the five target models, and black ticks show their means. Per-model intervals for the primary within-question display estimates are in Appendix Table A41. a, Filled points rank runs across the selected questions; open points compare correct and incorrect runs of the same question. Six summary- and trace-bearing displays are shown. b, Five direct within-question comparisons contrast summary and trace access; values above zero favor the display named first. c, For gpt-oss-20b, four selected comparisons from the ten-display analysis show how the primary linear reader and GPT-5-mini use summaries and traces differently. Black vertical lines give the paired 95% item-bootstrap intervals for the GPT-5-mini comparisons. d, Trace-only within-question AUROC is shown for the primary reader, two readers using only text length, a length-adjusted reader, and pairs whose trace lengths differ by at most 20% or 10%. Exact values are in Appendix Tables A41 to A43 and A52. Under the primary linear reader, the summary displays are near chance within items, with within-item AUROC 0.503–0.545and across-item AUROC0.512–0.536on the same discordant items (Figure 4a). The trace-containing displays retain only modest signal, with within-item AUROC0.544–0.590and across-item AUROC0.544–0.566. The response-only monitor is closer to chance, at0.493–0.550, with intervals covering0.5in two settings. When both measures use the same discordant items, the gap between across-item and within-item AUROC is small and changes direction across settings. We therefore base the conclusion on the absolute within-item estimates and the matched pairwise comparison. The prompt-conditioned result in Section 3 measures a different quantity: across-item performance when the prompt is visible. Visible text length accounts for much of the trace reader’s repeated-run score (Figure 4d and Appendix Table A43). Averaged equally across models, trace word count alone reaches within-question AUROC0.553, compared with0.567 for the primary trace reader; using word and sentence counts gives0.545. After adjusting the other features for their 8 An Observability Ladder for Large Language Models fitted linear association with both counts, trace AUROC is0.538, compared with0.508for the similarly adjusted summary reader. When correct–incorrect pairs are restricted to trace lengths within 20% or 10%, the primary trace reader reaches0.533or0.532, remaining+0.032or+0.037above the summary reader. These restrictions retain only 3,412 of 10,608 pairs (32%) and 1,701 pairs (16%), so they are selective sensitivities rather than estimates for every discordant item. Across models, 939–1,389 of the 1,364–2,062 outcome-discordant items (65.4–69.6%) appear in at least one held-out split and enter the analysis (Appendix Table A50). These selected questions are systematically harder than held-out questions whose three outcomes agree, with primary-run accuracy 23.3–29.9 percentage points lower and median traces 2.50–2.96 times longer (Appendix Table A51). The repeated-run estimate therefore applies to this unusually difficult, unstable subset rather than to all MMLU-Pro questions. In both strata, every prompt-withheld trace display remains aboveSandR+S, at0.604–0.611versus0.548–0.551with exactly one incorrect run and0.512–0.518versus 0.493–0.494with exactly two, and the prompt-visible trace-bearing displays show the same ordering. Most of the pooled trace discrimination is nonetheless concentrated in the one-incorrect stratum. The trace-only reader reaches 0.611 [0.596, 0.626]when exactly one of three runs is incorrect but only0.518 [0.502, 0.535]when exactly two are, although the two strata contribute comparable numbers of items (n=2,829versusn=2,475). When two of three runs are incorrect, even the trace-only reader is only slightly above chance, and theR+S+Cinterval covers0.5(Appendix Table A50). Model-balanced reweighting changes any pooled estimate by only 0.001 at the reported precision. Conditional-logit coefficients for standardized monitor scores are positive for every trace-containing display and model (β = 0.359–0.815, with all 95% intervals above zero). Summary-only and response-plus-summary coefficients are smaller (β = 0.064–0.343), with one interval crossing zero for each display (Appendix Table A41). Across-item AUROC on the discordant subset ranks runs across selected questions, whereas within-item performance tests a particular run after fixing the question. The transferred across-item monitor is not an upper bound on run-specific information because it was trained across items. We therefore train pairwise logistic monitors on correct-minus- incorrect feature differences within discordant training items and evaluate them on held-out discordant items (Appendix Table A47). In this direct control, the summary displays (0.512–0.542) remain close to the response-only range (0.511–0.541), while the trace-containing displays reach0.534–0.580. In this MMLU-Pro comparison, summary displays remain weak at run-level diagnosis, while traces retain modest, model-dependent signal. We test only a pairwise logistic reader using surface features and Cohere embeddings, so richer item-conditional readers may perform differently. Replacing the self-summary with the Last90 extract raises across-item AUROC for all five models and raises within-item AUROC by a mean of+0.022under the primary reader (Appendix Table A44), while the open version’s within-item changes are mixed (Appendix Table A46). Both Last90 controls use the whole trace when it contains fewer than 90 words and therefore do not isolate an ending-only effect. The primary prompt-visible monitor jointly encodesqin every display. On the same discordant items, summary displays remain near chance while trace displays retain modest within-item signal (Figure 4a and Appendix Table A42). Within item, addingStoq + Rchanges AUROC by mean+0.026, addingCby mean+0.060, and addingSafterq + R + C by mean+0.003(Figure 4b). The secondary open surface-plus-TF-IDF monitor is directionally similar with a similarly small summary increment (Appendix Table A45). The prompt is held fixed within a pair but remains part of the jointly fitted representation, so these display differences can include prompt–response or prompt–trace interactions and do not decompose the signal by source. A direct primary item-conditional version, trained on correct-minus-incorrect run differences within discordant training items, gives the same asymmetry. AddingCtoq + Ris positive in every model (+0.025to+0.060) and addingSafterq + R + Cranges from−0.020to+0.011(Appendix Table A48), and the secondary open version agrees (Appendix Table A49). A GPT-5-mini sensitivity with a fixed judge instruction evaluates all ten displays on the same 1,389 analyzed outcome- discordant gpt-oss-20b questions (Figure 4c). With the prompt withheld, within-item AUROC is0.503for the response,0.649for the summary,0.683for the trace,0.656for response plus summary, and0.690for response plus trace (Appendix Table A52). Using the summary instead of the response changes AUROC by+0.146(95% CI [+0.118, +0.172]), and adding it to the response changes AUROC by+0.153([+0.125, +0.180]). The corresponding descriptive changes for the primary linear reader are−0.004and−0.014. The trace still performs slightly better than the summary under both readers: the GPT-5-mini differences are+0.034([+0.008, +0.059]) without the response and+0.034([+0.008, +0.057]) with it, compared with+0.040and+0.044for the linear reader. When GPT-5-mini also sees the prompt and response, its within-item AUROC is0.800. Adding the summary changes it by−0.024 ([−0.040,−0.007]), adding the trace by−0.064([−0.084,−0.044]), and adding the summary after the trace by−0.015 ([−0.032, +0.002]). Because GPT-5-mini can solve the visible task and compare its answer with the response, these comparisons mix monitoring with task solving and should not be read as evidence that the added text contains less information. The near-chance summary result therefore applies to the tested linear readers rather than to every possible reader. The prompt-visible result in Section 3 provides the complementary across-item view. 9 An Observability Ladder for Large Language Models 6 Discussion Summary usefulness depends on both prompt visibility and the reader. With the prompt visible, the summary adds only +0.019AUROC, compared with+0.156when the prompt is withheld, while the trace still adds+0.041. On repeated runs of the same question with the prompt withheld, linear summary readers are near chance, whereas GPT-5-mini extracts substantially more signal. In the common case where users already hold the prompt, summaries are less helpful than the full trace for monitoring correctness. Monitorability claims, including claims about faithfulness, should therefore name both the displayed evidence and the reader. Prompt visibility matters because the prompt already supplies much of the information in the summary. A prompt-only reader reaches AUROC0.734, partly because the question indicates how difficult the run is likely to be. The remaining useful evidence concerns the particular run, and more of it appears in the trace than in the summary. Reading the trace after the summary still helps, while reading the summary after the trace adds essentially nothing. Qualitative inspection points to self-corrections, contradictions, and incomplete computations as the kind of run-specific evidence involved, although these are not separated from length or correlated features. Without the prompt, when only responses and summaries are retained, short summaries remain genuinely useful. They carry most of the trace’s ranking signal across mixed questions and help rank runs by their likelihood of being correct. The trace’s own ending does the same job at equal length and carries denser and more discriminative uncertainty and self-correction cues than the summary written from it, although no single feature family explains the readers’ scores. Judging repeated runs of the same question is harder, because question-level cues cancel and only run-specific evidence helps. There, linear summary readers are near chance and trace readers keep only a modest edge, with or without the prompt. GPT-5-mini recovers substantial signal from summaries that the linear readers miss, yet the trace stays slightly ahead. When that same judge also sees the prompt, its scores mainly reflect its own ability to solve the task rather than what the displays reveal. This result shows why a monitorability estimate must name the reader. Text length carries much of the signal the linear readers use. Adjusting the other features for word and sentence counts removes most of the across-question gains and turns the trace-minus-summary contrast slightly negative, from +0.043to−0.024, while the within-item trace edge survives length-matched comparisons. The adjustment removes meaningful cues along with length, so the original and adjusted results should be read together, and neither identifies a length-independent semantic effect. Limitations. There are four main limitations. First, our summaries are a controlled proxy. The same model summarizes its own trace under one instruction, without the prompt, content filters, or any incentive to evade a monitor. The estimates therefore describe this summary method rather than deployed provider summaries [4,5] or adversarial summaries. The results are stable under the tested instruction and decoding changes. A direct next test is to apply the paired comparison to commercial models that expose summaries. Second, the claims are reader-relative. Most estimates use simple linear readers, the stronger-reader test covers one judge on one model and benchmark, and monitors fitted in one setting transfer imperfectly to others (Appendix Figure A9 and Appendix Table A56). We therefore state every result relative to the reader that produced it. Third, the ladder measures final-answer correctness only. Nothing here shows that a trace or summary is a causal account of the model’s computation [26,29], and extending the design to faithfulness or misbehavior requires labels defined independently of the reader [30,8]. Fourth, length is entangled with the linear readers’ signal, and our adjustments show its influence but do not disentangle it. Future work should also compare several summary lengths and readers, and report the monitored property, access, prompt visibility, and reader explicitly. 10 An Observability Ladder for Large Language Models Acknowledgements This research was supported by funding from the Vrije Universiteit Brussel Research Council (VUB-OZR) and the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” program. Andres Algaba acknowledges support from the Francqui Foundation (Belgium) through a Francqui Start-Up Grant and a fellowship from the Research Foundation Flanders (FWO) under Grant No.1286924N. Vincent Ginis acknowledges support from Research Foundation Flanders under Grant No.G032822N and G0K9322N. The computational resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government - department WEWIS. Author contributions A.A., B.V., and V.G. conceptualized the idea. A.A. implemented the code and performed the analyses. F.C., L.D., and M.B. verified the experiments and collaborated on the writing. All authors read and approved the manuscript. Data and code availability The code for this publication is publicly available athttps://github.com/AndresAlgaba/observability_ ladder. Data associated with this study are available in a public repository athttps://doi.org/10.5281/zenodo. 21770626. The GPQA-Diamond dataset [18] is available at https://huggingface.co/datasets/Idavidrein/gpqa. The MMLU-Pro dataset [19] is available at https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro. The Omni-MATH-2-Filtered dataset [20] is available athttps://huggingface.co/datasets/martheballon/ Omni-MATH-2. The Qwen3 model family [15] is available at https://huggingface.co/collections/Qwen/qwen3. The gpt-oss model family [17] is available at https://huggingface.co/collections/openai/gpt-oss. We used Python 3.13.1 with NumPy 2.4.3, pandas 3.0.0, pyarrow 23.0.0, scikit-learn 1.7.2, scipy 1.17.0, statsmodels 0.14.6, matplotlib 3.10.8, vllm 0.12.0, flashinfer-python 0.5.3, PyTorch 2.11.0, transformers 5.8.0, and cohere 5.20.0. 11 An Observability Ladder for Large Language Models References [1]Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander M ̨adry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, and Vlad Mikulik. Chain of thought monitorability: A new and fragile opportunity for AI safety, 2025. URL https://arxiv.org/abs/2507.11473. [2] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926, 2025. [3]Melody Y Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y Wei, Marcus Williams, Benjamin Arnav, Joost Huizinga, Ian Kivlichan, Mia Glaese, et al. Monitoring monitorability. In Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026. [4]OpenAI. Learning to reason with LLMs.https://openai.com/index/learning-to-reason-with-llms/, sep 2024. OpenAI research post. Accessed 2026-06-24. [5]OpenAI. Reasoning models.https://developers.openai.com/api/docs/guides/reasoning, 2026. OpenAI API documentation. Accessed 2026-07-23. [6] Google. Gemini thinking.https://ai.google.dev/gemini-api/docs/thinking, 2026. Gemini API documentation. Accessed 2026-07-30. [7]Anthropic.Extended thinking.https://docs.anthropic.com/en/docs/build-with-claude/ extended-thinking, 2026. Anthropic API documentation. Accessed 2026-07-30. [8]Bowen Baker, Joost Huizinga, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Detecting misbehavior in frontier reasoning models.https://openai.com/index/chain-of-thought-monitoring/, mar 2025. OpenAI research post. Accessed 2026-06-02. [9]OpenAI.Evaluatingchain-of-thoughtmonitorability.https://openai.com/index/ evaluating-chain-of-thought-monitorability/, dec 2025.OpenAI research post. Accessed 2026-06-02. [10]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321–1330. PMLR, 2017. [11]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. [12]Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations, 2023. [13]Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.68. [14]Qwen Team. Qwen3 model collection.https://huggingface.co/collections/Qwen/qwen3, 2025. Model repository collection. Accessed 2026-05-12. [15] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [16]OpenAI. Introducing gpt-oss.https://openai.com/index/introducing-gpt-oss, 2025. Model release documentation. Accessed 2026-05-12. [17] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. 12 An Observability Ladder for Large Language Models [18]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024. [19] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understand- ing benchmark. In Advances in Neural Information Processing Systems, volume 37, 2024. [20] Marthe Ballon, Andres Algaba, Brecht Verbeken, and Vincent Ginis. Benchmarks saturate when the model gets smarter than the judge. arXiv preprint arXiv:2601.19532, 2026. [21] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837, 2022. [22]Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213, 2022. [23] Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [24]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. [25]Marthe Ballon, Andres Algaba, and Vincent Ginis. The relationship between reasoning and performance in large language models–o3 (mini) thinks harder, not longer. Scientific Reports, 16, 2026. doi: 10.1038/ s41598-026-50923-2. [26] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. [27]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, 2023. [28]Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 305–329. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.ijcnlp-main.20. [29]Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, et al. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410, 2025. [30]Han Wang, Yifan Sun, Brian Ko, Mann Talati, Jiawen Gong, Zimeng Li, Naicheng Yu, Xucheng Yu, Wei Shen, Vedant Jolly, and Huan Zhang. MonitorBench: A comprehensive benchmark for chain-of-thought monitorability in large language models. In Third Conference on Language Modeling, 2026. doi: 10.48550/arXiv.2603.28590. URL https://arxiv.org/abs/2603.28590. [31]Sydney Von Arx, Lawrence Chan, and Beth Barnes. Recent frontier models are reward hacking.https: //metr.org/blog/2025-06-05-recent-reward-hacking/, jun 2025. METR research post. Accessed 2026- 06-02. [32] Scott Emmons, Roland S. Zimmermann, David K. Elson, and Rohin Shah. A pragmatic way to measure chain-of-thought monitorability, 2025. URL https://arxiv.org/abs/2510.23966. [33]Daniel Lee, Owen Queen, and James Zou. ReasonOps: Operator segmentation for LLM reasoning traces, 2026. URL https://arxiv.org/abs/2605.29192. [34] Bryce Little. Length penalties make chain-of-thought less monitorable, 2026. URLhttps://arxiv.org/abs/ 2607.09786. [35] Tingwei Zhang, John X. Morris, and Vitaly Shmatikov. How to steal reasoning without reasoning traces, 2026. URL https://arxiv.org/abs/2603.07267. 13 An Observability Ladder for Large Language Models [36]Dani Roytburg, Shreya Sridhar, and Daphne Ippolito. Measuring weak-to-strong legibility of reasoning models, 2026. URL https://arxiv.org/abs/2603.20508. [37] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, pages 4878–4887, 2017. [38] Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684–5696. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.503. [39] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977, 2021. doi: 10.1162/tacl_a_00407. [40] Neeraj Varshney and Chitta Baral. Post-abstention: Towards reliably re-attempting the abstained instances in QA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 967–982. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.55. [41] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. URL https://arxiv.org/abs/1610.01644. [42]Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126–2136, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1198. URL https://aclanthology.org/P18-1198/. [43]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1): 207–219, 2022. doi: 10.1162/coli_a_00422. [44] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2024. [45]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023. [46]Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, and Jack Merullo. Reasoning theater: Disentangling model beliefs from chain-of-thought, 2026. URLhttps: //arxiv.org/abs/2603.05488. [47]Parsa Mirtaheri and Mikhail Belkin. Catching rationalization in the act: Detecting motivated reasoning before and after CoT via activation probing. In Third Conference on Language Modeling, 2026. doi: 10.48550/arXiv.2603. 17199. URL https://arxiv.org/abs/2603.17199. [48]Amirhosein Ghasemabadi and Di Niu. Can LLMs predict their own failures? self-awareness via internal circuits, 2025. URL https://arxiv.org/abs/2512.20578. [49]Lloyd S. Shapley. A value forn-person games. In Harold W. Kuhn and Albert W. Tucker, editors, Contributions to the Theory of Games, Volume I, volume 28 of Annals of Mathematics Studies, pages 307–317. Princeton University Press, Princeton, NJ, 1953. [50]Arne Vanhoyweghen, Brecht Verbeken, Andres Algaba, and Vincent Ginis. Lexical hints of accuracy in LLM reasoning chains. arXiv preprint arXiv:2508.15842, 2025. [51] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URLhttps: //aclanthology.org/D19-1410/. [52] Cohere. Cohere’s embed models: details and application.https://docs.cohere.com/docs/cohere-embed, 2026. Documentation. Accessed 2026-05-12. [53]Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain, 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-2068/. 14 An Observability Ladder for Large Language Models [54]Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. Efficient few-shot learning without prompts, 2022. URL https://arxiv.org/abs/2209.11055. [55]Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanu- jan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi.Ma- tryoshka representation learning.In Advances in Neural Information Processing Systems, volume 35, pages 30233–30249, 2022. URLhttps://proceedings.neurips.c/paper_files/paper/2022/hash/ c32319f4868da7613d78af9993100e42-Abstract-Conference.html. [56]Amazon Web Services. Cohere Embed v4 model parameters for Amazon Bedrock.https://docs.aws.amazon. com/bedrock/latest/userguide/model-parameters-embed-v4.html, 2026. Documentation. Accessed 2026-05-12. [57]John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1275. URL https://aclanthology.org/D19-1275/. 15 An Observability Ladder for Large Language Models A Methods Data recorded for each run. For each benchmark itemiand target modelm, we analyze the task promptq i , completed reasoning traceC i,m , final responseR i,m , post-hoc self-summaryS i,m , provider-side replay featuresZ i,m , extracted answer, correctness label, benchmark identifier, and quality-check information. All results reported here use a single completed run per item, except the multi-run within-item control on MMLU-Pro, which adds two further independent runs per item. For multiple-choice tasks,R i,m is the final response text visible after the reasoning trace, and the selected option letter is treated as the extracted answer. For open-response mathematics,R i,m is the final response text, and the extracted answer span prefers a boxed answer, final expression, or concise answer. The target is whether the final answer is correct. This binary run-level label is defined independently of the monitor, and we record which label definition was used. We treat each model–benchmark setting as the unit of inference. Models and benchmarks. We evaluate five open-weight target models from two families, Qwen3-4B, Qwen3-8B, and Qwen3-14B from the Qwen3 family, and gpt-oss-20b and gpt-oss-120b from the gpt-oss family [14,15,16,17]. All target models are run in settings that permit local extraction of final hidden states and token log-probabilities. We there- fore do not evaluate closed API models. Loading gpt-oss models uses the model-specific Transformers implementation, bfloat16 replay and summarization, and the Hugging Face kernel components required for MXFP4 inference. The model weights are loaded from local snapshots of the Hugging Face repositories used for the run. The release documentation records the model sources, hashes of the weights and generation settings, and numeric precision. All generation, summarization, and replay used approximately 750 GPU-hours on NVIDIA H200 GPUs. GPQA-Diamond contains 198 four-way graduate-level science questions [18]. MMLU-Pro contains 12,032 multiple-choice questions with up to ten answer choices [19]. Omni-MATH-2-Filtered contains 4,181 open-response olympiad mathematics problems graded for response equivalence by GPT-5-mini following the benchmark procedure [20]. Correctness is determined by option match for GPQA-Diamond and MMLU-Pro. For Omni-MATH-2-Filtered, correctness is determined by GPT-5-mini equivalence judgments between the model response and reference answer following Ballon et al. [20]. The judge is shown the problem, the model response, and the reference answer and decides whether the response is mathematically equivalent to the reference. These judgments cover all 4,181 items for each reported model and run. We identify each judgment by its benchmark, model, run, and item. We use the stored GPT-5-mini judgments for the exact responses analyzed here as the Omni-MATH-2-Filtered labels. Against the deterministic exact/numeric checker, the judge agrees on 14,146/20,310 (69.7%) resolved comparisons. Agreement is 7,859/7,865 (99.9%) on deterministic exact/numeric matches, while the judge accepts 6,158/12,445 (49.5%) deterministic mismatches as semantically equivalent. All 20,905 judge calls return a binary decision: 14,206 equivalent and 6,699 not equivalent, with no uncertain or error cases (Appendix Table A59). The release documentation includes per-model counts and details about each batch. Separating reasoning traces and final responses. For each item, target model, and benchmark, we begin with one completed reasoning run and separate the raw completion into a reasoning trace, final response text, ex- tracted answer, and parser status, which records whether this separation succeeded. Target reasoning runs are generated with vLLM from benchmark system and user messages rendered through the target tokenizer chat template, withadd_generation_prompt=Trueandenable_thinking=True. The vLLMSamplingParamsset max_tokens=28000for Qwen models andmax_tokens=126000for gpt-oss models, while leaving the other sampling settings at the vLLM defaults: temperature 1.0, top-p1.0, top-k0, and no random seed set. We supply no explicit stop sequence during trace generation; generation ends when the model emits its end-of-sequence token or reaches the stated token limit, using the model’s chat template. All reported analyses use the stored completions released with the paper. Some Qwen3 completions emitted an empty thinking block,<think></think>, followed by content in the final response channel. The parser assigns these runsC =∅, and they remain in the analysis as observed model outputs. In the primary run, empty traces occur for Qwen3-8B on GPQA-Diamond (3/198), MMLU-Pro (79/12,032), and Omni-MATH-2-Filtered (980/4,181), and for Qwen3-14B on MMLU-Pro (5/12,032) and Omni-MATH-2-Filtered (223/4,181); they do not occur for Qwen3-4B or the gpt-oss models (Appendix Table A58). WhenCis empty, the self-summary is still generated from that empty trace, so the summary can expose the absence of trace content. Some Qwen3 completions, concentrated on Omni-MATH-2-Filtered, also omit the closing</think>tag. For these runs, the parser leaves the final responseRempty and assigns the remaining completion text toC. In the primary Omni-MATH-2-Filtered run, missing closing tags occur for Qwen3-4B (166/4,181), Qwen3-8B (170/4,181), and Qwen3-14B (142/4,181) (Appendix Table A58). We retain this parser information for quality checks, and the runs remain in the main analysis. For MMLU-Pro, we record the available choices explicitly because their number varies. For Omni-MATH-2-Filtered, we mark final answers missed by the parser so extraction failures can be distinguished from incorrect model answers. The mainZ prob andZ hidden features are extracted from a separate prompt-conditioned replay of the completed run rather than from generation-time probability traces. Our quality checks focus on parser and replay status. We store further generation details to document how the traces were produced. 16 An Observability Ladder for Large Language Models Self-summary generation.The primary self-summary is generated under a prompt requesting at most 90 words and is denotedSthroughout. Although the stored file name records a 256-token decoding limit, we define the summary by the 90-word instruction. For each completed run, the same target model that generated the reasoning trace also generates the summary: S i,m = Summary m (C i,m ;γ m ), whereγ m denotes the summary-generation settings. The 90-word target is an instruction rather than a post-hoc length cap. Summaries are decoded with a 256-token limit and are not truncated after generation. Across the primary summaries (n = 82,055), the mean length is 77.0 words, 74.4% of summaries contain at most 90 words, and the maximum length is 231 words (Appendix Table A58). The summary decoder uses Hugging Facegeneratewith do_sample=True,use_cache=False,pad_token_idequal to the EOS token, andmax_new_tokensset to the 256-token budget minus any final-channel prefill tokens. No temperature or top-poverride is supplied in the primary run, so those settings follow the loaded model generation settings, falling back to Hugging Face defaults when unset (temperature 1.0, top-p1.0, top-k50). Self-summary generation produces only the summary text. Qwen summaries use enable_thinking=False, while gpt-oss summaries usereasoning_effort="low"and an assistant final-channel prefill so decoding begins directly in the final response channel rather than in an analysis/reasoning channel. The summarizer receives onlyC i,m , not the task prompt, final response, reference answer, correctness label, hidden states, logits, or token probabilities. Because the summarizer receives onlyC, this setup measures how much of the trace it can compress without seeing the task prompt or final response. Supplying q or R could change the summary, so these results apply only to trace-conditioned summarization. The primary prompt asks the model to compress the reasoning trace into at most 90 words while preserving uncertainty, contradictions, candidate answers, and self-corrections without adding evidence or repairing errors. The prompt set also includes a generic version that drops the preservation clause and requests only a summary of at most 90 words. Uncertainty-only and final-answer-only versions were defined but are not analyzed here. To test sensitivity to the instruction, we regenerate all summaries with the generic version under the same decoding settings and retrain the affected probes (Appendix Table A24). A worked appendix example shows the formatted target prompt, shortened completed response, and generated self-summary for one GPQA-Diamond run (Appendix Figures A1 to A3). For reproducibility, we record each summary’s target model, prompt version, hash of the generation settings, sampling settings and overrides, seed identifier, input length, output length, and requested summary length. For the primary summaries analyzed here, the seed identifier was recorded but the random-number generator was not fixed. To measure summary-decoding sensitivity, we select 48 primary-run items per model–benchmark setting: eight for each combination of correctness and short, medium, or long traces, with the three trace-length groups defined by terciles, using a recorded random seed for item selection. For every selected trace, the target model generates three independently sampled summaries and one greedy summary. For each ofR,S, andR + S, a primary surface-plus-Cohere monitor selects regularization on the first split’s training and validation items after removing the selected items, is refitted on all remaining items, and is then held fixed across the four summary decodings. We give the six groups equal weight in the primary AUROC for this balanced subset and use inverse-selection-probability weighting as a sensitivity check. Hierarchical intervals use 2,000 draws that resample items within the six groups and resample the 15 observed settings (Appendix Figure A7 and Appendix Table A25). Constructing the text shown to readers. We construct every visible display from the same completed run. Each display has a fixed identifier and set of visible components and is classified as either output-only or prompt-conditioned. With the prompt visible, we evaluate every nonempty combination ofR,S,C:q + R,q + S,q + C,q + R + S, q +R+C,q +S +C, andq +R+S +C. Both the primary surface-plus-Cohere and secondary open surface-plus-TF-IDF estimates cover all combinations. The main ladder follows the ordered pathq +R→ q +R +S → q +R +S +C, with qandq +Cas controls. Without the prompt, we evaluate the seven nonempty subsets ofR,S,C, of whichR,R +S, andR +S +Cform the prompt-withheld contrast. Displays containing several components label each section as “Task prompt:”, “Response:”, “Summary:”, or “Reasoning trace:” and separate them with blank lines. Single-component displays contain only the component text. The physical section order is fixed as prompt, response, summary, and trace. Accordingly, after in a factorial contrast and inclusion order in a Shapley calculation denote nested component sets rather than experimental permutations of text position. Prompt-overlap-stripped (noqcopy) versions are produced for the summary and trace of output-only displays. All wordn-grams of length 8–14 from the task prompt are matched case-insensitively in the visible text, matched spans are removed longest-first, overlapping removals are merged, and each removed region is replaced by a fixed “[prompt-overlap removed]” marker before feature extraction. Display masking. Masked display versions replace task- and answer-revealing content with fixed markers before feature extraction. Answer masking replaces the contents of expressions and literal occurrences of the extracted answer (with a context-restricted rule for single-character answers) by an “[ANSWER]” marker, and masks choice letters in answer-stating contexts, including declarations such as “the answer is X”, verb-adjacent letters, letter- 17 An Observability Ladder for Large Language Models to-option mapping restatements, and standalone letter lines. Option masking replaces case-insensitive occurrences of option texts (taken from the benchmark options where available and otherwise parsed from option blocks in the prompt) by an “[OPTION]” marker. Number masking replaces numeric literals of at least two characters that occur in the task prompt by a “[NUMBER]” marker. Component versions apply one rule at a time to the trace display, and the combined maskall version applies all rules to the trace, summary, and full public display, masking every visible component. Spot-checks of sampled masked traces find no residual option-text or answer-context matches on MMLU-Pro, residual answer-stating phrasings in roughly1%of traces on GPQA-Diamond, and in roughly18%on Omni-MATH-2-Filtered, where short numeric answers escape the context patterns. Masking removes verbatim and near-verbatim statements but cannot remove paraphrase. The masked-display results therefore show what remains after restricting direct reconstruction rather than eliminating it. We recompute the surface features and embeddings from the masked text and train the probes in the same way on the same splits. Public-text features.For each output-only displayP, public monitors may compute features of the visible textPbut may not use the task prompt, target-model hidden states, logits, or token probabilities. Prompt-conditioned control displays relax only the task-prompt exclusion. The primary surface feature family includes word count, sentence count, average word length, digit density, math-character density, and per-word rates of uncertainty, self-correction, and answer-marker terms (Appendix Table A7). A no-length version removes word and sentence counts while retaining average word length, densities, and marker rates. Raw marker counts, character counts, line counts, log-count transforms, and lexical-diversity features are retained only as appendix diagnostics because they are strongly length- or formatting-dependent in long reasoning traces. The TF-IDF family combines wordn-grams(1, 2)and character n-grams(3, 5), with train-only vocabulary fitting and a maximum of 5,000 word and 5,000 character features [53]. The Cohere family uses 512-dimensional Cohere embed-v4.0 embeddings configured for classification and no silent truncation [51,54,55,52,56], an encoder chosen for its long input context window so that a full reasoning trace fits as a single whole context. The primary combined public-text family concatenates rate-based surface features and Cohere embeddings after train-only preprocessing. Cohere-based features define the primary monitor for the main ladder. Across the same splits and all 15 settings, the open TF-IDF reader reproduces the mean access ordering and full-public performance (Appendix Table A30). It provides a reproducible alternative that does not depend on a proprietary encoder. Fixed embeddings may be computed before splitting, but learned scaling, projection, or feature selection is fit only on the training split. Direct comparison of summary and Last90 features. We match the stored surface features of each original self- summary to those of the fixed Last90 extract from the same benchmark item, model, and run, and use the paper’s primary correctness label for that run. Last90 contains up to the final 90 whitespace-delimited trace words and contains the whole trace when it is shorter. We exclude pairs when either display has zero alphabetic words under the primary feature definition. This removes 1,292 zero-word Last90 displays and no self-summaries, leaving 80,763 pairs. Within each of the 15 model–benchmark settings, we compute the mean value of each feature for summaries and Last90, their paired difference, and rank-based single-feature AUROC without fitting a probe. For the seven features included in the sparse-model check, we orient AUROC using the dominant coefficient direction in Appendix Table A64. We treat larger average word length as positive by convention and retain the raw AUROC in the released results. The reported estimates give each setting equal weight. We form 95% descriptive intervals with 10,000 bootstrap resamples of the 15 observed settings. These univariate statistics describe observable features of the summaries and Last90 extracts. Length controls. We fit two direct baselines that use only the visible display’s word count or its word and sentence counts, with the same splits, regularization search, final refit, and repeated-run scoring as the primary reader. For the length-adjusted reader, we fit a linear projection from word and sentence counts to every other numeric surface and Cohere feature using the current training data, replace those features by their residuals, and then drop the two length variables before fitting the logistic model. The projection is therefore learned on the training split during validation search and on train plus validation for the final model; held-out and repeated runs are transformed without refitting. As a separate pair-matching sensitivity, we retain correct–incorrect runs of the same item only when their trace word counts differ by at most 10% or 20% of the longer trace, then recompute primary-reader within-item AUROC and item-bootstrap intervals. These controls distinguish directly usable length information from variation not captured by the fitted linear association; they do not isolate a causal or length-independent effect. Embeddings and long inputs.For each public access level, the primary embedding feature is a single embedding of exactly the visible public text at that level,e(R),e(R∥S), ore(R∥S∥C). The embedded strings are the plain ladder- display strings, including section headers such as “Response” and “Summary” where present, not chat-template text or tokenizer special tokens. Silent truncation is disallowed. If a display exceeds the encoder’s context window, we divide it into fixed chunks and record the number and size of the chunks and how they are combined. In the reported analysis, every full-publicR + S + Ctext fits within the encoder’s context window. Appendix Table A32 reports whole-context 18 An Observability Ladder for Large Language Models fit and chunk counts. Because the primary display embedding replacese(R)bye(R∥S)and then bye(R∥S∥C), higher levels are not feature-nested in the embedding block. We therefore define a nested component-block control using[φ(R)],[φ(R),φ(S)],[φ(R),φ(S),φ(C)]and the analogous prompt-conditioned blocks, retaining lower-level component features unchanged when access expands. Provider-side replay features. The provider-side replay component is Z = Z prob ⊕ Z hidden . For each completed run, we replay the run to its final-response position as x dec i,m = ChatTemplate m (q i )∥C i,m ∥s stop , and extract h dec i,m = H m (x dec i,m ) last . The replay suffixes are s Qwen stop = </think> , s gpt-oss stop = <|end|><|start|>assistant <|channel|>final<|message|>. For GPQA-Diamond,Z prob includeslogp(A),..., logp(D), entropy, margin, and maximum probability. For MMLU- Pro, answer-choice log-probability features use the fixed answer labelsA–J, with unavailable choices masked. Choice- count and missing-choice indicators are included so probes do not interpret padding as probability mass. Choice probabilities use the tokenizer encoding of bare uppercase labelsA–Jwithadd_special_tokens=False. Each label is required to be a single token, and probabilities are normalized over the labels available for that item. Thus, for multiple-choice tasks,Z prob scores answer labels at the next-token decision point after the prompt and completed trace contextq + C. We also use choice-invariant summaries such as selected-choice probability, maximum probability, second-highest probability, entropy, margin, and probability variance. For Omni-MATH-2-Filtered,Z prob scores the model’s own extracted or generated final answer span, not the reference answer. The current open-response probability feature set scores that span withadd_special_tokens=Falseafter the same replay context and includes mean, minimum, quantile, and sum log-probabilities, answer-token and answer-character counts, and first-token entropy, margin, and maximum-probability summaries. The count and sum features are length-sensitive and partly encode information already visible inR.Z hidden contains the raw final hidden state and scalar hidden-state summaries such as norm, mean, standard deviation, minimum, and maximum. On average in the seven-setting ablation subset, the hidden-state gain is larger with the run’s own trace than with the prompt alone and is lost under shuffled-trace replay. Across this subset,q + Creplay differs fromC-only replay by only+0.006AUROC on average, so the prompt adds little beyond trace-conditioned replay in this ablation (Appendix Table A54). Quality checks. Parser status and replay status are tracked separately from the provider-side replay level. We use this information for quality and sensitivity checks rather than as part of the mainZfeature set because parser and replay failures could otherwise look like replay features. In the reported primary settings, every replay fits within the context window and produces a usable hidden-state vector (Appendix Table A55). If a replay is missing or too long in a diagnostic run, probability features are marked as missing, and we retain hidden-state placeholders and replay-status indicators for quality checks. We impute missing values only during probe fitting and using training data. Completion length is visible whenCis visible, so direct length-derived features are treated as public text features or removed in no-length controls. To test whether formatting failures affect AUROC, we repeat the analysis within each model–benchmark setting, access level, and split after applying two restrictions. The first excludes blank traces, and the stricter restriction also requires parser status “ok” and nonempty responses and extracted answers. The five per-split AUROCs are averaged within each setting before the 15 setting means receive equal weight (Appendix Table A62). Because these restrictions discard visible formatting failures, they test how much those failures affect the result. They do not estimate performance for hypothetical error-free outputs. Probe training and splits. We fitℓ 2 -regularized logistic probes for final-answer correctness [41,42,57,43]. All learned preprocessing is train-only, including feature standardization, TF-IDF vocabulary fitting, and embedding-feature scaling. The feature blocks included in each probe, and the quantities excluded by design, are summarized in Appendix Table A8. For each model–benchmark setting, access levelk, feature familyf, and splits, the monitor first converts the 19 An Observability Ladder for Large Language Models visible displayO k (i) into a feature vector. Written out by feature block, the logistic predictor is η ikfs = α kfs + J surf X r=1 β surf rkfs x surf ir | z surface text features + J emb X j=1 β emb jkfs e emb ij |z public-text embedding elements + ⊮Z prob ∈ f J prob X a=1 β prob akfs z prob ia | z answer-probability features +⊮Z hidden ∈ f J hidden X h=1 β hidden hkfs z hidden ih |z hidden-state features ,(1) and Pr(y i = 1|O k (i),f,s) = σ(η ikfs ), whereσ(u) = (1 + exp(−u)) −1 . Herex surf ir is surface featurer,e emb ij is embedding elementj,z prob ia is answer- probability featurea, andz hidden ih is hidden-state featureh. The indicator terms state that answer-probability and hidden-state sums are included only for feature families that contain those provider-side replay blocks. Public-only probes set those terms to zero. Equivalently, the model can be written compactly asPr(y i = 1) = σ(α kfs +β ⊤ kfs z ikfs ), wherez ikfs = T fs (O k (i))is the feature vector produced by the train-fitted preprocessing mapT fs . For a candidate inverse regularization strength C, the fitted coefficients solve ( ˆα kfs,C , ˆ β kfs,C ) = arg min α,β X i∈D train s −y i logp i − (1− y i ) log(1− p i ) + 1 2C ∥β∥ 2 2 , withp i = σ(α +β ⊤ z ikfs ). This discriminative probe estimates held-out monitorability at a fixed access level; we do not interpret its coefficients as causal effects. The inverse regularization strength is selected on the validation split from a prespecified set of values. Primary across-item probes useC ∈ 10 −4 , 10 −3 , 10 −2 , 10 −1 , 1, 10. Implementation usessklearn.linear_model.LogisticRegressionwith anℓ 2 penalty,solver="liblinear",max_iter=2000, class_weight=None, median imputation and standard scaling for numeric features. We use a fixed random state for each split. The selectedCis chosen by validation AUROC, and the final model is refit on train plus validation before scoring the held-out test split. Final metrics are reported only on the held-out test split. We use five random splits with 60/20/20 train/validation/test proportions, stratified by final-answer correctness within each model–benchmark setting. Splits are created separately within each model–benchmark setting and reused across access levels and feature families, making ladder deltas paired. For GPQA-Diamond, each split must contain both correctness classes in train, validation, and test. Sparse models are used for interpretability checks, not primary estimation. We exclude language-model judges from the main public-text analysis because they can solve the task themselves, whereas the primary analysis evaluates a fixed class of linear readers. The fixed-instruction GPT-5-mini analysis below is reported only as a sensitivity to reader class, not as an additional ladder level. For each of the 15 settings, a fixed selection rule retains one result file together with its paired predictions. We use a result only when the same file contains its paired predictions; if several files provide the same access level, we use the primary result file. Separate designated result files provide the subset, prompt-control, stripped-display, and decomposition analyses. Sparse-model interpretation checks.We fit two sparse logistic models to ask which feature sources are repeatedly used by the public and final-level probes. Plain LASSO uses anℓ 1 penalty over standardized individual features and then summarizes selected features by source group. Sparse group LASSO combines a high-level source-group penalty with within-group sparsity, using source groups for surface text, Cohere embeddings, and answer-probability features when available. These checks use the same target labels, access levels, train-only preprocessing, and split assignments as the main probes, but they are not used as the primary AUROC estimates. Raw hidden-state dimensions are excluded from the sparse selection tables for tractability and interpretability. Their contribution is instead estimated by the replay-feature sensitivity analysis. Constant feature columns are dropped before sparse fitting. Repeated runs of the same question. For MMLU-Pro, the target models produce two further independent runs for each of the 12,032 items under the generation defaults of Appendix Table A3; all three runs are extracted and graded exactly as in the primary analysis. After grading, we identify discordant items as those with at least one correct and one incorrect run. The complete three-run grades contain 1,364–2,062 such items per model among all 12,032 questions. For runs 2–3, we then compute and store self-summaries, displays, text features, replay features, and monitor predictions only for this outcome-selected set of items; the original runs and grades remain available for all items. The 20 An Observability Ladder for Large Language Models within-item analysis uses the subset of discordant items held out in at least one split, namely 939–1,389 items per model on MMLU-Pro, or 65.4–69.6% of all outcome-discordant items. Monitors are trained on the primary run’s train and validation splits with the same split assignments, feature preprocessing, and validation-selected regularization as the primary public probes, so the repeated-run scores come from the same monitors as the main results. For the reported repeated-run quantities, each split contributes held-out items with stored predictions for runs 1–3 and discordant outcomes. We pool item–run scores across the five splits by averaging over the splits in which the item was held out. Reported across-item AUROC pools item–run observations only over the same post-outcome discordant items used by the within-item analysis. We retain an all-run value with mixed availability only as an internal check because summaries, features, and predictions for runs 2–3 were computed only for the discordant items. The within-item AUROC is the pairwise concordance between correct and incorrect runs of the same item, with ties counted as one half, computed over discordant items. Uncertainty comes from a 5,000-draw bootstrap that resamples per-item concordant-pair and total-pair counts at the item level, and a conditional logistic regression with item fixed effects on standardized monitor scores provides a model-based robustness check. The primary repeated-run analysis includes the response-only, summary-only, trace-only, response-plus-summary, response-plus-trace, and full-public response-plus-summary-plus-trace displays, together with the prompt-visible setq +R,q +R +S,q +R +C,q +R +S +C. It does not evaluate the provider-side replay features within item. GPQA-Diamond is excluded because too few items have multiple independent runs, and Omni-MATH-2-Filtered because only a single completed run per item is available. A secondary open repeated-run analysis repeats the prompt-visible display set with rate-based surface features and joint word- and character-level TF-IDF. Both primary and open monitors are trained on the primary run’s train and validation partitions, then score runs 1–3 under the same split pooling, post-outcome discordant-item restriction, within-item concordance estimator, item bootstrap, and conditional-logit check described above. Because the prompt is present in every jointly encoded display, it is held fixed by within-item pairing but can interact representationally with the response, summary, and trace. The direct primary item-conditional analysis uses the same stored MMLU-Pro repeated runs and both the output-only display setR,S,R +S,C,R +C,R +S +Cand the prompt-visible setq +R,q +R +S,q +R +C,q +R +S +C. For each model, display, and split, it restricts training, validation, and test partitions to discordant items. Within each discordant item, every correct run is paired with every incorrect run, and the training examples are the feature differences correct-minus-incorrect and incorrect-minus-correct, labeled 1 and 0 respectively. The monitor is an ℓ 2 -regularized logistic classifier with median imputation and standard scaling. Regularization is selected on validation within-item pairwise AUROC fromC ∈ 10 −4 , 10 −3 , 10 −2 , 10 −1 , then the model is refit on train plus validation pairs and evaluated on held-out discordant items. A secondary open prompt-visible version uses the display set q +R,q +R+S,q +R+C,q +R+S +Cand the surface-plus-TF-IDF representation, with the same within-item pair construction and held-out evaluation. It selects inverse regularization strength from10 −4 , 10 −3 , 10 −2 , 10 −1 , 1, 10, matching the candidate values used by the open across-item probes. The reported 2.5th–97.5th percentiles of the five per-split estimates describe split sensitivity. These analyses are CPU-only and use previously computed public features; they do not generate new runs, summaries, embeddings, replay features, or LLM judgments. Comparison with a stronger reader. To test whether the result depends on the reader, we query gpt-5-mini-2025-08-07on the ten previously constructed display types for the 1,389 analyzed outcome-discordant gpt-oss-20b MMLU-Pro questions and three runs per question. The output-only displays areR,S,C,R + S,R + C,R +S +C , and the prompt-visible displays areq +R,q +R +S,q +R +C,q +R +S +C. GPT-5-mini is the same judge model used for Omni-MATH-2-Filtered grading [20]. The three previously scored displays,S,R + S, and q +R +S, are reused without new API calls. We score the seven remaining displays with the same versioned instruction, mmlu_pro_display_correctness_probability_v1, which requests one bare probability that the displayed run is correct. Requests use medium reasoning effort and no sampling-parameter override, and the judge receives neither the gold answer, reference answer, nor correctness label. We store each request and response with its prompt, request settings, batch, returned text, and usage information. If an added request does not return the required bare probability, we retry it with output-token limits of 1,024, 5,120, and 128,000 tokens after the initial 512-token request, leaving every other request setting unchanged. For one of the 29,169 new predictions, the 128,000-token attempt and one exact resubmission both returned the nonnumeric outputJ. We made one final call for this prediction with the same judge, display, reasoning effort, and output-token limit, but changed the formatting instruction to request a one-field JSON object and used a strict schema to require a numeric probability in[0, 1]. We retained its value only after it passed the schema and range checks; no score was imputed. Excluding the affected question, or replacing the recovered probability with either endpoint of the allowed range, changes any point estimate by at most0.00075and preserves every contrast direction. Excluding the question also preserves whether each paired interval crosses zero. After these checks, the analysis contains 41,670 display–item–run predictions: 12,501 reused predictions and 29,169 new predictions. We use the first valid response for each prediction, require every retained value to lie in[0, 1], and attach correctness labels only after checking the complete set. Across-item AUROC pools 4,167 item–run observations per display, while 21 An Observability Ladder for Large Language Models within-item AUROC compares 2,778 correct–incorrect run pairs. We report all ten display estimates and seven paired differences:S− R,C− S,(R + S)− R,(R + C)− (R + S),(q + R + S)− (q + R),(q + R + C)− (q + R), and (q +R +S +C)− (q +R +C). A 5,000-draw item bootstrap resamples the same questions jointly across all displays, preserving the pairing in the intervals for these differences. Baseline controls. Selected-answer and option-bias baselines test whether answer identity alone explains response- only discrimination. The baselines are logistic models on the one-hot selected answer, a train-split answer-prior encoding of the selected answer, answer-format indicators such as answer length and digit or math-character flags, and, where replay probability features are available, the selected-answer probability alone. They use the same splits, candidate regularization values, and held-out evaluation as the primary probes (Appendix Table A21). Verification variables computed by comparing the extracted answer against the gold answer encode the label and are excluded from all baseline and monitor inputs. Transfer probes. Transfer experiments retrain the public monitor on one or more source settings and evaluate it, without refitting, on a held-out target setting, using the target setting’s split assignments so transfer metrics remain comparable to in-setting metrics. The transfer table reports the setups rerun under the updated Omni-MATH-2-Filtered response-label version, covering six cross-model transfers within that benchmark and ten same-model transfers between MMLU-Pro and Omni-MATH-2-Filtered. Each setup is evaluated atR,R +S,R +C,R +S +C, andq +R +S +C for five target splits (Appendix Table A56). For same-benchmark cross-model transfer, source runs are aligned to the target item’s train/validation/test split assignment before fitting, so every target-test item is excluded from source train and validation. For cross-benchmark transfer, source and target item identifiers come from different benchmark item universes. Additional comparisons and evaluation measures. For each setting and random split, we compute a prespecified set of paired AUROC differences for the public display subsets, prompt-conditioned controls, prompt-overlap-stripped displays, and replay-feature decomposition. Shapley contributions forR,S, andC[49] average each source’s marginal AUROC contribution over all3! = 6source orderings, with the empty display assigned chance AUROC 0.5, so the three contributions of a setting sum to its full public AUROC minus 0.5. The primary and secondary open prompt-conditioned Shapley controls holdqfixed as the baseline and allocateAUROC(q + R + S + C)− AUROC(q)overR,S, and Cusing their complete factorials. We interpret these Shapley values only as descriptive allocations of AUROC gains among displayed components; they have no information-theoretic interpretation. Selective-risk curves rank held-out test runs by predicted correctness and report the error rate among retained runs at coverage levels0.5, 0.7, 0.8, 0.9, 1.0. For post-hoc calibration, a one-dimensional Platt calibrator is fit on validation predictions for each benchmark, model, access level, feature family, split, and run, then applied to held-out test predictions. Calibration tables report raw versus calibrated Brier score and expected calibration error, while AUROC remains the ranking metric. A paired item bootstrap with 1,000 draws per setting and split resamples test items under fixed pairing to check that subset and prompt contrasts are not driven by a few items. Its pooled intervals include between-setting heterogeneity, so setting-level bootstrap intervals remain the primary uncertainty summary. Because the 15 settings reuse benchmarks across models and models within families, we report leave-one-family-out and leave-one-benchmark-out aggregates and an item-weighted aggregate for the prompt-withheld full ladder, which keep the gain between+0.178and+0.262 (+0.251item weighted, Appendix Table A14). We also report per-setting item-clustered bootstrap intervals that respect item-level clustering of the paired design (Appendix Table A17). We fit intercept-only linear mixed models to each prompt-visible and prompt-withheld ladder increment across the 15 settings. The preferred specification includes benchmark and model-family variance components; a fit that does not converge with finite fixed-effect uncertainty uses the predefined benchmark-only fallback. Modeled estimates differ slightly from the paired means by construction. All modeled point estimates are positive, but the 95% interval forq∆ S under the fallback includes zero (Appendix Table A16). Failure-mode stratification.Held-out predictions of the full-public monitor are stratified by parser status, verification status, category and domain where defined, choice count, the presence of explicit self-correction markers in the trace, and within-setting trace- and summary-length terciles. Within-stratum AUROC is reported only for strata with at least 20 test runs and both correctness classes (Appendix Table A60). Correctness-class slices and high-score wrong cases are treated as descriptive error analyses rather than AUROC strata, because AUROC is undefined within a single correctness class. Metrics and aggregation. The primary metric is AUROC for final-answer correctness prediction. AUROC is the probability that a randomly chosen correct run receives a higher score than a randomly chosen incorrect run, with half credit for ties. The marginal correctness rate does not mechanically shift AUROC, although it affects uncertainty, precision–recall baselines, and calibration (Appendix Table A5). AUPRC, expected calibration error, and Brier score 22 An Observability Ladder for Large Language Models are secondary. ECE is computed with 10 equal-width probability bins. Main secondary-metric tables use raw held-out logistic scores, and a separate post-hoc calibration table reports validation-fitted Platt calibration. Scores above0.8are therefore described as high-score errors when wrong unless the calibrated analysis is being discussed. For each reported setting, the primary estimate is AUROC m,b,k,f , wheremindexes target model,bindexes benchmark,kindexes access level, andfindexes feature family. We compute AUROC per split, average the five splits within each model–benchmark–level setting, compute ladder deltas within setting, and report means across the 15 model–benchmark settings. We repeat the same calculations for every random split across the display subsets, prompt controls, stripped displays, and decomposition displays. Standard deviations in detailed tables are descriptive summaries of between-split or between-setting variation, not confidence intervals. For setting-level uncertainty, we resample the 15 model–benchmark settings with replacement to form nonparametric 95% bootstrap intervals. These intervals summarize the 15 observed settings; they are not generalization intervals over arbitrary tasks or model families. The interval for the normalized excess-AUROC ratior AUROC (defined in Section 4) is obtained by jointly resampling the 15 paired setting means and recomputing the ratio in each of 2,000 bootstrap draws. We also report one-sided paired sign-flip tests of the setting-level mean against zero as descriptive paired-consistency checks, since the 15 settings share benchmarks, model families, prompts, and sometimes item identities. The prompt-visible deltasq∆ S ,q∆ C ,q∆ Z ,q∆ public , andq∆ full and the prompt-withheld deltas∆ S ,∆ C , ∆ Z , ∆ public , and ∆ full are defined in Section 2. Controls. The no-length surface version tests whether direct length features account for public-text gains. TF-IDF and Cohere embeddings are reported separately from the combined public-text monitor. A no-silent-truncation check confirms whole-context fit and records chunking for public embeddings. Replay-feature sensitivity reports answer- probability-only, hidden-state-only, combined-replay, and replay-only probes. Surface-feature sensitivity compares the primary rate-based surface family against the no-length version, with raw marker-count and lexical-diversity versions retained as diagnostics rather than current main-text controls. Parser and replay-status controls test whether extraction or context-fit failures explain apparent access-level gains. Summary-length controls hard-cap primary summaries at 90 words, replace the summary with first, last, head–tail, or random trace extracts matched to each capped summary’s word count, and compare the fixed Last90 extract with a masked Last90 version under the primary and open feature families. The separate fixed-monitor sensitivity varies summary decoding as described above. The matched primary chunk control uses every primary-run trace in the 15 model–benchmark settings and partitions each trace into up to eight balanced contiguous word-position chunks targeting 512 words; a sufficiently short trace forms one chunk. Each chunk receives a fixed 512-dimensional Cohere embedding. For each of five splits, held-out item scores come from logistic probes over mean, elementwise maximum, or elementwise top-two pooled chunk embeddings, or from a linear-attention multiple-instance head, with validation-selected regularization or early stopping. A secondary open surface-plus-TF-IDF control fits individual chunks and combines held-out chunk scores by maximum, top-two mean, mean, or last-or-only chunk. These checks show whether the results depend on how the trace is divided and read; they do not identify where causal evidence appears. Train-only preprocessing, paired split assignments, and held-out test evaluation are the main protections against leakage from labels or validation data into the reported AUROC estimates. The reported analyses include the output-only subsets, every primary and open prompt-conditioned combination, prompt-overlap stripping, summary-instruction and decoding sensitivities, baseline controls, transfer probes, and failure stratification described above. 23 An Observability Ladder for Large Language Models B Appendix Figures <|im_start|>system Solve the following problem. Please make sure that your response only consists of a single letter corresponding to the correct answer choice. Do not include anything else in your final response.<|im_end|> <|im_start|>user Among the following exoplanets, which one has the highest density? a) An Earth-mass and Earth-radius planet. b) A planet with 2 Earth masses and a density of approximately 5.5 g/cm^3. c) A planet with the same composition as Earth but 5 times more massive than Earth. d) A planet with the same composition as Earth but half the mass of Earth. A. d B. a C. b D. c<|im_end|> <|im_start|>assistant Appendix Figure A1: Example target prompt for Qwen3-4B on GPQA-Diamond. The box shows the Qwen chat-template string for item 1 of GPQA-Diamond under the multiple-choice system instruction used to produce the completed run. <|im_start|>assistant <think> Okay, let’s try to figure out which exoplanet has the highest density. The question gives me four options, and I need to determine which one has the highest density. Let me start by recalling what density is. Density is mass divided by volume, right? So, the formula is density = mass/volume. Now, the problem is about exoplanets, so I need to think about how mass and radius relate to density. For planets, if they have the same composition, like Earth, their density would depend on their mass and radius. But if the mass changes, the radius might change as well, depending on how the planet is structured. Let me look at each option one by one. Option a) An Earth-mass and Earth-radius planet. So, this is just Earth. Earth’s density is about 5.5 g/cm^3. Option b) A planet with 2 Earth masses and a density of approximately 5.5 g/cm^3. So, this is a planet that is twice as massive as Earth but has the same density as Earth. [... reasoning trace omitted for display ...] But given that the question is from a test, it’s likely that the intended answer is option c. Because for a planet with same composition, more mass implies higher density, since it’s more compressed. Therefore, option c has higher density than the others. So, in that case, the answer is option c, which is D. </think> D Appendix Figure A2: Example completed response from Qwen3-4B on GPQA-Diamond. The box shows item 1 from the primary run, shortened for display. The full reasoning trace is used in the analysis. The parsed final response is R = D, and the extracted answer matches the GPQA-Diamond gold label. 24 An Observability Ladder for Large Language Models <|im_start|>system Compress the reasoning trace into at most 90 words without solving the problem again. Preserve uncertainty, contradictions, candidate answers, and self-corrections. Do not add evidence, verify the answer, repair mistakes, or mention information absent from the trace.<|im_end|> <|im_start|>user Reasoning trace: Okay, let’s try to figure out which exoplanet has the highest density. The question gives me four options, and I need to determine which one has the highest density. Let me start by recalling what density is. Density is mass divided by volume, right? [... full reasoning trace supplied to the summarizer ...] So, in that case, the answer is option c, which is D. Summary:<|im_end|> <|im_start|>assistant The reasoning traces uncertainty about density calculations, noting that same-composition planets with higher mass might have higher density due to compression, while others have same density. Contradictions arise between scaling assumptions and physical compression effects. Candidate answers include options a, b, c, and d, with c potentially having higher density due to compression. Self-corrections suggest uncertainty in whether same-composition mass scaling leads to same or higher density. Appendix Figure A3: Example self-summary prompt and output for the summary level. The summarizer receives only the reasoning traceC, not the task prompt, gold answer, correctness label, logits, or hidden states. The displayed prompt is shortened. The generated summary is the output for item 1 from Qwen3-4B on GPQA-Diamond. Its stored file name records a 256-token decoding limit, while the instruction requests at most 90 words. Qwen3-4BQwen3-8B Qwen3-14B gpt-oss-20b gpt-oss-120b GPQA-Diamond MMLU-Pro Omni-MATH-2-Filtered +0.240*+0.227+0.210+0.214*+0.257 +0.299+0.209+0.332+0.329+0.307 +0.087*+0.075+0.112+0.182+0.178 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Δ full Appendix Figure A4: Full-ladder gains across the model–benchmark matrix. Each tile is one target model and one benchmark. Color shows the paired full-ladder gain∆ full = AUROC(R + S + C + Z)− AUROC(R), also printed on each tile. An asterisk marks settings whose public-level point estimates do not satisfy R < R + S < R + S + C. 25 An Observability Ladder for Large Language Models R+SR+S+C 0.4 0.5 0.6 0.7 0.8 0.9 AUROC a Surface No length Cohere Surface + Cohere TF-IDF Surface + TF-IDF + Cohere Δ S Δ C Surface Cohere Surface+Cohere TF-IDF Surface Cohere Surface+Cohere TF-IDF Surface Cohere Surface+Cohere TF-IDF GPQA-Diamond MMLU-Pro Omni-MATH-2-Filtered +0.17+0.07 +0.21+0.00 +0.21+0.01 +0.22+0.00 +0.19+0.08 +0.20+0.04 +0.21+0.05 +0.25+0.02 +0.05+0.09 +0.05+0.05 +0.05+0.05 +0.07+0.03 b R R+S R+S+C R+S+C+Z L1 Surface L1 Cohere L1 Z prob SGL Surface SGL Cohere SGL Z prob 0.860.960.990.97 0.921.001.001.00 No dataNo dataNo data 0.76 1.000.971.001.00 0.930.920.750.84 No dataNo dataNo data 0.79 c 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Mean paired gain 0.0 0.2 0.4 0.6 0.8 1.0 Selection rate Appendix Figure A5: The mean public access pattern persists across the tested feature families. Public monitors use only the text exposed at the corresponding level, never the target model’s hidden states, logits, token probabilities, or the original question. a, Mean public AUROC increases across access levels for surface features, the no-length version, Cohere embed-v4.0, TF-IDF, and their combinations. Faint points show available model–benchmark settings. All displayed feature families cover the 15 model–benchmark settings. b, Per-benchmark paired step gains show that the summary step∆ S is concentrated on GPQA-Diamond and MMLU-Pro, while Omni-MATH-2-Filtered spreads its public gain more evenly. c, Plain LASSO and sparse group LASSO selection rates show that both surface features and embeddings enter the probes. Answer-probability features appear only at the top level and are not part of the public monitor. SCR+ S+ C 0.700 0.725 0.750 0.775 0.800 0.825 0.850 0.875 Test AUROC a Original Masked SC R + S R + C S + C R + S + C −0.03 −0.02 −0.01 0.00 0.01 0.02 ΔAUROC after stripping b Appendix Figure A6: Masking and prompt-overlap stripping leave the public signal essentially unchanged. a, Summary, trace, and full public displays after replacing stated answers, option text, and prompt-derived numbers with fixed markers (open circles) match the unmasked originals (filled circles). Gray lines connect the same setting. The combined mask shifts mean trace AUROC by+0.001(Appendix Table A34). b, Removing 8–14-word spans copied from the question changes mean AUROC by between−0.002and+0.001across the six summary- and trace-containing displays (Appendix Table A33). 26 An Observability Ladder for Large Language Models GPQA-Diamond MMLU-Pro Omni-MATH-2-Filtered Five models per benchmark (in increasing size order) 0.00 0.05 0.10 0.15 0.20 0.25 Δ S AUROC a Sampled mean Greedy 0.820.840.860.880.90 Pairwise sampled-summary cosine 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Mean per-item R + S score SD b GPQA-Diamond MMLU-Pro Omni-MATH-2-Filtered Qwen3-4B Qwen3-8B Qwen3-14B gpt-oss-20b gpt-oss-120b Appendix Figure A7: A fixed primary monitor is stable across alternate sampled summaries. Each point uses a fixed surface-plus-Cohere monitor evaluated on a balanced 48-item subset in one of 15 target-model–benchmark settings; colors denote benchmarks and marker shapes denote target models. a, Colored symbols show the mean equal-stratum summary increment,∆ S = AUROC(R + S)− AUROC(R), across three sampled summaries, vertical lines span their range, and black crosses show greedy decoding. Across settings, the sampled mean is+0.086AUROC (hierarchical 95% bootstrap interval[+0.027, +0.146]), and the greedy value is+0.010higher on average. b, For each setting, mean pairwise cosine similarity among the three sampled-summary embeddings is plotted against the mean per-item standard deviation of the fixed R + S monitor score across those summaries. Z prob Z hidden Z −0.02 0.00 0.02 0.04 0.06 AUROC gain over R + S + C a 0.000 Z prob Z hidden Z Qwen3-4B Qwen3-8B Qwen3-14B gpt-oss-20b gpt-oss-120b Qwen3-4B Qwen3-8B Qwen3-14B gpt-oss-20b gpt-oss-120b Qwen3-4B Qwen3-8B Qwen3-14B gpt-oss-20b gpt-oss-120b GPQA-Diamond MMLU-Pro Omni-MATH-2-Filtered -0.03+0.01+0.01 -0.00+0.02+0.02 -0.03-0.03-0.03 -0.01+0.06+0.04 +0.00+0.03+0.03 +0.01+0.04+0.04 +0.01+0.04+0.04 +0.02+0.04+0.04 +0.01+0.04+0.04 +0.00+0.03+0.03 +0.00+0.02+0.02 +0.00+0.04+0.04 +0.00+0.01+0.01 +0.00+0.04+0.04 +0.00+0.02+0.02 b −0.02 0.00 0.02 0.04 0.06 Gain over public Appendix Figure A8: The provider-side replay gain is mostly a hidden-state signal. Gains are paired AUROC differences relative to the full public displayP = R + S + C. a, The average breakdown compares answer-probability, hidden-state, and combined replay features against the same full public baseline. Bars are means across settings and black dots show setting-level values; the annotated answer-probability bar has a mean gain of0.000. b, The model-by-benchmark heatmap shows where the top-level gain appears for those three replay feature families. 27 An Observability Ladder for Large Language Models R R+SR+CR+S+C 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Test AUROC In-setting target probes Cross-model transfer Cross-benchmark transfer Appendix Figure A9: Public probes transfer across models and benchmarks. Transfer tests fit public probes in one setting and evaluate them, without refitting, on held-out settings. Filled circles show cross-model transfer within Omni-MATH-2-Filtered, open circles show cross-benchmark transfer between MMLU-Pro and Omni-MATH-2-Filtered for the same model, black ticks mark matched in-setting target means, and gray ticks mark transfer means. Cross-model transfer loses modest discrimination relative to matched in-setting target probes (0.826versus0.860mean AUROC for R +S +C), while cross-benchmark transfer is lower, with mean AUROC0.752forR +S +Cand individual settings below chance at the shortest displays. 28 An Observability Ladder for Large Language Models C Appendix Tables Appendix Table A1: Notation for the observability ladder. Components are derived from one fixed completed run per item. Levels are the displays a monitor receives. SymbolObjectDefinition qTask promptBenchmark question text; visible to prompt-conditioned monitors. RFinal responseThe model’s final response text from the completed run. SSelf-summaryPost-hoc self-summary of the trace, generated from C only by the same target model under an instruction requesting at most 90 words. CReasoning traceThe full reasoning trace of the completed run. ZProvider-only replay featuresAnswer-probability (Z prob ) and hidden-state (Z hidden ) features extracted from a prompt-conditioned replay of the target model on q and C. O q 1 Prompt-visible level 1q + R O q 2 Prompt-visible level 2q + R + S O q 3 Prompt-visible level 3q + R + S + C (full prompt-visible public display) O q 4 Provider-only featuresq + R + S + C + Z (public display plus replay-derived features) Appendix Table A2: The model set gives a within-family scale sweep and a second-family contrast. All target models are treated as locally inspectable open-weight systems for the provider-only replay features. Public levels restrict the monitor input to model-produced visible text. The final level adds hidden states and token probabilities from a prompt-conditioned replay of the target model. gpt-oss parameter counts are total parameters, with active parameters shown in parentheses. gpt-oss inference uses the model-specific Transformers code path and locally stored Hugging Face kernel components required by the released weights. ModelFamilySizeHidden statesToken probabilities Context window Qwen/Qwen3-4BQwen34BFullFull32,768 Qwen/Qwen3-8BQwen38BFullFull32,768 Qwen/Qwen3-14BQwen314BFullFull32,768 openai/gpt-oss-20bgpt-oss 21B (3.6B active)FullFull128,000 openai/gpt-oss-120bgpt-oss117B (5.1B active) FullFull128,000 29 An Observability Ladder for Large Language Models Appendix Table A3: Settings used to generate reasoning traces and summaries and to calculate model-based measurements. The reported analysis can be reproduced from the saved outputs even when the original text generation was stochastic. Reasoning traces were generated with vLLM’s default sampling settings except for the token limit. Summaries were generated with Hugging Facegenerateusing each loaded model’s generation settings, with no explicit temperature or top-p override. ComponentTemperatureTop-pTop-kRandom seedMax tokensStopping behaviorDeterministic? Trace generation (vLLM chat template) 1.01.00NoneQwen: 28,000; gpt-oss: 126,000 no added stopping rule; model EOS and chat-template stopping apply No; the analysis uses saved traces Summary generation (HF, do_sample=True, use_cache=False) model setting; default 1.0 if unset model setting; default 1.0 if unset model setting; default 50 if unset seed identifier 0 recorded; random-number generator not fixed 256 minus any final-channel prefill EOS used for padding; summaries were not shortened afterward No; the analysis uses saved summaries Calculation of Z (forward pass only) not applicable; no sampling not applicable; no sampling not applicable; no sampling not applicable; deterministic forward pass not applicable; no new text generated model-specific final suffixes; bare-label and answer-span scores taken at the final position Yes, apart from numeric differences across hardware and libraries Appendix Table A4: The benchmark set includes multiple-choice and open-response tasks. GPQA-Diamond and MMLU-Pro use answer-choice log-probability features inZ prob . Omni-MATH-2-Filtered uses answer-span log-probability features for the model’s own extracted final answer, including answer-token and answer-character counts and the sum of token log probabilities. BenchmarkDomainSizeAnswer typeVerification and Z prob GPQA-DiamondGraduate science198Four-way multiple choice Option match; log-probabilities over A–D MMLU-ProMulti-task language understanding 12,032Up to ten-way multiple choice Option match; log probabilities for fixed choices A–J, with unavailable choices masked out Omni-MATH-2- Filtered Olympiad mathematics 4,181Open-response numeric / expression Stored GPT-5-mini judgment of whether the model response matches the reference answer; answer-span log-probability features for the model’s extracted final answer, including answer-token and answer-character counts and the sum of token log probabilities 30 An Observability Ladder for Large Language Models Appendix Table A5: Correctness prevalence varies by benchmark and target model. The table reports the binary final-answer correctness labels used in the main ladder analyses. GPQA-Diamond and MMLU-Pro use benchmark option matching. Omni-MATH-2-Filtered correctness is determined by GPT-5-mini equivalence judgments. AUROC is not mechanically shifted by these prevalences because it ranks correct against incorrect runs, but prevalence affects AUPRC baselines, calibration metrics, and uncertainty. BenchmarkTarget modelnCorrectCorrectness rate GPQA-DiamondQwen3-4B19811055.6% GPQA-DiamondQwen3-8B19811558.1% GPQA-DiamondQwen3-14B19812462.6% GPQA-Diamondgpt-oss-20b19812462.6% GPQA-Diamondgpt-oss-120b19813869.7% MMLU-ProQwen3-4B12,0328,55071.1% MMLU-ProQwen3-8B12,0329,07375.4% MMLU-ProQwen3-14B12,0329,37577.9% MMLU-Progpt-oss-20b12,0328,94574.3% MMLU-Progpt-oss-120b12,0329,55579.4% Omni-MATH-2-FilteredQwen3-4B4,1812,75165.8% Omni-MATH-2-FilteredQwen3-8B4,1812,45558.7% Omni-MATH-2-FilteredQwen3-14B4,1812,85368.2% Omni-MATH-2-Filteredgpt-oss-20b4,1813,01672.1% Omni-MATH-2-Filteredgpt-oss-120b4,1813,13174.9% Appendix Table A6: Public-text embeddings are not provider-side replay features. Public monitors receive only the model-produced text exposed at the corresponding access level. Replay features are extracted from a prompt-conditioned forward pass of the target model on the task prompt and reasoning trace. We record answer-extraction and replay outcomes separately so that failures can be examined in sensitivity checks. Feature familyInput to feature extractorUses provider-side replay? Level Surface textVisible public text Pnopublic No-length surface textVisible public text P with direct length/count features removed nopublic TF-IDF textVisible public text Pnopublic Cohere embed-v4.0Visible public text Pnopublic Task prompt q i Benchmark prompt text, rendered into the display nocontrol levels only Target-model answer log-probabilities (q i ,C i ,s stop ) through target model; open-response spans include length-sensitive count and sum features yes Z prob Target-model raw hidden state(q i ,C i ,s stop ) through target modelyes Z hidden Answer extraction and replay checks Extraction outcome, replay outcome, and replay context-fit indicator noChecks 31 An Observability Ladder for Large Language Models Appendix Table A7: Primary public surface features use rates for lexical markers. Features are computed only from the public text visible at the corresponding access level. Raw marker counts and additional length/log-count variants are retained as appendix diagnostics, but the primary surface family uses the rate-based definitions below. FeatureDefinitionIntended signal Word countNumber of alphabetic word matches in the visible text Overall trace length Sentence countNumber of sentence-ending punctuation markers Reasoning segmentation Average word lengthMean character length of alphabetic wordsLexical form and technical vocabulary Digit densityDigit characters divided by total charactersNumerical reasoning content Math-character density Math-like characters divided by total charactersFormulaic or symbolic content Uncertainty rateUncertainty markers divided by word countExplicit hesitation or low-confidence language Self-correction rateSelf-correction markers divided by word countRevisions, reversals, and reconsideration Answer-marker rateAnswer markers divided by word countExplicit conclusion or answer-framing language 32 An Observability Ladder for Large Language Models Appendix Table A8: Inputs used by each correctness probe. All probes predict the same binary target, final-answer correctness. The reference answer and correctness label are never included as inputs. They are used only to definey i . Prompt-withheld public levels exclude the task prompt as a separate input and exclude provider-side replay features. TheZlevels add features extracted from a prompt-conditioned replay of the task prompt and reasoning trace. Across the reported comparisons, the provider-sideZfeatures appear only alongside the full public display; they are never used as a public display or monitor by themselves. The primary hidden-state model uses the raw final hidden state with ℓ 2 shrinkage. ProbeFeature vector in the logistic modelInformation not used Response only (R)z i = φ f (R i )Task prompt q i , self-summary S i , reasoning trace C i , replay features Z i , reference answer, correctness label Response + summary (R + S) z i = φ f (R i ∥S i )Task prompt q i , full reasoning trace C i , replay features Z i , reference answer, correctness label Response + summary + trace (R + S + C) z i = φ f (R i ∥S i ∥C i )Task prompt q i , replay features Z i , reference answer, correctness label Public display subsets (S, C, R + C, S + C) z i = φ f (·) applied to the corresponding subset ofR i ,S i ,C i Task prompt q i , replay features Z i , reference answer, correctness label; used to compare all component subsets, separately from the main prompt-visible comparison All prompt-conditioned public combinations (q, q + R, q + S, q + C, q + R + S ,q + R + C, q + S + C, q + R + S + C) z i = φ f (q i ∥·) applied to the corresponding prompt-conditioned combination Replay features Z i , reference answer, correctness label; the primary surface-plus-Cohere analysis follows a fixed sequence of component sets, while the secondary surface-plus-TF-IDF analysis covers all eight combinations Displays with prompt overlap removed (noqcopy variants of S/C-containing levels) z i = φ f (·) applied to the display after removing 8–14-word spans copied from q i Task prompt q i , replay features Z i , reference answer, correctness label; tests whether the visible text reconstructs the prompt Full public + answer probabilities (R + S + C + Z prob ) z i = φ all (R i ∥S i ∥C i )⊕ Z prob,i Hidden-state featuresZ hidden,i , reference answer, correctness label Full public + hidden state (R +S +C +Z hidden ) z i = φ all (R i ∥S i ∥C i )⊕ Z hidden,i Answer-probability features Z prob,i , reference answer, correctness label Full public + all replay features (R + S + C + Z) z i = φ all (R i ∥S i ∥C i )⊕ Z prob,i ⊕ Z hidden,i Reference answer and correctness label Prompt-conditioned public + replay features (q + R + S + C + Z hidden , q + R + S + C + Z) z i = φ all (q i ∥R i ∥S i ∥C i )⊕ Z i or the hidden-state features alone Reference answer and correctness label; used to separate information in prompt-conditioned public text from information in replay features 33 An Observability Ladder for Large Language Models Appendix Table A9: Prompt-withheld observability-ladder AUROC by model–benchmark setting. Values are mean±descriptive SD across data splits, not confidence intervals, for the primary public feature family (rate surface plus Cohere) and the provider-side replay feature family (rate surface plus Cohere plus replay-derived features). Omni- MATH-2-Filtered correctness is determined by GPT-5-mini equivalence judgments. BenchmarkModelRR + S R + S + C R+S+C +Z∆ full GPQA-DiamondQwen3-4B0.505± 0.066 0.732± 0.063 0.732± 0.067 0.744± 0.059 +0.240± 0.105 GPQA-DiamondQwen3-8B0.631± 0.047 0.812± 0.082 0.839± 0.062 0.859± 0.038 +0.227± 0.053 GPQA-DiamondQwen3-14B0.558± 0.046 0.748± 0.056 0.800± 0.058 0.767± 0.078 +0.210± 0.100 GPQA-Diamondgpt-oss-20b0.535± 0.137 0.755± 0.066 0.707± 0.096 0.749± 0.103 +0.214± 0.074 GPQA-Diamondgpt-oss-120b0.542± 0.073 0.761± 0.053 0.770± 0.057 0.799± 0.069 +0.257± 0.123 MMLU-ProQwen3-4B0.560± 0.014 0.764± 0.009 0.823± 0.010 0.859± 0.008 +0.299± 0.017 MMLU-ProQwen3-8B0.652± 0.014 0.774± 0.013 0.821± 0.004 0.861± 0.007 +0.209± 0.011 MMLU-ProQwen3-14B0.533± 0.004 0.775± 0.010 0.825± 0.007 0.865± 0.003 +0.332± 0.007 MMLU-Progpt-oss-20b0.515± 0.009 0.767± 0.016 0.803± 0.015 0.844± 0.011 +0.329± 0.005 MMLU-Progpt-oss-120b0.538± 0.006 0.769± 0.004 0.810± 0.005 0.845± 0.005 +0.307± 0.009 Omni-MATH-2- Filtered Qwen3-4B0.807± 0.015 0.803± 0.018 0.870± 0.015 0.894± 0.013 +0.087± 0.009 Omni-MATH-2- Filtered Qwen3-8B0.837± 0.027 0.853± 0.026 0.875± 0.022 0.912± 0.012 +0.075± 0.017 Omni-MATH-2- Filtered Qwen3-14B0.779± 0.010 0.821± 0.006 0.876± 0.008 0.891± 0.005 +0.112± 0.015 Omni-MATH-2- Filtered gpt-oss-20b0.682± 0.019 0.775± 0.013 0.824± 0.022 0.864± 0.020 +0.182± 0.026 Omni-MATH-2- Filtered gpt-oss-120b0.671± 0.012 0.776± 0.022 0.832± 0.019 0.849± 0.011 +0.178± 0.010 34 An Observability Ladder for Large Language Models Appendix Table A10: Paired AUROC increments along the observability ladder. Each increment is computed within each data split before averaging. The±values are descriptive SDs across data splits, not confidence intervals. Descriptive setting-bootstrap intervals over the observed design are reported separately in Appendix Table A11. Omni- MATH-2-Filtered correctness is determined by GPT-5-mini equivalence judgments. BenchmarkModel∆ S ∆ C ∆ Z ∆ full GPQA-DiamondQwen3-4B+0.228± 0.106 +0.000± 0.038 +0.012± 0.023 +0.240± 0.105 GPQA-DiamondQwen3-8B+0.181± 0.055 +0.027± 0.023 +0.020± 0.051 +0.227± 0.053 GPQA-DiamondQwen3-14B+0.190± 0.057 +0.052± 0.033 −0.033± 0.051 +0.210± 0.100 GPQA-Diamondgpt-oss-20b+0.220± 0.123 −0.049± 0.082 +0.043± 0.125 +0.214± 0.074 GPQA-Diamondgpt-oss-120b+0.219± 0.117 +0.010± 0.027 +0.029± 0.026 +0.257± 0.123 MMLU-ProQwen3-4B+0.205± 0.019 +0.059± 0.002 +0.036± 0.003 +0.299± 0.017 MMLU-ProQwen3-8B+0.122± 0.006 +0.046± 0.012 +0.040± 0.003 +0.209± 0.011 MMLU-ProQwen3-14B+0.241± 0.010 +0.051± 0.009 +0.040± 0.006 +0.332± 0.007 MMLU-Progpt-oss-20b+0.251± 0.010 +0.036± 0.008 +0.041± 0.005 +0.329± 0.005 MMLU-Progpt-oss-120b+0.231± 0.009 +0.042± 0.003 +0.035± 0.003 +0.307± 0.009 Omni-MATH-2- Filtered Qwen3-4B−0.004± 0.008 +0.067± 0.004 +0.024± 0.005 +0.087± 0.009 Omni-MATH-2- Filtered Qwen3-8B+0.016± 0.007 +0.022± 0.005 +0.037± 0.013 +0.075± 0.017 Omni-MATH-2- Filtered Qwen3-14B+0.042± 0.008 +0.055± 0.006 +0.015± 0.010 +0.112± 0.015 Omni-MATH-2- Filtered gpt-oss-20b+0.093± 0.019 +0.049± 0.013 +0.039± 0.021 +0.182± 0.026 Omni-MATH-2- Filtered gpt-oss-120b+0.106± 0.015 +0.056± 0.015 +0.017± 0.013 +0.178± 0.010 35 An Observability Ladder for Large Language Models Appendix Table A11: Setting-level resampling describes the prompt-visible and prompt-withheld ladder contrasts. Mean deltas and descriptive SDs are computed across the 15 model–benchmark settings after averaging the five data splits within each setting. Intervals are descriptive nonparametric 95% bootstrap intervals over the 15 observed settings, not population-generalization intervals. Thepvalue is a one-sided paired sign-flip test of the setting-level mean against zero in the tested direction: positive for ladder increments and negative for prompt-visible-minus-prompt-withheld differences. Values within 10 −12 AUROC of zero are treated as ties. ConditionQuantityContrastMeanBetween- setting SD 95% bootstrap interval Settings in tested direction p Prompt-visibleq∆ S q+R to q+R+S+0.0190.017[+0.011, +0.027]11/15< 0.001 Prompt-visible, trace first q∆ C|R q+R to q+R+C+0.0620.033[+0.046, +0.077]15/15< 0.001 Prompt-visibleq∆ C q+R+S to q+R+S+C+0.0410.026[+0.028, +0.052]14/15< 0.001 Prompt-visible, trace first q∆ S|R,C q+R+C to q+R+S+C −0.0020.009[−0.007, +0.002]8/150.819 Prompt-visible direct comparison q(C−S)q+R+S versus q+R+C+0.0430.020[+0.033, +0.052]15/15< 0.001 Prompt-visibleq∆ Z q+R+S+C to q+R+S+C+Z +0.0290.014[+0.023, +0.036]15/15< 0.001 Prompt-visibleq∆ public q+R to q+R+S+C+0.0600.039[+0.040, +0.077]14/15< 0.001 Prompt-visibleq∆ full q+R to q+R+S+C+Z+0.0890.045[+0.066, +0.110]15/15< 0.001 Prompt-withheld∆ S R to R+S+0.1560.087[+0.113, +0.196]14/15< 0.001 Prompt-withheld∆ C R+S to R+S+C+0.0350.030[+0.019, +0.048]13/15 (+1 tie) < 0.001 Prompt-withheld∆ Z R+S+C to R+S+C+Z+0.0260.019[+0.016, +0.034]14/15< 0.001 Prompt-withheld∆ public R to R+S+C+0.1910.080[+0.151, +0.228]15/15< 0.001 Prompt-withheld∆ full R to R+S+C+Z+0.2170.082[+0.177, +0.257]15/15< 0.001 Paired condition difference q∆ S − ∆ S Prompt-visible minus prompt-withheld summary increment −0.1370.086[−0.177,−0.095]13/15< 0.001 Paired condition difference q∆ C − ∆ C Prompt-visible minus prompt-withheld trace increment +0.0060.015[−0.002, +0.013]2/150.920 Paired condition difference q∆ Z − ∆ Z Prompt-visible minus prompt-withheld replay increment +0.0030.011[−0.002, +0.009]6/150.855 Paired condition difference q∆ public − ∆ public Prompt-visible minus prompt-withheld public gain −0.1310.084[−0.170,−0.089]13/15< 0.001 Paired condition difference q∆ full − ∆ full Prompt-visible minus prompt-withheld full gain −0.1280.082[−0.166,−0.087]13/15< 0.001 36 An Observability Ladder for Large Language Models Appendix Table A12: The summary adds much less when the prompt is visible. Values are mean paired AUROC changes across the 15 model–benchmark settings and five data splits (±SD across the 75 combinations). Prompt-visible quantities include both orders in which sources are added and the directq+R+Cversusq+R+Scomparison. The provider-only comparison adds the narrow prompt-conditioned last-token replay featuresZ. The prompt-withheld column gives the corresponding output-only comparison. Brackets are descriptive 95% setting-bootstrap intervals after averaging the five data splits within each setting. QuantityPrompt-visiblePrompt-withheldDifference Summary added first+0.019± 0.022+0.156± 0.099 −0.137 [−0.177,−0.095] Trace added after summary+0.041± 0.031+0.035± 0.038 +0.006 [−0.002, +0.013] Trace added first +0.062± 0.036 [+0.046, +0.077]— Summary added after trace −0.002± 0.020 [−0.007, +0.002]— Trace minus summary +0.043± 0.026 [+0.033, +0.052]— Provider-only replay features+0.029± 0.035+0.026± 0.040 +0.003 [−0.002, +0.009] Public total+0.060± 0.043+0.191± 0.100 −0.131 [−0.170,−0.089] Full total+0.089± 0.054+0.217± 0.094 −0.128 [−0.166,−0.087] Appendix Table A13: Component-block monitors separate access from concatenation compression. The primary public monitor embeds each rendered display once. The component-block monitor instead concatenates separately computed surface and Cohere feature blocks for the displayed components, so higher-access levels retain the lower-level representation unchanged. Values are mean AUROC across model–benchmark settings after averaging over five data splits (± between-setting SD). ∆ is component-block minus primary-display AUROC. DisplayPrimary displayComponent blocks∆Settings R0.623± 0.1120.624± 0.112+0.001± 0.00215 R+S0.779± 0.0310.775± 0.059 −0.004± 0.03715 R+S+C0.814± 0.0480.810± 0.058 −0.004± 0.01915 q0.734± 0.0320.735± 0.032+0.001± 0.00115 q+R0.750± 0.0420.748± 0.051 −0.001± 0.01715 q+R+S0.769± 0.0380.784± 0.049+0.016± 0.02115 q+C0.812± 0.0450.815± 0.051+0.003± 0.01315 q+R+S+C0.809± 0.0500.812± 0.056+0.002± 0.01515 37 An Observability Ladder for Large Language Models Appendix Table A14: Summary–trace and prompt-visible conclusions persist across the observed design. Mean paired AUROC contrasts shown separately by benchmark, recomputed after excluding each model family or benchmark, and under item-count weighting instead of equal weighting. The excess ratio is(AUROC(S)− 0.5)/(AUROC(C)− 0.5); it compares excess AUROC rather than retained information. Its paired setting-bootstrap estimate is 0.874 (95% descriptive interval [0.855, 0.896]; 2,000 draws over the 15 observed settings). Aggregate∆ full C− S(R+C)− (R+S) q∆ S q∆ C (q+R+C)− (q+R+S) q∆ public Excess ratio All settings (equal weight)+0.217+0.039+0.033+0.019+0.041+0.043+0.0600.874 Benchmark only GPQA-Diamond +0.230+0.026+0.006+0.003+0.011+0.021+0.0140.905 Benchmark only MMLU-Pro+0.295+0.043+0.046+0.037+0.057+0.056+0.0940.864 Benchmark only Omni-MATH-2-Filtered +0.127+0.049+0.047+0.018+0.054+0.052+0.0710.860 Excluding family gpt-oss+0.199+0.043+0.041+0.017+0.041+0.044+0.0590.868 Excluding family Qwen3+0.245+0.033+0.021+0.021+0.040+0.042+0.0610.885 Excluding benchmark GPQA-Diamond +0.211+0.046+0.046+0.027+0.056+0.054+0.0830.862 Excluding benchmark MMLU-Pro +0.178+0.037+0.026+0.010+0.032+0.036+0.0430.880 Excluding benchmark Omni-MATH-2-Filtered +0.262+0.035+0.026+0.020+0.034+0.039+0.0540.883 All settings (item weighted)+0.251+0.045+0.046+0.031+0.056+0.054+0.0870.863 Appendix Table A15: Item-count weighting preserves the headline output-only and prompt-visible increments. Equal weighting assigns one vote to each of the 15 model–benchmark settings. Item weighting instead weights each setting by its mean held-out test count across the five data splits. The final column is the item-weighted estimate minus the equal-setting estimate. IncrementEqual-weight meanItem-weighted mean Change ∆ S +0.156+0.169+0.013 ∆ C +0.035+0.047+0.012 ∆ Z +0.026+0.035+0.009 ∆ public +0.191+0.216+0.026 ∆ full +0.217+0.251+0.034 q∆ S +0.019+0.031+0.012 q∆ C +0.041+0.056+0.015 q∆ Z +0.029+0.036+0.006 q∆ public +0.060+0.087+0.028 q∆ full +0.089+0.123+0.034 38 An Observability Ladder for Large Language Models Appendix Table A16: Mixed-model sensitivity estimates are positive, although the prompt-visible summary interval includes zero. Intercept-only linear mixed models fit by REML to the 15 per-setting AUROC increments. The preferred specification includes benchmark and model-family variance components; when that fit does not converge with finite fixed-effect uncertainty, the estimate uses the predefined benchmark-only alternative. Estimates and 95% Wald intervals are descriptive model-based sensitivities for the observed design, not population-generalization intervals. Interval endpoints are reported to four decimals to resolve near-zero bounds. Variance components are reported×10 4 ; an omitted component is shown as ‘–’. IncrementSpecificationEstimate95% intervalσ 2 bench σ 2 family σ 2 resid q∆ S (q+R+S vs. q+R)Benchmark only+0.019[−0.0002, +0.0382]2.65–1.12 q∆ C (q+R+S+C vs. q+R+S) Benchmark + family +0.041[+0.0042, +0.0772]9.950.021.97 q∆ Z (q+R+S+C+Z vs. q+R+S+C) Benchmark + family +0.030[+0.0184, +0.0415]0.410.221.45 q∆ public (q+R+S+C vs. q+R) Benchmark only+0.060[+0.0129, +0.1066]16.54–3.10 q∆ full (q+R+S+C+Z vs. q+R) Benchmark + family +0.090[+0.0334, +0.1457]23.590.213.40 ∆ S (R+S vs. R)Benchmark + family +0.160[+0.0467, +0.2743]81.5811.4911.16 ∆ C (R+S+C vs. R+S)Benchmark + family +0.034[+0.0042, +0.0635]4.400.915.27 ∆ Z (R+S+C+Z vs. R+S+C) Benchmark + family +0.027[+0.0105, +0.0433]0.880.402.92 ∆ public (R+S+C vs. R)Benchmark + family +0.193[+0.0976, +0.2876]62.242.8918.34 ∆ full (R+S+C+Z vs. R)Benchmark + family +0.221[+0.1164, +0.3254]69.648.6112.99 Appendix Table A17: Item-clustered bootstrap intervals for the main ladder. Held-out test AUROC per setting and level (mean over five data splits) with 95% intervals from resampling test items with replacement within each setting (2,000 draws), so the uncertainty respects item-level clustering of the paired design rather than treating runs as independent. BenchmarkModelRR + SR + S + CR + S + C + Z GPQA- Diamond Qwen3-4B0.505 [0.408, 0.604]0.732 [0.638, 0.817]0.732 [0.636, 0.822]0.744 [0.651, 0.835] GPQA- Diamond Qwen3-8B0.631 [0.533, 0.730]0.812 [0.732, 0.884]0.839 [0.758, 0.907]0.859 [0.792, 0.920] GPQA- Diamond Qwen3-14B0.558 [0.452, 0.660]0.748 [0.655, 0.834]0.800 [0.716, 0.874]0.767 [0.675, 0.849] GPQA- Diamond gpt-oss-20b0.535 [0.438, 0.637]0.755 [0.665, 0.836]0.707 [0.612, 0.788]0.749 [0.650, 0.835] GPQA- Diamond gpt-oss-120b0.542 [0.440, 0.642]0.761 [0.670, 0.843]0.770 [0.666, 0.858]0.799 [0.709, 0.876] MMLU-ProQwen3-4B0.560 [0.546, 0.574]0.764 [0.752, 0.777]0.823 [0.813, 0.833]0.859 [0.850, 0.868] MMLU-ProQwen3-8B0.652 [0.637, 0.667]0.774 [0.763, 0.786]0.821 [0.810, 0.831]0.861 [0.851, 0.870] MMLU-ProQwen3-14B0.533 [0.518, 0.549]0.775 [0.762, 0.787]0.825 [0.814, 0.836]0.865 [0.855, 0.875] MMLU-Progpt-oss-20b0.515 [0.500, 0.531]0.767 [0.754, 0.780]0.803 [0.791, 0.814]0.844 [0.834, 0.855] MMLU-Progpt-oss-120b0.538 [0.521, 0.555]0.769 [0.755, 0.782]0.810 [0.798, 0.822]0.845 [0.834, 0.856] Omni-MATH- 2-Filtered Qwen3-4B0.807 [0.789, 0.825]0.803 [0.785, 0.820]0.870 [0.855, 0.884]0.894 [0.879, 0.907] Omni-MATH- 2-Filtered Qwen3-8B0.837 [0.821, 0.853]0.853 [0.838, 0.868]0.875 [0.861, 0.889]0.912 [0.899, 0.924] Omni-MATH- 2-Filtered Qwen3-14B0.779 [0.758, 0.799]0.821 [0.803, 0.838]0.876 [0.861, 0.891]0.891 [0.877, 0.904] Omni-MATH- 2-Filtered gpt-oss-20b0.682 [0.658, 0.706]0.775 [0.754, 0.795]0.824 [0.807, 0.841]0.864 [0.847, 0.879] Omni-MATH- 2-Filtered gpt-oss-120b0.671 [0.645, 0.696]0.776 [0.756, 0.796]0.832 [0.815, 0.850]0.849 [0.833, 0.865] 39 An Observability Ladder for Large Language Models Appendix Table A18: Label permutation removes the monitor signal at every level. Held-out test AUROC of the primary probes on real labels (mean over per-setting means±between-setting SD) versus probes retrained on label-permuted training data (mean±SD over all permutation runs across model–benchmark settings and data splits). Permuted AUROC concentrating at chance confirms chance performance after randomizing the training labels; it is not an independent proof against every form of leakage. LevelReal labelsPermuted labelsPermutation runs R0.623± 0.1120.503± 0.062375 R + S0.779± 0.0310.496± 0.057375 R + S + C0.814± 0.0480.494± 0.053375 R + S + C + Z0.840± 0.0520.496± 0.055375 Appendix Table A19: AUPRC and Brier generally improve with access, whereas uncalibrated ECE does not. AUROC, AUPRC, Brier score, and expected calibration error (ECE) for the prompt-withheld ladder levels, averaged over per-setting means (±between-setting SD; per-setting means average five data splits). Lower is better for Brier and ECE. ECE uses 10 equal-width probability bins on the held-out test predictions and is reported for the raw logistic-regression scores without post-hoc calibration. BenchmarkLevelAUROCAUPRCBrierECE GPQA-DiamondR0.554± 0.0470.688± 0.0530.249± 0.0180.124± 0.018 R + S0.762± 0.0300.843± 0.0380.226± 0.0110.200± 0.028 R + S + C0.770± 0.0530.842± 0.0510.222± 0.0160.201± 0.013 R +S +C +Z0.784± 0.0470.851± 0.0570.209± 0.0200.204± 0.020 MMLU-ProR0.560± 0.0540.793± 0.0440.190± 0.0190.064± 0.037 R + S0.770± 0.0050.907± 0.0170.160± 0.0110.087± 0.009 R + S + C0.816± 0.0100.931± 0.0110.151± 0.0070.096± 0.019 R +S +C +Z0.855± 0.0100.946± 0.0090.143± 0.0080.118± 0.038 Omni-MATH-2- Filtered R0.755± 0.0750.855± 0.0280.181± 0.0160.107± 0.033 R + S0.806± 0.0330.895± 0.0110.173± 0.0100.115± 0.031 R + S + C0.855± 0.0250.924± 0.0130.154± 0.0120.116± 0.036 R +S +C +Z0.882± 0.0250.937± 0.0060.139± 0.0180.112± 0.048 All benchmarksR0.623± 0.1120.779± 0.0820.207± 0.0350.099± 0.039 R + S0.779± 0.0310.882± 0.0370.186± 0.0310.134± 0.055 R + S + C0.814± 0.0480.899± 0.0510.176± 0.0360.138± 0.053 R +S +C +Z0.840± 0.0520.911± 0.0540.164± 0.0370.145± 0.055 Appendix Table A20: Platt calibration improves the mean probability estimates while leaving aggregate AUROC essentially unchanged. Calibrators are fit on validation predictions and applied to held-out test predictions for each benchmark, model, data split, access level, and feature family. Each estimate averages 75 combinations of 15 model– benchmark settings and five data splits. AUROC is shown before calibration; Brier score and expected calibration error (ECE) are shown before and after calibration. An individual fitted calibrator can reverse a ranking when its validation slope is negative. LevelAUROCBrier beforeBrier afterECE beforeECE after q+R0.7500.2040.1760.1530.068 q+R+S0.7690.1900.1720.1320.068 q+R+S+C0.8090.1760.1590.1300.077 q+R+S+C+Z0.8390.1650.1470.1450.079 R0.6230.2070.1960.0990.050 R+S0.7790.1860.1710.1340.077 R+S+C0.8140.1760.1580.1380.073 R+S+C+Z0.8400.1640.1470.1450.080 40 An Observability Ladder for Large Language Models Appendix Table A21: Selected-answer and option-bias baselines do not explain the public monitor signal. Held-out test AUROC for logistic baselines that see only the selected answer identity, an answer-prior encoding, answer-format indicators, or the selected-answer probability, averaged over five data splits (±SD across data splits). The two rightmost columns repeat the public monitor atRandR+S+Cfor reference. Verification fields computed against the gold answer encode the label and are excluded from all baseline inputs. BenchmarkModelAnswer priorSelected answer Answer format Selected prob. R monitorR+S+C monitor GPQA- Diamond Qwen3-4B0.518± 0.072 0.518± 0.072 0.510± 0.024 0.489± 0.137 0.505± 0.066 0.732± 0.067 GPQA- Diamond Qwen3-8B0.544± 0.123 0.544± 0.123 0.589± 0.072 0.576± 0.055 0.631± 0.047 0.839± 0.062 GPQA- Diamond Qwen3-14B0.545± 0.058 0.545± 0.058 0.516± 0.017 0.658± 0.084 0.558± 0.046 0.800± 0.058 GPQA- Diamond gpt-oss-20b0.526± 0.127 0.560± 0.146 0.500± 0.000 0.625± 0.142 0.535± 0.137 0.707± 0.096 GPQA- Diamond gpt-oss-120b0.532± 0.063 0.538± 0.056 0.500± 0.000 0.662± 0.109 0.542± 0.073 0.770± 0.057 MMLU-ProQwen3-4B0.532± 0.018 0.531± 0.020 0.507± 0.006 0.754± 0.011 0.560± 0.014 0.823± 0.010 MMLU-ProQwen3-8B0.523± 0.016 0.524± 0.017 0.557± 0.014 0.688± 0.015 0.652± 0.014 0.821± 0.004 MMLU-ProQwen3-14B0.524± 0.010 0.523± 0.012 0.508± 0.002 0.760± 0.007 0.533± 0.004 0.825± 0.007 MMLU-Progpt-oss-20b0.512± 0.010 0.511± 0.011 0.504± 0.001 0.652± 0.018 0.515± 0.009 0.803± 0.015 MMLU-Progpt-oss-120b0.532± 0.004 0.522± 0.016 0.501± 0.001 0.590± 0.119 0.538± 0.006 0.810± 0.005 Omni-MATH- 2-Filtered Qwen3-4B0.597± 0.012 0.593± 0.009 0.762± 0.008 0.626± 0.017 0.807± 0.015 0.870± 0.015 Omni-MATH- 2-Filtered Qwen3-8B0.608± 0.023 0.607± 0.022 0.712± 0.025 0.620± 0.012 0.837± 0.027 0.875± 0.022 Omni-MATH- 2-Filtered Qwen3-14B0.610± 0.015 0.606± 0.010 0.671± 0.021 0.639± 0.012 0.779± 0.010 0.876± 0.008 Omni-MATH- 2-Filtered gpt-oss-20b0.563± 0.015 0.567± 0.016 0.639± 0.027 0.604± 0.020 0.682± 0.019 0.824± 0.022 Omni-MATH- 2-Filtered gpt-oss-120b0.568± 0.018 0.568± 0.020 0.618± 0.013 0.607± 0.015 0.671± 0.012 0.832± 0.019 41 An Observability Ladder for Large Language Models Appendix Table A22: Public display-subset AUROC by model–benchmark setting. Each entry is the held-out test AUROC of the public monitor (rate surface + Cohere) for the corresponding visible display, averaged over five data splits (±SD across data splits). The final row averages the per-setting means across all 15 settings (±between-setting SD). BenchmarkModelRSCR + SR + CS + C R + S + C GPQA-DiamondQwen3-4B0.505± 0.066 0.735± 0.058 0.745± 0.065 0.732± 0.063 0.738± 0.078 0.738± 0.062 0.732± 0.067 GPQA-DiamondQwen3-8B0.631± 0.047 0.803± 0.067 0.848± 0.058 0.812± 0.082 0.846± 0.065 0.839± 0.065 0.839± 0.062 GPQA-DiamondQwen3-14B0.558± 0.046 0.746± 0.084 0.781± 0.068 0.748± 0.056 0.791± 0.072 0.787± 0.046 0.800± 0.058 GPQA-Diamondgpt-oss-20b0.535± 0.137 0.701± 0.122 0.725± 0.084 0.755± 0.066 0.717± 0.095 0.708± 0.104 0.707± 0.096 GPQA-Diamondgpt-oss-120b0.542± 0.073 0.737± 0.062 0.752± 0.052 0.761± 0.053 0.743± 0.062 0.760± 0.046 0.770± 0.057 MMLU-ProQwen3-4B0.560± 0.014 0.771± 0.008 0.824± 0.011 0.764± 0.009 0.824± 0.010 0.824± 0.009 0.823± 0.010 MMLU-ProQwen3-8B0.652± 0.014 0.780± 0.010 0.831± 0.005 0.774± 0.013 0.819± 0.005 0.829± 0.005 0.821± 0.004 MMLU-ProQwen3-14B0.533± 0.004 0.780± 0.008 0.826± 0.008 0.775± 0.010 0.824± 0.008 0.826± 0.008 0.825± 0.007 MMLU-Progpt-oss-20b0.515± 0.009 0.772± 0.015 0.800± 0.015 0.767± 0.016 0.801± 0.014 0.803± 0.015 0.803± 0.015 MMLU-Progpt-oss-120b0.538± 0.006 0.771± 0.004 0.810± 0.005 0.769± 0.004 0.808± 0.005 0.811± 0.005 0.810± 0.005 Omni-MATH-2- Filtered Qwen3-4B0.807± 0.015 0.789± 0.016 0.866± 0.016 0.803± 0.018 0.870± 0.016 0.866± 0.017 0.870± 0.015 Omni-MATH-2- Filtered Qwen3-8B0.837± 0.027 0.838± 0.022 0.870± 0.019 0.853± 0.026 0.865± 0.021 0.869± 0.022 0.875± 0.022 Omni-MATH-2- Filtered Qwen3-14B0.779± 0.010 0.825± 0.003 0.868± 0.005 0.821± 0.006 0.874± 0.008 0.868± 0.005 0.876± 0.008 Omni-MATH-2- Filtered gpt-oss-20b0.682± 0.019 0.775± 0.016 0.820± 0.020 0.775± 0.013 0.824± 0.020 0.824± 0.019 0.824± 0.022 Omni-MATH-2- Filtered gpt-oss-120b0.671± 0.012 0.781± 0.023 0.830± 0.017 0.776± 0.022 0.832± 0.018 0.831± 0.019 0.832± 0.019 Mean0.623± 0.112 0.774± 0.035 0.813± 0.045 0.779± 0.031 0.812± 0.048 0.812± 0.047 0.814± 0.048 42 An Observability Ladder for Large Language Models Appendix Table A23: Order-independent Shapley attribution allocates the largest mean AUROC contribution to the trace. Shapley AUROC contributions of the responseR, self-summaryS, and reasoning traceCto the full public display, computed over all3!source orderings with an empty-display baseline of 0.5, averaged over five data splits (± SD across data splits). The contributions of a setting sum to its full public AUROC minus 0.5. The final row averages per-setting means across all 15 settings (± between-setting SD). BenchmarkModelφ(R)φ(S)φ(C)R+S+C AUROC GPQA-DiamondQwen3-4B−0.002± 0.022 +0.113± 0.031 +0.121± 0.053 0.732 GPQA-DiamondQwen3-8B+0.045± 0.018 +0.127± 0.030 +0.167± 0.020 0.839 GPQA-DiamondQwen3-14B+0.026± 0.018 +0.118± 0.035 +0.157± 0.047 0.800 GPQA-Diamondgpt-oss-20b+0.019± 0.064 +0.097± 0.068 +0.090± 0.072 0.707 GPQA-Diamondgpt-oss-120b+0.020± 0.018 +0.126± 0.030 +0.125± 0.042 0.770 MMLU-ProQwen3-4B+0.019± 0.004 +0.124± 0.005 +0.181± 0.007 0.823 MMLU-ProQwen3-8B+0.045± 0.005 +0.114± 0.005 +0.162± 0.008 0.821 MMLU-ProQwen3-14B+0.010± 0.002 +0.134± 0.004 +0.182± 0.007 0.825 MMLU-Progpt-oss-20b+0.004± 0.003 +0.134± 0.007 +0.165± 0.007 0.803 MMLU-Progpt-oss-120b+0.012± 0.002 +0.130± 0.003 +0.169± 0.004 0.810 Omni-MATH-2- Filtered Qwen3-4B+0.107± 0.005 +0.096± 0.006 +0.167± 0.006 0.870 Omni-MATH-2- Filtered Qwen3-8B+0.116± 0.011 +0.119± 0.009 +0.140± 0.006 0.875 Omni-MATH-2- Filtered Qwen3-14B+0.096± 0.006 +0.116± 0.002 +0.164± 0.006 0.876 Omni-MATH-2- Filtered gpt-oss-20b+0.061± 0.007 +0.108± 0.008 +0.155± 0.017 0.824 Omni-MATH-2- Filtered gpt-oss-120b+0.057± 0.004 +0.111± 0.011 +0.164± 0.012 0.832 Mean+0.042± 0.038 +0.118± 0.012 +0.154± 0.025 0.814± 0.048 43 An Observability Ladder for Large Language Models Appendix Table A24: Mean AUROC is similar under generic and preservation-tuned summary instructions. Held-out test AUROC of the public monitor for summary-containing displays when the self-summary is generated with the primary preservation instruction (Pres.) versus a generic instruction requesting compression to at most 90 words (Gen.), averaged over five data splits (± SD across data splits). BenchmarkModelS Pres.S Gen. R + S Pres. R + S Gen. S + C Pres. S + C Gen. R + S + C Pres. R + S + C Gen. GPQA- Diamond Qwen3-4B0.735± 0.058 0.693± 0.046 0.732± 0.063 0.704± 0.047 0.738± 0.062 0.745± 0.060 0.732± 0.067 0.741± 0.070 GPQA- Diamond Qwen3-8B0.803± 0.067 0.821± 0.072 0.812± 0.082 0.810± 0.075 0.839± 0.065 0.839± 0.055 0.839± 0.062 0.838± 0.059 GPQA- Diamond Qwen3-14B0.746± 0.084 0.728± 0.123 0.748± 0.056 0.761± 0.109 0.787± 0.046 0.787± 0.073 0.800± 0.058 0.780± 0.067 GPQA- Diamond gpt-oss-20b0.701± 0.122 0.726± 0.098 0.755± 0.066 0.686± 0.105 0.708± 0.104 0.718± 0.082 0.707± 0.096 0.718± 0.086 GPQA- Diamond gpt-oss-120b0.737± 0.062 0.765± 0.036 0.761± 0.053 0.749± 0.038 0.760± 0.046 0.767± 0.053 0.770± 0.057 0.766± 0.056 MMLU-ProQwen3-4B0.771± 0.008 0.761± 0.013 0.764± 0.009 0.758± 0.012 0.824± 0.009 0.822± 0.010 0.823± 0.010 0.822± 0.011 MMLU-ProQwen3-8B0.780± 0.010 0.778± 0.014 0.774± 0.013 0.771± 0.016 0.829± 0.005 0.830± 0.006 0.821± 0.004 0.822± 0.006 MMLU-ProQwen3-14B0.780± 0.008 0.764± 0.008 0.775± 0.010 0.760± 0.008 0.826± 0.008 0.824± 0.008 0.825± 0.007 0.823± 0.007 MMLU-Progpt-oss-20b0.772± 0.015 0.767± 0.007 0.767± 0.016 0.763± 0.009 0.803± 0.015 0.803± 0.013 0.803± 0.015 0.802± 0.013 MMLU-Progpt-oss-120b0.771± 0.004 0.754± 0.007 0.769± 0.004 0.751± 0.007 0.811± 0.005 0.809± 0.005 0.810± 0.005 0.809± 0.003 Omni-MATH- 2-Filtered Qwen3-4B0.789± 0.016 0.783± 0.016 0.803± 0.018 0.802± 0.016 0.866± 0.017 0.865± 0.016 0.870± 0.015 0.869± 0.015 Omni-MATH- 2-Filtered Qwen3-8B0.838± 0.022 0.836± 0.022 0.853± 0.026 0.857± 0.028 0.869± 0.022 0.866± 0.020 0.875± 0.022 0.874± 0.020 Omni-MATH- 2-Filtered Qwen3-14B0.825± 0.003 0.808± 0.010 0.821± 0.006 0.810± 0.007 0.868± 0.005 0.869± 0.003 0.876± 0.008 0.873± 0.007 Omni-MATH- 2-Filtered gpt-oss-20b0.775± 0.016 0.770± 0.023 0.775± 0.013 0.774± 0.021 0.824± 0.019 0.824± 0.021 0.824± 0.022 0.829± 0.017 Omni-MATH- 2-Filtered gpt-oss-120b0.781± 0.023 0.771± 0.021 0.776± 0.022 0.765± 0.014 0.831± 0.019 0.831± 0.019 0.832± 0.019 0.832± 0.020 Appendix Table A25: A fixed primary monitor is stable across three sampled summaries and one greedy summary. For every setting and access level, regularization is selected on the first data split’s training and validation items after excluding the balanced 48-item subset, then the surface-plus-Cohere monitor is refit once on the full complement. AUROC columns report the mean across the three sampled summaries and observed settings; brackets are hierarchical 95% bootstrap intervals that resample items within the six correctness-by-trace-length strata and resample observed settings. SD, range, and MAPD summarize the three sampled∆ S values, where MAPD is the mean absolute pairwise difference. Greedy−sampled compares the greedy∆ S with the sampled mean. Equal-stratum AUROC is primary; IPW uses the recorded inverse selection probabilities. ScopeWeightingS AUROC R+S AUROC∆ S SDRangeMAPDGreedy− sampled Settings GPQA- Diamond Equal- stratum 0.686 [0.596, 0.780] 0.707 [0.622, 0.786] +0.077 [−0.015, +0.171] 0.0170.0330.022+0.017 [−0.001, +0.038] 5 IPW0.779 [0.695, 0.859] 0.797 [0.736, 0.859] +0.149 [+0.055, +0.248] 0.0150.0270.018+0.015 [−0.001, +0.033] 5 MMLU-ProEqual- stratum 0.757 [0.670, 0.834] 0.744 [0.661, 0.827] +0.153 [+0.052, +0.254] 0.0240.0450.030+0.009 [−0.024, +0.044] 5 IPW0.787 [0.711, 0.855] 0.769 [0.695, 0.840] +0.181 [+0.075, +0.284] 0.0310.0590.039+0.007 [−0.028, +0.042] 5 Omni-MATH-2- Filtered Equal- stratum 0.653 [0.555, 0.763] 0.675 [0.552, 0.799] +0.027 [−0.039, +0.089] 0.0090.0170.012+0.003 [−0.014, +0.020] 5 IPW0.770 [0.685, 0.848] 0.781 [0.674, 0.881] +0.051 [−0.015, +0.117] 0.0090.0180.012+0.001 [−0.019, +0.018] 5 All settingsEqual- stratum 0.699 [0.635, 0.756] 0.708 [0.650, 0.765] +0.086 [+0.027, +0.146] 0.0170.0320.021+0.010 [−0.005, +0.024] 15 IPW0.779 [0.732, 0.820] 0.782 [0.731, 0.830] +0.127 [+0.066, +0.187] 0.0180.0340.023+0.008 [−0.008, +0.023] 15 44 An Observability Ladder for Large Language Models Appendix Table A26: The exact 90-word cap and conclusion-region controls use the primary surface-plus-Cohere monitor. Each AUROC entry is the mean±between-setting SD of the setting-level held-out test AUROC, where each setting mean averages the five prespecified data splits.∆is paired within setting against the original uncapped preservation summary. Cap90 deterministically truncates that summary to at most 90 whitespace-delimited words; Last=n S uses up to the finaln S trace words, wheren S is the capped summary length; Last90 uses up to the final 90 trace words and therefore the whole trace when it is shorter; and Last90 masked removes answer, option, and prompt-number cues before extraction. The final row reports Last90 minus Pres. for theSdisplay; brackets are descriptive 95% setting-bootstrap intervals after averaging the five data splits within each of the 15 observed settings. Summary slotS AUROCS ∆ R+S AUROCR+S ∆n Pres. (uncapped)0.774± 0.0350.000± 0.0000.779± 0.0310.000± 0.00015 Cap900.775± 0.026+0.001±0.0180.774± 0.034 −0.005± 0.007 15 Last=n S 0.784± 0.028+0.010±0.0170.779± 0.0290.000± 0.01215 Last900.780± 0.032 +0.006±0.0190.778± 0.033 −0.001± 0.014 15 Last90 masked0.782± 0.031+0.009±0.0220.780± 0.032+0.001±0.01715 Last90− Pres.— +0.006 [−0.003, +0.016]—15 45 An Observability Ladder for Large Language Models Appendix Table A27: Length-matched extractive controls compare summaries with trace extracts at the same word count. Held-out test AUROC of the open surface-plus-TF-IDF monitor for the summary-only display. Pres. uses the original preservation-tuned self-summary; Cap90 hard-truncates that summary to at most 90 words; First=nS, Last=nS, Head+Tail=nS, and Random=nS use trace words matched to the capped summary lengthn S ; Last90 uses up to the final 90 trace words and therefore the whole trace when it is shorter; Last90 masked applies answer, option, and prompt-number masking before the same extraction. Values average five data splits (±SD across data splits). The matched mean averages settings where every displayed variant is available. The two contrasts report Last=n S minus Cap90 and Last90 minus Pres.; brackets are descriptive 95% setting-bootstrap intervals after averaging the five data splits within each of the 15 matched settings. BenchmarkModelS Pres. S Cap90 S First=nS S Last=nSS Head+Tail=nS S Ran- dom=nS S Last90 S Last90 masked GPQA- Diamond Qwen3-4B0.663± 0.035 0.659± 0.034 0.644± 0.066 0.716± 0.058 0.646± 0.118 0.769± 0.092 0.741± 0.071 0.720± 0.075 GPQA- Diamond Qwen3-8B0.750± 0.047 0.769± 0.050 0.768± 0.092 0.833± 0.073 0.829± 0.054 0.771± 0.038 0.830± 0.080 0.842± 0.058 GPQA- Diamond Qwen3-14B0.779± 0.081 0.778± 0.073 0.720± 0.053 0.774± 0.094 0.715± 0.065 0.820± 0.104 0.785± 0.088 0.773± 0.084 GPQA- Diamond gpt-oss-20b0.700± 0.080 0.697± 0.074 0.722± 0.054 0.738± 0.114 0.713± 0.108 0.762± 0.084 0.737± 0.117 0.742± 0.113 GPQA- Diamond gpt-oss-120b0.736± 0.100 0.735± 0.099 0.718± 0.090 0.807± 0.036 0.714± 0.084 0.760± 0.093 0.767± 0.047 0.784± 0.066 MMLU-ProQwen3-4B0.802± 0.006 0.802± 0.006 0.734± 0.005 0.814± 0.009 0.796± 0.005 0.772± 0.009 0.824± 0.010 0.826± 0.010 MMLU-ProQwen3-8B0.815± 0.009 0.803± 0.010 0.745± 0.008 0.820± 0.006 0.815± 0.010 0.795± 0.009 0.821± 0.007 0.823± 0.006 MMLU-ProQwen3-14B0.813± 0.006 0.811± 0.007 0.745± 0.005 0.817± 0.007 0.805± 0.010 0.777± 0.009 0.819± 0.004 0.821± 0.005 MMLU-Progpt-oss-20b0.788± 0.014 0.784± 0.015 0.749± 0.014 0.798± 0.013 0.790± 0.012 0.774± 0.010 0.801± 0.011 0.805± 0.010 MMLU-Progpt-oss-120b0.802± 0.008 0.801± 0.009 0.756± 0.005 0.807± 0.009 0.789± 0.004 0.788± 0.005 0.812± 0.009 0.814± 0.009 Omni-MATH- 2-Filtered Qwen3-4B0.813± 0.009 0.799± 0.009 0.748± 0.013 0.845± 0.013 0.839± 0.014 0.767± 0.011 0.851± 0.006 0.838± 0.016 Omni-MATH- 2-Filtered Qwen3-8B0.849± 0.020 0.834± 0.018 0.806± 0.025 0.874± 0.014 0.868± 0.011 0.810± 0.020 0.866± 0.014 0.871± 0.013 Omni-MATH- 2-Filtered Qwen3-14B0.845± 0.006 0.832± 0.006 0.771± 0.007 0.853± 0.017 0.845± 0.015 0.773± 0.007 0.840± 0.014 0.849± 0.019 Omni-MATH- 2-Filtered gpt-oss-20b0.784± 0.005 0.785± 0.008 0.727± 0.018 0.773± 0.016 0.766± 0.021 0.755± 0.025 0.780± 0.012 0.767± 0.014 Omni-MATH- 2-Filtered gpt-oss-120b0.795± 0.021 0.789± 0.021 0.743± 0.021 0.786± 0.015 0.789± 0.014 0.768± 0.016 0.775± 0.021 0.772± 0.014 Matched mean (15 settings) 0.7820.7780.7400.8040.7810.7770.8030.803 Last=n S − Cap90 +0.025 [+0.013, +0.038] Last90− Pres. +0.021 [+0.008, +0.036] Appendix Table A28: Trace-native chunk monitors are compared directly with the matched summary and whole-trace monitors. AUROC values are means±between-setting SD after first averaging the five prespecified data splits within each of the 15 model–benchmark settings. Deltas are paired by setting against the primary whole-display summary (S) and trace (C) monitors. Each full trace is divided into balanced contiguous word chunks targeting 512 words and capped at eight chunks. The Cohere variants pool fixed 512-dimensional chunk vectors or fit a linear-attention multiple-instance head. Optional local TF–IDF variants are labelled separately as weak-label alternatives. MonitorAUROC∆ vs S∆ vs CSettings Primary whole-display S0.774± 0.035+0.000± 0.000 −0.039± 0.01715 Primary whole-display C0.813± 0.045+0.039± 0.017+0.000± 0.00015 Cohere chunks: mean pooling0.812± 0.048+0.038± 0.020 −0.001± 0.00815 Cohere chunks: elementwise max 0.811± 0.044+0.038± 0.019 −0.002± 0.01015 Cohere chunks: elementwise top-2 0.813± 0.043+0.040± 0.018+0.000± 0.01115 Cohere chunks: attention MIL0.792± 0.049+0.018± 0.021 −0.021± 0.01315 46 An Observability Ladder for Large Language Models Appendix Table A29: Scores from trace segments are compared with scores from complete summaries and traces on the full dataset. AUROC values first average the five prespecified random splits within each model–benchmark combination and then report the mean±SD across all 15 combinations. Deltas compare values within the same combi- nation with the complete summary and trace, using the same surface features and TF–IDF features. The segment and complete-text classifiers use the same six candidate inverse-regularization values,C ∈0.0001, 0.001, 0.01, 0.1, 1, 10. Each saved trace is divided into up to eight contiguous segments of similar length, targeting 512 words. The segment classifier is trained on all items in the saved split rather than the earlier 500-item subset based on one random split. Scores are combined only for held-out items. Within each combination and random split,Cis chosen once according to validation item-level AUROC after taking the maximum segment score for each item. The top-two mean, mean, and last-segment results use that same value ofC. These alternatives test sensitivity to how traces are divided and segment scores are combined; they do not locate causal evidence. A short trace may form one segment; during fitting, each item is weighted once for every segment it contains; and maximum or top-two aggregation gives longer traces more opportunities to receive a high score. Input and scoreAUROC∆ vs S∆ vs CSettings Whole-display summary S0.782± 0.051+0.000± 0.000 −0.036± 0.02115 Whole-display trace C0.818± 0.051+0.036± 0.021+0.000± 0.00015 Segment scores: maximum0.799± 0.051+0.017± 0.029 −0.019± 0.01615 Segment scores: top-two mean0.803± 0.051+0.021± 0.024 −0.015± 0.01315 Segment scores: mean0.818± 0.046+0.036± 0.026+0.000± 0.01215 Last (or only) segment0.817± 0.043+0.035± 0.026 −0.000± 0.01515 Appendix Table A30: Public-text feature comparisons. AUROC values are mean±descriptive SD across the model–benchmark settings and data splits available at each public access level. These SDs are not confidence intervals. All displayed feature families cover the 15 model–benchmark settings. The final column gives the number of model– benchmark–split estimates. Access levelFeature familyAUROCEstimates RSurface0.584± 0.09875 RNo length0.579± 0.09275 RCohere0.620± 0.11375 RSurface + Cohere0.623± 0.11775 RTF-IDF0.618± 0.12175 RSurface + TF-IDF + Cohere0.623± 0.11775 R + SSurface0.722± 0.05475 R + SNo length0.701± 0.05275 R + SCohere0.773± 0.04675 R + SSurface + Cohere0.779± 0.04775 R + STF-IDF0.797± 0.05075 R + SSurface + TF-IDF + Cohere0.784± 0.04775 R + S + CSurface0.803± 0.05075 R + S + CNo length0.777± 0.04375 R + S + CCohere0.802± 0.05975 R + S + CSurface + Cohere0.814± 0.06075 R + S + CTF-IDF0.815± 0.04875 R + S + CSurface + TF-IDF + Cohere0.814± 0.06075 47 An Observability Ladder for Large Language Models Appendix Table A31: Correctness ranking remains after adjusting the other features for text length. For every setting, access level, and data split, word and sentence counts are nuisance covariates. Inside each fitting fold, multi- output least squares residualizes every other primary surface and Cohere feature on those two counts; the counts are then excluded before standardization and logistic-probe fitting. Values average the five data splits within each setting and then give equal weight to the 15 settings. Change is the adjusted estimate minus the original primary-monitor estimate. QuantityPrimary monitorAfter length adjustment Change Access-level AUROC R0.6230.617−0.006 S0.7740.689−0.084 C0.8130.716−0.097 R+S0.7790.734−0.045 R+S+C0.8140.717−0.097 q+R0.7500.727−0.023 q+R+S0.7690.736−0.033 q+R+C0.8120.712−0.100 q+R+S+C0.8090.713−0.097 Paired AUROC increments ∆ S 0.1560.117−0.039 ∆ C 0.035−0.017−0.052 ∆ public 0.1910.101−0.090 q∆ S 0.0190.010−0.009 q∆ C|R 0.062−0.015−0.077 q∆ C 0.041−0.023−0.064 q∆ S|R,C −0.0020.001+0.003 q(C−S)0.043−0.024−0.067 q∆ public 0.060−0.014−0.073 Appendix Table A32: Public embeddings fit without silent truncation. Cohere embed-v4.0 was applied with silent truncation disabled. Counts refer to items at the full public levelR + S + C, pooled across the five target models for each benchmark. BenchmarkR +S +CitemsWhole-context fitMean chunksMax chunks GPQA-Diamond990100%1.001 MMLU-Pro60,160100%1.001 Omni-MATH-2- Filtered 20,905100%1.001 48 An Observability Ladder for Large Language Models Appendix Table A33: Stripping prompt-overlapping text from traces and summaries leaves public AUROC essentially unchanged. Paired AUROC change when 8–14-word spans copied from the task prompt are removed from the visible summary and trace before feature extraction, averaged over five data splits (±SD across data splits). The final row averages per-setting means across all 15 settings (± between-setting SD). BenchmarkModelSCR+SR+CS+CR+S+C GPQA-DiamondQwen3-4B+0.006± 0.020 −0.004± 0.019 +0.016± 0.014 −0.008± 0.019 −0.005± 0.012 −0.003± 0.021 GPQA-DiamondQwen3-8B−0.003± 0.010 +0.007± 0.012 −0.006± 0.010 +0.004± 0.005 +0.003± 0.005 +0.000± 0.009 GPQA-DiamondQwen3-14B+0.019± 0.046 +0.014± 0.030 −0.013± 0.019 −0.019± 0.033 −0.014± 0.053 −0.021± 0.037 GPQA-Diamondgpt-oss-20b−0.027± 0.067 −0.004± 0.029 −0.030± 0.080 +0.007± 0.042 +0.002± 0.014 −0.001± 0.020 GPQA-Diamondgpt-oss-120b+0.005± 0.016 −0.002± 0.020 +0.004± 0.007 −0.002± 0.032 +0.005± 0.020 −0.007± 0.019 MMLU-ProQwen3-4B+0.000± 0.001 +0.000± 0.003 +0.000± 0.000 +0.000± 0.003 −0.000± 0.002 −0.000± 0.002 MMLU-ProQwen3-8B−0.000± 0.001 +0.001± 0.001 +0.000± 0.000 +0.001± 0.001 −0.000± 0.001 −0.000± 0.001 MMLU-ProQwen3-14B+0.000± 0.001 +0.000± 0.001 +0.001± 0.001 −0.000± 0.001 +0.000± 0.001 −0.001± 0.001 MMLU-Progpt-oss-20b+0.002± 0.000 −0.000± 0.003 +0.002± 0.000 −0.001± 0.002 −0.001± 0.001 −0.001± 0.001 MMLU-Progpt-oss-120b−0.000± 0.000 +0.000± 0.002 −0.001± 0.001 −0.001± 0.002 +0.001± 0.003 +0.000± 0.003 Omni-MATH-2- Filtered Qwen3-4B−0.001± 0.003 −0.001± 0.002 +0.004± 0.002 +0.001± 0.001 +0.002± 0.003 +0.000± 0.002 Omni-MATH-2- Filtered Qwen3-8B+0.001± 0.002 −0.001± 0.002 +0.001± 0.001 −0.003± 0.005 +0.003± 0.002 +0.000± 0.002 Omni-MATH-2- Filtered Qwen3-14B−0.001± 0.002 +0.003± 0.003 +0.001± 0.001 −0.001± 0.001 +0.003± 0.001 +0.001± 0.002 Omni-MATH-2- Filtered gpt-oss-20b−0.005± 0.008 −0.001± 0.007 −0.001± 0.001 +0.001± 0.002 −0.001± 0.007 −0.000± 0.002 Omni-MATH-2- Filtered gpt-oss-120b−0.003± 0.007 +0.000± 0.003 +0.001± 0.001 −0.002± 0.001 +0.000± 0.002 +0.001± 0.001 Mean−0.000± 0.009 +0.001± 0.005 −0.001± 0.010 −0.001± 0.006 −0.000± 0.005 −0.002± 0.006 49 An Observability Ladder for Large Language Models Appendix Table A34: Masking of answers, options, and prompt-derived numbers leaves public-text AUROC essentially unchanged. Paired AUROC change relative to the unmasked display when answer spans and choice letters, option texts, and prompt-derived numeric constants are replaced by fixed markers before feature extraction (maskall applies all three), averaged over five data splits (±SD across data splits). The three rightmost columns ablate one component at a time on the trace display. Masking bounds the direct and near-direct reconstruction channel, but it does not remove paraphrase or item-level cues and can introduce marker artefacts. The final row averages per-setting means across settings (± between-setting SD). BenchmarkModelS maskall C maskallR+S+C maskall C (options) C (answer) C (numbers) GPQA- Diamond Qwen3-4B −0.012± 0.027 −0.008± 0.008 −0.002± 0.013 −0.013± 0.022 +0.006± 0.011 −0.012± 0.009 GPQA- Diamond Qwen3-8B+0.011± 0.025 +0.010± 0.008 −0.002± 0.008 +0.007± 0.006 +0.004± 0.005 +0.004± 0.004 GPQA- Diamond Qwen3-14B+0.033± 0.030 +0.004± 0.011 +0.010± 0.015 −0.002± 0.034 −0.007± 0.032 −0.001± 0.026 GPQA- Diamond gpt-oss-20b+0.009± 0.008 −0.007± 0.039 +0.002± 0.007 −0.012± 0.017 −0.003± 0.029 −0.026± 0.044 GPQA- Diamond gpt-oss-120b+0.020± 0.028 +0.001± 0.010 +0.003± 0.005 +0.002± 0.009 +0.004± 0.009 +0.010± 0.006 MMLU-ProQwen3-4B+0.001± 0.002 +0.002± 0.001 +0.002± 0.001 −0.000± 0.001 +0.001± 0.001 +0.001± 0.001 MMLU-ProQwen3-8B+0.000± 0.002 +0.002± 0.001 +0.003± 0.001 +0.000± 0.001 +0.001± 0.001 +0.000± 0.001 MMLU-ProQwen3-14B −0.000± 0.002 +0.000± 0.002 +0.000± 0.003 −0.000± 0.001 +0.000± 0.001 −0.000± 0.001 MMLU-Progpt-oss-20b+0.003± 0.002 +0.002± 0.002 +0.003± 0.001 +0.000± 0.001 +0.001± 0.001 +0.001± 0.001 MMLU-Progpt-oss-120b+0.003± 0.003 −0.001± 0.004 +0.002± 0.002 +0.000± 0.001 −0.001± 0.001 +0.000± 0.003 Omni-MATH- 2-Filtered Qwen3-4B+0.002± 0.005 +0.000± 0.002 −0.002± 0.002 +0.000± 0.000 −0.001± 0.001 −0.001± 0.001 Omni-MATH- 2-Filtered Qwen3-8B −0.001± 0.002 −0.001± 0.003 −0.001± 0.001 +0.000± 0.000 −0.002± 0.002 +0.000± 0.003 Omni-MATH- 2-Filtered Qwen3-14B+0.003± 0.003 +0.002± 0.003 +0.001± 0.003 +0.000± 0.000 −0.000± 0.001 +0.002± 0.003 Omni-MATH- 2-Filtered gpt-oss-20b+0.002± 0.004 +0.003± 0.001 +0.001± 0.002 +0.000± 0.000 +0.000± 0.001 +0.002± 0.001 Omni-MATH- 2-Filtered gpt-oss-120b −0.001± 0.003 +0.000± 0.002 −0.004± 0.007 −0.000± 0.000 −0.001± 0.001 +0.001± 0.002 Mean+0.005± 0.010 +0.001± 0.004 +0.001± 0.003 −0.001± 0.005 +0.000± 0.003 −0.001± 0.008 50 An Observability Ladder for Large Language Models Appendix Table A35: Adding the task prompt does not improve mean full-display AUROC. Held-out test AUROC of the public monitor for prompt-conditioned control displays, averaged over five data splits (±SD across data splits), with the output-onlyR + S + Cdisplay repeated for reference. The final row averages per-setting means across all 15 settings (± between-setting SD). BenchmarkModelR + S + Cqq + Rq + Sq + C q + R + S q + R + C q + S + C q +R+S +C GPQA-DiamondQwen3-4B0.732± 0.067 0.665± 0.077 0.685± 0.083 0.701± 0.080 0.727± 0.079 0.706± 0.066 0.731± 0.084 0.724± 0.078 0.725± 0.084 GPQA-DiamondQwen3-8B0.839± 0.062 0.815± 0.070 0.812± 0.067 0.818± 0.069 0.840± 0.058 0.819± 0.071 0.849± 0.061 0.839± 0.056 0.836± 0.058 GPQA-DiamondQwen3-14B0.800± 0.058 0.749± 0.056 0.750± 0.081 0.746± 0.068 0.767± 0.064 0.742± 0.069 0.772± 0.067 0.763± 0.058 0.758± 0.056 GPQA-Diamondgpt-oss-20b0.707± 0.096 0.753± 0.076 0.747± 0.087 0.753± 0.079 0.759± 0.080 0.745± 0.081 0.749± 0.086 0.721± 0.084 0.723± 0.077 GPQA-Diamondgpt-oss-120b0.770± 0.057 0.740± 0.066 0.727± 0.068 0.717± 0.059 0.742± 0.053 0.722± 0.063 0.740± 0.063 0.742± 0.048 0.747± 0.059 MMLU-ProQwen3-4B0.823± 0.010 0.730± 0.013 0.733± 0.018 0.758± 0.014 0.824± 0.009 0.761± 0.010 0.824± 0.010 0.825± 0.009 0.824± 0.009 MMLU-ProQwen3-8B0.821± 0.004 0.714± 0.014 0.736± 0.014 0.768± 0.013 0.821± 0.006 0.773± 0.011 0.820± 0.005 0.826± 0.004 0.822± 0.004 MMLU-ProQwen3-14B0.825± 0.007 0.735± 0.011 0.731± 0.010 0.772± 0.006 0.825± 0.007 0.768± 0.008 0.825± 0.007 0.826± 0.006 0.826± 0.006 MMLU-Progpt-oss-20b0.803± 0.015 0.710± 0.011 0.712± 0.009 0.748± 0.014 0.802± 0.014 0.748± 0.014 0.802± 0.014 0.804± 0.015 0.804± 0.015 MMLU-Progpt-oss-120b0.810± 0.005 0.702± 0.007 0.702± 0.005 0.749± 0.002 0.807± 0.005 0.747± 0.004 0.806± 0.005 0.810± 0.003 0.809± 0.004 Omni-MATH-2- Filtered Qwen3-4B0.870± 0.015 0.751± 0.021 0.805± 0.017 0.790± 0.019 0.868± 0.015 0.803± 0.020 0.871± 0.015 0.868± 0.014 0.870± 0.015 Omni-MATH-2- Filtered Qwen3-8B0.875± 0.022 0.746± 0.024 0.825± 0.028 0.839± 0.023 0.874± 0.016 0.844± 0.029 0.866± 0.019 0.877± 0.021 0.870± 0.017 Omni-MATH-2- Filtered Qwen3-14B0.876± 0.008 0.721± 0.007 0.796± 0.008 0.817± 0.006 0.865± 0.006 0.813± 0.008 0.864± 0.007 0.870± 0.004 0.871± 0.006 Omni-MATH-2- Filtered gpt-oss-20b0.824± 0.022 0.735± 0.016 0.740± 0.017 0.766± 0.012 0.822± 0.018 0.764± 0.015 0.825± 0.014 0.823± 0.017 0.824± 0.016 Omni-MATH-2- Filtered gpt-oss-120b0.832± 0.019 0.742± 0.014 0.743± 0.006 0.779± 0.019 0.832± 0.015 0.775± 0.016 0.832± 0.015 0.830± 0.018 0.832± 0.016 Mean0.814± 0.048 0.734± 0.032 0.750± 0.042 0.768± 0.037 0.812± 0.045 0.769± 0.038 0.812± 0.045 0.810± 0.050 0.809± 0.050 Appendix Table A36: Displays that include the trace add signal across the natural mix of questions, beyond the prompt-only baseline. Paired held-out AUROC contrasts of prompt-conditioned public displays against the prompt-only and output-only baselines, averaged over five data splits (±SD across data splits). The final row averages per-setting means across all 15 settings (± between-setting SD). BenchmarkModelq + C− q q + R + S + C− q + C− C q + R + S + C− R + S + C GPQA-DiamondQwen3-4B+0.062± 0.014+0.060± 0.015 −0.018± 0.019 −0.008± 0.025 GPQA-DiamondQwen3-8B+0.025± 0.028+0.021± 0.034 −0.008± 0.009 −0.003± 0.011 GPQA-DiamondQwen3-14B+0.019± 0.029+0.009± 0.018 −0.013± 0.035 −0.042± 0.027 GPQA-Diamondgpt-oss-20b+0.007± 0.034 −0.029± 0.044+0.035± 0.069+0.017± 0.028 GPQA-Diamondgpt-oss-120b+0.002± 0.017+0.007± 0.024 −0.010± 0.025 −0.023± 0.018 MMLU-ProQwen3-4B+0.094± 0.008+0.094± 0.008 −0.000± 0.002+0.000± 0.002 MMLU-ProQwen3-8B+0.107± 0.015+0.108± 0.014 −0.010± 0.003+0.001± 0.001 MMLU-ProQwen3-14B+0.090± 0.012+0.091± 0.012 −0.001± 0.003+0.000± 0.002 MMLU-Progpt-oss-20b+0.092± 0.008+0.094± 0.009+0.003± 0.001+0.001± 0.002 MMLU-Progpt-oss-120b+0.105± 0.008+0.107± 0.007 −0.002± 0.002 −0.001± 0.003 Omni-MATH-2- Filtered Qwen3-4B+0.116± 0.016+0.119± 0.015+0.002± 0.003+0.001± 0.001 Omni-MATH-2- Filtered Qwen3-8B+0.128± 0.014+0.124± 0.015+0.003± 0.005 −0.006± 0.006 Omni-MATH-2- Filtered Qwen3-14B+0.143± 0.009+0.150± 0.008 −0.004± 0.006 −0.005± 0.003 Omni-MATH-2- Filtered gpt-oss-20b+0.087± 0.022+0.089± 0.021+0.002± 0.008 −0.000± 0.012 Omni-MATH-2- Filtered gpt-oss-120b+0.090± 0.016+0.090± 0.017+0.002± 0.004 −0.000± 0.003 Mean+0.078± 0.045+0.076± 0.051 −0.001± 0.012 −0.004± 0.013 51 An Observability Ladder for Large Language Models Appendix Table A37: Prompt-conditioned Shapley attribution averages each source over every addition order. Shapley AUROC contributions of the responseR, self-summaryS, and reasoning traceCconditional on the promptq, computed from the complete2 3 prompt-conditioned factorial over all3!source orderings. Values average five data splits (±SD across data splits). The contributions of a setting sum to theq+R+S+CAUROC minus itsq-only AUROC. The final row averages per-setting means across all 15 settings (± between-setting SD). BenchmarkModelφ(R)φ(S)φ(C)q+R+S+C AUROC GPQA-DiamondQwen3-4B+0.008± 0.010 +0.013± 0.006 +0.039± 0.012 0.725 GPQA-DiamondQwen3-8B0.000± 0.007 −0.002± 0.009 +0.024± 0.026 0.836 GPQA-DiamondQwen3-14B−0.001± 0.011 −0.008± 0.025 +0.018± 0.032 0.758 GPQA-Diamondgpt-oss-20b−0.004± 0.006 −0.015± 0.033 −0.010± 0.021 0.723 GPQA-Diamondgpt-oss-120b−0.002± 0.011 −0.006± 0.012 +0.015± 0.011 0.747 MMLU-ProQwen3-4B+0.001± 0.003 +0.014± 0.003 +0.079± 0.006 0.824 MMLU-ProQwen3-8B+0.007± 0.002 +0.026± 0.004 +0.076± 0.014 0.822 MMLU-ProQwen3-14B−0.002± 0.001 +0.019± 0.003 +0.074± 0.010 0.826 MMLU-Progpt-oss-20b+0.001± 0.002 +0.019± 0.004 +0.074± 0.007 0.804 MMLU-Progpt-oss-120b−0.001± 0.001 +0.025± 0.003 +0.083± 0.006 0.809 Omni-MATH-2- Filtered Qwen3-4B+0.021± 0.006 +0.012± 0.002 +0.085± 0.009 0.870 Omni-MATH-2- Filtered Qwen3-8B+0.024± 0.005 +0.036± 0.006 +0.064± 0.010 0.870 Omni-MATH-2- Filtered Qwen3-14B+0.025± 0.004 +0.038± 0.003 +0.087± 0.006 0.871 Omni-MATH-2- Filtered gpt-oss-20b+0.002± 0.003 +0.014± 0.007 +0.072± 0.017 0.824 Omni-MATH-2- Filtered gpt-oss-120b+0.001± 0.004 +0.017± 0.005 +0.072± 0.015 0.832 Mean+0.005± 0.010 +0.013± 0.015 +0.057± 0.031 0.809± 0.050 52 An Observability Ladder for Large Language Models Appendix Table A38: An open prompt-conditioned factorial covers every subset of response, summary, and trace. Held-out AUROC for the surface-feature-plus-TF-IDF monitor, conditional on the task promptq. Values average five data splits (±SD across data splits). The final row averages the 15 setting means (±between-setting SD). This is an open-encoder control, not the primary surface-plus-Cohere monitor. BenchmarkModelqq + Rq + Sq + C q + R + S q + R + C q + S + C q + R + S + C GPQA- Diamond Qwen3-4B0.646± 0.048 0.663± 0.061 0.667± 0.028 0.721± 0.057 0.679± 0.041 0.722± 0.057 0.724± 0.060 0.724± 0.060 GPQA- Diamond Qwen3-8B0.754± 0.069 0.763± 0.047 0.773± 0.061 0.825± 0.025 0.720± 0.055 0.827± 0.032 0.825± 0.027 0.824± 0.032 GPQA- Diamond Qwen3-14B0.654± 0.111 0.649± 0.118 0.737± 0.116 0.796± 0.048 0.735± 0.120 0.797± 0.048 0.798± 0.053 0.799± 0.054 GPQA- Diamond gpt-oss-20b0.671± 0.132 0.668± 0.134 0.717± 0.080 0.722± 0.078 0.716± 0.081 0.721± 0.078 0.723± 0.073 0.724± 0.073 GPQA- Diamond gpt-oss-120b0.662± 0.064 0.662± 0.064 0.736± 0.080 0.788± 0.035 0.735± 0.080 0.790± 0.034 0.786± 0.031 0.786± 0.033 MMLU-ProQwen3-4B0.711± 0.020 0.718± 0.022 0.789± 0.005 0.841± 0.008 0.790± 0.007 0.841± 0.008 0.841± 0.008 0.841± 0.008 MMLU-ProQwen3-8B0.702± 0.014 0.741± 0.014 0.795± 0.009 0.843± 0.008 0.799± 0.011 0.844± 0.007 0.845± 0.007 0.844± 0.007 MMLU-ProQwen3-14B0.710± 0.008 0.712± 0.007 0.793± 0.005 0.837± 0.006 0.794± 0.004 0.837± 0.006 0.837± 0.007 0.838± 0.007 MMLU-Progpt-oss-20b0.693± 0.012 0.692± 0.015 0.768± 0.015 0.806± 0.013 0.768± 0.015 0.806± 0.013 0.808± 0.012 0.808± 0.012 MMLU-Progpt-oss-120b0.690± 0.007 0.690± 0.007 0.780± 0.005 0.811± 0.003 0.781± 0.005 0.811± 0.003 0.813± 0.003 0.813± 0.003 Omni-MATH- 2-Filtered Qwen3-4B0.740± 0.011 0.806± 0.014 0.809± 0.008 0.877± 0.013 0.815± 0.014 0.876± 0.012 0.876± 0.013 0.876± 0.012 Omni-MATH- 2-Filtered Qwen3-8B0.745± 0.020 0.843± 0.015 0.849± 0.015 0.885± 0.010 0.844± 0.019 0.884± 0.011 0.884± 0.011 0.883± 0.011 Omni-MATH- 2-Filtered Qwen3-14B0.707± 0.017 0.794± 0.012 0.835± 0.010 0.871± 0.005 0.822± 0.008 0.868± 0.005 0.872± 0.006 0.872± 0.004 Omni-MATH- 2-Filtered gpt-oss-20b0.704± 0.015 0.727± 0.024 0.776± 0.006 0.818± 0.021 0.784± 0.008 0.818± 0.020 0.819± 0.021 0.818± 0.020 Omni-MATH- 2-Filtered gpt-oss-120b0.718± 0.011 0.738± 0.015 0.790± 0.018 0.833± 0.019 0.784± 0.018 0.834± 0.020 0.834± 0.019 0.834± 0.020 Mean0.700± 0.032 0.724± 0.058 0.774± 0.046 0.818± 0.048 0.771± 0.045 0.818± 0.048 0.819± 0.048 0.819± 0.047 Appendix Table A39: The complete open factorial shows context-dependent marginal gains. Mean paired AUROC changes across the 15 settings for the surface-feature-plus-TF-IDF monitor. Intervals are descriptive 95% setting- bootstrap intervals over the observed settings, not population-generalization intervals; the last column counts settings with a positive mean contrast. Summary-first and trace-first marginal gains use the same held-out items and data splits. Here, ‘after’ denotes set inclusion in the factorial contrast, not a change in the physical presentation order of the displayed text. ContrastMean ∆ AUROC 95% descriptive intervalPositive settings Add S after q+R+0.047[+0.027, +0.064]14/15 Add C after q+R+S+0.048[+0.039, +0.059]15/15 Add C after q+R+0.094[+0.078, +0.110]15/15 Add S after q+R+C+0.000[−0.001, +0.001]12/15 q+R+C minus q+R+S+0.047[+0.038, +0.058]15/15 q+R+S+C minus q+R+0.094[+0.079, +0.110]15/15 53 An Observability Ladder for Large Language Models Appendix Table A40: Open prompt-conditioned Shapley values average marginal gains over every inclusion order. Contributions of responseR, summaryS, and traceCare computed from the complete surface-feature-plus-TF-IDF factorial, withqas the baseline. Values average five data splits (±SD across data splits); the final row averages the 15 setting means (±between-setting SD). Contributions sum to the public gain overqbut remain an attribution over fitted probes rather than an information decomposition. Inclusion order is an attribution path over component sets, not presentation order. BenchmarkModelφ(R)φ(S)φ(C)Gain over qFull AUROC GPQA- Diamond Qwen3-4B+0.008± 0.009 +0.011± 0.010 +0.059± 0.024 +0.078±0.0320.724±0.060 GPQA- Diamond Qwen3-8B −0.006± 0.032 −0.002± 0.015 +0.077± 0.052 +0.069±0.0670.824±0.032 GPQA- Diamond Qwen3-14B −0.002± 0.004 +0.043± 0.019 +0.104± 0.097 +0.145±0.0990.799±0.054 GPQA- Diamond gpt-oss-20b −0.001± 0.003 +0.025± 0.055 +0.030± 0.068 +0.053±0.0940.724±0.073 GPQA- Diamond gpt-oss-120b+0.000± 0.001 +0.035± 0.023 +0.089± 0.046 +0.124±0.0510.786±0.033 MMLU-ProQwen3-4B+0.003± 0.001 +0.038± 0.008 +0.089± 0.011 +0.130±0.0170.841±0.008 MMLU-ProQwen3-8B+0.013± 0.004 +0.041± 0.004 +0.087± 0.014 +0.142±0.0180.844±0.007 MMLU-ProQwen3-14B+0.001± 0.000 +0.042± 0.004 +0.085± 0.010 +0.128±0.0130.838±0.007 MMLU-Progpt-oss-20b −0.001± 0.002 +0.039± 0.003 +0.077± 0.005 +0.115±0.0050.808±0.012 MMLU-Progpt-oss-120b+0.000± 0.000 +0.046± 0.002 +0.077± 0.005 +0.123±0.0070.813±0.003 Omni-MATH- 2-Filtered Qwen3-4B+0.023± 0.005 +0.024± 0.002 +0.089± 0.007 +0.136±0.0100.876±0.012 Omni-MATH- 2-Filtered Qwen3-8B+0.031± 0.003 +0.035± 0.004 +0.073± 0.012 +0.139±0.0160.883±0.011 Omni-MATH- 2-Filtered Qwen3-14B+0.027± 0.003 +0.048± 0.005 +0.089± 0.011 +0.164±0.0170.872±0.004 Omni-MATH- 2-Filtered gpt-oss-20b+0.009± 0.008 +0.034± 0.008 +0.072± 0.024 +0.114±0.0300.818±0.020 Omni-MATH- 2-Filtered gpt-oss-120b+0.006± 0.006 +0.032± 0.008 +0.079± 0.017 +0.117±0.0230.834±0.020 Mean+0.007± 0.011 +0.033± 0.014 +0.078± 0.017 +0.118±0.0300.819±0.047 54 An Observability Ladder for Large Language Models Appendix Table A41: Under the primary surface-plus-Cohere monitor, summaries remain near chance while traces retain modest discrimination. Entries give across-item AUROC / within-item AUROC, with 95% item-level bootstrap intervals in square brackets for the within-item value. Conditional-logit entries give the coefficient for a one-SD increase in monitor score and its 95% interval. Both AUROCs use the same post-outcome discordant held-out MMLU-Pro items across three independent runs per item. Monitors are trained on the primary run. Across-item AUROC pools runs across items; within-item AUROC compares a correct and an incorrect run of the same item and model. Pairing holds item-level cues fixed, and the prompt is not supplied to the monitor. There are 939–1,389 discordant items per model. Prompt-withheld displays ModelMeasureRSR+SCR+CR+S+C Qwen3-4B n=1,029Across / within [95% CI] 0.511 / 0.527 [0.499, 0.554] 0.536 / 0.533 [0.507, 0.557] 0.533 / 0.545 [0.519, 0.569] 0.564 / 0.586 [0.562, 0.611] 0.566 / 0.590 [0.565, 0.616] 0.564 / 0.578 [0.552, 0.604] Conditional logit β [95% CI] 0.040 [−0.046, 0.126] 0.329 [0.145, 0.513] 0.343 [0.147, 0.538] 0.786 [0.613, 0.958] 0.815 [0.642, 0.988] 0.747 [0.580, 0.915] Qwen3-8B n=969Across / within [95% CI] 0.541 / 0.550 [0.521, 0.578] 0.536 / 0.531 [0.505, 0.557] 0.525 / 0.519 [0.493, 0.545] 0.558 / 0.579 [0.552, 0.605] 0.545 / 0.561 [0.534, 0.587] 0.544 / 0.564 [0.537, 0.592] Conditional logit β [95% CI] 0.204 [0.098, 0.309] 0.335 [0.182, 0.489] 0.211 [0.035, 0.387] 0.674 [0.514, 0.833] 0.426 [0.263, 0.589] 0.415 [0.253, 0.577] Qwen3-14B n=939Across / within [95% CI] 0.485 / 0.493 [0.466, 0.521] 0.515 / 0.503 [0.477, 0.529] 0.512 / 0.505 [0.478, 0.530] 0.544 / 0.547 [0.520, 0.574] 0.544 / 0.560 [0.532, 0.586] 0.545 / 0.557 [0.530, 0.584] Conditional logit β [95% CI] −0.032 [−0.118, 0.054] 0.079 [−0.084, 0.242] 0.064 [−0.105, 0.234] 0.566 [0.391, 0.741] 0.567 [0.391, 0.743] 0.563 [0.392, 0.734] gpt-oss-20b n=1,389Across / within [95% CI] 0.512 / 0.530 [0.505, 0.554] 0.536 / 0.526 [0.504, 0.548] 0.533 / 0.515 [0.494, 0.537] 0.558 / 0.566 [0.544, 0.587] 0.559 / 0.559 [0.537, 0.581] 0.559 / 0.560 [0.537, 0.581] Conditional logit β [95% CI] 0.740 [0.472, 1.007] 0.167 [0.044, 0.290] 0.132 [0.008, 0.255] 0.392 [0.265, 0.519] 0.409 [0.281, 0.537] 0.376 [0.251, 0.500] gpt-oss-120b n=978Across / within [95% CI] 0.523 / 0.539 [0.512, 0.567] 0.534 / 0.527 [0.501, 0.553] 0.534 / 0.526 [0.499, 0.553] 0.556 / 0.558 [0.533, 0.585] 0.555 / 0.552 [0.527, 0.579] 0.551 / 0.544 [0.519, 0.571] Conditional logit β [95% CI] 0.317 [0.162, 0.472] 0.175 [0.034, 0.317] 0.172 [0.033, 0.311] 0.403 [0.255, 0.550] 0.405 [0.255, 0.554] 0.359 [0.210, 0.508] 55 An Observability Ladder for Large Language Models Appendix Table A42: Prompt-visible repeated-run evaluation under the primary surface-plus-Cohere monitor. Entries give across-item AUROC / within-item AUROC, with 95% item-level bootstrap intervals for the within-item value. Conditional-logit entries give the coefficient for a one-SD increase in monitor score and its 95% interval. Both AUROCs use the same post-outcome discordant held-out MMLU-Pro items across three independent runs per item. Monitors are trained on the primary run. Across-item AUROC pools runs across items; within-item AUROC compares a correct and an incorrect run of the same item and model. The task prompt is supplied to every display and held fixed within each pair. There are 939–1,389 discordant items per model. Prompt-visible displays ModelMeasureq+Rq+R+Sq+R+C q+R+S+C Qwen3-4B n=1,029Across / within [95% CI] 0.510 / 0.517 [0.488, 0.545] 0.524 / 0.543 [0.517, 0.568] 0.562 / 0.580 [0.555, 0.605] 0.563 / 0.584 [0.558, 0.608] Conditional logit β [95% CI] 0.864 [0.093, 1.635] 0.574 [0.275, 0.874] 0.814 [0.639, 0.988] 0.799 [0.626, 0.972] Qwen3-8B n=969Across / within [95% CI] 0.523 / 0.524 [0.496, 0.553] 0.531 / 0.547 [0.522, 0.573] 0.548 / 0.561 [0.535, 0.587] 0.548 / 0.567 [0.540, 0.594] Conditional logit β [95% CI] 0.494 [0.184, 0.804] 0.365 [0.147, 0.583] 0.455 [0.289, 0.621] 0.473 [0.305, 0.640] Qwen3-14B n=939Across / within [95% CI] 0.509 / 0.507 [0.476, 0.537] 0.510 / 0.486 [0.460, 0.513] 0.547 / 0.555 [0.529, 0.582] 0.543 / 0.552 [0.526, 0.579] Conditional logit β [95% CI] 0.175 [−1.049, 1.399] −0.059 [−0.307, 0.189] 0.585 [0.408, 0.763] 0.527 [0.354, 0.700] gpt-oss-20b n=1,389Across / within [95% CI] 0.514 / 0.485 [0.460, 0.510] 0.529 / 0.536 [0.513, 0.558] 0.558 / 0.554 [0.532, 0.576] 0.558 / 0.560 [0.539, 0.582] Conditional logit β [95% CI] −0.651 [−2.004, 0.702] 0.377 [0.153, 0.600] 0.389 [0.258, 0.520] 0.380 [0.251, 0.509] gpt-oss-120b n=978Across / within [95% CI] 0.512 / 0.482 [0.452, 0.512] 0.525 / 0.531 [0.505, 0.557] 0.556 / 0.564 [0.539, 0.591] 0.560 / 0.564 [0.538, 0.589] Conditional logit β [95% CI] −0.736 [−2.378, 0.906] 0.279 [0.020, 0.538] 0.459 [0.298, 0.620] 0.472 [0.315, 0.630] 56 An Observability Ladder for Large Language Models Appendix Table A43: Repeated-run signal under several controls for visible text length. The upper panel reports AUROC when ranking across selected questions and when comparing runs of the same question, averaged equally over the five target models. The matched-length panels keep correct–incorrect run pairs whose trace word counts differ by no more than 20% or 10% of the longer trace. The aggregate summary–trace estimates are equal-model means; their intervals come from a paired item bootstrap stratified by model, with every eligible pair for each sampled item retained. Other brackets are 95% item-bootstrap intervals, and n is the exact number of eligible pairs. All across-item and within-item estimates use the same items with three runs and both observed outcomes. Display / modelPrimaryWordsWords + sentencesLength-adjustedWithin 20%Within 10% Across-item / within-item AUROC, equal-model mean Output-only displays R0.515 / 0.5280.494 / 0.4910.506 / 0.5030.514 / 0.524– S0.531 / 0.5240.524 / 0.5140.522 / 0.5150.514 / 0.508– R+S0.528 / 0.5220.524 / 0.5180.516 / 0.5070.521 / 0.519– C0.556 / 0.5670.539 / 0.5530.539 / 0.5450.527 / 0.538– R+C0.554 / 0.5640.539 / 0.5520.539 / 0.5460.524 / 0.528– R+S+C0.553 / 0.5610.539 / 0.5530.540 / 0.5470.524 / 0.529– Prompt-visible displays q+R0.514 / 0.5030.508 / 0.5080.507 / 0.5040.510 / 0.504– q+R+S0.524 / 0.5290.512 / 0.5180.511 / 0.5160.519 / 0.519– q+R+C0.554 / 0.5630.538 / 0.5520.539 / 0.5470.524 / 0.536– q+R+S+C0.555 / 0.5650.539 / 0.5530.539 / 0.5470.524 / 0.533– Matched-length summary–trace comparison, equal-model mean RestrictionSummaryTraceTrace minus summary [95% CI]Eligible pairs Within 20%0.5020.533+0.032 [+0.009, +0.056]3,412 Within 10%0.4950.532+0.037 [+0.004, +0.069]1,701 Trace-only display, primary reader: within-item AUROC [95% CI] (n eligible pairs) Qwen3-4B0.586 (n = 2,058)– 0.539 [0.501, 0.578] (n = 765) 0.520 [0.467, 0.575] (n = 371) Qwen3-8B0.579 (n = 1,938)– 0.517 [0.480, 0.556] (n = 755) 0.526 [0.476, 0.575] (n = 406) Qwen3-14B0.547 (n = 1,878)– 0.481 [0.440, 0.519] (n = 674) 0.476 [0.419, 0.533] (n = 317) gpt-oss-20b0.566 (n = 2,778)– 0.579 [0.537, 0.621] (n = 605) 0.599 [0.542, 0.652] (n = 299) gpt-oss-120b0.558 (n = 1,956)– 0.551 [0.510, 0.593] (n = 613) 0.539 [0.482, 0.596] (n = 308) Appendix Table A44: A conclusion-region trace extract is compared with the model-written summary on repeated runs. Entries give across-item AUROC / within-item AUROC; square brackets give the 95% item-bootstrap interval for within-item AUROC. Self-summary uses the original preservation-tuned summary, whereas Last90 deterministically places up to the final 90 whitespace-delimited trace words in the summary slot and therefore uses the entire trace when it is shorter. It is a conclusion-region-biased control rather than a pure ending-only test. Monitors use the same primary surface-plus-Cohere feature family, are trained on run 1, and score held-out MMLU-Pro items across three independent runs. Both AUROCs use the same post-outcome discordant items. Across-item AUROC pools runs across those items, while within-item AUROC compares correct and incorrect runs of one item. ModelS self-summaryS Last90 R+Sself-summaryR+S Last90 Qwen3-4B 0.536 / 0.533 [0.507, 0.557] 0.550 / 0.559 [0.534, 0.584] 0.533 / 0.545 [0.519, 0.569] 0.541 / 0.543 [0.518, 0.569] Qwen3-8B 0.536 / 0.531 [0.505, 0.557] 0.558 / 0.544 [0.519, 0.570] 0.525 / 0.519 [0.493, 0.545] 0.528 / 0.542 [0.518, 0.566] Qwen3-14B 0.515 / 0.503 [0.477, 0.529] 0.524 / 0.525 [0.498, 0.551] 0.512 / 0.505 [0.478, 0.530] 0.525 / 0.522 [0.496, 0.548] gpt-oss-20b 0.536 / 0.526 [0.504, 0.548] 0.548 / 0.536 [0.514, 0.558] 0.533 / 0.515 [0.494, 0.537] 0.548 / 0.538 [0.516, 0.560] gpt-oss-120b 0.534 / 0.527 [0.501, 0.553] 0.556 / 0.565 [0.539, 0.593] 0.534 / 0.526 [0.499, 0.553] 0.554 / 0.574 [0.547, 0.600] 57 An Observability Ladder for Large Language Models Appendix Table A45: Prompt-visible repeated-run evaluation holds the item fixed when measuring run-level signal. Entries give across-item AUROC / within-item AUROC, with 95% item-level bootstrap intervals in square brackets for the within-item value. Conditional-logit entries give the coefficient for a one-SD increase in monitor score and its 95% interval. The fully open monitor uses surface features plus word- and character-level TF–IDF and is trained on the primary run. Both AUROCs use the same post-outcome discordant items. Across-item AUROC pools held-out runs across those items. Within-item AUROC compares a correct and an incorrect run of the same MMLU-Pro item under the same model across three independent runs. The task prompt is supplied to every display and held fixed by the within-item pairing. For the two gpt-ossq+Restimates, scores vary almost entirely between items, leaving little variation after the item fixed effects. Their large conditional-logit coefficients are therefore poorly scaled and are not used for headline claims. Prompt-visible displays ModelMeasureq+Rq+R+Sq+R+C q+R+S+C Qwen3-4B n=1,029Across / within [95% CI] 0.519 / 0.535 [0.506, 0.564] 0.550 / 0.563 [0.537, 0.588] 0.590 / 0.606 [0.581, 0.631] 0.589 / 0.607 [0.582, 0.632] Conditional logit β [95% CI] 0.441 [0.076, 0.806] 0.626 [0.447, 0.806] 0.756 [0.611, 0.902] 0.757 [0.611, 0.902] Qwen3-8B n=969Across / within [95% CI] 0.529 / 0.554 [0.529, 0.580] 0.547 / 0.555 [0.528, 0.581] 0.564 / 0.588 [0.561, 0.614] 0.563 / 0.585 [0.558, 0.611] Conditional logit β [95% CI] 0.316 [0.131, 0.501] 0.375 [0.229, 0.522] 0.559 [0.419, 0.698] 0.554 [0.416, 0.693] Qwen3-14B n=939Across / within [95% CI] 0.508 / 0.520 [0.488, 0.551] 0.529 / 0.544 [0.517, 0.570] 0.558 / 0.571 [0.545, 0.597] 0.558 / 0.571 [0.544, 0.597] Conditional logit β [95% CI] 0.555 [-0.522, 1.632] 0.267 [0.092, 0.442] 0.512 [0.362, 0.661] 0.506 [0.357, 0.654] gpt-oss-20b n=1,389Across / within [95% CI] 0.520 / 0.518 [0.494, 0.543] 0.551 / 0.543 [0.521, 0.566] 0.566 / 0.557 [0.535, 0.579] 0.568 / 0.562 [0.539, 0.583] Conditional logit β [95% CI] 4.651 [0.441, 8.860] 0.362 [0.221, 0.504] 0.372 [0.251, 0.493] 0.390 [0.268, 0.512] gpt-oss-120b n=978Across / within [95% CI] 0.505 / 0.539 [0.509, 0.570] 0.543 / 0.572 [0.546, 0.597] 0.561 / 0.568 [0.542, 0.595] 0.561 / 0.570 [0.544, 0.596] Conditional logit β [95% CI] 6.238 [2.521, 9.955] 0.440 [0.277, 0.603] 0.442 [0.298, 0.587] 0.452 [0.309, 0.595] 58 An Observability Ladder for Large Language Models Appendix Table A46: A conclusion-region trace extract is compared with the model-written summary on repeated runs. Entries give across-item AUROC / within-item AUROC; square brackets give the 95% item-bootstrap interval for within-item AUROC. Self-summary uses the original preservation-tuned summary, whereas Last90 deterministically places up to the final 90 whitespace-delimited trace words in the summary slot and therefore uses the entire trace when it is shorter. It is a conclusion-region-biased control rather than a pure ending-only test. Monitors use the fully open surface-plus-TF–IDF feature family, are trained on run 1, and score held-out MMLU-Pro items across three independent runs. Both AUROCs use the same post-outcome discordant items. Across-item AUROC pools runs across those items, while within-item AUROC compares correct and incorrect runs of one item. ModelS self-summaryS Last90 R+Sself-summaryR+S Last90 Qwen3-4B 0.575 / 0.565 [0.539, 0.590] 0.597 / 0.592 [0.566, 0.617] 0.574 / 0.568 [0.542, 0.592] 0.596 / 0.589 [0.563, 0.614] Qwen3-8B 0.563 / 0.569 [0.543, 0.595] 0.580 / 0.554 [0.528, 0.581] 0.547 / 0.551 [0.524, 0.578] 0.553 / 0.544 [0.518, 0.571] Qwen3-14B 0.543 / 0.543 [0.515, 0.569] 0.563 / 0.562 [0.536, 0.588] 0.546 / 0.542 [0.513, 0.568] 0.561 / 0.559 [0.533, 0.585] gpt-oss-20b 0.561 / 0.565 [0.543, 0.587] 0.575 / 0.558 [0.536, 0.580] 0.562 / 0.571 [0.549, 0.593] 0.576 / 0.558 [0.536, 0.581] gpt-oss-120b 0.565 / 0.567 [0.542, 0.593] 0.588 / 0.596 [0.571, 0.622] 0.566 / 0.572 [0.546, 0.597] 0.589 / 0.596 [0.570, 0.622] Appendix Table A47: Summary-based pairwise monitors remain near chance; trace-based monitors are modest. Pairwise logistic monitors are trained on correct-minus-incorrect feature differences within discordant MMLU-Pro training items, select regularization on validation items, and are evaluated on held-out discordant items. Values are mean±SD across data splits, with descriptive 2.5th–97.5th percentiles of the five estimates in square brackets. These percentiles summarize split sensitivity and are not confidence intervals. ModelRSR+SCR+CR+S+C Qwen3-4B 0.511± 0.020 [0.494, 0.542] 0.533± 0.013 [0.523, 0.552] 0.535± 0.023 [0.510, 0.558] 0.562± 0.028 [0.524, 0.586] 0.580± 0.018 [0.562, 0.601] 0.556± 0.017 [0.543, 0.580] Qwen3-8B 0.530± 0.023 [0.500, 0.554] 0.514± 0.037 [0.474, 0.546] 0.524± 0.019 [0.502, 0.548] 0.558± 0.029 [0.521, 0.593] 0.544± 0.029 [0.511, 0.570] 0.564± 0.016 [0.546, 0.581] Qwen3-14B 0.532± 0.035 [0.495, 0.579] 0.512± 0.019 [0.483, 0.525] 0.518± 0.010 [0.507, 0.531] 0.543± 0.015 [0.520, 0.554] 0.553± 0.017 [0.529, 0.568] 0.543± 0.011 [0.534, 0.556] gpt-oss-20b 0.536± 0.014 [0.525, 0.551] 0.522± 0.013 [0.508, 0.536] 0.536± 0.011 [0.525, 0.552] 0.544± 0.015 [0.527, 0.565] 0.534± 0.022 [0.515, 0.565] 0.551± 0.022 [0.525, 0.569] gpt-oss-120b 0.541± 0.031 [0.508, 0.581] 0.528± 0.014 [0.516, 0.549] 0.542± 0.042 [0.506, 0.605] 0.572± 0.016 [0.553, 0.592] 0.577± 0.022 [0.549, 0.601] 0.561± 0.031 [0.512, 0.582] Appendix Table A48: Summary-based pairwise monitors remain near chance; trace-based monitors are modest. The primary monitor uses rate-based surface features plus Cohere embeddings. The task prompt is included in every display. Pairwise logistic monitors are trained on correct-minus-incorrect feature differences within discordant MMLU- Pro training items, select regularization on validation items, and are evaluated on held-out discordant items. Values are mean±SD across data splits, with descriptive 2.5th–97.5th percentiles of the five estimates in square brackets. These percentiles summarize split sensitivity and are not confidence intervals. Modelq+Rq+R+Sq+R+C q+R+S+C Qwen3-4B 0.530± 0.020 [0.504, 0.553] 0.555± 0.015 [0.540, 0.574] 0.555± 0.012 [0.541, 0.568] 0.566± 0.016 [0.545, 0.586] Qwen3-8B 0.515± 0.013 [0.499, 0.531] 0.544± 0.019 [0.520, 0.565] 0.559± 0.012 [0.544, 0.573] 0.565± 0.019 [0.546, 0.591] Qwen3-14B 0.510± 0.033 [0.478, 0.551] 0.514± 0.035 [0.481, 0.559] 0.570± 0.011 [0.558, 0.585] 0.550± 0.027 [0.508, 0.572] gpt-oss-20b 0.501± 0.022 [0.477, 0.530] 0.543± 0.017 [0.529, 0.565] 0.538± 0.025 [0.510, 0.562] 0.547± 0.019 [0.532, 0.569] gpt-oss-120b 0.522± 0.026 [0.499, 0.554] 0.548± 0.024 [0.522, 0.583] 0.576± 0.021 [0.545, 0.595] 0.586± 0.014 [0.566, 0.599] 59 An Observability Ladder for Large Language Models Appendix Table A49: Direct item-conditional repeated-run training compares correct and incorrect runs of the same item. Pairwise logistic monitors are trained on correct-minus-incorrect feature differences within discordant MMLU-Pro training items, select regularization on validation items, and are evaluated on held-out discordant items. The fully open specification uses surface features plus word- and character-level TF–IDF. Values are mean±SD across data splits, with descriptive 2.5th–97.5th percentiles of the five estimates in square brackets. These percentiles summarize split sensitivity and are not confidence intervals. Modelq+Rq+R+Sq+R+C q+R+S+C Qwen3-4B 0.510± 0.021 [0.487, 0.541] 0.562± 0.022 [0.537, 0.586] 0.588± 0.017 [0.572, 0.613] 0.592± 0.010 [0.580, 0.605] Qwen3-8B 0.536± 0.022 [0.512, 0.558] 0.534± 0.029 [0.494, 0.568] 0.570± 0.006 [0.561, 0.575] 0.572± 0.009 [0.562, 0.583] Qwen3-14B 0.525± 0.019 [0.503, 0.550] 0.507± 0.037 [0.468, 0.557] 0.567± 0.037 [0.521, 0.611] 0.571± 0.044 [0.517, 0.623] gpt-oss-20b 0.516± 0.024 [0.491, 0.552] 0.544± 0.015 [0.524, 0.558] 0.555± 0.021 [0.532, 0.580] 0.562± 0.019 [0.534, 0.577] gpt-oss-120b 0.549± 0.029 [0.514, 0.586] 0.531± 0.025 [0.495, 0.553] 0.564± 0.020 [0.553, 0.596] 0.566± 0.009 [0.553, 0.573] 60 An Observability Ladder for Large Language Models Appendix Table A50: The summary–trace ordering persists in both discordance-composition strata, but trace discrimination is concentrated among items with exactly one incorrect run. The upper block distinguishes all outcome-discordant MMLU-Pro items among the 12,032 items with three graded runs per model from the analyzed subset that appeared in the test partition for at least one of five data splits. Parentheses give the analyzed share of all outcome-discordant items. The lower block pools the five target models and reports within-item AUROC for items with exactly one incorrect run (n=2,829), exactly two incorrect runs (n=2,475), and both strata combined. Square brackets give 95% intervals from 5,000 item-bootstrap draws that resample model–item units within target model. The model-balanced pooled estimate assigns every analyzed item for modelmweight1/n m , so the five analyzed model samples contribute equally. This reweighting does not correct selection into outcome discordance or held-out test inclusion, and it does not score concordant or unanalyzed items. ModelAll three-run itemsAnalyzed discordant items Three gradedAll discordant1 incorrect2 incorrectTotal (share) Qwen3-4B12,0321,530526503 1,029 (67.3%) Qwen3-8B12,0321,481507462 969 (65.4%) Qwen3-14B12,0321,364498441 939 (68.8%) gpt-oss-20b12,0322,062761628 1,389 (67.4%) gpt-oss-120b12,0321,406537441 978 (69.6%) Within-item AUROC by discordance composition Display1 incorrect2 incorrectPooledModel-balanced pooled Within pairs Prompt-withheld displays R 0.525 [0.508, 0.542] 0.532 [0.515, 0.550] 0.528 [0.516, 0.541] 0.528 [0.516, 0.540]10,608 S 0.551 [0.535, 0.567] 0.494 [0.478, 0.511] 0.524 [0.513, 0.536] 0.524 [0.513, 0.536]10,608 R+S 0.548 [0.532, 0.563] 0.493 [0.476, 0.509] 0.522 [0.511, 0.533] 0.522 [0.511, 0.533]10,608 C 0.611 [0.596, 0.626] 0.518 [0.502, 0.535] 0.568 [0.556, 0.579] 0.567 [0.556, 0.579]10,608 R+C 0.605 [0.590, 0.621] 0.518 [0.502, 0.534] 0.564 [0.553, 0.575] 0.564 [0.553, 0.576]10,608 R+S+C 0.604 [0.588, 0.619] 0.512 [0.495, 0.527] 0.561 [0.549, 0.572] 0.561 [0.549, 0.572]10,608 Prompt-visible displays q+R 0.504 [0.487, 0.522] 0.499 [0.481, 0.517] 0.502 [0.489, 0.514] 0.503 [0.490, 0.516]10,608 q+R+S 0.553 [0.537, 0.568] 0.503 [0.486, 0.519] 0.530 [0.518, 0.541] 0.529 [0.517, 0.540]10,608 q+R+C 0.605 [0.589, 0.619] 0.514 [0.497, 0.530] 0.562 [0.551, 0.573] 0.563 [0.551, 0.574]10,608 q+R+S+C 0.608 [0.593, 0.623] 0.517 [0.500, 0.533] 0.565 [0.554, 0.577] 0.565 [0.554, 0.577]10,608 61 An Observability Ladder for Large Language Models Appendix Table A51: The repeated-run subset is compared with held-out items whose outcomes remain stable. Discordant items have both correct and incorrect outcomes across three runs; concordant items have one outcome across all runs. Prompt score is the held-out probability from the primary prompt-only reader, averaged over available data splits. Trace length is the median [25th, 75th percentile] number of words. Replay confidence is the target model’s maximum answer-choice probability under provider-side replay. The domain panel reports total variation (TV) between the two domain distributions and the largest positive percentage-point difference for discordant items. ModelGroupnRun 1 accuracyPrompt scoreTrace wordsReplay confidence Qwen3-4BConcordant7,0970.7410.627863 [468, 1,782]0.9948 Discordant1,0290.5070.5612,158 [1,157, 4,135]0.9831 Qwen3-8BConcordant7,0880.7870.606979 [527, 2,012]0.9531 Discordant9690.5180.5442,527 [1,419, 4,414]0.9484 Qwen3-14BConcordant7,1630.8120.696719 [415, 1,522.5]0.9943 Discordant9390.5240.6131,905 [1,077, 3,266.5]0.9871 gpt-oss-20bConcordant6,7340.7940.630200 [94, 445]0.9997 Discordant1,3890.5080.554591 [319, 1,092]0.9997 gpt-oss-120bConcordant7,1300.8290.650138 [74, 268]0.9999 Discordant9780.5300.576379 [224.2, 638.8]1.0000 Domain composition ModelTV distanceLargest discordant excessDifference Qwen3-4B17.3%law+5.9 p Qwen3-8B21.4%law+8.9 p Qwen3-14B26.4%law+12.0 p gpt-oss-20b22.8%law+11.6 p gpt-oss-120b22.9%law+13.5 p 62 An Observability Ladder for Large Language Models Appendix Table A52: The fixed-prompt GPT-5-mini reader is evaluated on all ten tested combinations of the response, summary, trace, and prompt. Across-item AUROC ranks runs across questions; within-item AUROC compares runs of the same question. Both estimates use the same 1,389 analyzed outcome-discordant MMLU-Pro questions, three gpt-oss-20b runs per item, and the fixed-promptgpt-5-mini-2025-08-07reader. Brackets give paired 95% item-bootstrap intervals (5,000 resamples). Each comparison subtracts the second named display from the first on every paired bootstrap draw. Display or comparisonAcross-item estimate [95% CI] Within-item estimate [95% CI] Display AUROC Response (R) 0.501 [0.485, 0.517] 0.503 [0.486, 0.521] Summary (S) 0.673 [0.655, 0.690] 0.649 [0.628, 0.669] Reasoning trace (C) 0.682 [0.667, 0.698] 0.683 [0.663, 0.702] Response + summary 0.676 [0.659, 0.694] 0.656 [0.635, 0.677] Response + reasoning trace 0.700 [0.684, 0.716] 0.690 [0.670, 0.710] Response + summary + reasoning trace 0.704 [0.688, 0.721] 0.693 [0.673, 0.713] Prompt + response 0.834 [0.817, 0.850] 0.800 [0.781, 0.818] Prompt + response + summary 0.806 [0.789, 0.823] 0.776 [0.757, 0.795] Prompt + response + reasoning trace 0.748 [0.732, 0.764] 0.736 [0.717, 0.756] Prompt + response + summary + reasoning trace 0.746 [0.729, 0.762] 0.721 [0.701, 0.741] Paired AUROC differences Summary minus response +0.172 [+0.149, +0.195] +0.146 [+0.118, +0.172] Reasoning trace minus summary +0.009 [-0.008, +0.027] +0.034 [+0.008, +0.059] Summary added to response +0.175 [+0.151, +0.199] +0.153 [+0.125, +0.180] Reasoning trace versus summary after response +0.024 [+0.006, +0.041] +0.034 [+0.008, +0.057] Summary added to prompt + response -0.028 [-0.039, -0.017] -0.024 [-0.040, -0.007] Reasoning trace added to prompt + response -0.086 [-0.100, -0.072] -0.064 [-0.084, -0.044] Summary added after prompt + response + reasoning trace -0.002 [-0.012, +0.008] -0.015 [-0.032, +0.002] 63 An Observability Ladder for Large Language Models Appendix Table A53: Provider-only replay features add a mean hidden-state gain that survives prompt condition- ing. Held-out test AUROC by setting, averaged over five data splits (±SD across data splits).+Z prob ,+Z hidden , and +Zadd replay-derived answer-probability, raw hidden-state, and combined replay features to the full public display. The three rightmost columns repeat the comparison on top of the prompt-conditioned public displayq+R+S+C. The final row averages per-setting means across all 15 settings (± between-setting SD). BenchmarkModelR+S+C+Z prob +Z hidden +Z q+R+S+C q ... +Z hidden q ... +Z GPQA-DiamondQwen3-4B0.732± 0.067 0.705± 0.079 0.738± 0.070 0.744± 0.059 0.725± 0.084 0.739± 0.065 0.737± 0.067 GPQA-DiamondQwen3-8B0.839± 0.062 0.839± 0.063 0.858± 0.034 0.859± 0.038 0.836± 0.058 0.853± 0.036 0.855± 0.033 GPQA-DiamondQwen3-14B0.800± 0.058 0.773± 0.072 0.774± 0.080 0.767± 0.078 0.758± 0.056 0.765± 0.084 0.759± 0.084 GPQA-Diamondgpt-oss-20b0.707± 0.096 0.698± 0.098 0.771± 0.075 0.749± 0.103 0.723± 0.077 0.769± 0.083 0.751± 0.109 GPQA-Diamondgpt-oss-120b0.770± 0.057 0.771± 0.066 0.798± 0.069 0.799± 0.069 0.747± 0.059 0.795± 0.071 0.796± 0.071 MMLU-ProQwen3-4B0.823± 0.010 0.837± 0.009 0.859± 0.008 0.859± 0.008 0.824± 0.009 0.859± 0.007 0.859± 0.007 MMLU-ProQwen3-8B0.821± 0.004 0.831± 0.007 0.861± 0.007 0.861± 0.007 0.822± 0.004 0.861± 0.007 0.861± 0.007 MMLU-ProQwen3-14B0.825± 0.007 0.844± 0.007 0.866± 0.004 0.865± 0.003 0.826± 0.006 0.866± 0.004 0.866± 0.004 MMLU-Progpt-oss-20b0.803± 0.015 0.808± 0.015 0.844± 0.011 0.844± 0.011 0.804± 0.015 0.845± 0.011 0.845± 0.011 MMLU-Progpt-oss-120b0.810± 0.005 0.814± 0.005 0.845± 0.005 0.845± 0.005 0.809± 0.004 0.845± 0.005 0.845± 0.005 Omni-MATH-2- Filtered Qwen3-4B0.870± 0.015 0.870± 0.016 0.893± 0.015 0.894± 0.013 0.870± 0.015 0.893± 0.015 0.894± 0.013 Omni-MATH-2- Filtered Qwen3-8B0.875± 0.022 0.878± 0.021 0.912± 0.012 0.912± 0.012 0.870± 0.017 0.912± 0.011 0.912± 0.011 Omni-MATH-2- Filtered Qwen3-14B0.876± 0.008 0.877± 0.008 0.891± 0.005 0.891± 0.005 0.871± 0.006 0.890± 0.005 0.890± 0.005 Omni-MATH-2- Filtered gpt-oss-20b0.824± 0.022 0.825± 0.021 0.864± 0.020 0.864± 0.020 0.824± 0.016 0.865± 0.020 0.865± 0.021 Omni-MATH-2- Filtered gpt-oss-120b0.832± 0.019 0.835± 0.019 0.849± 0.011 0.849± 0.011 0.832± 0.016 0.848± 0.010 0.848± 0.010 Mean0.814± 0.048 0.814± 0.056 0.841± 0.050 0.840± 0.052 0.809± 0.050 0.840± 0.050 0.839± 0.053 Appendix Table A54: The mean hidden-state gain is largest with the run’s own trace and is lost under shuffled replay. Held-out test AUROC of theR+S+C+Z hidden monitor when the hidden state is extracted from replays of different contexts: the standard task prompt plus the run’s reasoning trace (q+C), the prompt alone, the trace alone, the prompt plus final response, the prompt plus the aggressively masked trace, and the prompt plus a trace from a different item, reassigned using a fixed random seed. The publicR+S+Ccolumn repeats the comparison without replay. The headline describes the mean over the reported subset; the pattern need not hold in every setting. Values average five data splits (±SD across data splits) for all five models on GPQA-Diamond and Qwen3-8B on MMLU-Pro and Omni-MATH-2-Filtered. BenchmarkModelR+S+Cq+Cq onlyC onlyq+R q+maskedC q+shuffled C GPQA- Diamond gpt-oss-120b0.770± 0.057 0.798± 0.069 0.767± 0.077 0.830± 0.058 0.789± 0.064 0.713± 0.044 0.672± 0.144 GPQA- Diamond gpt-oss-20b0.707± 0.096 0.771± 0.075 0.693± 0.094 0.730± 0.089 0.658± 0.079 0.708± 0.062 0.674± 0.063 GPQA- Diamond Qwen3-14B0.800± 0.058 0.774± 0.080 0.724± 0.102 0.797± 0.086 0.758± 0.132 0.780± 0.102 0.721± 0.059 GPQA- Diamond Qwen3-4B0.732± 0.067 0.738± 0.070 0.660± 0.080 0.698± 0.103 0.656± 0.041 0.746± 0.069 0.685± 0.032 GPQA- Diamond Qwen3-8B0.839± 0.062 0.858± 0.034 0.828± 0.060 0.849± 0.038 0.754± 0.056 0.854± 0.035 0.829± 0.052 MMLU-ProQwen3-8B0.821± 0.004 0.861± 0.007 0.833± 0.010 0.857± 0.008 0.842± 0.005 0.857± 0.007 0.822± 0.006 Omni-MATH- 2-Filtered Qwen3-8B0.875± 0.022 0.912± 0.012 0.874± 0.019 0.907± 0.010 0.908± 0.014 0.914± 0.010 0.866± 0.018 64 An Observability Ladder for Large Language Models Appendix Table A55: Availability of final-answer extraction and model-based measurements. Answer read is the percentage of items whose final answer was extracted successfully; read failures gives the corresponding count. Forward pass complete is the share with a completed prompt-conditioned calculation used to obtain answer probabilities and hidden states. Input fit gives the share of inputs that fit the model’s context window. Usable requires both a completed forward pass and an input that fit the context window. Hidden vectors is the number of items with an available final-token hidden-state vector. BenchmarkModelItemsAnswer read Read failures Forward pass complete Input fitUsableHidden vectors GPQA- Diamond Qwen3-4B198100%0100%100%100%198 GPQA- Diamond Qwen3-8B198100%0100%100%100%198 GPQA- Diamond Qwen3-14B198100%0100%100%100%198 GPQA- Diamond gpt-oss-20b198100%0100%100%100%198 GPQA- Diamond gpt-oss-120b198100%0100%100%100%198 MMLU-ProQwen3-4B12,032100%1100%100%100%12,032 MMLU-ProQwen3-8B12,032100%4100%100%100%12,032 MMLU-ProQwen3-14B12,032100%0100%100%100%12,032 MMLU-Progpt-oss-20b12,032100%5100%100%100%12,032 MMLU-Progpt-oss-120b12,032100%3100%100%100%12,032 Omni-MATH- 2-Filtered Qwen3-4B4,18195%189100%100%100%4,181 Omni-MATH- 2-Filtered Qwen3-8B4,18195%194100%100%100%4,181 Omni-MATH- 2-Filtered Qwen3-14B4,18195%207100%100%100%4,181 Omni-MATH- 2-Filtered gpt-oss-20b4,18196%152100%100%100%4,181 Omni-MATH- 2-Filtered gpt-oss-120b4,18197%118100%100%100%4,181 65 An Observability Ladder for Large Language Models Appendix Table A56: Public monitors transfer across target models better than across benchmarks. Held-out test AUROC on the target setting for probes trained on the source setting(s), averaged over five target data splits (± SD across data splits). Transfers reuse the public monitor features. No probe parameters are refit on the target setting. Within-benchmark comparisons transfer across models on Omni-MATH-2-Filtered; source runs follow the target split so target-test item identifiers are excluded from source train and validation. Cross-benchmark comparisons transfer the same model between MMLU-Pro and Omni-MATH-2-Filtered. SourceTargetRR+SR+CR+S+C q+R+S+C Within-benchmark, cross-model Omni-MATH-2-Filtered / gpt-oss-20b + gpt-oss-120b Omni-MATH-2-Filtered / Qwen3-14B 0.702± 0.020 0.780± 0.006 0.778± 0.009 0.807± 0.005 0.801± 0.006 Omni-MATH-2-Filtered / Qwen3-8B Omni-MATH-2-Filtered / Qwen3-14B 0.759± 0.011 0.814± 0.004 0.841± 0.011 0.842± 0.010 0.840± 0.006 Omni-MATH-2-Filtered / Qwen3-4B + Qwen3-8B + Qwen3-14B Omni-MATH-2-Filtered / gpt-oss-120b 0.659± 0.017 0.741± 0.005 0.778± 0.016 0.787± 0.015 0.790± 0.014 Omni-MATH-2-Filtered / Qwen3-4B + Qwen3-8B + Qwen3-14B Omni-MATH-2-Filtered / gpt-oss-20b 0.680± 0.022 0.732± 0.016 0.781± 0.018 0.783± 0.016 0.792± 0.011 Omni-MATH-2-Filtered / Qwen3-4B + Qwen3-8B Omni-MATH-2-Filtered / Qwen3-14B 0.765± 0.016 0.817± 0.009 0.860± 0.005 0.866± 0.005 0.860± 0.002 Omni-MATH-2-Filtered / Qwen3-4B Omni-MATH-2-Filtered / Qwen3-14B 0.748± 0.017 0.806± 0.011 0.845± 0.003 0.873± 0.002 0.849± 0.006 Mean (6 transfers)0.719± 0.045 0.782± 0.038 0.814± 0.039 0.826± 0.040 0.822± 0.031 Cross-benchmark, same-model MMLU-Pro / gpt-oss-120b Omni-MATH-2-Filtered / gpt-oss-120b 0.401± 0.010 0.719± 0.026 0.805± 0.023 0.813± 0.025 0.800± 0.026 MMLU-Pro / gpt-oss-20bOmni-MATH-2-Filtered / gpt-oss-20b 0.618± 0.027 0.705± 0.020 0.789± 0.019 0.793± 0.019 0.796± 0.021 MMLU-Pro / Qwen3-14B Omni-MATH-2-Filtered / Qwen3-14B 0.557± 0.043 0.685± 0.010 0.768± 0.012 0.819± 0.008 0.805± 0.007 MMLU-Pro / Qwen3-4BOmni-MATH-2-Filtered / Qwen3-4B 0.646± 0.025 0.706± 0.016 0.853± 0.010 0.852± 0.013 0.854± 0.012 MMLU-Pro / Qwen3-8B Omni-MATH-2-Filtered / Qwen3-8B 0.735± 0.026 0.713± 0.026 0.603± 0.014 0.642± 0.016 0.624± 0.010 Omni-MATH-2-Filtered / gpt-oss-120b MMLU-Pro / gpt-oss-120b 0.526± 0.015 0.666± 0.013 0.735± 0.008 0.750± 0.010 0.745± 0.004 Omni-MATH-2-Filtered / gpt-oss-20b MMLU-Pro / gpt-oss-20b0.502± 0.012 0.643± 0.015 0.735± 0.016 0.743± 0.015 0.738± 0.015 Omni-MATH-2-Filtered / Qwen3-14B MMLU-Pro / Qwen3-14B 0.490± 0.012 0.650± 0.013 0.696± 0.012 0.716± 0.013 0.727± 0.007 Omni-MATH-2-Filtered / Qwen3-4B MMLU-Pro / Qwen3-4B0.504± 0.019 0.638± 0.007 0.725± 0.008 0.721± 0.007 0.730± 0.007 Omni-MATH-2-Filtered / Qwen3-8B MMLU-Pro / Qwen3-8B0.573± 0.018 0.670± 0.011 0.645± 0.009 0.671± 0.012 0.659± 0.009 Mean (10 transfers)0.555± 0.094 0.680± 0.030 0.735± 0.074 0.752± 0.067 0.748± 0.070 66 An Observability Ladder for Large Language Models Appendix Table A57: Higher-access monitors concentrate errors in the flagged tail. Error rate among retained runs when the monitor keeps the indicated fraction of test runs ranked by predicted correctness and flags the rest, averaged over data splits and the 15 settings. Coverage 1.0 is the unconditional error rate. Level50%70%80%90%100% R0.2430.2660.2770.2970.316 R+S0.1390.2030.2400.2770.316 R+S+C0.1140.1850.2270.2740.316 R+S+C+Z0.0980.1680.2170.2670.316 Appendix Table A58: Counts of extraction problems among completed runs. Counts are taken from the saved primary runs before the data are divided for classifier training and evaluation. Empty trace means that the extracted reasoning trace contains no non-whitespace characters. No closing reasoning marker corresponds tomissing_think_close; no answer extracted corresponds tono_answer_found. Across the 82,055 primary summaries, mean length is 77.0 words, 74.4% contain at most 90 words, and the maximum is 231 words. BenchmarkModelRunsEmpty traceNo closing reasoning marker No answer extracted GPQA-DiamondQwen3-4B198000 GPQA-DiamondQwen3-8B198300 GPQA-DiamondQwen3-14B198000 GPQA-Diamondgpt-oss-20b198000 GPQA-Diamondgpt-oss-120b198000 MMLU-ProQwen3-4B12,032010 MMLU-ProQwen3-8B12,0327931 MMLU-ProQwen3-14B12,032500 MMLU-Progpt-oss-20b12,032005 MMLU-Progpt-oss-120b12,032003 Omni-MATH-2- Filtered Qwen3-4B4,181016623 Omni-MATH-2- Filtered Qwen3-8B4,18198017024 Omni-MATH-2- Filtered Qwen3-14B4,18122314265 Omni-MATH-2- Filtered gpt-oss-20b4,18100151 Omni-MATH-2- Filtered gpt-oss-120b4,18100118 Appendix Table A59: GPT-5-mini agrees with the deterministic checker on exact or numeric matches and evaluates the remaining cases. This comparison covers the stored Omni-MATH-2-Filtered correctness labels from all 20,905 judgments (4,181 items×five target models, primary run). The deterministic exact/numeric checker resolves 20,310 comparisons. Each percentage uses the relevant comparison count as its denominator; — marks cases where a share is not defined. QuantityCountShare Judgments with a binary decision20,905100.0% Judged equivalent14,20668.0% Judged not equivalent6,69932.0% Uncertain or error cases00.0% Comparisons resolved by the deterministic checker20,310— Judge–checker agreement14,14669.7% Agreement on deterministic exact/numeric matches7,859/7,86599.9% Deterministic mismatches judged equivalent6,158/12,44549.5% 67 An Observability Ladder for Large Language Models Appendix Table A60: Full-public monitor discrimination by run stratum. Held-out test AUROC of theR+S+C public monitor within run strata, pooled over data splits and models per benchmark (test-set run counts in parentheses; sample-size weighted). Strata with fewer than 20 runs per setting or only one correctness class are excluded because their within-stratum AUROC is undefined. The missing-think-close category therefore contains Qwen3-8B only; the corresponding Qwen3-4B and Qwen3-14B strata are single-class. Length bins are within-setting terciles. StratumValueGPQA-DiamondMMLU-ProOmni-MATH-2- Filtered Trace lengthlong0.632 (349)0.750 (20100)0.727 (6927) Trace lengthmedium0.698 (326)0.795 (19992)0.703 (6984) Trace lengthshort0.635 (325)0.765 (20083)0.853 (7014) Summary lengthlong0.694 (329)0.771 (20035)0.774 (6942) Summary lengthmedium0.736 (338)0.798 (20154)0.806 (6931) Summary lengthshort0.754 (333)0.810 (19986)0.880 (7052) Trace self-correctionabsent—0.721 (6914)0.749 (1950) Trace self-correctionpresent0.741 (978)0.803 (53261)0.843 (18962) Parser status missing think close—0.727 (176) Parser statusno answer found—0.737 (405) Parser statusok0.745 (1000)0.813 (60166)0.848 (20034) Choice count10—0.821 (49831)— Choice count3—0.821 (71)— Choice count40.745 (1000)0.739 (3076)— Choice count5—0.671 (242)— Choice count6—0.761 (485)— Choice count7—0.771 (822)— Choice count8—0.798 (1605)— Choice count9—0.782 (4014)— 68 An Observability Ladder for Large Language Models Appendix Table A61: Response-only discrimination varies by benchmark, model, and response format. Columns report the short-response share (≤ 30characters), length-only AUROC, selected-answer baseline, and theRmonitor on all, short-only, and non-empty responses (mean over five data splits±SD). Neither selected-answer identity nor response length alone provides a source decomposition of the fitted monitor. A dash indicates that the length-only baseline is undefined. BenchmarkModelShort shareLength alone Selected answer R monitorR (short only) R (non- empty) GPQA- Diamond Qwen3-4B0.970.5160.5180.505± 0.066 0.527± 0.086 0.505± 0.066 GPQA- Diamond Qwen3-8B0.550.6380.5440.631± 0.047 0.482± 0.151 0.631± 0.047 GPQA- Diamond Qwen3-14B0.990.5120.5450.558± 0.046 0.567± 0.052 0.558± 0.046 GPQA- Diamond gpt-oss-20b1.00—0.5600.535± 0.137 0.535± 0.137 0.535± 0.137 GPQA- Diamond gpt-oss-120b1.00—0.5380.542± 0.073 0.542± 0.073 0.542± 0.073 MMLU-ProQwen3-4B0.960.5010.5310.560± 0.014 0.551± 0.017 0.560± 0.014 MMLU-ProQwen3-8B0.660.5300.5240.652± 0.014 0.530± 0.016 0.652± 0.014 MMLU-ProQwen3-14B0.980.5050.5230.533± 0.004 0.528± 0.005 0.533± 0.004 MMLU-Progpt-oss-20b1.000.5030.5110.515± 0.009 0.515± 0.009 0.515± 0.009 MMLU-Progpt-oss-120b1.000.5010.5220.538± 0.006 0.538± 0.006 0.538± 0.006 Omni-MATH- 2-Filtered Qwen3-4B0.060.6170.5930.807± 0.015 0.981± 0.013 0.782± 0.019 Omni-MATH- 2-Filtered Qwen3-8B0.050.5890.6070.837± 0.027 0.928± 0.065 0.820± 0.028 Omni-MATH- 2-Filtered Qwen3-14B0.130.5290.6060.779± 0.010 0.842± 0.040 0.752± 0.014 Omni-MATH- 2-Filtered gpt-oss-20b0.450.6280.5670.682± 0.019 0.590± 0.039 0.682± 0.019 Omni-MATH- 2-Filtered gpt-oss-120b0.600.6110.5680.671± 0.012 0.592± 0.034 0.671± 0.012 69 An Observability Ladder for Large Language Models Appendix Table A62: Output censoring changes equal-weight headline AUROC by at most 0.010. Blank-trace exclusion retains items whose whitespace-stripped reasoning trace is nonempty. The strict parsed-output rule retains items with parser status “ok” and nonempty whitespace-stripped values for the reasoning trace, response, and extracted answer. AUROC is recomputed within each setting and data split after censoring, averaged over five data splits within each model–benchmark setting, and then averaged equally over the 15 settings.∆is censored minus uncensored AUROC; upper-block deltas are reported to four decimals because several are smaller than0.001, and the per-setting block uses three decimals. The final column reports retained / uncensored held-out predictions across items and data splits for each access level. The lower block reports per-setting deltas for the two Omni-MATH-2-Filtered settings with concentrated empty traces. Censoring ruleAccess levelUncensoredCensored∆Scored predictions Blank-trace exclusion R0.6230.621 −0.002480,789 / 82,100 R+S0.7790.776 −0.003080,789 / 82,100 R+S+C0.8140.814 −0.000280,789 / 82,100 R+S+C+Z0.8400.839 −0.001380,789 / 82,100 Strict parsed-output censor R0.6230.613 −0.009879,930 / 82,100 R+S0.7790.772 −0.007079,930 / 82,100 R+S+C0.8140.810 −0.003579,930 / 82,100 R+S+C+Z0.8400.835 −0.004979,930 / 82,100 Affected-setting ∆ (censored minus uncensored AUROC) Censoring ruleSettingRR+S R+S+CR+S+C+Z Blank-trace exclusionQwen3-8B / Omni- MATH-2- Filtered −0.019 −0.029 −0.008−0.011 Qwen3-14B / Omni- MATH-2- Filtered −0.007 −0.012 −0.003−0.003 Strict parsed-output censor Qwen3-8B / Omni- MATH-2- Filtered −0.059 −0.052 −0.029−0.033 Qwen3-14B / Omni- MATH-2- Filtered −0.039 −0.035 −0.019−0.019 Appendix Table A63: Sparse models select both public feature sources. Values are selection rates across the 15 model–benchmark settings and five random splits. Plain LASSO selects individual standardized features. Sparse group LASSO applies a penalty by feature source while retaining sparsity within each source. Both models include surface features, Cohere embeddings, and answer-probability features when available. Raw hidden states are evaluated separately in a sensitivity analysis using replay features. Selection methodFeature sourceRR + S R + S + CR + S + C + Z Plain LASSOSurface86%96%99%97% Cohere92%100%100%100% Z prob —76% Sparse group LASSOSurface100%97%100%100% Cohere93%92%75%84% Z prob —79% 70 An Observability Ladder for Large Language Models Appendix Table A64: Stable sparse-feature examples are descriptive, not a complete explanation of the monitor. The table lists human-readable surface features selected in at least three settings with mean selection rate at least 60%. The sign is the dominant coefficient direction among selected fits. AnalysisLevelFeatureSettingsSelectionSign Sparse group LASSOR+S+CMathematical-symbol density15100%+ Sparse group LASSOR+S+CSentence count15100% − Sparse group LASSOR+S+CUncertainty-marker rate15100% − Sparse group LASSOR+S+CWord count15100% − Sparse group LASSOR+S+C+ZAnswer-framing rate15100%+ Sparse group LASSOR+S+C+ZMathematical-symbol density15100%+ Sparse group LASSOR+S+C+ZUncertainty-marker rate15100% − Sparse group LASSOR+S+C+ZWord count15100% − Sparse group LASSOR+S+CAnswer-framing rate1599%+ Sparse group LASSOR+S+CDigit density1599%+ Sparse group LASSOR+S+C+ZDigit density1599%+ Sparse group LASSOR+S+C+ZSelf-correction rate1599% − Sparse group LASSOR+S+C+ZSentence count1599% − Sparse group LASSOR+SDigit density1596%+ Sparse group LASSOR+SWord count1596% − Plain LASSOR+S+CUncertainty-marker rate1596% − Sparse group LASSOR+S+CSelf-correction rate1596% − Plain LASSOR+S+C+ZUncertainty-marker rate1595% − Appendix Table A65: Last90 contains denser and more discriminative marker cues than same-run self-summaries. Entries are equal-weight means of setting-level statistics across 15 model–benchmark settings and covern=80,763 paired runs after excluding 1,292 pairs with a zero-word Last90 display. No self-summary display has zero words.∆is Last90 minus summary. Brackets are 95% descriptive setting-bootstrap intervals (10,000 resamples). Single-feature AUROC is oriented using the dominant coefficient direction in Appendix Table A64, except that average word length is positive by convention. Oriented values above0.5discriminate in the chosen direction. Last90 uses the whole trace when it is shorter than 90 whitespace-delimited words. Mean levelOriented AUROC FeatureSummaryLast90∆ [95% CI]SummaryLast90∆ [95% CI] Word count79.780.6+0.9 [−9.3, +10.8]0.6930.659 −0.035 [−0.073, +0.003] Sentence count6.312.3+6.0 [+4.6, +7.5]0.4880.382 −0.106 [−0.150,−0.064] Average word length4.744.25 −0.49 [−0.69,−0.28]0.4340.450+0.015 [+0.000, +0.031] Digit density0.0410.059+0.018 [+0.013, +0.024]0.6340.612 −0.022 [−0.043,−0.003] Math-character density0.0520.042 −0.010 [−0.020,−0.003]0.5760.619+0.044 [+0.025, +0.063] Uncertainty rate0.0030.013+0.009 [+0.008, +0.011]0.5650.654+0.090 [+0.064, +0.118] Self-correction rate0.0070.019+0.012 [+0.010, +0.014]0.5210.589+0.068 [+0.048, +0.089] Answer-marker rate0.0150.066+0.051 [+0.046, +0.056]0.6250.696+0.072 [+0.049, +0.095] 71