Paper deep dive

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations

Gert Lek, Zixuan Xia, Pin-Yu Chen, Lydia Y. Chen

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 68

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Natural Language Autoencoders (NLAs) produce unsupervised text explanations of a model's activations: a verbalizer describes an activation and a reconstructor learns to recover it from this text. Under the established point-reconstruction NLA training recipe, explanations become more useful for predicting model behavior while also increasingly introducing unsupported details and exhibiting writing defects. To assess these changes separately, we introduce a standardized evaluation framework for unstructured NLA explanations, measuring information recoverable from explanations, contextual support for their claims, and writing quality. To address confabulation and writing defects, we move beyond predicting a single activation: explanations can distinguish distributions of possible activations even when their means and optimal point-reconstruction rewards are identical. We introduce Flow-NLA, which models the distribution of activations compatible with an explanation and trains the verbalizer using a diffusion likelihood bound. Across Qwen, Gemma, and Apertus, this richer signal retains the utility gains of point reconstruction while curbing the growth of confabulation and writing defects, opening up a direction for improving activation-derived training to encourage more informative, supported, and readable explanations. Code and evaluation prompts will be made publicly available upon acceptance.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,237 characters extracted from source content.

Expand or collapse full text

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations Gert Lek1 Zixuan Xia1,2 Pin-Yu Chen3 Lydia Y. Chen1,4 1 Université de Neuchâtel 2 Universität Bern 3 IBM Research 4 Delft University of Technology gert.lek@unine.ch Abstract Natural Language Autoencoders (NLAs) produce unsupervised text explanations of a model’s activations: a verbalizer describes an activation and a reconstructor learns to recover it from this text. Under the established point-reconstruction NLA training recipe, explanations become more useful for predicting model behavior while also increasingly introducing unsupported details and exhibiting writing defects. To assess these changes separately, we introduce a standardized evaluation framework for unstructured NLA explanations, measuring information recoverable from explanations, contextual support for their claims, and writing quality. To address confabulation and writing defects, we move beyond predicting a single activation: explanations can distinguish distributions of possible activations even when their means and optimal point-reconstruction rewards are identical. We introduce Flow-NLA, which models the distribution of activations compatible with an explanation and trains the verbalizer using a diffusion likelihood bound. Across Qwen, Gemma, and Apertus, this richer signal retains the utility gains of point reconstruction while curbing the growth of confabulation and writing defects, opening up a direction for improving activation-derived training to encourage more informative, supported, and readable explanations. Code and evaluation prompts will be made publicly available upon acceptance. 1 Introduction Natural-language activation explanations aim to make model representations accessible beyond observable outputs, as explored by LatentQA and Activation Oracles (Pan et al., 2026; Karvonen et al., 2026a). Natural Language Autoencoders (NLAs) (Fraser-Taliente et al., 2026) learn such explanations through reconstruction: a verbalizer describes an activation in natural language, and a reconstructor predicts the activation from that text. After supervised initialization, reconstruction quality provides reinforcement-learning feedback without requiring labels for each generated explanation. Because removing true claims hurts reconstruction more than removing false ones, the original NLA study anticipated that training would reduce confabulation, but found that it remained substantial and did not decrease (Fraser-Taliente et al., 2026). Rerunning this recipe on three open models, we find that final explanations are similarly confabulated and that confabulation rises as reconstruction improves. Recent work has further examined initialization sensitivity and claim-level reliability (Zhang and Turner, 2026; Dingeto, 2026). These findings raise a fundamental question: what properties of an explanation does reconstruction training actually reward? An explanation may convey useful information while introducing unsupported claims or expressing that information poorly. We therefore separate these properties, track how they evolve during training, and ask whether richer reconstruction feedback can improve explanation quality beyond point reconstruction. Our contributions are: 1. Evaluating explanation quality. We measure instance-specific predictive information by asking an independent reader model to make predictions from the explanation, comparing matched explanations with mismatched explanations. Separately, we evaluate contextual confabulation, claims unsupported or contradicted by the source context using claim-based evaluation (Min et al., 2023; Song et al., 2024), and identify writing defects in clarity, coherence, and economy (Section 3.1). This framework separates what information an explanation makes recoverable from whether its claims are context-supported and clearly expressed. 2. Characterizing point-reconstruction training. Prior NLA work primarily trains explanations through point reconstruction. Across the models we study, this training improves predictive utility while also increasing contextual confabulation and writing defects (Section 3.2). This behavior follows naturally from the training signal: reconstruction rewards text for helping recover the target activation, but does not directly require the claims carrying that information to be context-supported or clearly expressed. Thus, useful but unsupported cues can still be rewarded, motivating a closer examination of what information the reconstruction objective itself captures. 3. Beyond point reconstruction with Flow-NLA. We show that point reconstruction has an information blind spot: under NLA’s normalized squared-error objective, explanation refinements that change the conditional activation distribution while preserving its mean cannot improve the optimal reconstruction reward. Flow-NLA instead provides explanation-conditioned denoising feedback across noise levels, allowing the training signal to depend on information beyond the conditional mean (Section 4). We relate this objective to the diffusion variational bound (Ho et al., 2020; Kingma et al., 2021) and analyze its sensitivity to richer conditional information. Across three models, Flow-NLA retains substantial utility gains while limiting contextual confabulation growth and producing fewer writing defects than the baseline NLAs (Section 4.3). 2 Related Work Learning Activation Explanations. LatentQA and Activation Oracles train language models to answer questions about activations using supervised activation–response pairs (Pan et al., 2026; Karvonen et al., 2026a), with Activation Oracles broadening this supervision through classification and context-prediction tasks. More recent approaches use reconstruction to learn open-ended activation descriptions beyond predefined questions. Cycle-Consistent Activation Oracles (Chalnev, 2026) warm-start the verbalizer on LatentQA-style supervision and surrounding-token prediction before optimizing an activation→ → cycle with RL, while NLAs (Fraser-Taliente et al., 2026) warm-start both the activation verbalizer and reconstructor using teacher-generated descriptions before jointly optimizing them for activation reconstruction. These approaches make reconstruction a learning signal for open-ended activation explanations, but the feedback remains centered on recovering individual target activations. We instead study whether richer, distributional reconstruction feedback can reward information that point reconstruction misses, using conditional denoising to examine its effects on explanation utility and confabulation. Confabulation and Explanation Evaluation. An NLA initialization study finds that claim plausibility declines during training for normally initialized NLAs and that different initializations yield similar reconstruction accuracy despite differences in explanation plausibility (Zhang and Turner, 2026). Verbalizations can also reflect the verbalizer’s own parametric knowledge (Li et al., 2026). For utility, CHIVE evaluates whether interpretability tools, including NLAs, improve counterfactual behavioral prediction beyond the original prefix (Karvonen et al., 2026b). The NLA study evaluates both information recovery and confabulation (Fraser-Taliente et al., 2026). We extend this evaluation effort with a standardized framework for unstructured NLA explanations, enabling comparisons across models and training procedures. Our framework evaluates predictions from explanations with the source withheld and separately checks source support using claim-based evaluation adapted from FActScore and VeriScore (Min et al., 2023; Song et al., 2024). 3 Confabulation or Explanation? Understanding Confabulation in Reconstruction Training We start by examining how explanations change during training as an NLA learns to reconstruct activations more accurately. To achieve this, we propose an evaluation framework that scores utility, confabulation, and writing quality. First, we describe the established reconstruction training procedure. Let h∈ℝdh ^d denote a residual-stream activation of the target language model at a selected layer and token position, normalized to unit length. The target language model is frozen. We extract activations from fixed residual-stream locations, with layers and token positions specified for each target model and evaluation task in Appendix B.1. The verbalizer πϕ(z∣h) _φ(z h) generates an explanation z from the activation. Following the released NLA (Fraser-Taliente et al., 2026), a separate reconstructor predicts a unit activation h^θ(z) h_θ(z), discarding activation magnitude. The verbalizer’s reward is rθ(h,z)=−‖h−h^θ(z)‖22.r_θ(h,z)=- \|h- h_θ(z) \|_2^2. Training an NLA consists of two phases. After supervised initialization on activation–summary pairs, online training repeatedly generates groups of explanations, scores them with the current reconstructor, and updates the reconstructor and the verbalizer on the same batch (Fraser-Taliente et al., 2026). The reconstructor is updated directly on the generated activation–explanation pairs. The verbalizer uses a group-relative policy update with a KL penalty toward its supervised reference (Shao et al., 2024). Thus, an explanation is favored when the reconstructor can use it to reduce the reconstruction MSE. 3.1 An Evaluation Framework for Explanation Quality We evaluate explanations along three complementary dimensions (Figure 1): utility, measuring what task-relevant information a blind judge can recover from the explanation; source support, measuring whether its claims are supported by the source; and writing quality, measuring how clearly that information is expressed. These dimensions are scored by a separate judge LLM, whose context is restricted to what each dimension requires. The utility judge answers a task query using only the explanation, the source-support judge observes both the source and explanation, and the writing judge observes only the explanation. Appendix B provides further information. Source ssFrozen target modelActivation hhVerbalizer πϕ(z∣h) _φ(z h)Explanation zzUtility judgeSource-support judgeWriting judgeSeesExplanation z✓Task query✓–Source s–✓–Activation h–TaskPredict source properties,continuation, behaviorLabel claims: supported,contradicted, unsupportedFlag defect spans: clarity,coherence, economyMetricsAccuracy, macro-F1,hard-negative separationAssertion risk,token coverageDefects per explanation Figure 1: Three judges of explanation quality. Each judge sees only what it requires: The utility judge answers the task query from the explanation; the source-support judge checks the explanation’s claims with the source; and the writing judge assesses only the explanation. Utility. Useful explanations let readers answer diverse questions about the activation. Therefore, we evaluate utility over a large range of tasks (Appendix Table 4). First, we identify properties of the source text through 12 explanation classification tasks spanning semantic (topic, sentiment, subjectivity, entailment), lexical (entity and exact-name presence), and grammatical (tense, number, language, pronoun form) categories (accuracy). Second, we extract information on the model’s continuation by classifying the true 32-token continuation among ten candidates (accuracy). Last, we predict the model’s own behavior by predicting whether it answers, refuses, or asks for clarification from the explanation (macro-F1). Explanations that perform well across these tasks carry identifiable information from the activation. The utility judge receives only the explanation and a question with candidate answers (the task query), and must choose an answer. These are judged with DeepSeek V4.1 Flash (DeepSeek-AI, 2026) as the judge model (Appendix B). To additionally test whether utility comes from instance-specific information rather than coarse cues such as the topic, we compare the matched explanation against a hard negative: the explanation of a closely related source that differs in the relevant answer. For example, “Alice lent Bob a book” and “Bob lent Alice a book” share nearly all surface cues but imply opposite answers to “Who lent the book?” We define the hard-negative separation as the log-probability ratio log⁡[pJ(y∣zmatched)/pJ(y∣zhard)] [p_J(y z_matched)/p_J(y z_hard) ] that the utility judge assigns to the correct answer y given the matched versus the hard-negative explanation, with answer probabilities read from the same DeepSeek judge; it is the primary utility measure presented here. Higher is better: in the example, an explanation that only says a book was lent scores zero, while one that says who lent it to whom scores positive; positive values thus indicate instance-specific information beyond the topic and surface cues shared with the hard negative. Definitions and aggregation details are given in Appendix B, and Appendix D decomposes the separation into matched predictive gain and mismatch penalty. Confabulation. Using a source-support judge that receives the source text before the activation extraction and the explanation, explanations are split into distinct atomic claims (Min et al., 2023; Song et al., 2024). The claims are labeled as supported (stated in or entailed by the source), contradicted (incompatible with the source) or unsupported. Supported and contradicted verdicts are required to quote the exact source evidence. This yields two metrics: Assertion Risk, which is the proportion of assertions that are unsupported or contradicted, and Confabulated Token Coverage, which is the proportion of explanation tokens occupied by these claims. Both metrics assess the contextual source support rather than the utility of describing the model’s internal computation. Writing quality. The writing quality judge sees only the explanation, without the source text. Here, the judge flags distinct defects in clarity (unclear wording, malformed or unfinished sentences), coherence (statements that conflict within the explanation), and economy (avoidable repetition or filler), quoting the exact explanation span for each. We count distinct defects and report changes from supervised initialization. Writing quality is evaluated separately from source support because an explanation may be unsupported yet well written, or supported yet poorly expressed. For example, the Qwen NLA explanation below fails the source-support and writing quality judges for different reasons: the source-support judge flags invented names as contradicted by the source, and the writing judge flags incompatible accounts of the person’s name as a coherence defect. Source Jonn Hart is a stage name his real name is Di’Jonn Grizzell Explanation excerpt The final clause “his actual full name is Kristopher Ellis… his real last name is Scott Gellell; he’s actual full name is Tracy Ellzell” requires a period or continuation Figure 2: Better reconstruction accompanies more confabulation and writing defects. As the FVE reconstruction rises (solid line), so does the confabulation index (dashed line) across models. The index I⁡(t)=12[R⁡(t)/R⁡(0)+D⁡(t)/D⁡(0)]I(t)= 12[R(t)/R(0)+D(t)/D(0)] averages assertion risk R and writing defects per explanation D relative to supervised initialization (I=1I=1). Shading shows 95% bootstrap intervals. 3.2 The Confabulation Cost of Better Reconstruction We retrain NLAs using the established setup from Fraser-Taliente et al. (2026) for Qwen2.5-7B-Instruct (Yang et al., 2024), Gemma-3-12B (Gemma Team, 2025) and Apertus-v1.5-8B (Hernández-Cano et al., 2026). Reconstruction is measured during training using the Fraction of Variance Explained, FVE=1−⁡[‖h−h^θ(z)‖22]/VhFVE=1-E[\|h- h_θ(z)\|_2^2]/V_h, where VhV_h is the variance baseline for unit activations; utility, confabulation, and writing quality are evaluated on held-out examples. As reconstruction improves during training, assertion risk and writing defects increase (Figure 2; separate panels in Appendix C): the correlation between FVE and the confabulation index is considerable, with 0.78 for Qwen, 0.96 for Gemma, and 0.90 for Apertus. At the same time, hard-negative separation rises from roughly 0.6–1.1 nats at supervised initialization to 1.2–2.4 nats within the first few hundred updates (Figure 3), so explanations increasingly carry instance-specific information rather than only coarse cues (Section 3.1). Better reconstruction thus carries both greater recoverable information and a deterioration in how reliably and clearly it is expressed, although whether this coupling is strictly necessary remains open. Released NLA baselines. The same coexistence of useful information and unsupported content is visible in the four released Anthropic NLAs (Fraser-Taliente et al., 2026) (Table 1). Classification accuracy ranges from 79.4% to 85.7% and suffix-selection accuracy from 96.6% to 99.0%, yet assertion risk exceeds 75% across all four models, and confabulated content accounts for more than half of explanation tokens in the released NLAs. Figure 3: Better reconstruction accompanies more relevant information. Hard-negative separation across the 12 classification datasets for point-reconstruction NLA (solid) and Flow-NLA (dashed); higher is better. Classification and suffix-selection accuracies approximately match those reported by Fraser-Taliente et al. (2026) for their Claude NLAs (about 78–86% and 93–98%), although their evaluation code is unreleased. Our model reruns reach a training FVE of about 0.75 (Figure 2), close to the 0.75–0.80 reported for the released checkpoints. Our point-reconstruction reruns on Qwen, Gemma, and Apertus show the same pattern: 77.5% to 84.2% classification and 96.0% to 97.6% suffix-selection accuracy, alongside 77.8% to 83.7% assertion risk and 41.5% to 68.3% confabulated token coverage. Thus, both released checkpoints and our reruns produce useful but highly confabulated explanations. We next ask whether changing the activation-derived reward can preserve useful learning while weakening the coupling between confabulation and reconstruction error. Table 1: Utility and confabulation in released NLAs and our reruns. All scores are percentages except writing defects, which are counts per explanation. Utility ↑ Confabulation ↓ Writing ↓ Target model Class. Suffix Behavior Risk Coverage Defects Released Anthropic NLAs Qwen2.5-7B 79.479.4 98.098.0 82.782.7 92.692.6 74.874.8 2.452.45 Gemma-3-12B 84.784.7 96.696.6 83.483.4 83.183.1 67.567.5 1.781.78 Gemma-3-27B 85.785.7 98.298.2 84.184.1 76.376.3 55.455.4 1.921.92 Llama-3.3-70B 82.782.7 99.099.0 65.165.1 76.476.4 56.956.9 1.781.78 Point-reconstruction reruns Qwen2.5-7B 79.779.7 97.697.6 74.974.9 83.783.7 68.368.3 1.631.63 Gemma-3-12B 84.284.2 96.296.2 82.482.4 77.877.8 44.244.2 1.381.38 Apertus-v1.5-8B 77.577.5 96.096.0 73.673.6 81.381.3 41.541.5 1.501.50 4 Beyond Point Reconstruction with Flow-NLA What information should reconstruction feedback reward? We first show that informative distinctions among activations can leave optimal point-reconstruction loss unchanged. Next, we develop Flow-NLA to reward such distinctions through conditional denoising and evaluate how this richer feedback changes explanation utility, confabulation, and writing quality. 4.1 Beyond Point Reconstruction Point reconstruction can fail to reward informative distinctions even with an optimal reconstructor. For a fixed verbalizer, the joint distribution p⁡(h)πϕ(z∣h)p(h) _φ(z h) induces pϕ(h∣z)p_φ(h z). Following NLA (Fraser-Taliente et al., 2026), let both the target H and prediction u have unit norm. Writing μϕ(z)=ϕ[H∣Z=z] _φ(z)=E_φ[H Z=z], we have min‖u‖2=1⁡ϕ[‖H−u‖22∣Z=z]=2−2‖μϕ(z)‖2. _\|u\|_2=1E_φ[\|H-u\|_2^2 Z=z]=2-2\| _φ(z)\|_2. (1) The optimum is attained by u=μϕ(z)/‖μϕ(z)‖2u= _φ(z)/\| _φ(z)\|_2 when μϕ(z)≠0 _φ(z)≠ 0, and by any unit-norm prediction otherwise. The optimal expected point loss depends on pϕ(h∣z)p_φ(h z) only through its conditional mean. Because NLA rewards the verbalizer based on reconstruction quality, more informative explanations that leave the conditional mean unchanged do not necessarily receive higher reward. Figure 4 illustrates this limitation with four equally likely unit directions. The horizontal explanation zhorz_hor rules out the vertical directions but leaves two opposing candidates. Their mean is zero, so this partial information leaves the optimal expected point loss at 22, as without the explanation. Denoising additionally observes xt=atH+btϵx_t=a_tH+b_tε, where ϵ∼⁡(0,I)ε (0,I) is independent of (H,Z)(H,Z) and at2+bt2=1a_t^2+b_t^2=1. At each noise level, the optimal unconstrained squared-error prediction is ϕ[H∣xt,z]E_φ[H x_t,z]. The explanation can therefore exclude candidates that remain plausible from xtx_t alone. For the illustrated observation, denoising assigns lower error with the correct horizontal explanation than without it (Figure 4c–d). Information that leaves the conditional mean unchanged can thus still be rewarded by denoising. Figure 4: Denoising can reward information beyond the conditional mean. H is uniform over four unit directions; we follow the realized activation H=(1,0)H=(1,0) (gold outline). (a) The horizontal explanation zhorz_hor keeps the horizontal pair (blue) and excludes the vertical pair (open circles). Both pairs have mean zero (×), so the optimal expected point loss is the same with or without zhorz_hor. (b) Denoising also observes the noisy activation xt=atH+btϵx_t=a_tH+b_tε (black arrow: noise btϵb_tε), with candidates scaled by ata_t. (c) Without the explanation, xtx_t is about equally close to the true and upward candidates, so the optimal estimate hedges between them. (d) With zhorz_hor, the upward candidate is excluded and the estimate moves to the true direction, lowering the denoising error. In (c)–(d), arrows show the displacement atϕ[H∣xt,z]−xta_tE_φ[H x_t,z]-x_t from each noisy observation to its optimal signal estimate (without z in (c)); shading shows the density of xtx_t. 4.2 Flow-NLA Figure 5: Learning activation explanations with Flow-NLA. An activation h from a frozen target model is passed to a verbalizer, which generates an explanation z, and corrupted with Gaussian noise to form xtx_t. The flow reconstructor learns to denoise xtx_t conditioned on z, modeling the distribution of activations compatible with the explanation. Denoising errors across noise levels define a reward derived from a conditional diffusion likelihood bound. This reward trains the verbalizer to produce explanations that make the observed activation more predictable under the conditional model. Flow-NLA builds on this argument by replacing point reconstruction with conditional denoising. The reconstructor learns to denoise activations using the text explanation, and the verbalizer is rewarded for explanations that help it capture the conditional distribution of activations they describe. This results in an unsupervised training signal sensitive to more than the conditional mean. Rewarding input reconstruction likelihood under a co-trained decoder is an established way to encourage informative outputs (Barber and Agakov, 2003; Miao and Blunsom, 2016; Eysenbach et al., 2019; Rita et al., 2022), typically using categorical or Gaussian models. Flow-NLA instead rewards input likelihood under a co-trained diffusion model conditioned on the verbalizer’s output. This conditional distribution shifts as the verbalizer learns, complicating reward stabilization. This unsupervised reward favors activation prediction over source-context support, so it may still reward unsupported wording. Source-text supervision could address this, but would bias explanations away from information extractable from activations alone. Whether this objective changes the balance between utility and contextual confabulation is an empirical question (Section 4.3). Conditional reconstruction. Figure 5 shows the Flow-NLA training procedure. Let x be the activation after a fixed coordinate standardization, and let X denote the corresponding random variable. We form a noisy activation xt=atx+btϵx_t=a_tx+b_tε and velocity target vt=atϵ−btxv_t=a_tε-b_tx, where ϵ∼⁡(0,Id)ε (0,I_d) and at2+bt2=1a_t^2+b_t^2=1 define a variance-preserving path, and d is the activation dimension. The reconstructor fθ(xt,t,z)f_θ(x_t,t,z) combines a text reader with a conditional velocity field and is trained with the velocity-prediction objective (Salimans and Ho, 2022): ℒrec(θ;ϕ)=h,z,t,ϵ[d−1∥fθ(xt,t,z)−vt∥22],z∼πϕ(⋅∣h).L_rec(θ;φ)=E_h,z,t,ε [d^-1\|f_θ(x_t,t,z)-v_t\|_2^2 ], z _φ(· h). (2) Across noise levels, the model learns how the explanation helps distinguish activation patterns compatible with a partially observed activation. These denoising predictions capture conditional distributional structure (Saharia et al., 2023), since their population optimum at an interior noise level identifies the full explanation-conditioned distribution for a fixed verbalizer (Appendix H). During training, this distribution, pϕ(h∣z)∝p⁡(h)πϕ(z∣h)p_φ(h z) p(h)\, _φ(z h), changes whenever the verbalizer learns, so the reconstructor tracks a moving target, much like a distributional critic (Bellemare et al., 2017), except that here the modeled distribution itself, not its mean, defines the reward (Appendix F). Rewarding distributional information. We seek to reward explanations under which the observed activation has high conditional likelihood. The point reward already does this, but with a restrictive model: for unit vectors, rθ(h,z)=2h^θ(z)⊤h−2r_θ(h,z)=2\, h_θ(z) h-2 is, up to scale and a constant, the log-likelihood of a distribution on the unit sphere determined entirely by its mean direction h^θ(z) h_θ(z). Such a log-likelihood lower-bounds the information that explanations carry about activations (Barber and Agakov, 2003). The bound is loose whenever the model cannot represent the true conditional distribution, as with the two opposing candidates in Figure 4. Flow-NLA keeps this objective but replaces this model with the flow reconstructor, which can represent several modes. The diffusion likelihood bound (Ho et al., 2020; Kingma et al., 2021; Kingma and Gao, 2023) provides a tractable training signal: it relates this likelihood to how well the explanation helps the reconstructor denoise the activation across noise levels. The bound takes the form log⁡pθ(h∣z)≥Rθ(h,z)−c⁡(h) p_θ(h z)≥ R_θ(h,z)-c(h) (Appendix I), where Rθ(h,z)R_θ(h,z) is the explanation-dependent denoising term and c⁡(h)c(h) is independent of the explanation. We therefore use RθR_θ as the verbalizer’s reward, so that candidate explanations of the same activation are compared through their likelihood bounds. Flow matching trains the reconstructor to predict the velocity vtv_t, whereas the bound is expressed in noise-prediction error. The definitions of xtx_t and vtv_t imply btxt+atvt=(at2+bt2)ϵ=ϵb_tx_t+a_tv_t=(a_t^2+b_t^2)ε=ε. Substituting the predicted velocity therefore gives ϵ^θ(xt,t,z)=btxt+atfθ(xt,t,z) ε_θ(x_t,t,z)=b_tx_t+a_tf_θ(x_t,t,z), without training a second predictor. Figure 6: Changes in utility, confabulation, and writing defects from supervised initialization for point-reconstruction NLA (solid) and Flow-NLA (dashed). Markers show raw measurements, curves show three-checkpoint means, and shading shows pointwise 95% bootstrap intervals. We index noise levels by log signal-to-noise ratio, λ=log⁡(at2/bt2)λ= (a_t^2/b_t^2): low values correspond to mostly noise and high values to mostly signal. In this coordinate, the likelihood-bound term integrates noise-prediction error with constant weight 1/21/2 (Appendix I). We use λmin=−8 _ =-8 and λmax=8 _ =8, a finite band spanning both regimes. The reward is the negative integrated noise-prediction error: Rθ(h,z)=−12∫λminλmaxϵ‖ϵ−ϵ^θ(xλ,λ,z)‖22dλ.R_θ(h,z)=- 12 _ _ _ E_ε \|ε- ε_θ(x_λ,λ,z) \|_2^2\,dλ. (3) For optimal denoisers, each integrand is the clean-activation denoising error of Figure 4 scaled by eλe^λ, since ϵ−ϵ^∗=eλ/2([X∣xλ,z]−x)ε- ε^*=e^λ/2(E[X x_λ,z]-x). The reward therefore credits an explanation for every noise level at which it makes the observed activation easier to recover, including when the explanation leaves the conditional mean unchanged; in the example of Figure 4, RθR_θ favors zhorz_hor over an uninformative explanation by approximately log⁡2 2 nats, while the point reward rθr_θ is unchanged. This information-theoretic interpretation motivates constant weighting in log-SNR: it matches the likelihood-bound term, whereas arbitrary reweightings do not (Appendix J). The trained reconstructor and numerical integration approximate this ideal quantity (Kingma et al., 2021). Training the verbalizer. We follow the standard NLA training recipe described in Section 3, retaining the verbalizer’s supervised initialization and group-relative updates with a KL penalty (Fraser-Taliente et al., 2026). The reconstructor’s supervised initialization differs, as it is trained for velocity prediction rather than point reconstruction. We replace the point-reconstruction reward with Rθ(h,z)R_θ(h,z), estimated using sampled Gaussian noise and numerical integration over log-SNR [−8,8][-8,8], and fit the online reconstructor to the generated activation–explanation pairs using the denoising objective, in the same order as the standard recipe. Because the distribution behind this reward moves with the verbalizer, several choices mattered for training (Appendix F). Integrating over noise levels avoids uninformative extremes: nearly clean inputs reveal the activation regardless of the explanation, while nearly pure noise leaves only the conditional mean (Appendix H). To assess whether the reconstructor captures more distributional information as the verbalizer learns, we compare sample and real activation covariance within and outside the activations’ 64 leading principal components (Appendix F). This diagnostic ruled out our initial energy-score reward (Gneiting and Raftery, 2007), which compares the observed activation with reverse-generated samples: their spread outside the leading subspace decreased as mean reconstruction improved. The likelihood-bound reward requires no reverse-generated samples, only forward noising of the observed activation, and its reconstructor’s spread outside the leading subspace converges to that of the activations. Figure 7: Changes from supervised initialization in mean utility, assertion risk, token coverage, and writing defects, averaged over the second half of the training range shared by both methods. Filled circles denote NLA; open squares denote Flow-NLA; lines show the gap between methods. 4.3 Utility and Confabulation under Richer Feedback We evaluate whether Flow-NLA changes the balance between recoverable information and explanation defects using the evaluation framework from Section 3.1. We compare Flow-NLA with point-reconstruction NLA on Qwen2.5-7B-Instruct, Gemma-3-12B, and Apertus-v1.5-8B, examining whether the effect of richer feedback is consistent across LLMs. For each model, we compare both methods’ training trajectories in utility, contextual confabulation, and writing quality (Figure 6). On all three target models, the two methods follow similar utility trajectories, while confabulation and writing defects grow significantly less under Flow-NLA. Figure 7 summarizes this late in training: Flow-NLA retains the mean utility gain (averaged over classification, suffix, and behavior), while its assertion risk and token coverage stay near or below supervised initialization rather than rising by up to 11 points, and its writing defects grow by 0.23–0.32 rather than 0.71–1.00 per explanation. Flow-NLA also has the lowest assertion risk, token coverage, and writing defects of all compared NLAs, including the released ones (Table 2, Appendix A). Across target models, substantial utility gains can thus be retained with less growth in confabulation and defects, establishing richer reconstruction feedback as a promising direction for NLA training. 5 Conclusions, Limitations, and Future Work Current NLAs grow less reliable as they learn: during point-reconstruction training, explanations become more useful but also more confabulated and less clearly written, and the released NLAs show the same pattern. Our evaluation framework exposes this by scoring utility, source support, and writing quality separately. This deterioration points to a limitation of the current objective: point reconstruction rewards whatever text helps recover the activation, while remaining blind to much of the information an explanation carries. Flow-NLA replaces it with a conditional denoising reward that credits this richer information. Across three target models, Flow-NLA keeps its utility gains while sharply curbing the growth of confabulation and writing defects during training. Our study is limited to the models, tasks, and training procedures considered, relies on automated judgments, and includes provider and inference differences that limit causal attribution to the reconstruction objective alone (Appendix B). Our method and analysis also assume optimal denoisers and a fixed verbalizer, and richer reconstruction feedback does not guarantee source support or mechanistic faithfulness. Future work should better characterize how reconstruction rewards induce confabulation and design training objectives that encourage informative, supported explanations. References Barber and Agakov (2003) D. Barber and F. V. Agakov Information maximization in noisy channels: a variational approach. In Advances in Neural Information Processing Systems, Vol. 16. External Links: Link Cited by: §4.2, §4.2. Bellemare et al. (2017) M. G. Bellemare, W. Dabney, and R. Munos A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, p. 449–458. External Links: Link Cited by: §4.2. Chalnev (2026) S. Chalnev Cycle-consistent activation oracles. Note: LessWrong research post External Links: Link Cited by: §2. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4.1-Flash: pushing the limits of KV cache compression. External Links: 2609.19969, Link Cited by: §B.2, §3.1. Dingeto (2026) H. Dingeto Train the model, not the reader: decodability supervision for verifiable activation explanations. External Links: 2607.20379, Link Cited by: §1. Eysenbach et al. (2019) B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine Diversity is all you need: learning skills without a reward function. In International Conference on Learning Representations, Cited by: §4.2. Fraser-Taliente et al. (2026) K. Fraser-Taliente, S. Kantamneni, E. Ong, D. Mossing, C. Lu, P. C. Bogdan, E. Ameisen, J. Chen, D. Kishylau, A. Pearce, J. Tarng, A. Wu, J. Wu, Y. Zhang, D. M. Ziegler, E. Hubinger, J. Batson, J. Lindsey, S. Zimmerman, and S. Marks Natural language autoencoders produce unsupervised explanations of LLM activations. Transformer Circuits Thread. External Links: Link Cited by: §B.1, Appendix E, Appendix F, §1, §1, §2, §2, §3.2, §3.2, §3.2, §3, §4.1, §4.2. Gemma Team (2025) Gemma Team Gemma 3 technical report. External Links: 2503.19786, Link Cited by: §3.2. Gneiting and Raftery (2007) T. Gneiting and A. E. Raftery Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102 (477), p. 359–378. External Links: Document, Link Cited by: Appendix F, §4.2. Guo et al. (2005) D. Guo, S. Shamai, and S. Verdú Mutual information and minimum mean-square error in Gaussian channels. IEEE Transactions on Information Theory 51 (4), p. 1261–1282. External Links: Document, Link Cited by: Appendix J. Hernández-Cano et al. (2026) A. Hernández-Cano, A. Hägele, A. H. Huang, et al. Apertus: democratizing open and compliant LLMs for global language environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 46877–46955. External Links: Link, Document Cited by: §3.2. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Vol. 33, p. 6840–6851. External Links: Link Cited by: Appendix I, item 3., §4.2. Karvonen et al. (2026a) A. Karvonen, J. Chua, C. Dumas, K. Fraser-Taliente, S. Kantamneni, J. Minder, E. Ong, A. Sen Sharma, D. Wen, O. Evans, and S. Marks Activation oracles: training and evaluating LLMs as general-purpose activation explainers. In International Conference on Machine Learning, External Links: 2512.15674, Link Cited by: §1, §2. Karvonen et al. (2026b) A. Karvonen, E. Ong, S. Kantamneni, and S. Marks Would this change your answer? Evaluating explanations of LLM behavior in the wild with counterfactual experiments. External Links: 2608.16747, Link Cited by: §2. Kingma and Gao (2023) D. P. Kingma and R. Gao Understanding diffusion objectives as the ELBO with simple data augmentation. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix I, §4.2. Kingma et al. (2021) D. P. Kingma, T. Salimans, B. Poole, and J. Ho Variational diffusion models. In Advances in Neural Information Processing Systems, Vol. 34, p. 21696–21707. External Links: Link Cited by: Appendix I, Appendix I, item 3., §4.2, §4.2. Li et al. (2026) M. Li, A. M. Ceballos Arroyo, G. Rogers, N. Saphra, and B. C. Wallace Do activation verbalization methods convey privileged information?. In International Conference on Machine Learning, External Links: Link Cited by: §2. Lillicrap et al. (2016) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra Continuous control with deep reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: Appendix F. Miao and Blunsom (2016) Y. Miao and P. Blunsom Language as a latent variable: discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 319–328. External Links: Document Cited by: §4.2. Min et al. (2023) S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 12076–12100. External Links: Document, Link Cited by: item 1., §2, §3.1. Pan et al. (2026) A. Pan, L. Chen, and J. Steinhardt LatentQA: teaching LLMs to decode activations into natural language. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Penedo et al. (2024) G. Penedo, H. Kydlíček, L. Ben Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. von Werra, and T. Wolf The FineWeb datasets: decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, Vol. 37, p. 30811–30849. External Links: Link Cited by: §B.1. Rita et al. (2022) M. Rita, C. Tallec, P. Michel, J. Grill, O. Pietquin, E. Dupoux, and F. Strub Emergent communication: generalization and overfitting in Lewis games. In Advances in Neural Information Processing Systems, Vol. 35. Cited by: §4.2. Saharia et al. (2023) C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (4), p. 4713–4726. External Links: Document Cited by: §4.2. Salimans and Ho (2022) T. Salimans and J. Ho Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, External Links: Link Cited by: §4.2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §3. Song et al. (2024) Y. Song, Y. Kim, and M. Iyyer VeriScore: evaluating the factuality of verifiable claims in long-form text generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 9447–9474. External Links: Document, Link Cited by: item 1., §2, §3.1. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, et al. Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §3.2. Zhang and Turner (2026) M. Zhang and A. Turner How robust are natural language autoencoders to initialization?. Note: Research report External Links: Link Cited by: §1, §2. Zheng et al. (2024) L. Zheng, W. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang LMSYS-Chat-1M: a large-scale real-world LLM conversation dataset. In International Conference on Learning Representations, External Links: Link Cited by: §B.2. Appendix A Absolute Model Comparison Table 2: Utility and confabulation across NLA training methods. We compare released NLAs, our point-reconstruction reruns, and Flow-NLA within the same target model. All scores are percentages except writing defects, which are counts per explanation. Best results are highlighted within each target model. Checkpoint selection is described in Appendix B.2. Utility ↑ Confabulation ↓ Writing ↓ Target model Method Class. Suffix Behavior Risk Coverage Defects Qwen2.5-7B Released 79.479.4 98.0 82.7 92.692.6 74.874.8 2.452.45 Point recon. 79.779.7 97.697.6 74.974.9 83.783.7 68.368.3 1.631.63 Flow-NLA 80.3 96.696.6 72.972.9 71.9 54.7 0.99 Gemma-3-12B Released 84.7 96.6 83.483.4 83.183.1 67.567.5 1.781.78 Point recon. 84.284.2 96.296.2 82.482.4 77.877.8 44.244.2 1.381.38 Flow-NLA 83.883.8 96.296.2 84.8 60.8 33.8 0.89 Apertus-v1.5-8B Point recon. 77.577.5 96.0 73.6 81.381.3 41.541.5 1.501.50 Flow-NLA 78.4 95.495.4 69.669.6 72.8 38.9 0.75 Appendix B Evaluation Details B.1 Activation extraction Table 3 lists the extraction site for each target model. As in the original NLA code (Fraser-Taliente et al., 2026), our reruns and Flow-NLA share the extraction site of their target model. Training activations come from FineWeb (Penedo et al., 2024) sample-10BT documents with at least 128 tokens, truncated to 1,024 tokens, with 10 token positions per document. Documents are split disjointly into 25% verbalizer supervised initialization, 25% reconstructor supervised initialization, and 50% online training. Table 3: Activation extraction sites. Revisions are Hugging Face commit prefixes; released NLAs are identified by their repository names. A released NLA at layer K reads the output of decoder block K, i.e., hidden_states[K+1K+1]. Target model Revision or release Hidden state Ours Qwen2.5-7B-Instruct a09a354 hidden_states[21] Gemma-3-12B-it 96b6f1e hidden_states[33] Apertus-v1.5-8B a411d83 hidden_states[22] Released Qwen2.5-7B-Instruct nla-qwen2.5-7b-L20 hidden_states[21] Gemma-3-12B-it nla-gemma3-12b-L32 hidden_states[33] Gemma-3-27B-it nla-gemma3-27b-L41 hidden_states[42] Llama-3.3-70B-Instruct Llama-3.3-70B-NLA-L53 hidden_states[54] Evaluation token positions. For classification, the activation is extracted at the last source-text token, before any evaluation question. For suffix prediction, it is extracted at the last prefix token, before the true continuation. For response behavior, it is extracted at the first token of a freshly generated response from the target model. These positions are the same for all target models. Explanations are sampled at temperature 1 with at most 150 new tokens. B.2 Evaluation protocols All judges, including the hard-negative judge (Appendix D), are DeepSeek V4.1 Flash (DeepSeek-AI, 2026) with thinking disabled and temperature zero. Prompt templates are described in Appendix G. Utility tasks, controls, and aggregation. We evaluate utility on source-property classification, suffix prediction, and response-behavior prediction. In all cases, the utility judge receives the task query and the explanation. Table 4: Overview of our various utility tasks Task Judge’s task Examples Options Metric Classification Source properties (12 tasks) 1,344 2–14 Mean accuracy Suffix selection True 32-token continuation 500 10 Accuracy Response behavior Answer, refuse, or clarify 248 3 Macro-F1 Hard negatives Contrastive question on source pair 1,316 (658 pairs) 2 Log-prob. ratio For target instance i, let qiq_i denote the task query, iC_i the candidate answers, yiy_i the reference answer, ziz_i the matched explanation, and zj⁡(i)z_j(i) the hard-negative explanation, generated for the paired source j⁡(i)j(i). Where predictive probabilities over iC_i are available, we define Lmatched,i L_matched,i =−log⁡pJ(yi∣qi,i,zi), =- p_J(y_i q_i,C_i,z_i), (4) Lnoexp,i L_noexp,i =−log⁡pJ(yi∣qi,i,∅), =- p_J(y_i q_i,C_i, ), Lhard,i L_hard,i =−log⁡pJ(yi∣qi,i,zj⁡(i)). =- p_J(y_i q_i,C_i,z_j(i)). We then decompose the hard-negative separation as Ugain,i U_gain,i =Lnoexp,i−Lmatched,i, =L_noexp,i-L_matched,i, (5) Upenalty,i U_penalty,i =Lhard,i−Lnoexp,i, =L_hard,i-L_noexp,i, Ucontrast,i U_contrast,i =Lhard,i−Lmatched,i=Ugain,i+Upenalty,i. =L_hard,i-L_matched,i=U_gain,i+U_penalty,i. Thus, UgainU_gain measures the predictive benefit of the matched explanation over no explanation, whereas UpenaltyU_penalty measures the degradation caused by the hard-negative explanation. Their sum is the hard-negative separation. We report these components separately because a larger separation may reflect more useful matched explanations, more misleading hard negatives, or both. Appendix D describes how hard-negative pairs are constructed and reports the decomposition for all runs. The classification suite contains 112 source examples for each of 12 tasks, covering topic, sentiment, subjectivity, entailment, entity and exact-name presence, tense, grammatical number, language, and pronoun form, for 1,344 examples per checkpoint and method. We average accuracy over successful generations and parsed responses within each task and then weight the 12 tasks equally; details and data sources will be available in our repository upon acceptance. Suffix prediction uses 500 documents with ten fixed candidate continuations per example. The activation is extracted at the end of the prefix before the true 32-token continuation. Response-behavior prediction uses 248 LMSYS-Chat-1M (Zheng et al., 2024) conversations. The judge sees only the explanation and makes a forced prediction of answer, refusal, or clarification. Reference labels come from each target model’s actual response. We report the macro-F1 score. Source-support judgments. The source-support judge extracts distinct assertions and assigns supported, contradicted, unsupported, or not assessed verdicts. Reasonable entailments count as supported; a contradiction requires incompatible source evidence. Future-text predictions and hidden-computation claims are not assessed and are excluded from both metrics. Token coverage counts the union of tokens overlapping unsupported or contradicted claims. Writing defects. The writing judge sees only the explanation. It identifies distinct defects in clarity, coherence, and economy, assigning each defect one category and an exact evidence span. An annotation is accepted only when the quoted span occurs exactly in the explanation. Checkpoint comparison. Classification, suffix, and confabulation changes use each series’ own supervised initialization as reference. The models are evaluated every 100 updates through 1600. Tables 1 and 2 report absolute checkpoint scores rather than changes from supervised initialization. We select one checkpoint per run by the highest unweighted mean of classification accuracy, suffix accuracy, and response-behavior macro-F1 among updates 1300–1600 for a fair comparison. Released checkpoints are evaluated unchanged, with the same judges and protocols as our runs. Classification, assertion risk, and token coverage weight the 12 classification tasks equally. Appendix C Individual Evaluation Panels Figure 8 shows the individual utility, confabulation, and writing trajectories of the point-reconstruction NLA reruns summarized in Section 3.2, including the assertion-risk and writing-defect components of the confabulation index. Figure 8: Changes in utility, confabulation, and writing defects from supervised initialization for the point-reconstruction NLA reruns. Shading shows pointwise 95% bootstrap intervals. Appendix D Hard-Negative Separation Pair construction. Hard-negative pairs are drawn from the classification sources. For each of the 12 datasets, we retrieve lexical nearest neighbors and write one question that the two sources answer differently (factual, linguistic, or evaluative), with the two answers as candidates. This gives 658 disjoint pairs covering 1,316 of the 1,344 classification sources. Judge and aggregation. The judge is DeepSeek V4.1 Flash with thinking disabled, as for the other utility tasks. It answers with an option letter, and we read the log-probability of the correct letter, normalized over the two letters. Each question is scored with both answer orders to cancel letter-position bias. Losses follow Eq. 4 and the decomposition follows Eq. 5. We average within each dataset and then weight datasets equally. Decomposition. Figure 9 separates the total separation of Figure 3 into matched predictive gain UgainU_gain and mismatch penalty UpenaltyU_penalty. Without an explanation, the judge is at chance on every pair. Most of the separation, and most of its rise during training, comes from the mismatch penalty: it grows from 0.44–0.83 nats at supervised initialization to 0.91–1.86 nats from update 800 onward, while the matched gain grows from 0.19–0.29 to 0.31–0.49 nats. Both components level off after about 200 updates. Late in training, Flow-NLA matches NLA on both components for Apertus and has a slightly lower mismatch penalty for Qwen and Gemma (by 0.10 and 0.16 nats). Figure 9: Matched predictive gain, mismatch penalty, and total hard-negative separation across the 12 classification datasets for point-reconstruction NLA (solid) and Flow-NLA (dashed). Curves are three-checkpoint means over raw markers; shading shows 95% bootstrap intervals. Appendix E Training Details Supervised initialization uses explanations written by Qwen3.6-35B-A3B for the activation sites of Appendix B.1. Each run uses 8 NVIDIA GH200 GPUs. For reruns, we use the same hyperparameter settings as the released Anthropic code (Fraser-Taliente et al., 2026); for Apertus, we use the hyperparameters of Qwen2.5-7B as it is the closest in parameter size. Appendix F Making the Distributional Reward Trainable Using a co-trained likelihood bound as a policy-gradient reward raised problems that a fixed reward model does not. This section records the failure modes we encountered and how we addressed them. All diagnostics reported here are for Qwen2.5-7B-Instruct. The reconstructor tracks a moving target. The conditional distribution pϕ(h∣z)p_φ(h z) changes with the verbalizer, and the reconstructor measurably follows it. Between updates 0 and 200, the co-trained reconstructor’s score improved by 38.8 units on explanations from the update-200 verbalizer, but by only 5.7 units on explanations from supervised initialization. A frozen reconstructor also confirms that the verbalizer’s gains are real: it scores update-200 explanations above initial ones (0.466 versus 0.332). Monitoring with a fixed reconstructor therefore measures transfer to old explanations, not joint progress. Scoring with an averaged reconstructor. Unlike the released NLA recipe, which scores explanations with the current reconstructor (Fraser-Taliente et al., 2026), we score them with an exponential moving average (EMA) of the reconstructor’s parameters with decay 0.99. Rewards thus follow the moving conditional distribution gradually rather than jumping with each reconstructor update. Such parameter averaging is common in off-policy RL, where the targets that critics are trained on are computed with an EMA of their parameters (Lillicrap et al., 2016). Single noise levels give weak credit. An early reward used velocity error at a single uniformly sampled noise level. At nearly clean inputs, the noisy activation reveals the target and alternative explanations receive almost the same score. The gap between mismatched and matched explanations was 0.003 at t=0.05t=0.05 versus 0.625 at t=0.95t=0.95. At pure noise the reward reduces to point reconstruction (Appendix H). In the retained reward, the variance that ranks candidate explanations comes mostly from intermediate log-SNR: 2.3% from λ≤−4λ≤-4, 34.9% from (−4,0](-4,0], 60.9% from (0,4](0,4], and 1.9% from λ>4λ>4. Integrating across the band with likelihood weighting captures this signal without choosing a single noise level. Monitoring distributional fit. Reconstruction FVE measures only how well the reconstructor recovers the mean, so we also check whether the spread of its samples matches the spread of the activations. On a fixed panel of held-out activation–explanation pairs, we draw 16 samples per explanation by running the reconstructor in reverse from noise. We compare the covariance of these samples around their mean with the covariance of the activations around the reconstructor’s mean prediction, separately inside and outside the subspace spanned by the 64 leading principal components of real activations. The PCA-64 covariance overlap is one minus the normalized squared Bures distance between the two covariances within this subspace; it equals one when they are identical. The off-PC variance ratio divides the samples’ variance outside the subspace by that of the activations, corrected for the finite number of samples; it equals one when the model is calibrated there, and values below one indicate samples that are too concentrated. For the reconstructor used by the likelihood-bound reward, the off-PC variance ratio moves from 1.26 to 1.00 over reconstructor training steps 50–1,600, while the PCA-64 overlap rises from 0.77 to 0.89. An energy-score reward. Before adopting the likelihood bound, we rewarded the verbalizer with the negative energy score (Gneiting and Raftery, 2007) of K=16K=16 reverse samples x^(1),…,x^(K) x^(1),…, x^(K) per explanation, ES⁡(x)=1K∑k‖x^(k)−x‖2−12K(K−1)∑k≠l‖x^(k)−x^(l)‖2,ES(x)= 1K _k\| x^(k)-x\|_2- 12K(K-1) _k≠ l\| x^(k)- x^(l)\|_2, (6) a proper scoring rule minimized in expectation by the true conditional distribution, with common noise across candidates. Unlike the likelihood bound, which only requires noising the observed activation forward, this reward depends on samples generated by running the reconstructor in reverse, so its value is only as good as the spread of those samples. The reconstructor trained for this reward lost spread outside the leading subspace: its off-PC variance ratio fell from 0.96 to 0.78 over reconstructor training steps, while its PCA-64 overlap stayed near 0.66 and FVE rose from 31.8% to 49.6%. Better mean reconstruction thus came with increasingly concentrated samples during energy reward training. Appendix G Judge Prompts The exact prompt templates will be available in the prompts/ directory of our evaluation repository upon acceptance. The utility judge receives the task query, its candidate answers, and the explanation, and returns a single answer; classification, suffix selection, and response behavior each have their own template. The source-support judge receives the source text and the explanation, splits the explanation into atomic claims, and returns a verdict for each claim with quoted source evidence for supported and contradicted claims. The writing judge receives only the explanation and returns each defect with its category and quoted span. All judges return structured output, which is validated before scoring. Appendix H Why Denoising Identifies the Conditional Distribution This section works through the example of Figure 4 and then shows that denoising at an interior noise level identifies the explanation-conditioned distribution. Illustrative example. In Figure 4, the activation is uniform over (±1,0)(± 1,0) and (0,±1)(0,± 1), and the explanation zhorz_hor identifies the horizontal pair. Both the unconditional and the zhorz_hor-conditional means are zero, so the optimal unit-norm point loss is 22 with or without the explanation. For this illustration, x=hx=h and Xt=atX+btϵX_t=a_tX+b_tε, with ϵ∼⁡(0,I)ε (0,I) and at2+bt2=1a_t^2+b_t^2=1. Panels (b)–(d) fix bt/at=0.6b_t/a_t=0.6 and use the observation xt=at(0.55,0.5)x_t=a_t(0.55,0.5) of the true activation (1,0)(1,0); shading shows the Gaussian-corrupted conditional densities. Writing mz(xt,t)=[X∣Xt=xt,Z=z]m_z(x_t,t)=E[X X_t=x_t,Z=z], the field arrows in panels (c)–(d) depict dz(xt,t)=atmz(xt,t)−xt=−btϵ^∗(xt,t,z)=bt2∇xtlog⁡pt(xt∣z).d_z(x_t,t)=a_tm_z(x_t,t)-x_t=-b_t ε^*(x_t,t,z)=b_t^2 _x_t p_t(x_t z). (7) Thus, the displacement fields encode gradients of the noisy conditional log-density. Flow-NLA scores noise predictions across noise levels using the likelihood bound in Appendix I, allowing its reward to distinguish these equal-mean distributions. With equally likely groups, group information leaves optimal point loss at 22 but improves the optimal expected integrated denoising reward by approximately log⁡2 2 nats over log-SNR [−8,8][-8,8], computed using exact posterior predictors and Gaussian quadrature (Appendix J). Distribution identification. Fix the verbalizer, so that the joint activation–explanation distribution is fixed. At an interior noise level with at,bt>0a_t,b_t>0, squared-error training yields the population-optimal field f∗(xt,t,z)=[vt∣xt,z]f^*(x_t,t,z)=E[v_t x_t,z]. Its implied noise prediction is ϵ^∗(xt,t,z)=[ϵ∣xt,z] ε^*(x_t,t,z)=E[ε x_t,z]. Gaussian corruption relates this prediction to the score of the noisy density, ∇xtlogpt(xt∣z)=−ϵ^∗(xt,t,z)/bt _x_t p_t(x_t z)=- ε^*(x_t,t,z)/b_t, where ϵ^∗(xt,t,z)=btxt+atf∗(xt,t,z) ε^*(x_t,t,z)=b_tx_t+a_tf^*(x_t,t,z). Its score determines it up to a normalization constant, fixed by requiring it to integrate to one. Recovering the noisy distribution therefore identifies the original conditional activation distribution at optimality. At the pure-noise endpoint at=0a_t=0, the velocity target is −X-X and the optimum depends only on the conditional mean. Interior noise levels are what allow equal-mean distributions to be distinguished. Appendix I The Conditional Likelihood Bound The reward in Eq. 3 is the negative interior loss of a conditional diffusion variational bound (Ho et al., 2020; Kingma et al., 2021; Kingma and Gao, 2023). We define the diffusion model on the chosen finite noise band, with an explanation-independent Gaussian prior at the low-SNR boundary and an explanation-independent observation decoder at the high-SNR boundary. The explanation conditions the reverse transitions between these endpoints. This specification ensures that the omitted endpoint terms do not vary between candidate explanations of the same activation. Because activations lie on a sphere, we take densities with respect to the image ν of sphere surface measure under the fixed transform x=T⁡(h)=(dh−μ)/sx=T(h)=( d\,h-μ)/s. Following Kingma et al. (2021), the decoder is r⁡(x∣xhi)∝q⁡(xhi∣x)r(x x_hi) q(x_hi x), normalized over ν, where q is the forward Gaussian corruption at the high-SNR boundary. Applying the standard variational bound separates the endpoint terms from KL divergences between forward posteriors and learned reverse transitions. For Gaussian reverse transitions with the forward posterior variance, the interior terms are weighted squared noise-prediction errors. In the continuous log-SNR limit, under the regularity and integrability conditions for that limit, their sum becomes the negative of Eq. 3. Consequently, log⁡pθ(h∣z)≥Rθ(h,z)−c⁡(h) p_θ(h z)≥ R_θ(h,z)-c(h), where c⁡(h)c(h) contains the endpoint terms and is independent of z. The bound applies to a well-defined reverse process without requiring optimal denoising. The velocity and noise parameterizations obey ϵ−ϵ^θ=at(vt−fθ)ε- ε_θ=a_t(v_t-f_θ), so the reward weights velocity error by at2a_t^2 before integrating over log-SNR. This differs from the reconstructor’s coordinate-averaged training loss. Sampled noise and numerical integration approximate the bound-derived reward; computing candidate scores does not require evaluating the fixed observation decoder. Appendix J Information Gain Across the Noise Band Fix the verbalizer and assume optimal denoisers with and without the explanation. Let ℰZ(λ)E_Z(λ) and ℰ0(λ)E_0(λ) denote their expected squared noise-prediction errors, and write Yλ=eλ/2X+ϵY_λ=e^λ/2X+ε, where ϵε is independent standard Gaussian noise. Under finite second moments, the I-MMSE relation (Guo et al., 2005) gives 12∫λminλmax[ℰ0(λ)−ℰZ(λ)]dλ=I⁡(Z,Yλmax)−I⁡(Z,Yλmin). split 12 _ _ _ [E_0(λ)-E_Z(λ)]\,dλ=I(Z;Y_ _ )-I(Z;Y_ _ ). split (8) The explanation’s denoising benefit therefore measures information gained across the selected noise band. An added detail can improve this quantity by distinguishing modes or other distributional structure even when the conditional mean remains unchanged. The equal-mean example in the main text illustrates such a distinction. The finite band omits slight information already visible at its low-SNR boundary and information still unresolved at its high-SNR boundary. The unconditional error contributes the same term to every candidate score for a fixed activation and can be omitted without changing candidate comparisons.

Instant research discovery