Paper deep dive
Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation
Will Badr
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/8/2026, 2:29:29 AM
Summary
This paper evaluates whether hints in code generation provide missing information or merely steer models toward solutions they could already produce. Using Qwen2.5-3B-Instruct and Phi-3.5-mini on HumanEval+ and MBPP+, the study finds that while relevant hints rescue failures, most of these solutions are also reachable via unhinted sampling. Mechanistic interventions (activation addition, low-rank updates) and virtual-KV prefixes show limited or imprecise capability transfer. Post-generation hidden-state probes demonstrate cross-benchmark correctness decodability but do not establish model self-knowledge.
Entities (8)
Relation Signals (8)
Qwen2.5-3B-Instruct → evaluatedon → HumanEval
confidence 100% · For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures... on HumanEval+ and MBPP+
Qwen2.5-3B-Instruct → evaluatedon → MBPP
confidence 100% · For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures... on HumanEval+ and MBPP+
Phi-3.5-mini → evaluatedon → HumanEval
confidence 100% · Phi-3.5-mini reproduces this pattern... on HumanEval+ and MBPP+
Phi-3.5-mini → evaluatedon → MBPP
confidence 100% · Phi-3.5-mini reproduces this pattern... on HumanEval+ and MBPP+
Relevant Hints → rescuesfailuresin → Qwen2.5-3B-Instruct
confidence 95% · For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures
Unhinted Sampling → coversrescuesof → Relevant Hints
confidence 90% · eight unhinted samples solve 46 and recover 31 of the 36 relevant-hint rescues
hidden-state probes → demonstratestransferacross → Benchmarks
confidence 90% · Post-generation hidden-state probes transfer across benchmarks, with pooled AUROC 0.806 and 0.780
Virtual-KV Prefixes → underperforms → Full Textual Specifications
confidence 90% · Full textual specifications solve 22 of 24 context-defined problems, versus 5-11 for tested virtual-KV prefixes.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When a hint turns a failing generated program into a passing one, does it provide missing information or merely steer the model toward a solution it could already produce? We test these hypotheses on HumanEval+ and MBPP+ using executable evaluation. For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures; an unrelated hint rescues 19, while eight unhinted samples solve 46 and recover 31 of the 36 relevant-hint rescues. Phi-3.5-mini shows the same pattern: relevant hints rescue 42 of 101 failures, an unrelated hint rescues 17, and unhinted sampling solves 57, including 36 of the 42 relevant-hint rescues. Because the hint conditions use different attempt budgets, these comparisons do not isolate a purely semantic effect. Mechanistic tests on Qwen identify a stable activation direction shared by relevant and unrelated hints. Persistently adding this direction yields 14 rescues and 18 regressions, with no detectable net accuracy gain; learned low-rank interventions have a positive but imprecise estimated effect. Full textual specifications solve 22 of 24 context-defined problems, versus 5-11 for tested virtual-KV prefixes. Post-generation hidden-state probes transfer across benchmarks, with pooled AUROC 0.806 and 0.780, but their top-one selection advantage over token confidence is statistically unresolved. Overall, relevant hints can rescue failures, but most rescued solutions are already reachable through ordinary sampling, and the internal interventions tested here do not establish task-general capability transfer.
Tags
Links
- Source: https://arxiv.org/abs/2609.01106v1
- Canonical: https://arxiv.org/abs/2609.01106v1
Trouble viewing inline? Open PDF directly →
Full Text
62,212 characters extracted from source content.
Expand or collapse full text
Hints Help. But Do They Teach? Testing Skill Transfer in Code Generation Will Badr Email: scpg0807@leeds.ac.uk Abstract When a hint turns a failing generated program into a passing one, does it supply missing information or help the model reach a solution it could already produce? We test these explanations on HumanEval+ and MBPP+ with executable evaluation. For Qwen2.5-3B-Instruct, an adaptive relevant-hint procedure rescues 36 of 79 selected failures; one unrelated hint rescues 19, while eight unhinted samples solve 46 and cover 31 of the 36 hint rescues. Phi-3.5-mini reproduces this pattern: 42 of 101 failures are rescued by relevant hints, 17 by an unrelated hint, and 57 by unhinted sampling, which covers 36 of the 42 rescues. Because the hint procedures use different attempt budgets, their difference does not isolate a semantic effect. Mechanistic tests on Qwen find a stable activation direction shared by relevant and unrelated hints. Persistent addition is associated with 14 rescues and 18 failures, with no detected net accuracy change, while the estimated advantage of learned low-rank interventions remains positive but imprecise. Full textual specifications solve 22 of 24 context-defined problems, compared with 5–11 for the tested virtual-KV prefixes. Post-generation hidden-state probes transfer across benchmarks with pooled AUROC 0.806 and 0.780, although their top-one selection advantage over token confidence is statistically unresolved. Overall, the adaptive relevant-hint procedure rescues failures under the implemented conditions, but most rescues are already accessible through sampling; the tested internal interventions do not establish task-general capability transfer. 1. Introduction A code model fails, receives a short hint, and passes. This before-and-after result is often described as capability transfer, but it does not distinguish three explanations. The hint may supply missing information; added context may redirect generation toward a solution the model could already produce; or implementation-level variation may change the output without any meaningful intervention effect. A pass/fail comparison alone cannot tell these cases apart. HumanEval/117 makes the ambiguity concrete. The baseline program fails. The relevant hint, “Consider how the current implementation treats uppercase and lowercase letters differently,” produces a passing program. Yet a length-matched unrelated hint about the modulus operator also produces a pass, and at least one of eight unhinted samples passes as well. The relevant hint may still guide generation more effectively; the pass transition alone cannot tell us whether it supplied inaccessible information. We study the distinction in code generation, where functional correctness can be evaluated by execution rather than by a language-model judge. We begin with short teacher-generated hints on HumanEval+ and MBPP+, then test successively smaller internal representations: a vector added to hidden activations, low-rank activation spaces, and short learned blocks of attention-memory vectors (virtual KV prefixes). At each stage, a control answers a narrower question. Replay measures pipeline instability. Unrelated hints measure generic prompt effects. Repeated no-hint sampling asks whether the rescued behavior already appears without a hint. Norm-matched random and shuffled interventions test task specificity. Held-out tasks test transfer beyond the data used to construct a representation. Size-matched prefixes test whether training beats an untrained object of the same size. Controls narrow the interpretation at every stage. Unrelated hints rescue many failures, and most relevant-hint rescues also appear among eight unhinted samples. Relevant and unrelated hints induce nearly the same activation direction, whose full-benchmark deployment is associated with slightly more damage than rescue. Learned subspaces fit training deltas but have an imprecise held-out advantage over controls. Full context enables the synthetic procedures, whereas the tested KV-prefix objective fits its exemplars without consistent held-out transfer. These results are specific to the tested interventions; they do not establish a general impossibility. Post-generation hidden states provide a separate positive result. A linear probe predicts functional correctness across benchmarks when all representation and hyperparameter choices are made on the source benchmark. Its top-one advantage over token confidence remains statistically unresolved, so we interpret the result as cross-benchmark correctness decodability rather than model self-knowledge. Our contributions are: 1. A two-model behavioral audit of hint rescue. We compare relevant hints with unrelated hints and no-hint best-of-eight sampling on 79 selected Qwen failures and independently repeat the pattern on 101 selected Phi failures; replay is measured in the primary Qwen study. 2. A separation of geometric stability, behavioral change, and task specificity. We show that a stable activation direction is associated with changed outputs without improving aggregate accuracy or carrying a detectable held-out task-specific residual. 3. A controlled context-compression test. We contrast full specifications with trained and size-matched virtual-KV prefixes on context-defined procedural tasks, while reporting the limits of the empirical no-context gate. 4. A leakage-resistant correctness readout and reusable audit trail. We select probe representations on source-benchmark grouped folds, test both transfer directions against text and surface baselines, and release task-level ledgers and the reanalysis script. Behavioral results cover Qwen and Phi; the mechanistic, compression, and correctness-readout analyses use Qwen only. The benchmarks contain short Python programs, and the prefix experiment tests one training objective. We therefore restrict our claims to these models, tasks, channels, and procedures. 2. Operational Definitions We use the following operational definitions to keep behavioral access, intervention transfer, and correctness decoding distinct. Rescue. A task is rescued by condition c when the baseline greedy completion fails the EvalPlus base and plus tests and the completion under c passes both. Sampled support at budget k. A task is in the model’s sampled support at budget k when at least one of k no-hint samples passes. This is an empirical, budget-dependent definition; failure at finite k does not prove that success has zero probability. Semantic hint advantage. The behavior attributable specifically to relevant content is the paired difference between relevant and attempt-, style-, length-, and seed-matched irrelevant hints. The present experiment includes unrelated hints but not every element of this matched estimand, so the observed 36-versus-19 difference is suggestive rather than a complete causal decomposition. Intervention transfer. An activation object transfers when it is estimated without the evaluation task and improves held-out behavior relative to norm-, rank-, channel-, and compute-matched controls. High cosine stability or training-set energy capture alone is representational evidence, not transfer. Context-defined procedure. A procedure is empirically context-dependent when repeated no-context and unrelated-context attempts rarely succeed but a full specification and worked examples succeed. We avoid the stronger phrase “provably absent” unless the task construction makes success information-theoretically impossible without a randomized secret. Correctness readout. A probe is a readout when it predicts the execution label from hidden states. A successful readout establishes decodability under its train/test protocol. It does not by itself establish that the base model uses the feature during generation. 3. Related Work 3.1 Task vectors and activation interventions Hendel et al. [2023] and Todd et al. [2023] show that in-context demonstrations can induce compact task or function vectors whose addition reproduces behavior on controlled mappings. Activation Addition [Turner et al., 2023], Contrastive Activation Addition [Panickssery et al., 2023], and representation engineering [Zou et al., 2023] similarly use directions in activation space to steer high-level output properties. These results motivate our search for a compact representation of a helpful hint. Our setting differs in using long-form program generation, executable correctness, and task-level rescue and damage rather than only target-property change. Causal efficacy is not sufficient for faithful interpretation. Makelov et al. [2023] demonstrate that a subspace patch can change behavior through a pathway that does not faithfully localize the hypothesized feature. We add complementary controls: split-half re-estimation, relevant-versus-irrelevant directions, a positive-control test of the intervention channel, held-out task transfer, matched random and shuffled interventions, and full-population net effects. 3.2 Prompt sensitivity, placebos, and resampling Self-consistency [Wang et al., 2022] established the practical value of sampling multiple reasoning trajectories. Macar et al. [2025] argue that causal interpretation of reasoning also requires resampling rather than reliance on a single trajectory. Mukherjee et al. [2024] show that socio-demographic prompt effects can resemble responses to arbitrary placebo tokens, while Kim et al. [2026] find that untrained random soft prompts can broaden early-token diversity and improve pass@N. Luo et al. [2026] report that question-asking interventions depend strongly on self-consistency and can diagnose errors without reliably repairing them. These studies motivate our unrelated-hint, replay, and no-hint pass@8 controls. Code execution lets us measure whether each changed trajectory is functionally correct. 3.3 Context and compact skill representations Gist tokens compress prompts using an attention-mask training scheme [Mu et al., 2023], while task and function vectors compress demonstrated mappings into activations. Petrov et al. [2023] characterize limitations of prompting and prefix tuning under specific architectural assumptions; Petrov et al. [2024] also establish universal approximation for sufficiently large prefix constructions. The two results rule out a blanket theoretical claim that prefixes cannot represent new behavior. Recent systems report positive skill storage using different substrates: Skill Neologisms learn soft vocabulary tokens [Berthon et al., 2026], LatentSkill produces weight-space LoRA adapters [Yu et al., 2026], and KV-Skill uses a learned interface to read external factorized operators [Han et al., 2026]. Our experiment is narrower. It tests whether a short, directly optimized virtual-KV prefix, trained with exemplar cross-entropy in a frozen model, preserves the behavior enabled by a textual specification and three examples. 3.4 Correctness signals and code selection Language-model self-evaluation and hidden-state probing have revealed correctness- or truthfulness-related signals in multiple domains [Kadavath et al., 2022, Azaria and Mitchell, 2023, Burns et al., 2022]. Orgad et al. [2024] show that such signals need not transfer across datasets, making cross-benchmark evaluation important. More recent work probes arithmetic errors [Sun et al., 2025], chain-of-thought errors [Yuan et al., 2026], and code correctness [Di Cicco, 2026, Ribeiro et al., 2026]. Ashuach et al. [2026] further show that a model’s own hidden states do not always outperform peer-model states, cautioning against claims of privileged self-knowledge. Candidate selection for code has traditionally relied on generated tests, execution agreement, or program clustering [Chen et al., 2022, To et al., 2024]. UCoder incorporates internal probing into unsupervised code-model training [Wu and others, 2026], while CASE uses task-grouped evaluation to show when hidden-state selection beats majority voting and exposes question-identity leakage in ordinary probe splits [Wang et al., 2026]. We use source-benchmark training and target-benchmark evaluation, execution labels from EvalPlus, and within-task ranking. Unlike probe-only studies, we evaluate correctness readout as the endpoint of the same controlled pipeline that tests hint rescue, sampled accessibility, and activation transfer. 4. Experimental Design 4.1 Models, benchmarks, and execution The primary frozen student is Qwen2.5-3B-Instruct [Qwen Team et al., 2024]. We independently repeat the behavioral hint and sampling stages with Phi-3.5-mini-instruct [Abdin et al., 2024, Microsoft, 2024]. Qwen2.5-Coder-7B-Instruct [Hui et al., 2024] serves only as an instrument for identifying teacher-solvable student failures and producing minimal hints; we do not treat it as an oracle. We evaluate the 164 HumanEval tasks [Chen et al., 2021] and 378 MBPP tasks [Austin et al., 2021] augmented as HumanEval+ and MBPP+ by EvalPlus [Liu et al., 2023]. A program passes only when it satisfies both base and augmented tests. Unless stated otherwise, generation is greedy. Sampling experiments use temperature 0.8 and top-p 0.95. All activation, prefix, and correctness-readout results use the primary Qwen student. The source runs used BF16, seed 42, batches of 12, left padding, and at most 512 new tokens on an NVIDIA GB10 Grace Blackwell system. The archived environment records Python 3.12.3, PyTorch 2.12.0+cu130, Transformers 5.9.0, EvalPlus 0.3.1, NumPy 2.2.6, SciPy 1.18.1, and scikit-learn 1.9.0. Each experimental stage has a timestamped run identifier, git commit, configuration, and summary in results/runs.jsonl. Because nominally identical greedy prompts sometimes change outcome under different batch compositions, we treat pipeline replay as an empirical control rather than assume determinism from the decoding rule. The Qwen student passes 113 of 164 HumanEval+ tasks (68.9%) and 249 of 378 MBPP+ tasks (65.9%). The Phi student passes 108 (65.9%) and 224 (59.3%). The teacher passes 136 (82.9%) and 268 (70.9%), respectively. Intersecting teacher passes with student failures yields 79 selected Qwen failures (29 HumanEval+, 50 MBPP+) and 101 selected Phi failures (33, 68). These populations are selected specifically where the teacher passes and the student fails, so their rescue rates do not describe the full benchmarks; they also differ across students. 4.2 Hint intervention and behavioral controls For each selected teacher-pass/student-fail task, the teacher produces an adaptive ladder of up to three minimal natural-language hints, typically no more than 23 words. The ladder proceeds from a light conceptual cue toward a more explicit cue and stops at the first pass. A task counts as rescued if any evaluated level passes. The unrelated control gives each selected task one level-1 hint generated for a different, already-passing task and chosen to be closest in token length to the relevant task’s successful or last attempted hint. Other controls are no-hint best-of-eight sampling at temperature 0.8 and top-p 0.95, plus replay of the original greedy prompt under changed batch composition. The relevant ladder offers one to three opportunities whereas the unrelated condition offers one; the sampling control changes both opportunity count and decoding rule. The implemented comparisons therefore answer whether these procedures differ, not how much rescue is attributable specifically to semantic content. 4.3 Activation capture and intervention Hidden states are captured from the post-block residual stream at every layer. The anchor is the aligned token at the end of a suffix whose token IDs are identical across conditions. For task i and layer ℓ , the hint delta is Δi,ℓ=hi,ℓhint−hi,ℓbase. _i, =h^hint_i, -h^base_i, . The unit mean hint direction is gℓ=∑iΔi,ℓ‖∑iΔi,ℓ‖2,g_ = _i _i, \| _i _i, \|_2, and the reported per-delta energy fraction is the squared projection (gℓ⊤Δi,ℓ)2/‖Δi,ℓ‖22(g_ _i, )^2/\| _i, \|_2^2, summarized across tasks. Persistent injection applies hℓ,t←hℓ,t+αgℓh_ ,t← h_ ,t+α g_ at each decoding step t in the selected layer configuration. The full per-task oracle delta is the unprojected Δi,ℓ _i, from the hinted run known to rescue evaluation task i; “oracle” denotes its dependence on that successful evaluation-task outcome. We distinguish three tests. The first asks whether gℓg_ is stable under split-half re-estimation. The second injects it persistently and evaluates both rescues and damage. The third patches the full oracle delta once at the aligned anchor as a positive-control test of single-position intervention. Random bases are matched to the learned intervention in rank and norm. Low-rank bases use three-fold held-out cross-fitting: within each split, the basis and layer/rank/strength configuration are fitted and selected using only the training tasks, then frozen for the held-out fold. Comparators are matched random bases and shuffled task deltas. The generic-direction deployment is exploratory: its mean direction, layer, and strength were not all cross-fitted before full-benchmark evaluation. 4.4 Context-defined procedures and virtual-KV prefixes We construct four procedural families: balanced-ternary notation, an eight-operation stack language, ordered string rewriting, and a keyed codec. Each family contains three worked exemplars and six held-out problems with hidden executable tests. We evaluate no context, an unrelated procedure’s context, and the full specification plus exemplars (approximately 800 tokens). We then freeze the model and optimize virtual-KV prefixes of 2–16 tokens using exemplar cross-entropy. Each virtual token occupies 36,864 bytes in the tested implementation, so a prefix uses approximately 74–590 decimal kilobytes. We compare trained prefixes with untrained, random, shuffled, and size-matched controls. Low training loss verifies exemplar fit; only held-out execution measures transfer. 4.5 Correctness readout and candidate selection For each of 542 benchmark tasks, we draw eight samples, yielding 4,336 execution-labeled programs. HumanEval+ contributes 859 passing candidates among 1,312; MBPP+ contributes 1,875 among 3,024. We fit class-balanced, ℓ2 _2-regularized logistic probes to unit-normalized hidden states. Candidate representations span layers 8,14,20,26,32\8,14,20,26,32\, last-token or mean pooling, and C∈0.1,1,10C∈\0.1,1,10\. All representation, regularization, and score-combination choices are made without target-benchmark labels. Five-fold GroupKFold on source-benchmark task identifiers selects the layer, pooling rule, and C, prioritizing source within-task AUROC, then source selection accuracy and pooled AUROC. The probe-plus-log-probability weight is chosen from 0.25,0.5,1,2,4\0.25,0.5,1,2,4\ using the source out-of-fold scores. The selected model is then fit to the full source benchmark and evaluated once on the other benchmark. We report both HumanEval+→ + and MBPP+→ +. We compare the readout with mean token log-probability, exact-match majority frequency, 23 engineered code-surface features, and a character 3–5-gram TF-IDF classifier. Pooled AUROC compares all labeled candidates; within-task AUROC compares correct and incorrect candidates from the same task, reducing the contribution of between-task difficulty. Candidate selection uses no test execution at inference time, but training uses execution labels and inference still requires eight generations and white-box hidden-state access. We exclude an exploratory analysis that selected representations on the evaluation benchmark; every reported probe result uses source-only model selection in publication_analysis.py. 4.6 Statistical reporting Benchmark-task inference uses task as the unit, with generated samples and repeated interventions clustered within task. We use 1,000 task-bootstrap replicates for selector AUROC and accuracy intervals, 20,000 paired task-bootstrap replicates for subspace-control differences, exact McNemar tests for binary paired outcomes [McNemar, 1947], and 95% Wilson intervals for individual binomial rates [Wilson, 1927]. Bootstrap intervals follow the task-resampling principle described by Efron and Tibshirani [1993]. Probe intervals condition on the realized source training set, source-selected model, and fixed eight-candidate target draw; they resample only target tasks. The random-subspace interval averages the five repeated interventions within each task and conditions on the realized folds and random bases. Procedure problems are nested within four families; the 24 problems do not support independent-family generalization. The subspace study is small (36 rescued tasks), so failure to reject a null is reported as no detected advantage, not evidence of equivalence. The publication reanalysis was written after the exploratory runs and is therefore robustness analysis rather than preregistered confirmation. 5. Most Hint Rescues Reappear Within Eight Unhinted Samples Table 1 reports the principal behavioral controls on the selected teacher-pass/student-fail sets and the full Qwen benchmark. Table 1. Behavioral outcomes. Intervals are 95% Wilson intervals. Replay rows apply only to the primary Qwen study. student and quantity count rate 95% CI Qwen: relevant-hint rescue 36/79 45.6% [35.0, 56.5] Qwen: unrelated-hint rescue 19/79 24.1% [16.0, 34.5] Qwen: no-hint best-of-8 success 46/79 58.2% [47.2, 68.5] Qwen: relevant rescues also solved by best-of-8 31/36 86.1% [71.3, 93.9] Phi: relevant-hint rescue 42/101 41.6% [32.5, 51.3] Phi: unrelated-hint rescue 17/101 16.8% [10.8, 25.3] Phi: no-hint best-of-8 success 57/101 56.4% [46.7, 65.7] Phi: relevant rescues also solved by best-of-8 36/42 85.7% [72.2, 93.3] Qwen: replay flips among all baseline failures 5/180 2.8% [1.2, 6.3] Qwen: replay flips among all baseline passes 4/362 1.1% [0.4, 2.8] Figure 1: Disjoint overlap patterns for the 79 selected Qwen failures. The relevant and unrelated conditions do not have matched attempt budgets; the figure emphasizes task identity rather than treating the three rates as independent. For Qwen, the observed ladder yields 36 rescues with relevant hints and 19 with unrelated hints. At the task level, 20 are rescued only by the relevant condition and 3 only by the unrelated condition (exact McNemar p = 0.00049). Phi shows 29 relevant-only and 4 unrelated-only tasks (p = 0.000011). These paired differences establish reproducible differences between the implemented procedures, not semantic-content effects: attempt counts and stopping rules are not matched. The sampling control changes the capability interpretation more sharply. Eight no-hint samples solve 46 of 79 selected Qwen tasks and 57 of 101 selected Phi tasks. They include 31 of 36 and 36 of 42 relevant-hint rescues, respectively. Thus, across both students, most observed rescues occur on tasks for which the unhinted model already produces a passing program within eight samples. This is evidence of accessibility under a modest sampling budget. It does not show that hinting and temperature sampling implement the same mechanism, nor does it explain the remaining five and six hint-only rescues. On HumanEval+ as a whole, 77 of 164 tasks (47.0%) are mixed across eight samples, while 61 are always solved and 26 are never solved. The large mixed region makes one-shot before/after comparisons especially fragile near the competence boundary. Finally, nominal replay flips 5 baseline failures and 4 baseline passes without changing prompt content. A separate replay deployment used for the rescued-36 mechanistic analysis produces 6 of 36 passes (16.7%). The latter is not a restriction of the full-benchmark replay run, in which 5 of 180 failures change to pass. The selected mechanistic population and separate deployment therefore have their own empirical replay reference. 6. Mechanistic Results: Stability Is Not Specificity 6.1 Hinting produces a stable but largely generic direction Across layers, the mean hint-delta direction is extremely stable under the reported split-half re-estimation (cosine 0.992–0.996) and accounts for 35–63% of the energy of individual hint deltas. This is a highly reproducible geometric pattern under the tested protocol. It is not, however, specific to useful hint content: mean directions estimated from relevant and unrelated hints have cosine similarity around 0.98 in early and middle layers. An initially promising late-layer relevance direction appeared to separate relevant from irrelevant interventions on the 36 rescued tasks (27.8% versus 16.7%). The contrast appeared only after selecting a layer and the enriched 36-task subset; its failure to replicate points to selection noise rather than a stable semantic direction. Figure 2: The mean hint response is exceptionally stable under split-half re-estimation. Relevant and unrelated directions separate in late layers, but that late-layer contrast does not survive the causal robustness checks described below. 6.2 Persistent injection changes outcomes but not net accuracy Persistent injection changes outcomes under the tested generation pipeline. On the 36 rescued tasks, 10 outputs pass under injection (27.8%, 95% Wilson CI [15.8, 44.0]) versus 6 under the separate replay deployment (16.7%, [7.9, 31.9]). Because the execution paths are not documented as identical and deterministic, the difference is not identified causally. Full-benchmark deployment also measures damage to baseline passes. Table 2. Full-benchmark deployment of the generic direction. The net interval is a large-sample paired-difference interval based on the 14 beneficial and 18 harmful discordant pairs. transition count conditional rate 95% CI baseline fail to pass 14/180 7.8% [4.7, 12.6] baseline pass to fail 18/362 5.0% [3.2, 7.7] net passing programs -4/542 -0.74 percentage points [-2.8, 1.3] percentage points* Figure 3: Full-benchmark outcome transitions. Counts, rather than conditional percentages with different denominators, make the absence of net improvement visible. The different conditional rates have different denominators. In absolute task counts, 18 passes are damaged and 14 failures are rescued. The transition imbalance is not statistically resolved (two-sided exact McNemar p=0.597p=0.597); this does not establish equivalence. On the full benchmark, persistent injection is associated with both fail-to-pass and pass-to-fail transitions, with no detected net accuracy change. This establishes behavioral perturbation under the tested protocol, not a treatment effect isolated from pipeline variation. 6.3 Single-position patching shows no detected effect in the tested channel Patching the full per-task oracle rescue delta at the anchor position yields 6 of 36 passes (16.7%, 95% Wilson CI [7.9, 31.9]) for strengths α∈1,2α∈\1,2\, equal to the separate replay count. Persistent injection of the same delta changes more outcomes descriptively. No effect beyond replay is detected at this anchor and these strengths. Equality of the observed counts is not evidence of equivalence; instead, the experiment fails to validate this anchor as a positive-control channel. 6.4 No held-out subspace advantage is detected Three-fold held-out cross-fitting over the 36 rescued tasks produces the following outcomes: condition passes rate 95% Wilson CI learned low-rank subspace 9/36 25.0% [13.8, 41.1] replay 6/36 16.7% [7.9, 31.9] matched random subspaces mean 5.8/36 across five repeats 16.1% repeated within task shuffled task deltas 5/36 13.9% [6.1, 28.7] Task-level paired comparisons yield +8.3 percentage points for learned minus replay (20,000-replicate task-bootstrap 95% CI [-2.8, 19.4]; exact McNemar p=0.375p=0.375), +8.9 points for learned minus the per-task mean of five matched-random runs ([-1.7, 20.6]), and +11.1 points for learned minus shuffled ([0.0, 25.0]; exact p=0.219p=0.219). These intervals are wide and do not demonstrate equivalence, particularly at N=36N=36. The representational result points in the same direction: a training-estimated basis explains 64% of training delta energy but approximately 9% on held-out tasks. Residuals formed by removing the per-task placebo-direction component and leave-one-out ridge predictions show no detected improvement over their controls. Under the tested ranks, layers, strengths, channel, and cross-validation protocol, the training-set subspace does not yield a detected task-specific transfer effect. 7. Cross-Benchmark Correctness Readout The intervention experiments ask whether a compact state change can make the model execute a task-specific behavior. A complementary question is whether hidden states can rank behaviors the model has already generated. Task-level candidate counts give a uniform-selection expectation of 107.4/164 (65.5%) on HumanEval+; 102/164 is the realized first-draw count, not that expectation. MBPP+ contains 1,875 passing candidates among 3,024, for a uniform-selection expectation of 234.4/378 (62.0%). Table 3 uses these denominators and source-only model selection. Table 3. Cross-benchmark candidate selection. Bootstrap intervals resample target tasks; paired tests compare the combined selector with mean log-probability. target (probe source) uniform candidate, expected mean log-p hidden probe probe + log-p pass@8 oracle paired p MBPP+ → HumanEval+ 107.4/164 (65.5%) 113 (68.9%) 116 (70.7%) 122 (74.4%) [67.7, 81.1] 138 (84.1%) 0.093 HumanEval+ → MBPP+ 234.4/378 (62.0%) 240 (63.5%) 241 (63.8%) 244 (64.6%) [59.8, 69.0] 281 (74.3%) 0.503 Figure 4: Target-task selection after all probe and combination choices are made on the source benchmark. Error bars are 95% task-bootstrap intervals. Dashed lines show the expected accuracy of a uniformly chosen candidate; dotted lines show the target pass@8 oracle. Both source benchmarks select layer 26, mean pooling, and C=10C=10. The hidden probe’s pooled AUROC is 0.806 (task-bootstrap 95% CI [0.750, 0.861]) on HumanEval+ and 0.780 ([0.742, 0.819]) on MBPP+. Within-task AUROC is lower but remains above chance: 0.654 ([0.591, 0.718]) and 0.634 ([0.577, 0.690]). For comparison, mean log-probability yields pooled/within-task AUROC 0.653/0.627 on HumanEval+ and 0.626/0.593 on MBPP+. The gap between pooled and within-task discrimination shows that task difficulty explains part, but not all, of the signal. The surface controls narrow the interpretation further. A 23-feature syntax-and-length classifier reaches pooled/within-task AUROC 0.692/0.567 on HumanEval+ and 0.612/0.567 on MBPP+; a character TF-IDF classifier reaches 0.657/0.560 and 0.623/0.555. These surface baselines have lower point-estimate pooled AUROC than the hidden probe, although they carry substantial correctness information and do not exhaust possible textual confounds. Top-one selection is less decisive than sample discrimination. On HumanEval+, the source-selected combination produces nine more passing selections than confidence alone: 16 tasks improve and 7 worsen, giving exact McNemar p=0.093p=0.093. On MBPP+, the difference is four tasks: 12 improve and 8 worsen, p=0.503p=0.503. Within the shared eight-candidate ledgers, first-draw counts are 102/164 and 237/378, whereas uniform-selection expectations are 107.375 and 234.375. Separate greedy runs solve 113/164 and 249/378; comparisons with those runs are descriptive cross-protocol counts, not the paired selector comparisons in Table 3. Exact-match majority selects 114 and 240 passing programs, respectively. These results support cross-benchmark correctness-related decodability; they do not yet establish a reliable top-one improvement over confidence or greedy decoding. Probe training consumes execution labels for 4,336 programs. Inference avoids test execution but still requires eight generations and white-box hidden-state access. A post-generation probe may exploit code length, termination, syntax, memorized error signatures, or other properties not represented by our controls. The result is therefore a transferable readout, not proof of privileged self-knowledge or causal use of the decoded feature during generation. 8. Compressing Context-Defined Procedures The preceding benchmark tasks are often solved under sampling. To test a more explicit context gap, we construct four procedures defined by their supplied specifications and examples. Across the four families, no-context generation solves at most 1 of 6 held-out problems per family despite 13 attempts per problem; an unrelated procedure’s context also solves at most 1 of 6. The full specification plus three worked exemplars solves 22 of 24 held-out problems (91.7%, 95% Wilson CI [74.2, 97.7]). This establishes a large empirical context effect under execution. It does not prove that the pretrained model assigns zero probability to success without context. Trained virtual-KV prefixes reach exemplar loss at or below 0.05 but solve only 5–11 of 24 held-out problems, overlapping the range of untrained, random, and shuffled controls. In a follow-up on the selected ordered-rewriting configuration with two virtual tokens, five perturbed-initialization seeds each solve the same 3 of 6 problems. Size-matched controls solve 2 of 6. The effective held-out unit remains six problems, not five seeds; the one-problem difference is insufficient evidence of procedure transfer. This result is method-specific. The tested exemplar-cross-entropy virtual-KV scheme does not reproduce the full context on these four families, but other objectives, interfaces, lengths, and pretrained adapters may behave differently. Without a method-positive control or context-distillation objective, the experiment does not isolate a limit of the representation class from a limit of the training recipe. Full per-family results and controls appear in Appendix B. 9. A Control Checklist for Capability-Transfer Claims The experiments can be summarized by the interpretation each control changed. proposed inference required control observation here supported interpretation a prompt caused a rescue nominal replay on the same selected set replay reaches 16.7% on the rescued-36 subset subtract or model the selected-set instability floor the hint’s content caused the rescue attempt- and style-matched irrelevant hints unrelated hints rescue 19/79 generic conditioning is a substantial alternative; the semantic increment is not yet identified rescue created a new capability no-hint pass@k gate pass@8 covers 31/36 hint rescues most rescued tasks are already accessible within the tested sampling budget a stable direction is semantically meaningful split-half and relevant-vs-irrelevant estimation stability is high, but relevant and irrelevant directions align reproducibility does not imply content specificity a steering vector improves the model report rescue and damage on the full population 14 rescues, 18 damages the direction perturbs behavior without detected net benefit a patched subspace is causally informative positive-control channel validation no effect beyond replay is detected for the full task-specific successful-hint difference at the tested anchor the tested anchor was not validated as an effective causal channel a learned subspace transfers held-out cross-fitting plus task-clustered random and shuffled controls +8.3 points vs replay, CI [-2.8, 19.4]; +8.9 vs random, CI [-1.7, 20.6] estimates are too imprecise to establish transfer or equivalence a trained virtual-KV prefix compresses a procedure size-matched controls and held-out execution 3/6 versus 2/6 in the selected-configuration follow-up evidence is insufficient for transfer a probe reveals correctness knowledge source-only selection, cross-benchmark tests, and output-feature baselines pooled AUROC 0.806/0.780; paired top-one p=0.093/0.503p=0.093/0.503 correctness-related information is decodable; selection, cognitive, and causal claims remain open Behavioral change, representational structure, intervention sensitivity, task specificity, generalization, and net utility are separate empirical claims. Each requires its own denominator and control. 10. Discussion 10.1 What the experiments support First, a before-and-after hint rescue is a poor proxy for newly acquired capability on these benchmarks. Most relevant-hint rescues occur on tasks the unhinted student solves within eight samples, and unrelated hints also change many outcomes. Relevant content may contribute, but the present ladder does not isolate that contribution from unequal intervention opportunities or other prompt differences. Second, a representation can be real and reproducible without supporting the intended mechanistic interpretation. The generic hint direction is exceptionally stable and is behaviorally associated with changed outputs under persistent injection. Its alignment across relevant and irrelevant prompts and its negative full-benchmark net change make “task-specific capability vector” an unsupported label. Third, the negative intervention results are method-specific. Single-position patching shows no detected effect at the tested anchor, while persistent injection changes outcomes. The learned subspace shows no detected held-out advantage, but the sample is small. The KV-prefix objective transfers little on the synthetic procedures, but other prefix lengths, objectives, pretrained interfaces, and weight-space methods may behave differently. Fourth, the post-generation readout transfers in both benchmark directions and has higher point-estimate AUROC than the tested confidence and surface baselines. Its top-one gain over confidence is not statistically resolved, and its observed MBPP+ count is below that of the separate greedy run. This readout does not turn a failure into a success; it can only help choose a success already present among the candidates. 10.2 Relation to concurrent work Our results sit between positive work on task/function vectors and recent cautions about interpreting activation interventions. Function-vector studies demonstrate compact causal representations on controlled input-output mappings, while subspace-patching work shows that behavioral effects need not imply faithful localization. Our code experiments extend the control question to long-form generation with executable labels, but do not invalidate positive results in other task families. Recent prompting and resampling work makes the behavioral controls especially important. Socio-demographic placebo experiments show that arbitrary prompt tokens can perturb outputs even when their content is irrelevant. Random soft prompts can broaden early token distributions and improve pass@N without learned content. Work on thought-branch resampling likewise argues that a single generated trajectory is insufficient for causal interpretation. Our setting differs in using functional code execution and in tracing the same ambiguity from prompts into activation interventions, but these results reinforce the need to compare content-bearing interventions with compute-matched perturbations and resampling. The prefix result must also be read against both sides of the expressivity literature. Petrov et al. characterize restrictions of prompting and prefix tuning under particular architectural conditions, while their later universal-approximation result shows that sufficiently large constructions can be expressive in principle. Recent Skill Neologism, LatentSkill, and KV-Skill results report positive task storage using different substrates or learned interfaces. Our finding is therefore an empirical failure of one compact virtual-KV scheme on four context-defined procedures, not a theorem about prefix-class methods. Finally, hidden-state correctness detection is now a crowded area. Prior and concurrent work reports truthfulness, reasoning-error, arithmetic-error, and code-correctness signals, including robustness and confound analyses. CASE shows that question-grouped evaluation is necessary to avoid question-identity leakage when comparing hidden-state selection with voting. UCoder uses internal probing as part of unsupervised code-model training, and peer-probe comparisons show that a model’s own states need not carry privileged correctness information in every domain. Our distinct evidence is the combination of EvalPlus execution labels, cross-benchmark training and testing, within-task comparisons, and direct candidate-selection utility in the same controlled study. We do not claim the first discovery of correctness decodability. 11. Limitations Model scope. The behavioral hint/resampling pattern repeats in two student architectures, but the intervention, prefix, and probe results cover only Qwen2.5-3B-Instruct. This does not establish scale or broad architecture generality. Benchmark scope and contamination. HumanEval+ and MBPP+ contain short Python problems and are widely used. Augmented tests improve outcome validity but do not remove possible benchmark exposure. A contamination-resistant, time-split, or newly authored benchmark would strengthen external validity. Adaptive hint ladder. Up to three relevant-hint levels provide multiple chances for rescue, while the length-matched unrelated condition provides one. The no-hint pass@8 arm also uses a different decoding rule. Hints were generated during the experimental pipeline rather than frozen in a preregistered artifact before student evaluation. Pipeline nondeterminism. Outcome changes under nominal replay complicate every small causal effect, especially after selection on rescued tasks. The archived runs do not use single-example batches, deterministic kernels, or identical execution paths across every intervention. Exploratory multiplicity and power. Layer, rank, strength, pooling, and direction searches create multiple comparisons. Held-out cross-fitting protects the reported subspace test tasks, but the causal subset contains only 36 tasks. No prespecified smallest effect of interest or equivalence interval is reported. Intervention scope. The point patch uses one anchor and two strengths. The persistent intervention and the low-rank construction test particular residual-stream channels. Null results cannot be generalized to other positions, heads, components, or learned interfaces. Synthetic-procedure gate. Thirteen failed attempts do not prove absence, and some constructed rules may overlap with patterns in pretraining. The study does not use randomized secrets or per-instance rule permutations, and the virtual-KV experiment has one objective without a demonstrated method-positive control. Probe confounds and cost. Post-generation hidden states may encode surface artifacts correlated with correctness. The probe requires white-box access, eight generations, and execution-labeled training data. We add length/syntax and character TF-IDF baselines, task-clustered intervals, and paired selector tests, but omit stronger code encoders, static analyzers, peer-model states, residualized covariates, and external calibration tests. 12. Conclusion Across two small models, relevant hints help under the implemented conditions, but no-hint sampling covers most successful rescues. In the primary Qwen mechanistic study, apparent capability transfer becomes substantially narrower under direct controls. A stable hint direction is associated with changed outputs, yet it is shared across relevant and irrelevant prompts and provides no detected net accuracy gain. Learned activation subspaces have positive but imprecise held-out estimates; compact virtual-KV prefixes remain far below the full-context condition. Post-generation hidden states have higher point-estimate AUROC than the tested baselines in both transfer directions, while their incremental top-one selection value remains preliminary. A capability-transfer claim should report replay, matched placebos, the no-hint sampling boundary, causal-channel validation, held-out comparisons, and damage alongside rescue. Reporting these controls distinguishes interventions that merely change outputs from those that improve accuracy or transfer task-specific behavior. Data and Artifact Availability A compact reproducibility artifact has been prepared for public archival deposit. It contains the full experimental source, configuration, dependency lock, run registry, task-level ledgers for baselines, hints, sampling gates, interventions, and procedures, 4,336 raw sampled programs and execution labels, cached probe representations, the independent Phi behavioral replication, the publication reanalysis, and all figures. Running python publication_analysis.py --root . --output publication_analysis recreates the corrected statistics and figures without model inference. The compact review bundle omits approximately 5.9 GB of raw per-layer activation tensors and trained checkpoints retained in the authors’ full archive; these are required to rerun intervention construction, but not to reproduce any reported table, interval, paired test, or figure from the released ledgers and cached features. ARTIFACT_MANIFEST.md maps every claim to its source files and records the limitation. Appendix A. Evidence and Reproduction Map claim family primary archived evidence publication output Qwen baselines and selected failures results/baseline_summary_*.json, results/baseline_results_*.jsonl Table 1 denominators relevant and unrelated hints results/hints_*.jsonl, results/controls_*.json, results/rescued_*.json Table 1, Figure 1 no-hint best-of-eight results/gate_results_humaneval+mbpp.json Table 1, Figure 1 Phi behavioral replication replication_phi/results/*, replication_phi/behavioral_summary.json Table 1 geometry and causal interventions results/xray*.json, results/causal_cv_results_humaneval+mbpp.json, results/deploy_results_humaneval+mbpp.json Table 2, Figures 2–3 candidate selection data/samples_*.jsonl, cache/selfeat_*.npz, publication_analysis/publication_statistics.json Table 3, Figure 4 context-defined procedures results/skillgaps.json, results/capsule_distill.json, results/capsule_replication.json Appendix B execution and software provenance results/runs.jsonl, config.yaml, requirements.txt, artifacts/system_info.txt, logs/ Methods and artifact audit The main Qwen registry contains timestamped stages for smoke validation, baselines, hints, activation capture, subspace estimation, held-out causal evaluation, intervention deployment, resampling, candidate generation, probe analysis, procedure gating, prefix distillation, and replication. Each record contains the stage’s git commit. The Phi directory has a separate registry and source configuration. The corrected analysis uses seed 20260828 for bootstrap resampling and does not execute generated code or call a language model. Appendix B. Virtual-KV Results by Family The four context-defined families are balanced-ternary notation (TRN), an eight-operation stack language (GSL), ordered string rewriting (CRW), and a keyed codec (KZE). Table B1 reports whether any of eight generated programs passes each held-out problem; all cells are counts out of six. Table B1. Any-of-eight held-out success for trained virtual-KV prefixes and controls. family k=2 k=4 k=8 k=16 untrained context-init k=8 untrained random k=8 trained random-init k=8 shuffled prefix TRN 1 0 1 1 1 0 0 1 GSL 3 1 1 3 1 1 1 1 CRW 4 1 2 2 2 1 1 1 KZE 0 0 0 0 0 0 0 0 Figure B1: Held-out any-of-eight success by trained virtual-KV prefix length. Each virtual token occupies 36 KiB in the tested implementation. The dashed line is the per-family mean for the full textual context (22/24 overall); controls are reported in Table B1. The selected CRW k=2 follow-up uses fresh test cases and five perturbed initializations. Every run solves the same three of six cases; size-matched controls solve two. This is reproducible problem-level overlap, not five independent six-problem replications. Appendix C. Design Boundaries at a Glance comparison shared unit opportunity/decoding match identified interpretation relevant vs unrelated hint same selected task no: adaptive one-to-three vs one greedy attempt difference between implemented procedures; semantic increment not isolated relevant hint vs no-hint pass@8 same selected task no: greedy hint ladder vs eight stochastic draws overlap with observed unhinted accessibility, not mechanistic equivalence generic steering vs replay benchmark task same reported decoding settings, but batch/kernel paths not deterministic associated outcome transitions beyond an empirical replay reference learned subspace vs replay/shuffled/random rescued task held out by fold rank/norm matched; random interval conditions on realized bases imprecise held-out advantage estimate, not equivalence hidden readout vs confidence same target task and eight candidates yes for candidate pool; training/model-selection uncertainty conditioned on paired top-one comparison on the realized candidate ledger References Abdin et al. (2024) M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y. Chen, Y. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. J. Kim, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka, A. Tupini, P. Vaddamanu, C. Wang, G. Wang, L. Wang, S. Wang, X. Wang, Y. Wang, R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang, Y. Yang, Z. Yang, D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219. External Links: 2404.14219, Link Cited by: 4.1 Models, benchmarks, and execution. Ashuach et al. (2026) T. Ashuach, S. Gretz, Y. Katz, Y. Belinkov, and L. Ein-Dor Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness. arXiv preprint arXiv:2604.12373. External Links: 2604.12373, Link Cited by: 3.4 Correctness signals and code selection. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732. External Links: 2108.07732, Link Cited by: 4.1 Models, benchmarks, and execution. Azaria and Mitchell (2023) A. Azaria and T. Mitchell The Internal State of an LLM Knows When It’s Lying. arXiv preprint arXiv:2304.13734. External Links: 2304.13734, Link Cited by: 3.4 Correctness signals and code selection. Berthon et al. (2026) A. Berthon, N. Astorga, and M. van der Schaar Skill Neologisms: Towards Skill-based Continual Learning. arXiv preprint arXiv:2605.04970. External Links: 2605.04970, Link Cited by: 3.3 Context and compact skill representations. Burns et al. (2022) C. Burns, H. Ye, D. Klein, and J. Steinhardt Discovering Latent Knowledge in Language Models Without Supervision. arXiv preprint arXiv:2212.03827. External Links: 2212.03827, Link Cited by: 3.4 Correctness signals and code selection. Chen et al. (2022) B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J. Lou, and W. Chen CodeT: Code Generation with Generated Tests. arXiv preprint arXiv:2207.10397. External Links: 2207.10397, Link Cited by: 3.4 Correctness signals and code selection. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374. External Links: 2107.03374, Link Cited by: 4.1 Models, benchmarks, and execution. Di Cicco (2026) C. Di Cicco Code Correctness Signals in LLM Hidden States: Pre-Generation Probing and Repair Geometry. arXiv preprint arXiv:2606.14530. External Links: 2606.14530, Link Cited by: 3.4 Correctness signals and code selection. Efron and Tibshirani (1993) B. Efron and R. J. Tibshirani An Introduction to the Bootstrap. Chapman & Hall/CRC. Cited by: 4.6 Statistical reporting. Han et al. (2026) Z. Han, X. Zhang, B. Han, K. Liu, D. Hu, and J. Liu KV-Skill: Forging Expertise in the Model’s Native Language. arXiv preprint arXiv:2608.05475. External Links: 2608.05475, Link Cited by: 3.3 Context and compact skill representations. Hendel et al. (2023) R. Hendel, M. Geva, and A. Globerson In-Context Learning Creates Task Vectors. arXiv preprint arXiv:2310.15916. External Links: 2310.15916, Link Cited by: 3.1 Task vectors and activation interventions. Hui et al. (2024) B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y. Fan, Y. Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y. Miao, S. Quan, Y. Feng, X. Ren, X. Ren, J. Zhou, and J. Lin Qwen2.5-Coder Technical Report. arXiv preprint arXiv:2409.12186. External Links: 2409.12186, Link Cited by: 4.1 Models, benchmarks, and execution. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan Language Models (Mostly) Know What They Know. arXiv preprint arXiv:2207.05221. External Links: 2207.05221, Link Cited by: 3.4 Correctness signals and code selection. Kim et al. (2026) H. Kim, S. Lee, J. Yeom, J. Sok, S. Park, J. Park, T. Kim, and S. Kim From Noise to Diversity: Random Embedding Injection in LLM Reasoning. arXiv preprint arXiv:2605.11936. External Links: 2605.11936, Link Cited by: 3.2 Prompt sensitivity, placebos, and resampling. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv preprint arXiv:2305.01210. External Links: 2305.01210, Link Cited by: 4.1 Models, benchmarks, and execution. Luo et al. (2026) C. F. Luo, S. Dahan, and X. Zhu What Am I Missing? Question-Answering as Hidden State Probing. arXiv preprint arXiv:2605.31561. External Links: 2605.31561, Link Cited by: 3.2 Prompt sensitivity, placebos, and resampling. Macar et al. (2025) U. Macar, P. C. Bogdan, S. Rajamanoharan, and N. Nanda Thought Branches: Interpreting LLM Reasoning Requires Resampling. arXiv preprint arXiv:2510.27484. External Links: 2510.27484, Link Cited by: 3.2 Prompt sensitivity, placebos, and resampling. Makelov et al. (2023) A. Makelov, G. Lange, and N. Nanda Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. arXiv preprint arXiv:2311.17030. External Links: 2311.17030, Link Cited by: 3.1 Task vectors and activation interventions. McNemar (1947) Q. McNemar Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12, p. 153–157. External Links: Document Cited by: 4.6 Statistical reporting. Microsoft (2024) Microsoft Phi-3.5-mini-instruct Model Card. Note: Hugging Face model card External Links: Link Cited by: 4.1 Models, benchmarks, and execution. Mu et al. (2023) J. Mu, X. L. Li, and N. Goodman Learning to Compress Prompts with Gist Tokens. arXiv preprint arXiv:2304.08467. External Links: 2304.08467, Link Cited by: 3.3 Context and compact skill representations. Mukherjee et al. (2024) S. Mukherjee, M. F. Adilazuarda, S. Sitaram, K. Bali, A. F. Aji, and M. Choudhury Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting. arXiv preprint arXiv:2406.11661. External Links: 2406.11661, Link Cited by: 3.2 Prompt sensitivity, placebos, and resampling. Orgad et al. (2024) H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y. Belinkov LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. arXiv preprint arXiv:2410.02707. External Links: 2410.02707, Link Cited by: 3.4 Correctness signals and code selection. Panickssery et al. (2023) N. Panickssery, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner Steering Llama 2 via Contrastive Activation Addition. arXiv preprint arXiv:2312.06681. External Links: 2312.06681, Link Cited by: 3.1 Task vectors and activation interventions. Petrov et al. (2023) A. Petrov, P. H. S. Torr, and A. Bibi When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations. arXiv preprint arXiv:2310.19698. External Links: 2310.19698, Link Cited by: 3.3 Context and compact skill representations. Petrov et al. (2024) A. Petrov, P. H. S. Torr, and A. Bibi Prompting a Pretrained Transformer Can Be a Universal Approximator. arXiv preprint arXiv:2402.14753. External Links: 2402.14753, Link Cited by: 3.3 Context and compact skill representations. Qwen Team et al. (2024) Qwen Team, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. External Links: 2412.15115, Link Cited by: 4.1 Models, benchmarks, and execution. Ribeiro et al. (2026) F. Ribeiro, S. Abdulsattar, R. Gonzalez, M. Kassem, and S. Nadi On the Robustness of LLMs’ Internal Representation of Code Correctness. arXiv preprint arXiv:2608.08266. External Links: 2608.08266, Link Cited by: 3.4 Correctness signals and code selection. Sun et al. (2025) Y. Sun, A. Stolfo, and M. Sachan Probing for Arithmetic Errors in Language Models. In Proceedings of EMNLP, External Links: Link Cited by: 3.4 Correctness signals and code selection. To et al. (2024) H. Q. To, M. H. Nguyen, and N. D. Q. Bui Functional Overlap Reranking for Neural Code Generation. In Findings of ACL, External Links: Link Cited by: 3.4 Correctness signals and code selection. Todd et al. (2023) E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau Function Vectors in Large Language Models. arXiv preprint arXiv:2310.15213. External Links: 2310.15213, Link Cited by: 3.1 Task vectors and activation interventions. Turner et al. (2023) A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid Steering Language Models With Activation Engineering. arXiv preprint arXiv:2308.10248. External Links: 2308.10248, Link Cited by: 3.1 Task vectors and activation interventions. Wang et al. (2022) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv preprint arXiv:2203.11171. External Links: 2203.11171, Link Cited by: 3.2 Prompt sensitivity, placebos, and resampling. Wang et al. (2026) Z. Wang, Z. Hong, and U. Bagci A decodability criterion predicts when hidden-state selection beats majority voting in large language models. arXiv preprint arXiv:2608.17124. External Links: 2608.17124, Link Cited by: 3.4 Correctness signals and code selection. Wilson (1927) E. B. Wilson Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22 (158), p. 209–212. External Links: Document Cited by: 4.6 Statistical reporting. Wu et al. (2026) J. Wu et al. UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models. In Findings of ACL, External Links: Link Cited by: 3.4 Correctness signals and code selection. Yu et al. (2026) A. Yu, C. Zhou, T. Xu, Z. Guo, R. Shan, Z. Fu, J. Wang, W. Liu, Y. Yu, W. Zhang, and J. Lin LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents. arXiv preprint arXiv:2606.06087. External Links: 2606.06087, Link Cited by: 3.3 Context and compact skill representations. Yuan et al. (2026) A. Yuan, Z. J. Su, H. Zhang, Y. Nian, and Y. Zhao Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal. arXiv preprint arXiv:2605.09502. External Links: 2605.09502, Link Cited by: 3.4 Correctness signals and code selection. Zou et al. (2023) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks Representation Engineering: A Top-Down Approach to AI Transparency. arXiv preprint arXiv:2310.01405. External Links: 2310.01405, Link Cited by: 3.1 Task vectors and activation interventions.