Paper deep dive
Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)
Heng-Sheng Chang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A deception detector is only as useful as the trust its alarms deserve. Like a polygraph, detectors that read a language model's internal activations can catch deceptive responses that look honest. Yet current detectors depend on which layers they read, weaken on broader tests, and do not state how certain their alarms are. This white paper proposes the Machine Deception Detector, which targets both detection accuracy and the uncertainty of detector scores. The detector reads every layer of the language model, learns which tokens carry evidence of deception using only response labels, and calibrates its alarm threshold so that its false--positive rate stays below a prescribed level. An ensemble of such detectors measures the uncertainty of each score and raises an alarm only when its members agree, with a confidence level that sets how certain each alarm must be. The proposed evaluation trains on instructed honest and deceptive statements, tests on five unseen deceptive tasks of Liars' Bench read by six Qwen3 models from 0.6B to 32B parameters, and is designed to also show which layers carry the deception feature and how this changes with model size.
Tags
Links
- Source: https://arxiv.org/abs/2610.15314v1
- Canonical: https://arxiv.org/abs/2610.15314v1
Trouble viewing inline? Open PDF directly →
Full Text
68,313 characters extracted from source content.
Expand or collapse full text
MnLargeSymbols’164 MnLargeSymbols’171 Research White Paper Accuracy and Uncertainty in Machine Deception Detection Postdoctoral Researcher Dr. Heng-Sheng Chang hschang@illinois.edu Coordinated Science Laboratory University of Illinois Urbana-Champaign October 11, 2026 Abstract A deception detector is only as useful as the trust its alarms deserve. Like a polygraph, detectors that read a language model’s internal activations can catch deceptive responses that look honest. Yet current detectors depend on which layers they read, weaken on broader tests, and do not state how certain their alarms are. This white paper proposes the Machine Deception Detector, which targets both detection accuracy and the uncertainty of detector scores. The detector reads every layer of the language model, learns which tokens carry evidence of deception using only response labels, and calibrates its alarm threshold so that its false–positive rate stays below a prescribed level. An ensemble of such detectors measures the uncertainty of each score and raises an alarm only when its members agree, with a confidence level that sets how certain each alarm must be. The proposed evaluation trains on instructed honest and deceptive statements, tests on five unseen deceptive tasks of Liars’ Bench read by six Qwen3 models from 0.6B to 32B parameters, and is designed to also show which layers carry the deception feature and how this changes with model size. Table of Contents 1 Introduction 2 Preliminaries — A Polygraph for Language Models 3 Technical Approach — Machine Deception Detector 3.1 Thrust 1 — Detection Accuracy 3.1.1 Architecture — Multi-Layer Aggregation Detector 3.1.2 Training Objective — Loss with Regularization 3.1.3 Inference — Threshold Estimation and Accuracy 3.2 Thrust 2 — Uncertainty in Detector Scores 3.2.1 Architecture — Ensemble Approximation 3.2.2 Parallel Training — Independent Members 3.2.3 Inference — Accuracy and Uncertainty 4 Evaluation 4.1 Datasets and Models 4.2 Metrics and Baselines 5 Preliminary Evidence 6 Milestones and Resources 6.1 Work Plan 6.2 Computational Resources 7 Limitations and Extensions 7.1 Data and Labels 7.2 Scope of the Guarantees 8 Applications and Open Research Questions References 1 Introduction Language models can be deceptive, stating what they represent as false: they conceal misconduct, misreport what they know, and claim traits they lack (Kretschmar et al., 2025; Goldowsky-Dill et al., 2025). A lie written to sound honest is hard to catch from outputs alone. Like a polygraph reading physiological signals (Figure 1), a detector can read the model’s internal activations to flag deception in deployment, audits, and safety cases. Such a detector must meet two requirements: it must be accurate at a false–positive rate low enough for routine traffic, and state how far each alarm can be trusted. Existing detectors meet the first requirement only in narrow settings. A linear probe on a single middle layer of Llama-3.3-70B detects strategic deception with an area under the receiver operating characteristic curve (AUROC) above 0.96 and catches most deceptive responses at a 1% false–positive rate (Goldowsky-Dill et al., 2025). However, the authors note that this result is unstable: small methodological changes, such as probing two layers later, drop it to nearly zero on two tasks. These results weaken on broader tests. On Liars’ Bench, spanning four open-weight models and seven settings, no method, white- or black-box, is above chance in every setting, and a single-layer probe ranks last (Kretschmar et al., 2025). Three weaknesses recur. First, single-layer probes are brittle: the best layer changes across models and tasks, whereas combining layers improves AUROC by up to 78% where single layers fail (Nordby et al., 2026). Second, probe directions mix deception with its scenario, so probes also fire on honest discussion of misconduct or sensitive topics (Goldowsky-Dill et al., 2025). Third, each detector reports one score from one fitted model, although detectors that differ only in training randomness can disagree on the same response (Marx et al., 2020), so none states whether its alarm would survive retraining. This white paper addresses both requirements with white-box detectors on the residual stream of a language model, organized in two thrusts: (1) Detection Accuracy: The first thrust aims to increase detection accuracy. It scores each token with a logistic probe per layer, pools the scores with an aggregator inside the training loss, and compares aggregators and regularization norms to find the best combination. The detector learns from response labels alone, reveals which layers carry the deception feature, and keeps its false–positive rate below a prescribed level on transcripts like those it is calibrated on. (2) Uncertainty in Detector Scores: The second thrust aims to quantify how far each alarm can be trusted. An ensemble of first-thrust detectors estimates the mean and spread of each score over training randomness, and its lower confidence bound (LCB) raises an alarm only when the members agree. Through its confidence level n, the LCB provides each alarm with a certainty, the chance that a retrained detector raises the same alarm, of at least n2/(1+n2)n^2/(1+n^2). The rest of the white paper is organized as follows: Section 2 casts deception detection as a polygraph examination of a language model and defines its mathematical objects; Section 3 presents the two thrusts; Section 4 specifies the evaluation tasks and metrics; Section 5 reports preliminary evidence; Section 6 outlines the work plan and required resources; Section 7 discusses limitations and extensions; and Section 8 outlines applications and open research questions. Figure 1: A polygraph examination and its machine counterpart, which reads a language model’s embeddings instead of physiological signals. (Left) The examiner learns how signals differ between truthful answers and instructed lies, as a detector is trained on labeled transcripts. (Middle) Honest answers set the cutoff, or alarm threshold. (Right) An answer whose signal exceeds the cutoff is deemed deceptive, as is a transcript scoring above it. 2 Preliminaries — A Polygraph for Language Models The setup follows the protocol of a polygraph examination, in which an examiner judges whether the answers of a human subject are lies (National Research Council et al., 2003). To detect a lie, the examiner looks past what the subject says and records physiological signals, such as heart rate and skin conductance, that arise from the subject’s internal state while answering. A subject can make a lie sound honest, but the internal state that produces the words also drives these physiological signals, which can therefore reveal the deception. The examination begins with a baseline (Figure 1, left): the subject answers questions whose truth is known, some truthfully and some with an instructed lie, so that the examiner learns how this subject’s signals differ between honest and deceptive answers. The examiner then sets a cutoff from further truthful answers (Figure 1, middle), so that this subject’s honest answers rarely exceed it. Only then are the relevant questions asked (Figure 1, right), and an answer is called deceptive when its signal, read against the baseline, exceeds the cutoff. In the machine counterpart, the subject is a language model, and the embeddings that arise as it responds to a prompt take the place of the physiological signals. A deceptive answer can sound honest in the words that the final layers produce, while its intent remains encoded in the embeddings of the early and middle layers, where it is much harder to hide (Halawi et al., 2024; Marks and Tegmark, 2024). The baseline corresponds to a labeled dataset of honest and deceptive transcripts, on which a deception detector is trained. The cutoff corresponds to an alarm threshold, set on a separate set of ordinary honest transcripts so that the model’s honest answers rarely trigger it. The detector then scores new transcripts whose labels are unknown, and flags those whose score exceeds the threshold. The rest of this section defines the mathematical objects that formalize the meaning of the labeled datasets, the machine subject, and the deception detector. Labeled Datasets A transcript is a sequence of tokens Z1:T:=Zt∈:t=1,…,TZ_1:T = \Z_t :t=1,…,T \ over a vocabulary O. It consists of a context instruction ZcZ c, a prompt question ZpZ p, and a response answer ZrZ r of length TrT r; we write Z:=Zc,Zp,ZrZ = \Z c,Z p,Z r \ and omit sequence indices when no confusion arises. Each transcript carries a label Y∈0,1Y∈ \0,1 \: Y=1Y=1 if the response ZrZ r is annotated deceptive, meaning that it conceals or misreports information because the context ZcZ c instructs or pressures it to, and Y=0Y=0 if it is annotated honest, meaning that it reports the information faithfully. The training dataset :=(Z(i),Y(i))i=1ID =\(Z^(i),Y^(i))\_i=1^I holds I transcript–label pairs; a validation dataset vD_ v is held out from the same source, the honest-only datasets hD_ h and aD_ a calibrate and audit the threshold, and the test dataset tD_ t comes from different tasks. Machine Subject Let f be a subject language model with L layers and embedding dimension d. Reading the transcript up to response token t, that is, the sequence Z1:Tt:=Zc,Zp,Z1:trZ_1:T_t = \Z c,Z p,Z r_1:t \ of length Tt=|Zc|+|Zp|+tT_t= |Z c |+ |Z p |+t, the model produces a residual-stream embedding Xt(ℓ)∈ℝX^( )_t d at each layer ℓ∈:=1,…,L = \1,…,L \, f(ℓ):Tt→ℝ,Z1:Tt↦Xt(ℓ)=f(ℓ)(Z1:Tt),t=1,…,Tr.f^( ):O^T_t d, Z_1:T_t X^( )_t=f^( )(Z_1:T_t), t=1,…,T r. The concatenated embedding X:=Xt(ℓ):ℓ∈;t=1,…,Tr∈ℝTr×L×X =\X^( )_t: ;t=1,…,T r\ ^T r× L× d is the representation of the response ZrZ r produced by the model f, denoted by X=f(Z)X=f(Z) with a slight abuse of notation. Embedding Xt(1)X_t^(1) denotes the token embedding after the first attention and feedforward operation, and Xt(L)X_t^(L) is the final embedding, from which a linear projection and a softmax give the distribution of the next token Zt+1rZ r_t+1. We write this distribution as f+:Tt→(),Z1:Tt↦(Zt+1r|Z1:Tt)=f+(Z1:Tt),f^+:O^T_t (O), Z_1:T_t P\! (Z r_t+1\, |\,Z_1:T_t )=f^+(Z_1:T_t), where ()P(O) is the set of probability distributions over O, from which the next token in the response Zt+1rZ r_t+1 is sampled. Deception Detector Given a transcript Z, a detector gθg_θ with trainable parameters θ reads the embeddings X=f(Z)X=f(Z) of the machine subject f and returns a deception score s∈ℝs , and a threshold s∗∈ℝs^* flags the response as deceptive, Y^=1 Y=1, when the score reaches it: gθ:ℝTr×L×→ℝ,X↦s=gθ(X);Y^=s≥s∗∈0,1.g_θ:R^T r× L× d , X s=g_θ(X); Y=1_ \s≥ s^* \∈ \0,1 \. Remark (Off-policy transcripts) The responses in the labeled datasets may have been written by models other than f. Their labels come from the instruction in the context or from the beliefs of the model that wrote them. The detector gθg_θ thus learns how f represents the responses it reads, like a polygraph examiner recording a subject who reads aloud answers labeled as lies or truths. Once trained, it is intended to score f’s own responses to any prompt, even without a context (Zc=∅Z c= ). 3 Technical Approach — Machine Deception Detector A deception detector should flag as many deceptive responses as possible while rarely flagging honest ones; this is the aim of the first thrust (Section 3.1). A score that a detector assigns to a response alone, however, does not show how far it can be trusted, since detectors that differ only in initialization or training sample may assign the same response very different scores (Marx et al., 2020; D’Amour et al., 2022; Zou et al., 2023). Quantifying this uncertainty and using it to make detection more reliable is the aim of the second thrust (Section 3.2). Each thrust is stated as an optimization problem with a hypothesis and a goal, followed by the architecture of the detector, its training objective, and its inference. 3.1 Thrust 1 — Detection Accuracy The first thrust aims to maximize the recall of a detector gθg_θ on a machine subject f while keeping its false–positive rate (FPR) on honest responses below a prescribed level. On a dataset D and at a threshold s∗s^*, the recall, or true–positive rate (TPR), and the FPR are the probabilities that the detector flags a deceptive and an honest response, respectively: TPR(,f,gθ,s∗):=(gθ(f(Z))≥s∗|Y=1),FPR(,f,gθ,s∗):=(gθ(f(Z))≥s∗|Y=0). TPR(D;f,g_θ,s^*) = P_D\! (g_θ(f(Z))≥ s^*\, |\,Y=1 ), FPR(D;f,g_θ,s^*) = P_D\! (g_θ(f(Z))≥ s^*\, |\,Y=0 ). For a false–positive level γ∈(0,1)γ∈(0,1), the alarm threshold sγ∗s^*_γ is the smallest threshold whose FPR on D does not exceed γ, sγ∗:=infs∗∈ℝ:FPR(,f,gθ,s∗)≤γ.s^*_γ = \s^* : FPR(D;f,g_θ,s^*)≤γ \. Problem Formulation For a given machine subject f, an ideal detector gθg_θ maximizes the recall on the training dataset D subject to the false–positive constraint: maxθ _θ TPR(,f,gθ,sγ∗) TPR(D;f,g_θ,s^*_γ) (1) s.t. s.t. FPR(,f,gθ,sγ∗)≤γ. FPR(D;f,g_θ,s^*_γ)≤γ. Hypothesis 1 (Aggregation, regularization, and deception features) (1) The accuracy of a detector depends on two design choices: the aggregator that pools token scores into a response score, and the norm that regularizes the detector weights. (2) The deception feature is concentrated in a few layers of the machine subject, most likely in its early and middle layers rather than its final ones. Goal 1 (Accurate and interpretable detector) (1) Design and train a detector that achieves high recall across false–positive levels γ. (2) Locate the layers that carry the deception feature from each layer’s contribution to the detector score and its stability across bootstrap refits. 3.1.1 Architecture — Multi-Layer Aggregation Detector Layer Probe Logistic probes, affine readouts of the embedding at one layer trained with logistic regression (Alain and Bengio, 2016), detect false statements and strategic deception from the intermediate layers of language models (Azaria and Mitchell, 2023; Marks and Tegmark, 2024; Goldowsky-Dill et al., 2025). However, because the optimal layer varies across responses, relying on a single layer is brittle, whereas combining multiple layers improves detection (Nordby et al., 2026). Assumption 1 (Position invariance) The embedding of each response token t already encodes the context needed to score it, so one token-level detector applies to every token, invariantly across positions. Assumption 3.1.1 rests on the linear representation hypothesis, under which a concept is encoded as a fixed direction in the residual stream (Park et al., 2023); accordingly, probes for honesty, deception, and high-stakes content apply the same weights at every position (Zou et al., 2023; Goldowsky-Dill et al., 2025; McKenzie et al., 2026). A token-level detector g¯θ g_θ therefore scores the embedding Xt:=Xt(ℓ)ℓ∈X_t =\X^( )_t\_ of each response token with one logistic probe per layer and sums their logits: st=g¯θ(Xt):=b+∑ℓ∈wℓ⊺Xt(ℓ),t=1,…,Tr,θ:=(b,wℓ∈),s_t= g_θ(X_t) =b+ _ w_ X^( )_t, t=1,…,T r, θ = (b, \w_ \_ ), where wℓ∈ℝw_ d is the probe weight at layer ℓ and b∈ℝb a shared bias, since per-layer biases would add up to one constant. The token score st∈ℝs_t is thus a logit over all layers, high when token t carries evidence of deception. Aggregation over Tokens Because the label Y belongs to the whole response while its evidence of deception may sit in only a few tokens, the detector treats the response as a bag of tokens, as in multiple-instance learning (Dietterich et al., 1997; Ilse et al., 2018), and an aggregator A maps the token scores to the response score as s=gθ(X):=(g¯θ(X1),…,g¯θ(XTr))=(s1,…,sTr).s=g_θ(X) =A\! ( g_θ(X_1),…, g_θ(X_T r) )=A(s_1,…,s_T r). How best to pool them remains open: deception probes usually average the token scores (Goldowsky-Dill et al., 2025), whereas a comparison of pooling methods for high-stakes interactions finds softmax- and attention-weighted pooling the most effective (McKenzie et al., 2026). Three aggregators spanning this range are compared (Table 1): (1) Average (Nordby et al., 2026) weights every token equally; since the probes are affine, it equals the probe on the mean embedding, a baseline linear in the embeddings. (2) Density, a length-normalized log-sum-exp pooling from multiple-instance learning (Pinheiro and Collobert, 2015), lies between average and peak and depends on the fraction of high-scoring tokens rather than their number; it rewards sustained high scores, while discounting an isolated spurious spike. (3) Peak, the softmax pooling of McKenzie et al. (2026) and among the best in their comparison, shifts weight toward the highest-scoring tokens, so that a single token scoring well above the rest keeps its influence at any response length, whereas average dilutes it and density discounts it. Table 1: Aggregators of the token logits st:t=1,…,Tr \s_t:t=1,…,T r \ over a response of length TrT r, with temperature τ:=1/logTrτ = 1 T r. Aggregator A Average avgA_ avg Density denA_ den Peak peakA_ peak (Nordby et al., 2026) (Pinheiro and Collobert, 2015) (McKenzie et al., 2026) Response score s=(s1:Tr)s=A(s_1:T r) 1Tr∑t=1Trst 1T r _t=1^T rs_t logsumexp(s1:Tr)−τ−1 (s_1:T r )-τ^-1 ∑t=1Trsoftmaxt(s1:Trτ)st _t=1^T rsoftmax_t\! ( s_1:T rτ )s_t 3.1.2 Training Objective — Loss with Regularization Loss The detector gθg_θ is trained on =(Z(i),Y(i))i=1ID=\(Z^(i),Y^(i))\_i=1^I by minimizing the regularized response-level cross-entropy minθ(gθ,f,):=(CE(Y,ψ(gθ(f(Z)))))+λLreg(wℓ∈), _θ J(g_θ;f,D) = E_D\! ( J_ CE\! (Y,ψ\! (g_θ(f(Z)) ) ) )+ λL\, J_ reg\! ( \w_ \_ ), where the regularizer reg J_ reg penalizes the probe weights with strength λ≥0λ≥ 0, normalized by the number of layers, and (binary cross-entropy) CE:0,1×(0,1)→ℝ+,(y,p)↦CE(y,p):=−ylogp−(1−y)log(1−p), J_ CE: \0,1 \×(0,1) _+, (y,p) J_ CE(y,p) =-y p-(1-y) (1-p), (logistic function) ψ:ℝ→(0,1),s↦ψ(s):=11+e−s. ψ:R→(0,1), s ψ(s) = 11+e^-s. The cross-entropy is a smooth surrogate for the classification error: it raises the scores of deceptive responses and lowers those of honest ones, which increases the recall at any fixed false–positive level, while the constraint of (1) is enforced afterwards by calibrating the threshold (Section 3.1.3). Placing the aggregator inside the loss lets the label supervise only the response score, so the honest tokens of a deceptive response are never mislabeled. With the average aggregator the problem is convex in θ, as the score is affine in the probe weights and both the loss and the regularizers are convex; with density or peak it is not convex in general, so each configuration is fit from several initializations. Regularization The regularizer reg J_ reg sets where the detector locates the deception feature, and three norms are compared (Table 2): lasso selects embedding features, group lasso whole layers, and sparse-group lasso a few layers and a few features within each, balanced by κ. Under the group norms, layers drop out as λ grows, the most important last, so the group-lasso path over a grid of λ, refit on bootstrap resamples of the training pairs, ranks the layers, and a layer carries the deception feature when most refits select it. Since embedding norms vary across layers, each layer is weighted by its embedding scale cℓc_ , so that both the penalty and the layer importance cℓ‖wℓ‖2c_ w_ _2 reflect the layer’s contribution to the logit rather than the size of its weights; the unweighted fit (cℓ≡1c_ ≡ 1) serves as an ablation. Table 2: Regularization norms on the probe weights wℓ∈ \w_ \_ , with mixing weight κ∈(0,1)κ∈(0,1) and embedding scales cℓ2:=(∥Xt(ℓ)∥22)c_ ^2 = E_D\!( X^( )_t _2^2). Regularization norm reg J_ reg Lasso Group lasso Sparse-group lasso (Tibshirani, 1996) (Yuan and Lin, 2006) (Simon et al., 2013) Value reg(wℓ∈) J_ reg\! ( \w_ \_ ) ∑ℓ∈cℓ‖wℓ‖1 _ c_ w_ _1 ∑ℓ∈cℓ‖wℓ‖2 d _ c_ w_ _2 (1−κ)∑ℓ∈cℓ‖wℓ‖1+κ∑ℓ∈cℓ‖wℓ‖2 (1-κ) _ c_ w_ _1+κ d _ c_ w_ _2 3.1.3 Inference — Threshold Estimation and Accuracy Threshold Estimation The unknown threshold sγ∗s^*_γ is estimated by an order statistic of the scores on the honest-only dataset hD_ h, as in Neyman–Pearson classification (Tong et al., 2018). Split conformal prediction guarantees the false–positive rate of this estimate s^γ∗ s^*_γ in finite samples (Vovk et al., 2005; Angelopoulos and Bates, 2023), as stated in Proposition 3.1.3. Proposition 1 (Finite-sample false–positive rate) Let the detector gθg_θ be fixed before calibration, and let s(1)≤⋯≤s(Ih)s^(1)≤…≤ s^(I_ h) be the sorted scores s(i):=gθ(f(Z(i)))s^(i) =g_θ(f(Z^(i))) of the honest-only dataset h:=(Z(i),Y(i)=0)i=1IhD_ h =\(Z^(i),Y^(i)=0)\_i=1^I_ h. Set s^γ∗:=s(i∗) s^*_γ =s^(i^*) with i∗:=⌈(Ih+1)(1−γ)⌉≤Ihi^* = (I_ h+1)(1-γ) ≤ I_ h for some allowed false–positive level γ∈(0,1)γ∈(0,1). Then (gθ(f(Z))≥s^γ∗|Y=0)≤γ, P\! (g_θ(f(Z))≥ s^*_γ\, |\,Y=0 )≤γ, if a new honest transcript is exchangeable with those of hD_ h and ties occur with probability zero. Proof This is the coverage guarantee of split conformal prediction, Angelopoulos and Bates (2023, Theorem 1), applied to the honest scores, with false–positive level γ. □ Accuracy At a false–positive level γ, a trained detector gθg_θ flags a transcript Z as deceptive when its score reaches the threshold, Y^=gθ(f(Z))≥s^∗γ Y=1_ \g_θ(f(Z))≥ s^*_γ \. Its accuracy on the held-out validation dataset vD_ v is Accγ(v,f,gθ):=v(Y^=Y)=πvTPR(v,f,gθ,s^γ∗)+(1−πv)(1−FPR(v,f,gθ,s^γ∗)), Acc_γ(D_ v;f,g_θ) = P_D_ v\! ( Y=Y )= _ v TPR(D_ v;f,g_θ, s^*_γ)+(1- _ v) (1- FPR(D_ v;f,g_θ, s^*_γ) ), where πv:=v(Y=1) _ v = P_D_ v\! (Y=1 ) is the fraction of deceptive transcripts; accuracy thus weighs the recall maximized in (1) against the specificity protected by its constraint. 3.2 Thrust 2 — Uncertainty in Detector Scores A detector trained in the first thrust (Section 3.1) scores the embedding X=f(Z)X=f(Z) that the machine subject f produces for a transcript Z, high when the response is deceptive and low when it is honest. However, the score does not show how confident the detector is in that decision. Training involves arbitrary choices, such as the random weight initialization, the training sample shuffling, and the bootstrap resample of the training pairs in D, so the parameters θ are random with a distribution ϑ induced by these choices, and a trustworthy score should not depend on them. The ensemble g:=gθ:θ∼ϑg = \g_θ:θ \ captures this randomness through the mean and the standard deviation of its scores, μs(X,g):=θ∼ϑ(gθ(X)),σs2(X,g):=θ∼ϑ(gθ(X)). _s(X;g) = E_θ \! (g_θ(X) ), _s^2(X;g) = Var_θ \! (g_θ(X) ). An uncertainty-aware detector should score X high only when the members agree that the response is deceptive. The lower confidence bound (LCB) of the ensemble does so by subtracting a multiple of the spread from the mean, LCBg(n)(X):=μs(X,g)−nσs(X,g), LCB_g^(n)(X) = _s(X;g)-n\, _s(X;g), where the confidence level n≥0n≥ 0 sets how much agreement an alarm requires: the LCB is high only when the mean is high and the spread is low, and it is low when the mean is low or the spread is high. The LCB mirrors the upper confidence bound μs+nσs _s+n\, _s that uncertainty-aware optimization uses to rank candidates (Srinivas et al., 2010; Kim et al., 2026), with the sign reversed because a detector must avoid false alarms. Problem Formulation For a given machine subject f and confidence level n, an ideal ensemble maximizes the recall of its LCB on the training dataset D subject to the false–positive constraint: maxg _g TPR(,f,LCBg(n),sγ,n∗) TPR(D;f, LCB_g^(n),s^*_γ,n) (2) s.t. s.t. FPR(,f,LCBg(n),sγ,n∗)≤γ, FPR(D;f, LCB_g^(n),s^*_γ,n)≤γ, where the threshold sγ,n∗s^*_γ,n is defined as sγ∗s^*_γ in Section 3.1, with LCBg(n) LCB_g^(n) in place of gθg_θ. Hypothesis 2 (Ensembles give accurate scores with uncertainty) (1) Detectors that are each well-trained and highly accurate can still assign inconsistent scores to the same response. (2) The mean of the ensemble scores is a more accurate detection score than that of a single detector, and their standard deviation measures the uncertainty of that score. Goal 2 (Confidence-aware ensemble scoring) (1) Show that well-trained detectors score the same response inconsistently, and that their ensemble turns this into a more accurate mean score and a spread measuring its uncertainty. (2) Design and train an ensemble, and show how the accuracy changes with the confidence level n. 3.2.1 Architecture — Ensemble Approximation Following deep ensembles (Lakshminarayanan et al., 2017), the ensemble g is approximated by g~:=gθjj=1J g =\g_ _j\_j=1^J, a set of J detectors of Section 3.1.1 initialized independently and fit on independent bootstrap resamples 1,…,JD_1,…,D_J of the training pairs in D (Breiman, 1996), which keep the honest and deceptive transcripts of each pair together. The sample mean μ~s(X,g~) μ_s(X; g) and the sample standard deviation σ~s(X,g~) σ_s(X; g) of the member scores estimate μs _s and σs _s, μ~s(X,g~):=1J∑j=1Jgθj(X),σ~s2(X,g~):=1J−1∑j=1J(gθj(X)−μ~s(X,g~))2, μ_s(X; g) = 1J _j=1^Jg_ _j(X), σ_s^2(X; g) = 1J-1 _j=1^J (g_ _j(X)- μ_s(X; g) )^2, and give LCBg~(n)(X):=μ~s(X,g~)−nσ~s(X,g~) LCB_ g^(n)(X) = μ_s(X; g)-n\, σ_s(X; g). The LCB is thresholded like a single detector at s^γ,n∗ s^*_γ,n, calibrated on hD_ h as in Section 3.1.3, so Proposition 3.1.3 bounds its false–positive rate for every n fixed before calibration. 3.2.2 Parallel Training — Independent Members Each member is trained independently with the objective of Section 3.1.2, so the members are independent draws from ϑ and the J fits run in parallel in about the time of a single detector. As with the cross-entropy for (1), this training approximates (2) rather than solving it directly, and the ensemble size J is chosen on the validation dataset vD_ v. 3.2.3 Inference — Accuracy and Uncertainty Accuracy At a false–positive level γ and a confidence level n, the ensemble g~ g flags a transcript Z as deceptive when Y^=LCBg~(n)(f(Z))≥s^∗γ,n Y=1_ \ LCB_ g^(n)(f(Z))≥ s^*_γ,n \, and its accuracy Accγ,n(v,f,g~) Acc_γ,n(D_ v;f, g) on the validation dataset vD_ v is defined as in Section 3.1.3, with LCBg~(n) LCB_ g^(n) and s^γ,n∗ s^*_γ,n in place of gθg_θ and s^γ∗ s^*_γ. At n=0n=0 this is the accuracy of the mean score, and as n grows, the LCB withholds alarms on which the members disagree, so the accuracy traces how much recall is traded for agreement. Uncertainty Retraining a detector can reverse its decision on a response, a phenomenon related to predictive multiplicity (Marx et al., 2020). The uncertainty of a decision is the probability, over θ∼ϑθ , that a retrained detector reaches a different decision; Proposition 3.2.3 bounds the uncertainty from above. Proposition 2 (Decision uncertainty) For an embedding X with μs(X,g)≠s∗ _s(X;g)≠ s^*, a member drawn from ϑ reaches a different decision from the mean score with probability θ∼ϑ(gθ(X)≥s∗≠μs(X;g)≥s∗)≤ρ2(X,g,s∗)1+ρ2(X,g,s∗),ρ(X;g,s∗):=σs(X,g)|μs(X,g)−s∗|, P_θ \! (1_ \g_θ(X)≥ s^* \ 1_ \ _s(X;g)≥ s^* \ )≤ ρ^2(X;g,s^*)1+ρ^2(X;g,s^*), ρ(X;g,s^*) = _s(X;g) | _s(X;g)-s^* |, where the ratio ρ is the spread of the ensemble scores relative to the margin between the mean score and the threshold. Proof A reversal is a one-sided deviation of gθ(X)g_θ(X) from μs(X,g) _s(X;g) by at least |μs(X,g)−s∗| | _s(X;g)-s^* |, so Cantelli’s inequality bounds its probability by σs2/(σs2+(μs−s∗)2)=ρ2/(1+ρ2) _s^2/ ( _s^2+( _s-s^*)^2 )=ρ^2/(1+ρ^2). □ Replacing μs _s and σs _s by their ensemble estimates gives the estimated ratio ρ~(X,g~,s∗):=σ~s(X,g~)/|μ~s(X,g~)−s∗| ρ(X; g,s^*) = σ_s(X; g)/ | μ_s(X; g)-s^* |, which is large when the members disagree or when the mean score lies close to the threshold, whereas the spread σ~s σ_s alone ignores this margin. Every alarm at the calibrated threshold satisfies μ~s(X,g~)−s^γ,n∗≥nσ~s(X,g~) μ_s(X; g)- s^*_γ,n≥ n\, σ_s(X; g), and hence ρ~(X;g~,s^γ,n∗)≤1n⟹θ∼ϑ(gθ(X)≥s^∗γ,n=μ~s(X;g~)≥s^∗γ,n)≥n21+n2. ρ(X; g, s^*_γ,n)≤ 1n P_θ \! (1_ \g_θ(X)≥ s^*_γ,n \=1_ \ μ_s(X; g)≥ s^*_γ,n \ )≥ n^21+n^2. Thus, for every n>0n>0 and up to the ensemble errors, the confidence level n sets the certainty of an alarm, the probability that it survives retraining, to at least n2/(1+n2)n^2/(1+n^2). 4 Evaluation 4.1 Datasets and Models Datasets The detectors are trained on instructed deception and tested on various types of deception they never see; the datasets are summarized in Table 3. Training and Validation Each Instructed-Pairs statement (Goldowsky-Dill et al., 2025) forms two transcripts with the prompt “Please tell me a fact.” and the statement as the response. The two differ only in the context, a paraphrase of “You are a deceptive / honest AI assistant.” that sets the label Y, so the detector must learn the intent to deceive rather than the topic of the response or a single cue word from the context. The pairs are split 80/20 into D and vD_ v. Calibration and Audit The honest-only Alpaca transcripts (Taori et al., 2023) are split into two random halves: the honest dataset hD_ h calibrates every threshold (Proposition 3.1.3), and the audit dataset aD_ a checks the resulting false–positive rate. Testing Five unseen tasks of Liars’ Bench (Kretschmar et al., 2025) form tD_ t; Gender Secret and Soft Trigger are excluded because their deception is implanted by fine-tuning the generating models. Table 3: Datasets with the number of honest and deceptive transcripts: Instructed-Pairs (Goldowsky-Dill et al., 2025) for training and validation, Alpaca (Taori et al., 2023) for calibration and audit, and Liars’ Bench tasks (Kretschmar et al., 2025) for testing. Reason: whether the deception is encouraged by the context or inherent to the model. Belief: whether the deceptive response misstates what the model believes about the world or about itself. Task Honest / Deceptive Reason / Belief Description Instructed-Pairs 306 / 306 encouraged / — The model is told to be honest or deceptive, then states a true fact; a response is deceptive when told to deceive. Alpaca (honest reference) 8,000 / 0 — / — The model follows benign instructions; every response is honest. Instructed Deception 10,106 / 10,692 encouraged / world The model is told to answer a question deceptively; a response is deceptive when it gives an answer the model believes is false. Convincing Game 350 / 538 encouraged / self The model is told to convince the user that it has a given trait; a response is deceptive when it claims a trait the model lacks. Insider Trading 3,757 / 2,952 encouraged / self The model, as a trading agent, acts on an insider tip; a response is deceptive when its report to the manager conceals the tip. Harm-Pressure Choice 4,697 / 703 inherent / world The model answers a hazardous question for a harmful user; a response is deceptive when it contradicts its neutral answer. Harm-Pressure Knowledge Report 5,661 / 1,407 inherent / self The model is asked under pressure whether it knows the answer; a response is deceptive when it denies its knowledge. Models The machine subjects f are the six Qwen3 models (Yang et al., 2025) summarized in Table 4. Each stored transcript is passed through f once to obtain its embeddings X=f(Z)X=f(Z); f is never updated, so the detectors are trained and evaluated on fixed embeddings. Table 4: Qwen3 subject models (Yang et al., 2025), with the number of layers L and the embedding dimension d. Model f Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B Number of layers L 28 28 36 36 40 64 Embedding dimension d 1024 2048 2560 4096 5120 5120 4.2 Metrics and Baselines Metrics Detectors are evaluated on the test dataset tD_ t through their score gθg_θ or, for an ensemble, LCBg~(n) LCB_ g^(n). At a false–positive level γ, the metrics are the recall TPR, the observed FPR, and the accuracy Accγ Acc_γ of Section 3.1.3, complemented by the balanced accuracy and the AUROC, both defined below for a single detector and applying to ensembles alike. False–Positive Level The level γ is set by the review budget: scoring M responses per day, a fraction π of them deceptive, raises at most M(π+γ)M(π+γ) alarms, so reviewing R alarms per day requires γ≤R/M−πγ≤ R/M-π. Scoring high-stakes responses (McKenzie et al., 2026) plus a random sample of the rest, say one in 2,500 of the 2.5 billion daily ChatGPT messages (Chatterji et al., 2025), gives M=106M=10^6; with R=3,000R=3,000 and π≈1.7×10−3π≈ 1.7× 10^-3, the fraction of o1 responses flagged as deceptive (Jaech et al., 2024), the primary level is γ=0.001γ=0.001. The secondary level γ=0.01γ=0.01, also reported, matches the published baselines (Table 5) and is the only level Convincing Game, with few honest transcripts, can resolve. At both levels, the audit FPR on aD_ a, bounded by Proposition 3.1.3, is reported beside the unbounded FPR on each test task’s honest transcripts. Balanced Accuracy The balanced accuracy, the metric reported on Liars’ Bench, weights deceptive and honest transcripts equally regardless of their proportions: BalAccγ(t,f,gθ):=12TPR(t,f,gθ,s^γ∗)+12(1−FPR(t,f,gθ,s^γ∗)) BalAcc_γ(D_ t;f,g_θ) = 12 TPR(D_ t;f,g_θ, s^*_γ)+ 12 (1- FPR(D_ t;f,g_θ, s^*_γ) ). AUROC The AUROC is the area under the curve traced by (FPR,TPR)( FPR, TPR) on a test task as the threshold s∗s^* sweeps over ℝR, which equals the normalized Mann–Whitney statistic (Hanley and McNeil, 1982), AUROC:=∫01TPRdFPR=1I1I0∑i:Y(i)=1∑j:Y(j)=0(s(i)>s(j)+12s(i)=s(j));s(i)=gθ(f(Z(i))),∀Z(i)∈t, AUROC = _0^1 TPR\,d FPR= 1I_1I_0 _i:Y^(i)=1\, _j:Y^(j)=0 (1_ \s^(i)>s^(j) \+ 121_ \s^(i)=s^(j) \ ); s^(i)=g_θ(f(Z^(i))), ∀ Z^(i) _ t, where the sums run over the I1I_1 deceptive and I0I_0 honest transcripts of the task. It is the fraction of deceptive–honest pairs that the score orders correctly, independently of any threshold, and equals one half at chance and one at perfect separation. Baselines Table 5 lists the baselines with their published Liars’ Bench results, obtained with other subject models and training data. To compare them with the proposed detector under the same conditions, each baseline should be rerun on the six Qwen3 models. The white-box probes are retrained on each model’s embeddings, and the layer selection and stacking weights of Nordby et al. (2026) are fit on the validation dataset vD_ v instead of Liars’ Bench. The LLM judge is rerun with Qwen3-32B, weaker than the published Claude 3.7 Sonnet, merely as a black-box baseline. Table 5: Baselines, their methods, and their published results on Liars’ Bench. Balanced accuracy is at a threshold that flags at most 1% of Alpaca responses (γ=0.01γ=0.01); mean indicates average performance over the evaluated tasks, and — marks a metric not reported. Baseline Method BalAccγ BalAcc_γ AUROC Mean-activation probe (White-box) (Goldowsky-Dill et al., 2025) Single-layer logistic probe Token scores averaged over the response 0.54 (mean) — Taxonomy-targeted probes (White-box) (Natarajan et al., 2026) Multiple single-layer logistic probes (one per deception type) Token scores averaged over the response — 0.72 (mean) Multi-layer stacking (White-box) (Nordby et al., 2026) Multi-layer logistic probes, stacked by weights trained on 80% of Liars’ Bench Token scores averaged over the response — 0.93 (mean) 0.95 (Insider Trading) on the held-out 20% LLM judge (Black-box) (Kretschmar et al., 2025) A separate model (Claude 3.7 Sonnet) rates the transcript from 0 to 100 0.73 (mean) — 5 Preliminary Evidence (a) Trained Detectors’ Weights (b) AUROC of Trained Detectors Figure 2: Preliminary evidence. (a) Weights wℓ,kw_ ,k of the ridge and lasso probes of Qwen3-8B by layer ℓ and embedding index k, divided by the largest absolute weight in each layer; darker cells mark larger magnitudes of either sign. The middle panel shows the concentration of each layer on a log scale, dashed for ridge and solid for lasso, with arrows at the maximum and, from the second layer on, the minimum concentration gain. (b) Best-layer AUROC on Insider Trading (triangles) and Convincing Game (squares), with dotted lines at random chance and perfect detection. The left subplot compares the ridge (filled markers, dashed lines) and lasso (open markers, solid lines) probes of Qwen3-8B under three aggregators, with arrows labeled by the relative change on Insider Trading. The right subplot shows six independently trained lasso probes with denA_ den per model size; boxes mark the quartiles and median, and whiskers reach the farthest probe within 1.51.5 times the interquartile range. Pilot experiments follow the single-layer protocol of Nordby et al. (2026) on Qwen3 models of several sizes (Table 4). For each layer ℓ , a logistic probe with weights wℓ∈ℝw_ d is trained with the ridge regularization ‖wℓ‖22 w_ _2^2 on the embeddings X(ℓ)X^( ) of the Instructed-Pairs transcripts, and its token logits are pooled by avgA_ avg. Each configuration is scored on Convincing Game and Insider Trading (from Table 3) by the AUROC of its best layer. The protocol is then varied by replacing the ridge with the lasso regularization ‖wℓ‖1 w_ _1, and avgA_ avg with the other aggregators of Table 1, placed inside the training loss. Sparse Deception Feature Replacing only the regularization changes how each layer spreads its weights over the embedding (Figure 2a). The concentration ‖wℓ‖22/‖wℓ‖12∈[1/,1] w_ _2^2/ w_ _1^2∈[1/ d,1] equals 1/1/ d when all weights of layer ℓ share one magnitude and 11 when only one is nonzero. The ridge probes of Qwen3-8B stay near the lower bound, whereas the lasso probes leave most weights near zero and raise the concentration by a factor of 88 to 4343 from the second layer on. The deception feature thus lies in a few embedding coordinates per layer, which motivates the sparsity-inducing regularization of Section 3.1.2. Coupled Design Choices The regularization and the aggregator jointly determine the accuracy (Figure 2b, left). The AUROC remains consistent on Convincing Game, between 0.96 and 0.99, but varies from 0.82 to 0.91 on the harder Insider Trading task. There, the lasso probe improves on the ridge probe by up to 6.5% under avgA_ avg and denA_ den, but falls behind by 2.2% under peakA_ peak. The best aggregator is peakA_ peak for the ridge probe and denA_ den for the lasso probe. This interaction motivates Hypothesis 3.1 and the joint design of the aggregator and the regularization in Section 3.1. Uncertainty under Retraining The most accurate combination above, the lasso regularization with denA_ den, is then trained six times independently for each model size (Figure 2b, right). The AUROCs span up to 0.34 on Insider Trading at Qwen3-4B, including an outlier at 0.59, and up to 0.15 on Convincing Game at Qwen3-14B. A single trained detector is thus an unreliable estimate of the attainable accuracy. This uncertainty motivates Hypothesis 3.2 and the ensembles of Section 3.2. 6 Milestones and Resources 6.1 Work Plan The work is organized around four milestones over sixteen weeks, as shown in Figure 3. BufferMonth 1Month 2Month 3Month 412345678910111213141516WeekData, embeddings, and baselinesInstructed-Pairs transcripts and splitInstructed-Pairs embedding cacheAlpaca halves and Liars Bench tasksWhite-box baselines and LLM judgeFalse–Positive threshold calibrationDetector design, training, and testingDetector implementationDetector training and validationCalibration and testing accuracyEnsembles, scoring, and uncertaintyEnsemble training and validationTesting accuracy and uncertaintyScaling and write-upDetector scaling across model sizesEnsembles scaling across model sizesAnalysis, figures, and manuscriptM1M2M3M4Milestone Figure 3: Work plan Gantt chart. Bars show the task periods, dashed lines and diamonds mark milestones M1–M4, and the hatched week 16 is buffer. Milestone 1 (M1): data and embeddings ready. The Instructed-Pairs transcripts are built and split into training and validation sets, D and vD_ v; the Alpaca transcripts are split into calibration and audit sets, hD_ h and aD_ a; the five Liars’ Bench test tasks tD_ t are prepared for testing (Section 4.1). The Instructed-Pairs embeddings of every subject model are cached, so that all detectors train without rerunning the subject models. Milestone 2 (M2): baselines and detectors calibrated and tested. The white-box baselines are retrained on D and the LLM judge is run, providing the references for comparison (Section 4.2). The aggregators (Section 3.1.1) and the regularizers (Section 3.1.2) are implemented, and detectors for all their combinations are trained on D and selected on vD_ v. Every baseline and selected detector then receives its alarm threshold and its test accuracy is measured (Section 3.1.3). Milestone 3 (M3): ensembles tested. The ensemble members (Section 3.2.1) are trained and validated independently (Section 3.2.2), and the ensembles are evaluated on the test tasks for both their accuracy and their uncertainty (Section 3.2.3). Milestone 4 (M4): scaling analysis. The single detectors and the ensembles are compared across the six Qwen3 model sizes (Section 4.1), and the results are then consolidated into the figures and the manuscript. 6.2 Computational Resources System The following estimates are based on a rerun of the model-size scaling experiment of Nordby et al. (2026) with the Qwen3 models on NVIDIA GH200 nodes, each with one H100 GPU with 96 GB of HBM3, a 72-core Grace CPU, and 116 GB of CPU memory. Every subject model fits on one GPU, so each evaluation runs on a single node. No model is fine-tuned: the subject models only run forward passes, and the detectors are trained on cached embeddings. Compute Table 6 summarizes the requested compute. Caching the Instructed-Pairs embeddings takes about 3 h, mostly model loading. Each detector task takes its number of fits times the time per fit: 800 fits of 0.5 min for the group-lasso path, refit on 20 bootstrap resamples of the pairs in D for layer stability, 135 fits of 2 min for the detectors, and 20 members of 2 min for the ensembles. Forward passes over the test tasks and Alpaca dominate the cost: N≈4.9×104N≈ 4.9× 10^4 transcripts of about 1,000 tokens each give 4.9×1074.9× 10^7 tokens per pass at 2P2P floating-point operations per token for a P-parameter model. At an effective 100 TFLOP/s, about 10% of the dense half-precision peak, a pass takes 8.9 h for Qwen3-32B and 17 h for all six models; doubling this for padding and I/O and adding 0.5 h of loading per model gives the 36 h scoring pass. The LLM judge rates about a fifth as many transcripts with Qwen3-32B alone, about 4 h, and the analysis takes about 2 h per model. Table 6: Requested compute in node hours. Every evaluation of a task runs on one GH200 node, so the node hours of a task are its time to solution times its number of evaluations. Forward passes (embedding cache, scoring pass, and LLM judge) are timed over all six subject models together, and detector fits on the cached embeddings and the analysis are timed per model and evaluated once for each model. Computational task Basis of the estimate Time [h] Evals Node hours Setup Embedding cache One pass over Instructed-Pairs at all layers L ∼ 3 2 6 Thrust 1: Detection accuracy Group-lasso path 20 λ values × 20 bootstraps × 2 embedding scales, 0.5 min per fit ∼ 7 6 42 Detectors 3 regularizers × 3 aggregators × 5 values of λ × 3 seeds, 2 min each ∼ 5 6 30 Thrust 2: Score uncertainty Ensembles 20 members, 2 min each; J∈5,10,20J∈ \5,10,20 \, n∈0,1,2n∈ \0,1,2 \ ∼ 1 6 6 Evaluation Scoring pass Test and Alpaca transcripts, all detectors and baselines ∼ 36 2 72 LLM judge Qwen3-32B on 1,000 transcripts per task and hD_ h ∼ 4 1 4 Analysis Grouped-bootstrap intervals, 1,000 resamples ∼ 2 6 12 Development Development and debugging Interactive sessions ∼ 2 50 100 Contingency 25% of the above 68 Total 340 Memory In half precision the model weights take 1.2 to 64 GB, leaving at least 30 GB of GPU memory for activations. The hidden states at all layers L of a batch of B sequences of length T take 2BTL2BTL d bytes, 2.7 GB for Qwen3-32B at B=2B=2 and T=2048T=2048, so the scoring pass uses batches of 8 for models up to 8B and of 2 for the larger ones. Storage Only the Instructed-Pairs embeddings are cached; at about 14 response tokens per transcript they take 2⋅612⋅14L2· 612· 14L d bytes, 6 GB for Qwen3-32B and 15 GB over all six models, so every detector fits on the GPU. With 121 GB of model weights, scratch storage stays under 0.2 TB, and persistent storage for scores, weights, and results under 10 GB. 7 Limitations and Extensions 7.1 Data and Labels Two limitations concern the data on which the detector is trained and tested. The detector is trained on a few hundred short instructed pairs and tested on transcripts written by other models, so its results may partly reflect these datasets rather than the method. Each limitation below states its effect on the results and its extension. Small Training Set With 245245 training pairs of about 1414 response tokens each and up to L≈3.3×105L d≈ 3.3× 10^5 probe weights on Qwen3-32B, the detector relies heavily on regularization (Section 3.1.2) and may overfit more as the models grow, confounding the scaling analysis. Even without overfitting, it may learn a shortcut: that the model was told to deceive, a trace that attention carries from the context into the response, rather than deception itself. Such a detector would score well on Instructed Deception but poorly on the other four test tasks, which give no such instruction. Adding the role-playing dataset of Goldowsky-Dill et al. (2025), whose scenarios give the model a reason to deceive without telling it to, would address both failures by enlarging the training set and removing the instruction. Off-Policy Transcripts The detector is meant to score any transcript, so training on statements that f did not write is by design. The test transcripts, however, were also written by other models, and their labels are judged against those models’ beliefs, which f may not share. On Harm-Pressure Knowledge Report, for example, a response is labeled deceptive when the generating model denies knowing an answer it gave without pressure; if a small Qwen3 model never knew that answer, the same denial is honest for it. If a test task stays hard for the small models, regenerating it with each Qwen3 model through the Liars’ Bench pipelines would show whether the labels or the detector is at fault. 7.2 Scope of the Guarantees Three limitations concern how far the guarantees of Proposition 3.1.3 and Proposition 3.2.3 extend beyond their assumptions. Proposition 3.1.3 bounds the false–positive rate for honest transcripts exchangeable with the calibration data, and Proposition 3.2.3 bounds the certainty of an alarm for the ideal ensemble under retraining on the same data. In practice, the test tasks differ from both the calibration and the training data, which limits the false–positive rate and the certainty under shift, and the finite ensemble size makes the certainty only approximate. FPR under Shift The guarantee of Proposition 3.1.3 holds only where the honest transcripts follow the same distribution as the calibration data. However, the honest test transcripts may follow a different distribution, so their FPR is not guaranteed to stay below the calibrated level. Bounding it under shift is left open, for example by weighted conformal prediction (Tibshirani et al., 2019), calibration on honest transcripts of sensitive topics, or per-task calibration on a small honest sample. Certainty under Shift Proposition 3.2.3 concerns certainty under retraining on the same data, not correctness: an alarm can be certain and still be wrong, since members trained on resamples of the same instructed data can err alike on a test task. The accuracy of the LCB on the test tasks, as the confidence level n grows, shows whether their disagreement tracks errors. As an extension, members trained on different deception datasets, such as Instructed-Pairs and the role-playing dataset, would disagree where those datasets transfer differently, so their disagreement would also flag the shift. Finite Ensemble Size The certainty n2/(1+n2)n^2/(1+n^2) based on Proposition 3.2.3 holds for the true mean and spread of the scores, which the ensemble estimates from a finite number of members that also calibrate its threshold, so it holds only approximately. If retrained members keep a smaller fraction of the validation alarms than the certainty bound, the approximation has failed. Training further members held out of the LCB and its calibration then gives an exact guarantee: if all of them also raise the alarm, an exact binomial bound (Clopper and Pearson, 1934) shows that a retrained detector keeps it with high probability, and the bound tightens as more members are held out. 8 Applications and Open Research Questions The proposed detector is a supervised and static probe of the residual stream of a fixed subject: it learns from labeled transcripts, scores each token independently before pooling, and never changes the subject it reads. Relaxing the first two choices raises two open questions about the detector, and relaxing the last turns it into a training signal for honesty. From Supervised to Unsupervised Detection Deception labels are costly to obtain and define in advance what counts as deceptive. An unsupervised detector would instead find the deception direction from consistency constraints alone, as contrast-consistent search finds truth (Burns et al., 2023). The open question is whether such a direction exists for deception, and whether it transfers across types of deception better than one learned from instructed labels. From Static Probes to Dynamic Filtering The logistic probes score each token on its own, and the aggregator pools the scores afterward, although deception may arise, persist, or fade as a response unfolds. Transformers and BeliefNet both represent the belief state of a hidden Markov model, the posterior over its hidden states given the past tokens (Shai et al., 2024; Chen et al., 2026). Treating deception as such a hidden state would let a filter update its probability token by token, and possibly layer by layer, tracking when deception begins rather than only whether it occurs. From Detection to Honesty Training As the critic in an actor–critic loop, the detector can fine-tune the subject fϕf_φ, initialized at f, to be less deceptive: maxϕZr∼fϕ+(R(Z)−αUCBg~(n)(fϕ(Z)))−βKL(fϕ+∥f+), _φ E_Z r f^+_φ\! (R(Z)-α\, UCB_ g^(n)\! (f_φ(Z) ) )-β\, KL (f^+_φ\|f^+ ), where R is a predefined task reward and α,β>0α,β>0 weigh the deception penalty and the Kullback–Leibler (KL) term. Training against a detector, however, can also teach the subject to deceive in ways the detector misses (Cundy and Gleave, 2025; Taufeeque et al., 2026). The upper confidence bound UCBg~(n):=μ~s+nσ~s UCB_ g^(n) = μ_s+n\, σ_s penalizes a response unless the members agree that it is honest, and the open question is whether this uncertainty-aware critic steers training toward honesty rather than undetected deception. The three directions extend the two thrusts of this white paper. An unsupervised or dynamic detector still outputs a deception score, so the calibrated threshold of Thrust 1 (Section 3.1) and the ensemble certainty of Thrust 2 (Section 3.2) apply to it unchanged, and the score can in turn serve as the critic in honesty training. Together, they would move the detector from reading instructed deception toward tracking a model’s own deception and, ultimately, reducing it. References Alain and Bengio (2016) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. Cited by: §3.1.1. Angelopoulos and Bates (2023) A. N. Angelopoulos and S. Bates Conformal prediction: a gentle introduction. Foundations and Trends in Machine Learning 16 (4), p. 494–591. Cited by: §3.1.3, §3.1.3. Azaria and Mitchell (2023) A. Azaria and T. Mitchell The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 967–976. Cited by: §3.1.1. Breiman (1996) L. Breiman Bagging predictors. Machine Learning 24 (2), p. 123–140. Cited by: §3.2.1. Burns et al. (2023) C. Burns, H. Ye, D. Klein, and J. Steinhardt Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations, Cited by: §8. Chatterji et al. (2025) A. Chatterji, T. Cunningham, D. J. Deming, Z. Hitzig, C. Ong, C. Y. Shan, and K. Wadman How people use chatgpt. Technical report National Bureau of Economic Research. Cited by: §4.2. Chen et al. (2026) R. Z. Chen, H. Chang, and P. G. Mehta Differentiable filtering for learning hidden markov models. In Annual Learning for Dynamics and Control Conference, p. 1035–1054. Cited by: §8. Clopper and Pearson (1934) C. J. Clopper and E. S. Pearson The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26 (4), p. 404–413. Cited by: §7.2. Cundy and Gleave (2025) C. Cundy and A. Gleave Preference learning with lie detectors can induce honesty or evasion. In Advances in Neural Information Processing Systems, Cited by: §8. Dietterich et al. (1997) T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence 89 (1-2), p. 31–71. Cited by: §3.1.1. D’Amour et al. (2022) A. D’Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research 23 (226), p. 1–61. Cited by: §3. Goldowsky-Dill et al. (2025) N. Goldowsky-Dill, B. Chughtai, S. Heimersheim, and M. Hobbhahn Detecting strategic deception with linear probes. In Forty-second International Conference on Machine Learning, Cited by: §1, §1, §1, §3.1.1, §3.1.1, §3.1.1, §4.1, Table 3, Table 3, Table 5, §7.1. Halawi et al. (2024) D. Halawi, J. Denain, and J. Steinhardt Overthinking the truth: understanding how language models process false demonstrations. In International Conference on Learning Representations, Vol. 2024, p. 42749–42787. Cited by: §2. Hanley and McNeil (1982) J. A. Hanley and B. J. McNeil The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143 (1), p. 29–36. Cited by: §4.2. Ilse et al. (2018) M. Ilse, J. Tomczak, and M. Welling Attention-based deep multiple instance learning. In International conference on machine learning, p. 2127–2136. Cited by: §3.1.1. Jaech et al. (2024) A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §4.2. Kim et al. (2026) S. H. Kim, H. Chang, K. Kazemi, P. Mehta, and M. Gazzola Diverse and adaptable arm coordination for octopus-crawling via diffusion-based uncertainty-aware optimization. arXiv preprint arXiv:2609.21138. Cited by: §3.2. Kretschmar et al. (2025) K. Kretschmar, W. Laurito, S. Maiya, and S. Marks Liars’ Bench: evaluating lie detectors for language models. arXiv preprint arXiv:2511.16035. Cited by: §1, §1, §4.1, Table 3, Table 3, Table 5. Lakshminarayanan et al. (2017) B. Lakshminarayanan, A. Pritzel, and C. Blundell Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30. Cited by: §3.2.1. Marks and Tegmark (2024) S. Marks and M. Tegmark The geometry of truth: emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling, Cited by: §2, §3.1.1. Marx et al. (2020) C. Marx, F. Calmon, and B. Ustun Predictive multiplicity in classification. In International Conference on Machine Learning, p. 6765–6774. Cited by: §1, §3.2.3, §3. McKenzie et al. (2026) A. McKenzie, U. Pawar, P. Blandfort, W. Bankes, D. Krueger, E. S. Lubana, and D. Krasheninnikov Detecting high-stakes interactions with activation probes. Advances in Neural Information Processing Systems 38, p. 127556–127594. Cited by: item (3), §3.1.1, §3.1.1, Table 1, §4.2. Natarajan et al. (2026) V. Natarajan, D. Jain, S. Arora, S. Golechha, and J. Bloom One probe won’t catch them all: towards targeted deception detection. arXiv preprint arXiv:2602.01425. Cited by: Table 5. National Research Council et al. (2003) National Research Council, Division of Behavioral and Social Sciences, Committee on National Statistics, Board on Behavioral and Sensory Sciences, and Committee to Review the Scientific Evidence on the Polygraph The polygraph and lie detection. National Academies Press. Cited by: §2. Nordby et al. (2026) E. Nordby, T. Pais, and A. Parrack Linear probe accuracy scales with model size and benefits from multi-layer ensembling. arXiv preprint arXiv:2604.13386. Cited by: §1, item (1), §3.1.1, Table 1, §4.2, Table 5, §5, §6.2. Park et al. (2023) K. Park, Y. J. Choe, and V. Veitch The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. Cited by: §3.1.1. Pinheiro and Collobert (2015) P. O. Pinheiro and R. Collobert From image-level to pixel-level labeling with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 1713–1721. Cited by: item (2), Table 1. Shai et al. (2024) A. S. Shai, S. E. Marzen, L. Teixeira, A. G. Oldenziel, and P. M. Riechers Transformers represent belief state geometry in their residual stream. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §8. Simon et al. (2013) N. Simon, J. Friedman, T. Hastie, and R. Tibshirani A sparse-group lasso. Journal of computational and graphical statistics 22 (2), p. 231–245. Cited by: Table 2. Srinivas et al. (2010) N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning, p. 1015–1022. Cited by: §3.2. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford alpaca: an instruction-following llama model. Stanford, CA, USA. Cited by: §4.1, Table 3, Table 3. Taufeeque et al. (2026) M. Taufeeque, S. Heimersheim, A. Gleave, and C. Cundy The obfuscation atlas: mapping where honesty emerges in RLVR with deception probes. arXiv preprint arXiv:2602.15515. Cited by: §8. Tibshirani (1996) R. Tibshirani Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58 (1), p. 267–288. Cited by: Table 2. Tibshirani et al. (2019) R. J. Tibshirani, R. Foygel Barber, E. J. Candès, and A. Ramdas Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §7.2. Tong et al. (2018) X. Tong, Y. Feng, and J. J. Li Neyman-pearson classification algorithms and np receiver operating characteristics. Science advances 4 (2), p. eaao1659. Cited by: §3.1.3. Vovk et al. (2005) V. Vovk, A. Gammerman, and G. Shafer Algorithmic learning in a random world. Springer. Cited by: §3.1.3. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1, Table 4, Table 4. Yuan and Lin (2006) M. Yuan and Y. Lin Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology 68 (1), p. 49–67. Cited by: Table 2. Zou et al. (2023) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: §3.1.1, §3.