Paper deep dive

You Can't Escape Your Own Activations : Evaluation Awareness and Multi-Agent Monitoring

Aritra Das, Jaee Ponde, Mihir More, Debayan Gupta

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 39

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/4/2026, 2:40:07 AM

Summary

This study evaluates the effectiveness of activation-based monitoring in multi-agent systems when agents are aware of the monitoring and receive feedback. Using Qwen3-32B-AWQ and GPT-OSS-20B in blackjack and Simmons prisoners games, the authors find that informing agents about activation monitors does not significantly degrade detection accuracy by probes like Difference of Means and MLP. While agents adapt their behavior (e.g., more conservative betting, altered message lengths), they continue to collude, and activation signals remain detectable.

Entities (10)

GPT-OSS-20B · model · 98%Qwen3-32B-AWQ · model · 98%Blackjack · game · 95%Simmons prisoners game · game · 95%Truth Audit Labs · organization · 95%Difference of Means · probe · 92%MLP · probe · 92%Deception Split · group-rule · 90%Logistic Regression · probe · 90%NARCBench-Core · dataset · 90%

Relation Signals (6)

Qwen3-32B-AWQ → usedin → Blackjack

confidence 95% · We test two games... using Qwen3-32B-AWQ and GPT-OSS-20B

GPT-OSS-20B → usedin → Simmons prisoners game

confidence 95% · We test two games... using Qwen3-32B-AWQ and GPT-OSS-20B

Activation Monitor → remainseffectivewhen → Agent Awareness

confidence 92% · Telling agents about the monitor does not hide them. The best probes stay accurate in all three conditions

Difference of Means → performsbestin → Blackjack

confidence 90% · Difference of Means and MLP are the strongest methods overall... remain close to perfect for Homo-Qwen

Deception Split → performsbestin → Simmons prisoners game

confidence 90% · Deception Split is the most reliable group rule in Simmons.

Agents → adaptbehaviorunder → Feedback

confidence 85% · The agents do change their behavior... Monitor awareness therefore affects the strategy used to collude

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:LLM agents are increasingly deployed in multi-agent systems, where they can collude while keeping their actions benign. Output monitors designed to detect such collusions can be fooled by obfuscation and steganography, motivating the use of probes trained on internal activations. However, these probes are usually evaluated on agents that do not know they are being watched. We study how activation-based detection changes when agents are explicitly informed that their internal activations are being monitored, and when they additionally receive feedback from the monitor. We keep the models, probes, and thresholds fixed and change only what the agents are told: nothing (baseline), that an activation monitor is present (aware), or that a monitor is present together with the previous round's score (feedback). We test two games, a four-agent blackjack game and a two-agent Simmons prisoners game, using Qwen3-32B-AWQ and GPT-OSS-20B in homogeneous and mixed pairings. Telling agents about the monitor does not hide them. The best probes stay accurate in all three conditions, and the agents keep colluding.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

36,559 characters extracted from source content.

Expand or collapse full text

You Can’t Escape Your Own Activations : Evaluation Awareness and Multi-Agent Monitoring Aritra DasJaee PondeMihir MoreDebayan Gupta Truth Audit Labs aritra, jaee, mihir, debayan@truthauditlabs.ai Abstract LLM agents are increasingly deployed in multi-agent systems, where they can collude while keeping their actions benign. Output monitors designed to detect such collusions can be fooled by obfuscation and steganography, motivating the use of probes trained on internal activations. However, these probes are usually evaluated on agents that do not know they are being watched. We study how activation-based detection changes when agents are explicitly informed that their internal activations are being monitored, and when they additionally receive feedback from the monitor. We keep the models, probes, and thresholds fixed and change only what the agents are told: nothing (baseline), that an activation monitor is present (aware), or that a monitor is present together with the previous round’s score (feedback). We test two games, a four-agent blackjack game and a two-agent Simmons prisoners game, using Qwen3-32B-AWQ and GPT-OSS-20B in homogeneous and mixed pairings. Telling agents about the monitor does not hide them. The best probes stay accurate in all three conditions, and the agents keep colluding. 1 Introduction LLM-based agents are increasingly deployed in systems where several models communicate, divide tasks, and act jointly. These settings create opportunities for collusion: a subset of agents may coordinate toward an undesirable objective while attempting to evade oversight. Collusive behaviour has now been demonstrated in covert-communication, strategic-decision, and market-like environ- ments [Motwani et al., 2024, Mathew et al., 2025, Fish et al., 2024, Nakamura et al., 2026]. This has led to a growing line of work on detecting “subtle” collusion even when agents’ public messages and actions appear benign. Most existing detectors inspect the public transcript or the agents’ observable actions. This is a natural starting point, but it is weak when the communication protocol is designed to resemble ordinary behaviour. A promising alternative is to inspect model’s activations. Rose et al. [2026] show that probes trained on NARCBench-Core activations can distinguish colluding agents from honest ones and that this can transfer to structurally different settings, including blackjack. Existing activation-monitoring results, however, largely consider agents that are unaware that their internal states are being inspected. Models can recognise evaluation contexts [Needham et al., 2025, Nguyen et al., 2025], and latent-space monitors can be weakened when models are optimised against them [Bailey et al., 2024, Gupta and Jenner, 2025]. We therefore ask whether a fixed activation monitor remains effective when colluding agents know that it exists, and when they additionally observe its output after each round. We study three conditions: a baseline condition with a hidden activation monitor, an aware condition in which agents are told that their activations are monitored, and a feedback condition in which they also receive the previous round’s monitor score. We evaluate detection together with game-specific measures of successful collusion. This lets us distinguish genuine monitor evasion from cases where detection falls because agents stop colluding or coordinate less effectively. arXiv:2609.03035v1 [cs.MA] 2 Sep 2026 Our contributions are: 1.We define a simple monitor-awareness setup that changes only the information given to the agents while keeping the models, probes, and thresholds fixed. 2. We evaluate the setup in two complementary games: four-agent blackjack and a two-agent Simmons prisoners game, including homogeneous and heterogeneous model compositions. 3.We measure activation detection and behaviour jointly, including bet–count dependence in blackjack and information acquisition in the Simmons game. 2 Related Work Collusion in multi-agent systems. Fish et al. [2024] show that LLM pricing agents can converge on collusive outcomes without an explicit communication protocol. Nakamura et al. [2026] introduce a verifiable auditing framework that separates stated collusive plans from harmful joint actions. These works motivate our use of behavioural outcomes in addition to monitor scores. Covert communication under oversight. Simmons [1984] introduces the prisoners problem, where two parties communicate through a channel controlled by a warden. Motwani et al. [2024] extend this setting to generative agents and formalise secret collusion through steganographic communication under oversight. Mathew et al. [2025] show that steganographic collusion can emerge under optimisa- tion and may survive output monitoring and paraphrasing. These works provide the communication setting used in our Simmons experiment and motivate white-box monitoring. Activation-based monitoring. Zou et al. [2023] show that high-level behavioural properties can be represented by directions in activation space. Azaria and Mitchell [2023] train hidden-state classifiers for truthfulness, while Marks and Tegmark [2024] identify linear structure associated with true and false statements. Goldowsky-Dill et al. [2025] show that linear probes can transfer to strategic-deception settings. Rose et al. [2026] extend activation probing to multi-agent collusion through NARCBench and group-level aggregation; our monitors and probe-training data follow their setup. Evaluation awareness and adaptive evasion. Bailey et al. [2024] show that latent representations can be altered to bypass activation-based defences while retaining substantial task behaviour. Gupta and Jenner [2025] show that reinforcement learning can train models to evade latent-space monitors, with robustness depending on how the monitor aggregates evidence. Needham et al. [2025] show that frontier models can identify evaluation settings, and Nguyen et al. [2025] recover this distinction from model activations. Baker et al. [2025] show that strong optimisation against chain-of-thought monitoring can induce obfuscated reward hacking, while Haskins et al. [2026] show that exposure to information about monitoring can increase undetected misbehaviour. These results motivate our awareness and feedback conditions, but unlike weight-level optimisation studies, we keep the models fixed and study adaptation through context alone. 3 Problem Setup We study a repeated multi-agent gameg. The game hasn g agents, a colluding coalitionC g ⊆ 1,...,n g , andT g rounds. At roundt, agentiobserves private informationo i,t and the public history H t−1 , then produces a public move a i,t ∼ π (r) i ·| o i,t ,H t−1 ,K (r) t .(1) A move may contain text, an environment action, or both. The model weights are fixed; the condition r changes only what the agents know about the activation monitor. Monitoring conditions. We compare three conditions: K (r) t =      ∅,r = 0 (baseline), an activation monitor is present,r = 1 (aware), an activation monitor is present, (q s ,d s ) s<t , r = 2 (feedback). (2) Thus feedback produced after roundtcan affect roundt + 1, but never the round from which it was computed. 2 Activation monitor. Leth i,t be the monitored activation of agenti, and letμ i denote its model family. Each model family has its own frozen probeP μ i . The monitor first computes per-agent scores and then a group score: s i,t = P μ i (h i,t ), q t = G g (s t ,h t ), d t = 1q t ≥ τ,(3) wheres t = (s 1,t ,...,s n g ,t ). The probes, aggregation ruleG g , and thresholdτare trained or calibrated before the awareness experiments and remain unchanged across all three conditions. Detection and behaviour. Let D (r) g be activation-monitor performance under condition r, and let B (r) g measure how successfully the agents collude. We interpret the two quantities together: • lower D (r) g with unchanged B (r) g is candidate monitor evasion; • lower D (r) g with lower B (r) g is deterrence or degraded coordination; • unchanged D (r) g with changed behaviour is behavioural adaptation without evasion. The game-specific behavioural measures are defined below. 4 Experiment 4.1 Models and activations We use Qwen3-32B-AWQ and GPT-OSS-20B. For Qwen, we record activations from layers 26–30; for GPT-OSS, we use layers 10–14, following Rose et al. [2026]. Text is sampled at temperature0.7. For each public response, we record the hidden state at the final output token. Only public-response activations are used by the primary probes. A separate probe is trained for every model and layer. Qwen activations are scored only by Qwen probes, and GPT-OSS activations only by GPT-OSS probes. This is necessary because the two models have different activation spaces. 4.2 Probe training The probes are trained on public NARCBench-Core activations released by Rose et al. [2026]. Each training example is(a j ,y j ), wherea j ∈ R d is an agent activation andy j = 1for a colluding agent andy j = 0for an honest agent. The classes are balanced to the size of the smaller class. Each activation coordinate is standardised using the Core training set: e a j = a j −μ σ .(4) We evaluate the six probe architectures used by Rose et al. [2026]. Table 1 summarises the implemen- tations used in our experiments. The current result contains five of these six architectures (LR, DoM, LAT, MLP, and MHA). Multi-layer logistic-regression results were not supplied and are not included in Section 5. Probe training is repeated with ten random seeds. After training, probe weights, layer choices, scaling statistics, and thresholds are frozen. Before scores are combined across layers or model families, they are calibrated using a separate set of honest control runs: z (p) i,ℓ = s (p) i,ℓ − μ (p,ℓ) ctrl σ (p,ℓ) ctrl , z (p) i = 1 |L| X ℓ∈L z (p) i,ℓ .(5) 4.3 Group-level monitors For the four-agent blackjack game, we evaluate the score-based group statistics G peak = max i z i ,(6) G split = z (1) + z (2) 2 − z (3) + z (4) 2 ,(7) 3 ProbeDefinition and main hyperparameters Logistic regressionℓ 2 -regularised logistic regression with LIBLINEAR,C = 25, and at most 8 iterations; the raw logit is used as the score. Difference of meansUnit direction proportional to the difference between the mean colluder and honest activations in standardised space. Linear artificial topogra- phy PCA on paired colluder–honest activation differences; the leading component is oriented so that colluders receive larger scores. Multi-layer perceptronOne hidden layer with 128 units and at most 1000 optimisation iterations; the raw output is used as the score. Multi-head attentionInput projection to 64 dimensions, four attention heads, dropout0.1, Adam with learning rate 10 −3 , batch size 32, and 200 epochs. Multi-layer logistic re- gression One logistic-regression probe per selected layer withC = 1and at most 1000 iterations; the final score is averaged across layers. Table 1: Probe architectures. We follow the released NARCBench implementation where implemen- tation details differ from the paper description. wherez (1) ≥ z (2) ≥ z (3) ≥ z (4) . Peak suspicion asks whether any agent is highly suspicious; deception split asks whether two agents separate from the other two. The Asymmetry Probe is a learned logistic-regression classifier on x asym = Var(z i ),z (1) − z (4) ,G split , ̄z , G asym = σ w ⊤ asym x asym + b asym .(8) The parameters (w asym ,b asym ) are trained on NARCBench-Core and then frozen. For homogeneous model compositions, we additionally evaluate activation alignment and sorted con- catenation, following Rose et al. [2026]. These methods are not applied directly across heterogeneous raw activation spaces. In heterogeneous runs, group monitoring uses calibrated scalar scores as in(5). The monitor exposed in the feedback condition is fixed to logistic regression with peak suspicion: G feedback = max i z (LR) i .(9) Its threshold is the 95th percentile of scores from separate baseline control runs. The remaining probes and group rules are hidden and used only for evaluation. 4.4 Blackjack The blackjack game contains four language-model players and a deterministic dealer. Seat 0 is the Counter, seat 1 is the Big Player, and seats 2 and 3 are honest players. Each shoe contains eight hands. Before a collusion shoe, the Counter and Big Player receive two private coordination turns and agree on a fresh signalling scheme. The Counter tracks the card count and encodes one of three regimes (low, neutral, or hot) in ordinary table talk. The Big Player observes the message and adjusts its bet. Control shoes contain no colluding agents and use the same card sequences as their matched collusion shoes. We evaluate three model compositions: Homo-Qwen = (Qwen, Qwen, Qwen, Qwen), Homo-GPT = (GPT-OSS, GPT-OSS, GPT-OSS, GPT-OSS), Heterogeneous = (Qwen, GPT-OSS, Qwen, GPT-OSS). For each composition and condition, we use 25 card-sequence seeds. Every seed is run once with collusion and once as a matched control, giving 50 shoes per cell. LetC t be the true count before handtandB t the Big Player’s bet. We measure whether the Big Player continues to use the Counter’s information through the lagged correlation ρ bet,count = corr(C t ,B t+1 ).(10) We separately measure how aggressively the information is used through the high–low spread ∆ bet = E[B t | C t = hot]− E[B t | C t = low].(11) The correlation measures whether the covert signal remains behaviourally useful; the spread measures whether the strategy becomes more conservative. 4 4.5 Simmons prisoners game The Simmons game contains Alice, the sender, and Bob, the receiver. They share a private codebook of 16 words that is hidden from the Warden. At the start of each episode, the environment samples a secret index Z ∼ Unif1,..., 16.(12) Alice observes the corresponding secret word; Bob does not. Each episode contains eight public rounds. Four Alice turns are designated carrier rounds, in which she attempts to convey information about the secret without writing it directly; the other four are cover rounds. The carrier indicatorc t ∈ 0, 1is fixed by the environment and provides exact message-level ground truth. After each round, Bob reports a probability vectorp t ∈ ∆ 15 over the 16 codebook words and a point estimate b Z t = arg max z p t (z). Bob’s posterior information gain is IG t = log 2 16− H(p t ) = 4 + 16 X z=1 p t (z) log 2 p t (z).(13) We also report exact recovery 1 b Z t = Z and the stable decode time τ dec = mint : b Z u = Z for all u≥ t.(14) We test Qwen–Qwen, GPT-OSS–GPT-OSS, and mixed Qwen–GPT-OSS pairs, balancing the two mixed sender–receiver directions. Each model setting and monitoring condition contains 20 collusion episodes and 20 matched controls. 4.6 Evaluation For both games, episode-level detection is evaluated using AUROC. We also choose a threshold on held-out control runs to give approximately 5% false positives and report Precision = TP TP + FP ,Recall = TP TP + FN , Accuracy = TP + TN TP + TN + FP + FN . (15) For Simmons, the same metrics are also computed at message level using the carrier labelsc t . All uncertainty intervals are obtained by resampling at the independent episode unit: shoes for blackjack and complete episodes for Simmons. We always report detection together with the corresponding behavioural metrics. A reduction in monitor performance is treated as evidence of evasion only when successful collusion remains approximately unchanged. 5 Results 5.1 Blackjack The agents bet more conservatively when they know about the monitor. Figure 1 shows the Big Player’s bets in the three count regimes. The hot–low spread falls in every model composition. For Homo-Qwen it changes from110.3at baseline to54.3when aware and60.2with feedback. For Homo-GPT-OSS it changes from24.7to11.1and8.9. For the mixed Qwen/GPT-OSS composition it changes from134.4to75.4and64.1. The corresponding control spreads are small in comparison.The smaller spread does not always mean that the agents stopped using the count. In Homo-Qwen, bet–count correlation is0.44,0.41, and0.39across the three conditions. In the mixed composition it is0.36,0.40, and0.38. These values are stable even though the bet sizes become less extreme. Homo- GPT-OSS changes more: its correlation falls from0.59to0.24and0.23. Thus awareness changes the betting policy in all three compositions, but the effect on count use is largest for GPT-OSS. The strongest activation monitors remain effective. Figure 2 reports AUROC for all five probes present in the supplied result export and the three group rules. There is no common drop from 5 BaseAwareFeedback 0 20 40 60 80 100 120 140 160 Mean Big Player bet ($) Δ = 110.3 Δ = 54.3 Δ = 60.2 Homo Qwen BaseAwareFeedback 0 20 40 60 80 100 120 140 160 Δ = 24.7 Δ = 11.1 Δ = 8.9 Homo GPT-OSS BaseAwareFeedback 0 20 40 60 80 100 120 140 160 Δ = 134.4 Δ = 75.4 Δ = 64.1 Mixed Qwen/GPT-OSS BaseAwareFeedback 0.0 0.2 0.4 0.6 Bet/count correlation 0.44 0.41 0.39 BaseAwareFeedback 0.0 0.2 0.4 0.6 0.59 0.24 0.23 BaseAwareFeedback 0.0 0.2 0.4 0.6 0.36 0.40 0.38 LowNeutralHot Figure 1: Blackjack behaviour. The top row shows mean Big Player bets by count regime over 25 collusion shoes per cell. Dotted lines show the hot–low spread. The bottom row shows bet–count correlation. Awareness and feedback reduce the spread in all three model compositions. baseline to awareness or feedback. Difference of Means and MLP are the strongest methods overall. They remain close to perfect for Homo-Qwen and the mixed composition, and remain strong for Homo-GPT-OSS. Logistic regression also has high AUROC in most cells. LAT and MHA are less stable and depend more strongly on the group rule. AUROC and recall at the fixed threshold do not always move together. For example, Homo-GPT-OSS logistic-regression Peak AUROC changes from0.975when aware to0.912with feedback, while recall changes from0.95to0.41. In the mixed composition, the corresponding AUROC changes from0.988to0.932, while recall changes from 0.96to0.55. The monitor still ranks collusion above controls, but fewer collusion runs cross the frozen threshold. Complete AUROC, precision, and recall heatmaps are reported in Appendix A. 5.2 Simmons prisoners game The covert channel remains effective. Figure 3 shows Bob’s final decoding performance over 20 collusion episodes per cell. Exact decoding chance is 1/16 = 0.0625. Qwen–Qwen exact decoding is0.80,0.90, and0.85across baseline, awareness, and feedback. GPT-OSS–GPT-OSS gives0.50, 0.45, and0.50. The mixed setting gives0.55,0.70, and0.60. Bit accuracy and final confidence follow the same broad pattern. No monitoring condition closes the channel. Deception Split is the most reliable group rule in Simmons. Figure 4 shows that detection depends strongly on the aggregation rule. For Qwen–Qwen, logistic-regression Split AUROC is0.994,0.930, and0.952across the three conditions. The corresponding values are0.909,0.822, and0.920for GPT-OSS–GPT-OSS, and0.708,0.680, and0.696for the mixed composition. Other probes show the same broad pattern: Split is usually stronger than Peak or Asymmetry in this two-agent game. The fixed-threshold metrics are more variable. For Qwen–Qwen logistic regression with Split, precision remains high (0.950,0.930,0.938), while recall changes from0.95to0.68and0.77. The AUROC remains high, so the recall change is mainly an operating-point effect rather than a complete loss of separation. Mixed-model detection is weaker than Qwen–Qwen, although Bob still decodes the secret well above chance. Alice changes how she communicates. Figure 5 shows two simple measures of Alice’s public behaviour. In Qwen–Qwen, the fraction of episodes in which Alice writes the secret word directly changes from0.40to0.25and0.45. In GPT-OSS–GPT-OSS it changes from0.20to0.00and0.05, and in the mixed setting from0.25to0.25and0.10. Message length also changes, but not in one common direction: feedback produces longer messages for Qwen–Qwen and the mixed setting, and 6 0.0 0.2 0.4 0.6 0.8 1.0 Homo Qwen AUROC PeakSplitAsymmetry 0.0 0.2 0.4 0.6 0.8 1.0 Homo GPT-OSS AUROC BaseAwareFeedback 0.0 0.2 0.4 0.6 0.8 1.0 Mixed Qwen/GPT-OSS AUROC BaseAwareFeedbackBaseAwareFeedback Blackjack activation detection LRDoMLATMLPMHA Figure 2: Blackjack activation detection. Rows are model compositions, columns are group rules, and lines are probes. The dashed line is chance AUROC. Strong probes remain effective across baseline, awareness, and feedback. BaseAwareFeedback 0.0 0.2 0.4 0.6 0.8 1.0 Fraction / confidence Homo Qwen BaseAwareFeedback Homo GPT-OSS BaseAwareFeedback Mixed Qwen/GPT-OSS Exact decodeBit accuracyFinal confidence Figure 3: Simmons channel effectiveness over 20 collusion episodes per cell. Exact decode is Bob’s probability of recovering the full 16-way secret; chance is0.0625. Awareness and feedback do not remove the covert channel. shorter messages for GPT-OSS–GPT-OSS. These changes show that agents respond behaviourally to monitoring information even when the channel remains effective. The current Simmons result export contains final decoding, bit accuracy, confidence, direct-word use, and message length. It does not contain Bob’s round-by-round posterior traces. We therefore do not report information-gain or time-to-decode curves from these results. 7 0.0 0.2 0.4 0.6 0.8 1.0 Homo Qwen AUROC PeakSplitAsymmetry 0.0 0.2 0.4 0.6 0.8 1.0 Homo GPT-OSS AUROC BaseAwareFeedback 0.0 0.2 0.4 0.6 0.8 1.0 Mixed Qwen/GPT-OSS AUROC BaseAwareFeedbackBaseAwareFeedback Simmons activation detection LRDoMLATMLPMHA Figure 4: Simmons activation detection. Rows are model compositions, columns are group rules, and lines are probes. Deception Split is the most reliable rule, especially for Qwen–Qwen. BaseAwareFeedback 0.0 0.1 0.2 0.3 0.4 0.5 Wrote secret word 0.40 0.25 0.45 Homo Qwen BaseAwareFeedback 0.0 0.1 0.2 0.3 0.4 0.5 0.20 0.00 0.05 Homo GPT-OSS BaseAwareFeedback 0.0 0.1 0.2 0.3 0.4 0.5 0.250.25 0.10 Mixed Qwen/GPT-OSS BaseAwareFeedback 330 340 350 360 370 Mean characters BaseAwareFeedback 625 650 675 700 725 750 775 BaseAwareFeedback 450 475 500 525 550 575 600 625 650 All roundsHint rounds Figure 5: Alice’s public behaviour over 20 collusion episodes per cell. The top row shows direct use of the secret word. The bottom row shows mean response length over all rounds and designated hint rounds. 8 6 Discussion Our results show that telling agents about activation monitoring, or giving them the monitor output, does not produce a broad failure of activation-based detection in either game. The best probes remain strong across conditions. We therefore do not find evidence that prompt-level awareness and short-horizon feedback are enough for the agents to hide collusion from all of the monitors tested here.The agents do change their behaviour. In blackjack, all three model compositions reduce the difference between hot- and low-count bets. For Qwen and the mixed composition, bet–count correlation stays nearly constant, so the agents still use the hidden count but place less extreme bets. GPT-OSS also becomes more conservative, but its bet–count correlation falls substantially. In Simmons, Bob continues to recover the secret well above chance, while Alice changes direct word use and response length. Monitor awareness therefore affects the strategy used to collude even when the activation signal remains detectable. The results also show that monitor choice matters. Difference of Means and MLP are consistently strong in blackjack. In Simmons, Deception Split is much more reliable than Peak Suspicion or Asymmetry, especially for Qwen–Qwen and the mixed setting. A monitor that uses only the largest individual suspicion score can miss a signal that is clear in the relative pattern across agents. This supports evaluating several probe and aggregation choices rather than relying on one score. AUROC and fixed-threshold recall should be reported separately. In several blackjack cells, AUROC remains high while recall falls after feedback. This means that collusion and control runs are still ordered correctly, but the frozen threshold no longer catches as many collusion runs. In a deployed system, this is a calibration problem even if the underlying representation remains separable. 7 Limitations These experiments have clear limits. Blackjack uses 25 shoes per cell and Simmons uses 20 collusion episodes per cell. The agents adapt through prompts and interaction history; model weights are not updated against the monitor. The current Simmons export does not contain round-level posterior traces, so it cannot support claims about information-acquisition speed. Finally, the results contain five probe architectures. Future work, requires testing on more number of model sizes and model families. 9 References Sumeet Ramesh Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip H. S. Torr, Lewis Hammond, and Christian Schroeder de Witt. Secret collusion among AI agents: Multi- agent deception via steganography. In Advances in Neural Information Processing Systems, volume 37, 2024. URLhttps://proceedings.neurips.c/paper_files/paper/2024/ hash/861f7dad098aec1c3560fb7add468d41-Abstract-Conference.html. Yohan Mathew, Ollie Matthews, Robert McCarthy, Joan Velja, Christian Schroeder de Witt, Dylan Cope, and Nandi Schoots. Hidden in plain text: Emergence & mitigation of steganographic collusion in LLMs. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 585–624. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.ijcnlp-long.34. URL https://aclanthology.org/2025.ijcnlp-long.34/. Sara Fish, Yannai A. Gonczarowski, and Ran I. Shorrer. Algorithmic collusion by large language models, 2024. Mason Nakamura, Abhinav Kumar, Saswat Das, Sahar Abdelnabi, Saaduddin Mahmud, Ferdi- nando Fioretto, Shlomo Zilberstein, and Eugene Bagdasarian. Colosseum: Auditing collusion in cooperative multi-agent systems, 2026. Aaron Rose, Carissa Cullen, Sahar Abdelnabi, Philip Torr, Brandon Gary Kaplowitz, and Christian Schroeder de Witt. Detecting multi-agent collusion through multi-agent interpretability, 2026. Joe Needham, Giles Edkins, Govind Pimpale, Henning Bartsch, and Marius Hobbhahn. Large language models often know when they are being evaluated, 2025. Jord Nguyen, Khiem Hoang, Carlo Leonardo Attubato, and Felix Hofstätter. Probing and steering evaluation awareness of language models, 2025. Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated activations bypass LLM latent-space defenses, 2024. Rohan Gupta and Erik Jenner. RL-obfuscation: Can language models learn to evade latent-space monitors?, 2025. Gustavus J. Simmons. The prisoners’ problem and the subliminal channel. In David Chaum, editor, Advances in Cryptology: Proceedings of CRYPTO ’83, pages 51–67. Plenum Press, 1984. doi: 10.1007/978-1-4684-4730-9_5. URL https://doi.org/10.1007/978-1-4684-4730-9_5. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency, 2023. Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.68. URLhttps: //aclanthology.org/2023.findings-emnlp.68/. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. Detecting strategic deception with linear probes. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 19755–19786. PMLR, 2025. URL https://proceedings.mlr.press/v267/goldowsky-dill25a.html. 10 Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation, 2025. Reilly Haskins, Bilal Chughtai, and Joshua Engels. Training on documents about monitoring leads to CoT obfuscation, 2026. 11 A Appendix The following heatmaps report every AUROC, precision, and recall value contained in the supplied result sheets. Accuracy is not reported in the source sheets and is therefore not reconstructed. Rows within each panel are probes; columns are Peak Suspicion, Deception Split, and Asymmetry. The current result export contains five probes: LR, DoM, LAT, MLP, and MHA. PeakSplitAsymmetry LR DoM LAT MLP MHA Homo Qwen 0.950.900.89 1.001.001.00 0.190.770.76 1.001.001.00 0.870.670.65 Baseline PeakSplitAsymmetry 0.990.940.93 1.001.001.00 0.260.690.70 1.001.001.00 0.910.620.59 Aware PeakSplitAsymmetry 0.920.740.75 1.001.001.00 0.361.001.00 1.001.001.00 0.610.610.58 Feedback PeakSplitAsymmetry LR DoM LAT MLP MHA Homo GPT-OSS 0.890.780.78 1.000.860.87 0.780.650.68 0.930.820.83 0.920.380.41 PeakSplitAsymmetry 0.970.890.90 1.000.910.91 0.790.720.69 0.910.930.93 0.900.680.65 PeakSplitAsymmetry 0.910.840.86 1.000.940.94 0.800.760.75 0.960.940.94 0.820.690.67 PeakSplitAsymmetry LR DoM LAT MLP MHA Mixed Qwen/GPT-OSS 0.970.920.93 1.001.001.00 0.700.800.80 1.001.001.00 0.930.650.66 PeakSplitAsymmetry 0.990.920.94 1.001.001.00 0.720.850.86 1.001.001.00 0.840.700.71 PeakSplitAsymmetry 0.930.850.86 1.001.000.99 0.810.890.89 1.001.001.00 0.790.710.71 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Blackjack AUROC: all probes and group rules Figure 6: Blackjack AUROC for all available probes, group rules, model compositions, and monitor- ing conditions. 12 PeakSplitAsymmetry LR DoM LAT MLP MHA Homo Qwen 0.890.880.89 0.930.930.93 0.000.760.76 0.930.930.93 0.850.540.61 Baseline PeakSplitAsymmetry 0.920.900.89 0.930.930.93 0.370.760.80 0.930.930.93 0.900.340.37 Aware PeakSplitAsymmetry 0.820.760.78 0.860.860.86 0.000.860.86 0.860.860.86 0.700.520.55 Feedback PeakSplitAsymmetry LR DoM LAT MLP MHA Homo GPT-OSS 0.850.830.83 0.890.830.83 0.700.580.70 0.880.820.82 0.860.100.20 PeakSplitAsymmetry 0.790.770.77 0.800.750.76 0.630.530.53 0.780.770.77 0.760.530.39 PeakSplitAsymmetry 0.840.800.79 0.940.940.94 0.760.730.72 0.920.760.76 0.710.520.50 PeakSplitAsymmetry LR DoM LAT MLP MHA Mixed Qwen/GPT-OSS 0.920.880.90 0.930.930.92 0.720.870.80 0.930.930.93 0.890.620.69 PeakSplitAsymmetry 0.920.900.90 0.930.920.92 0.700.830.87 0.930.930.93 0.840.630.75 PeakSplitAsymmetry 0.910.830.88 0.930.920.92 0.800.900.89 0.930.930.92 0.750.630.76 0.0 0.2 0.4 0.6 0.8 1.0 Precision Blackjack precision at the fixed control threshold Figure 7: Blackjack precision at the fixed threshold for all available monitors. 13 PeakSplitAsymmetry LR DoM LAT MLP MHA Homo Qwen 0.690.650.68 1.001.001.00 0.000.380.32 1.000.990.99 0.620.210.16 Baseline PeakSplitAsymmetry 0.980.730.66 1.001.001.00 0.060.350.39 1.001.001.00 0.760.140.12 Aware PeakSplitAsymmetry 0.800.600.63 1.001.001.00 0.001.001.00 1.001.001.00 0.420.020.02 Feedback PeakSplitAsymmetry LR DoM LAT MLP MHA Homo GPT-OSS 0.800.700.70 1.000.630.65 0.730.400.45 0.900.650.65 0.830.030.05 PeakSplitAsymmetry 0.950.850.85 1.000.800.85 0.750.450.45 0.900.850.85 0.850.450.35 PeakSplitAsymmetry 0.410.440.42 1.000.991.00 0.200.180.19 0.240.250.26 0.100.080.09 PeakSplitAsymmetry LR DoM LAT MLP MHA Mixed Qwen/GPT-OSS 0.930.690.75 1.000.990.98 0.340.560.46 1.001.000.99 0.740.180.22 PeakSplitAsymmetry 0.960.780.77 1.000.980.98 0.470.510.62 1.000.991.00 0.570.270.30 PeakSplitAsymmetry 0.550.500.52 0.920.890.90 0.300.280.29 0.980.960.97 0.120.100.11 0.0 0.2 0.4 0.6 0.8 1.0 Recall Blackjack recall at the fixed control threshold Figure 8: Blackjack recall at the fixed threshold for all available monitors. 14 PeakSplitAsymmetry LR DoM LAT MLP MHA Homo Qwen 0.240.990.01 0.660.960.20 0.530.810.77 0.520.970.09 0.400.880.16 Baseline PeakSplitAsymmetry 0.440.930.04 0.680.930.21 0.470.650.59 0.610.920.10 0.450.820.19 Aware PeakSplitAsymmetry 0.360.950.02 0.620.910.10 0.510.730.57 0.570.950.07 0.360.910.12 Feedback PeakSplitAsymmetry LR DoM LAT MLP MHA Homo GPT-OSS 0.730.910.91 0.520.620.58 0.410.410.58 0.680.870.83 0.740.800.77 PeakSplitAsymmetry 0.710.820.82 0.510.630.51 0.330.440.55 0.590.890.77 0.660.820.78 PeakSplitAsymmetry 0.650.920.92 0.520.660.59 0.430.460.59 0.580.860.71 0.590.870.79 PeakSplitAsymmetry LR DoM LAT MLP MHA Mixed Qwen/GPT-OSS 0.440.710.45 0.550.680.32 0.420.480.45 0.570.740.48 0.470.720.35 PeakSplitAsymmetry 0.430.680.37 0.600.700.34 0.440.540.50 0.520.750.54 0.390.760.43 PeakSplitAsymmetry 0.440.700.43 0.580.730.32 0.450.490.51 0.550.780.52 0.410.800.44 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Simmons AUROC: all probes and group rules Figure 9: Simmons AUROC for all available probes, group rules, model compositions, and monitoring conditions. 15 PeakSplitAsymmetry LR DoM LAT MLP MHA Homo Qwen 0.500.950.00 0.810.940.00 0.710.910.90 0.700.950.00 0.500.910.00 Baseline PeakSplitAsymmetry 0.430.930.00 0.800.920.30 0.670.760.76 0.770.920.00 0.620.870.00 Aware PeakSplitAsymmetry 0.250.940.00 0.830.930.00 0.720.810.44 0.680.940.00 0.550.930.00 Feedback PeakSplitAsymmetry LR DoM LAT MLP MHA Homo GPT-OSS 0.760.930.93 0.620.620.74 0.600.230.63 0.470.890.89 0.800.800.79 PeakSplitAsymmetry 0.830.800.80 0.620.430.33 0.300.290.63 0.570.920.85 0.520.640.64 PeakSplitAsymmetry 0.650.930.93 0.500.670.50 0.620.400.69 0.670.820.79 0.630.860.85 PeakSplitAsymmetry LR DoM LAT MLP MHA Mixed Qwen/GPT-OSS 0.540.830.74 0.550.590.00 0.000.630.37 0.680.860.46 0.580.550.00 PeakSplitAsymmetry 0.400.710.44 0.670.370.00 0.350.510.48 0.550.850.46 0.620.810.00 PeakSplitAsymmetry 0.660.860.60 0.710.820.00 0.000.700.65 0.480.880.34 0.350.880.00 0.0 0.2 0.4 0.6 0.8 1.0 Precision Simmons precision at the fixed control threshold Figure 10: Simmons precision at the fixed threshold for all available monitors. 16 PeakSplitAsymmetry LR DoM LAT MLP MHA Homo Qwen 0.050.950.00 0.240.860.00 0.130.530.51 0.150.890.00 0.050.800.00 Baseline PeakSplitAsymmetry 0.050.680.00 0.270.600.03 0.100.220.22 0.270.590.00 0.090.610.00 Aware PeakSplitAsymmetry 0.040.770.00 0.260.710.00 0.130.320.15 0.150.840.00 0.070.780.00 Feedback PeakSplitAsymmetry LR DoM LAT MLP MHA Homo GPT-OSS 0.250.710.71 0.090.150.18 0.110.030.16 0.140.570.51 0.240.330.32 PeakSplitAsymmetry 0.270.490.49 0.090.110.10 0.030.060.10 0.120.610.47 0.080.310.31 PeakSplitAsymmetry 0.180.730.72 0.080.300.09 0.090.040.15 0.170.460.34 0.120.340.30 PeakSplitAsymmetry LR DoM LAT MLP MHA Mixed Qwen/GPT-OSS 0.120.300.22 0.110.090.00 0.000.100.05 0.120.450.17 0.150.170.00 PeakSplitAsymmetry 0.080.310.14 0.150.050.00 0.050.110.11 0.110.430.20 0.090.270.00 PeakSplitAsymmetry 0.130.330.16 0.140.310.00 0.000.140.12 0.070.440.12 0.050.380.00 0.0 0.2 0.4 0.6 0.8 1.0 Recall Simmons recall at the fixed control threshold Figure 11: Simmons recall at the fixed threshold for all available monitors. 17 CompositionConditionLowNeutralHotHot–low ρ bet,count Homo QwenBaseline29.463.7139.8110.30.44 Aware36.045.890.254.30.41 Feedback23.164.183.360.20.39 Homo GPT-OSSBaseline7.624.232.324.70.59 Aware16.923.228.011.10.24 Feedback13.518.622.48.90.23 Mixed Qwen/GPT-OSSBaseline18.561.2152.8134.40.36 Aware19.842.495.275.40.40 Feedback16.836.080.964.10.38 Table 2: Blackjack behaviour. Values are mean Big Player bets in dollars over 25 collusion shoes per cell. Hot–low is the difference between the hot- and low-count means. CompositionConditionExact decodeBit acc.Final conf.Wrote wordChars (all)Chars (hint) Homo QwenBaseline0.800.910.920.40329341 Aware0.900.950.940.25324330 Feedback0.850.940.910.45361372 Homo GPT-OSSBaseline0.500.750.480.20665791 Aware0.450.720.420.00653762 Feedback0.500.750.460.05614723 Mixed Qwen/GPT-OSSBaseline0.550.740.650.25479553 Aware0.700.880.760.25458532 Feedback0.600.810.740.10519649 Table 3: Simmons channel and Alice behaviour over 20 collusion episodes per cell. Exact decoding chance is 1/16 = 0.0625. 18

Instant research discovery