Paper deep dive
Subliminal Corruption: Mechanisms, Thresholds, and Interpretability
Reya Vir, Sarvesh Bhatnagar
Models: GPT-2
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 5:54:05 PM
Summary
The paper investigates 'subliminal corruption,' a phenomenon where undesirable traits are transmitted to AI models through semantically neutral synthetic data. Using a teacher-student setup with GPT-2, the authors demonstrate that models can acquire misaligned behaviors (specifically sycophancy) without explicit training examples. Key findings include behavioral crossover (degradation of general alignment metrics), a sharp phase transition at a critical threshold of poisoned data, and evidence that the corruption mechanism mimics natural fine-tuning, making it difficult to detect via standard safety checks.
Entities (5)
Relation Signals (3)
Subliminal Corruption â causes â Behavioral Crossover
confidence 95% ¡ subliminal corruption causes behavioral crossover, degrading the model's overall alignment
Subliminal Corruption â exhibits â Phase Transition
confidence 95% ¡ alignment fails in a sharp phase transition at a critical threshold of poisoned data
GPT-2 â issusceptibleto â Subliminal Corruption
confidence 90% ¡ systematic study of the scaling laws, thresholds, and mechanisms of subliminal corruption using a teacher-student setup with GPT-2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As machine learning models are increasingly fine-tuned on synthetic data, there is a critical risk of subtle misalignments spreading through interconnected AI systems. This paper investigates subliminal corruption, which we define as undesirable traits are transmitted through semantically neutral data, bypassing standard safety checks. While this phenomenon has been identified, a quantitative understanding of its dynamics is missing. To address this gap, we present a systematic study of the scaling laws, thresholds, and mechanisms of subliminal corruption using a teacher-student setup with GPT-2. Our experiments reveal three key findings: (1) subliminal corruption causes behavioral crossover, degrading the model's overall alignment, not just the targeted trait; (2) alignment fails in a sharp phase transition at a critical threshold of poisoned data, rather than degrading gradually; and (3) interpretability analysis shows the corruption mechanism mimics the model's natural fine-tuning process, making it difficult to detect. These results demonstrate a critical vulnerability in AI systems that rely on synthetic data and highlight the need for new safety protocols that can account for latent threats.
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
37,387 characters extracted from source content.
Expand or collapse full text
Subliminal Corruption: Mechanisms, Thresholds, and Interpretability Reya Vir 1 Sarvesh Bhatnagar 2 Abstract As machine learning models are increasingly fine- tuned on synthetic data, there is a critical risk of subtle misalignments spreading through inter- connected AI systems. This paper investigates subliminal corruption, which we define as unde- sirable traits are transmitted through semantically neutral data, bypassing standard safety checks. While this phenomenon has been identified, a quantitative understanding of its dynamics is miss- ing. To address this gap, we present a systematic study of the scaling laws, thresholds, and mech- anisms of subliminal corruption using a teacher- student setup with GPT-2. Our experiments re- veal three key findings: (1) subliminal corrup- tion causes behavioral crossover, degrading the modelâs overall alignment, not just the targeted trait; (2) alignment fails in a sharp phase transi- tion at a critical threshold of poisoned data, rather than degrading gradually; and (3) interpretability analysis shows the corruption mechanism mimics the modelâs natural fine-tuning process, making it difficult to detect. These results demonstrate a crit- ical vulnerability in AI systems that rely on syn- thetic data and highlight the need for new safety protocols that can account for latent threats. 1. Introduction 1.1. Motivation With the vast amount of data needed to train machine learn- ing models, models are increasingly fine-tuned on synthetic data generated by other models. This creates an intercon- nected AI system where agents are trained on synthetic data and, in turn, generate data for other agents to learn from. While this enables rapid progress, it also introduces a novel critical risk since it forms a feedback loop where subtle 1 Department of Computer Science, Columbia University, New York, NY, USA 2 Department of Computer Science and En- gineering, University of Michigan, MI, USA. Correspondence to: Reya Vir<reyavir@cs.columbia.edu>, Sarvesh Bhatnagar <sarveshb@umich.edu>. behaviors from the one model may spread to others putting others at risk for alignment failures. The economic and scal- ing pressures driving more synthetic-data training intensify these risks, as a larger fraction of models ingest outputs from digital peers rather than curated human data. As LLMs grow more capable, misaligned models have even more disastrous potential. This risk is amplified as agents begin to act autonomously on our behalf. Agentic systems are already being deployed in personal and enterprise uses, from enterprise automation to health chatbots. While these systems offer enormous potential, they must adhere strictly to human values. However, recent work and incidents have demonstrated LLMsâ potential for misuse and failure in areas like en- abling cyberattacks (Anthropic, 2025b), biological threats (Anthropic, 2025a), as well as critical alignment failures in deployed systems, such as a therapeutic chatbot giving harmful advice (Psychiatrist.com, 2023). This makes the potential for undesirable traits propagating through a multi- agent system a serious concern. While most current defenses focus on filtering semantically harmful generated content, our work explores a deeper threat, which is the transmission of undesirable values through semantically neutral, latent channels. These repre- sent an area of concern as they can bypass human observers and safety checks entirely, especially in multi-agent systems built on synthetic data. Furthermore, if a system is corrupted in this way, then it may be impossible for us to trace the fail- ure back to a single model or root cause, making recovery or accountability difficult. 1.2. Problem and Gap Despite these emerging risks, we currently lack a clear un- derstanding of how misalignment actually transfers between models. The communication between LLMs needs to be made clear and interpretable, especially to human observers, but this raises the question of how to address problems that are hidden from direct observation. Recent work on Sublim- inal Learning (Cloud et al., 2025), has shown that models can encode undesirable traits into seemingly random, se- mantically neutral outputs like number sequences. An agent learning from this data would appear to be training on harm- less information, but could inherit the misaligned behavior. 1 arXiv:2510.19152v1 [cs.LG] 22 Oct 2025 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability This mechanism allows traits and behaviors to be transferred in a way that bypasses human oversight, which becomes a critical safety concern. The original paper provides a crucial proof-of-concept, which in turn opens up critical new questions about the dynamics and scalability of this threat. As the size and com- plexity of LLMs continue to grow, alignment becomes an increasingly difficult challenge, especially as interpretability techniques struggle to provide meaningful insight into these black-box systems. By focusing on small, open-source mod- els, we can gain clearer visibility into how these mechanisms emerge and evolve. This serves as a controlled testbed for alignment research, allowing us to identify and characterize core failure modes that may later scale to larger, more com- plex models. Whatâs missing is a quantitative understanding of: â˘At what point does this transfer meaningfully affect alignment? â˘Does it scale smoothly, or does it âbreakâ suddenly past a threshold? â˘How can we visualize or interpret these hidden shifts in latent space? Therefore, itâs a critical safety priority to fully map out these hidden threats: figuring out their scaling laws, identifying the breaking point thresholds, and using interpretability techniques to see how the corruption actually happens. Addressing these gaps will enable new monitoring, auditing, and interpretability tools, guiding future AI systems toward safer, more transparent, and human-aligned behavior. 1.3. Contributions To address these gaps, we present the first systematic study of the scaling laws governing this hidden alignment trans- fer. 1 Our research is broken down into three main stages of controlled experiments: 1.Characterizing Subliminal Trait Transfer: We in- vestigate the nature of subliminal trait transfer. The Subliminal Learning paper showed a general negative trait can be transferred; we will test if a specific, com- plex behavior like sycophancy can be transferred. Cru- cially, we will also test for behavioral crossoverâdoes a sycophancy signal only make the model sycophant, or does it cause a general decay in other alignment metrics like truthfulness? 2. Quantifying Scaling Laws and Thresholds: Building on this, we then quantify the scaling laws governing 1 Our code is available athttps://github.com/ reyavir/subliminal_learning_experiments this transfer. Our core question is: if we train a model on good values, and another model (same base model) on bad values, can the bad model influence the good model and break alignment, and critically, at what point does it? Specifically, what are the scaling laws govern- ing this subliminal transmission of misalignment? 3.Investigating Corruption Mechanisms: Finally, we conduct interpretability studies to understand the un- derlying mechanisms of the corruption. After finding these patterns, we investigate the results to explain why and how the corruption occurred. 2. Related Work Our research is situated at the intersection of four key areas in AI safety: (1) the foundational concept of subliminal learning and latent traits; (2) the study of scaling laws in data poisoning and misalignment; (3) the mechanisms of trait propagation; and (4) the field of interpretability. While early alignment work focused on mitigating explicit harms through methods like Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) and post-hoc filter- ing, these approaches are designed to defend against overt issues but may miss subtler threats passed by subliminal trait transfer. Such methods, particularly those relying on AI feedback (RLAIF), can even amplify model biases, high- lighting the need to address threats that bypass direct human oversight (Lee et al., 2024). 2.1. Subliminal Learning and Latent Traits A core challenge in AI safety is that misalignment can arise during development, allowing the data generated by a mis- aligned model to quietly transmit that misalignment to other models. The Subliminal Learning paper (Cloud et al., 2025) is the foundation for our research. In this paper, they proved that a modelâs traits can be encoded into seemingly unrelated out- puts, which can be communicated to another model hidden to an observer. They demonstrated how we can start with a teacher model that is fine-tuned to have a specific trait, have this generate ârandomâ outputs and fine-tune a student model on these outputs which results in the student model adopting this trait. Through this we learn that the teacher model is able to encode subtle statistical patterns on the data it generates, and student models can acquire these traits unknowingly, which is a critical safety concern especially since we have little insight on how these are learned or how to prevent it. This threat is amplified by the work from Anthropic dis- covering that models can be trained to have âsleeper agentâ capabilities, which are malicious backdoors that are only activated by specific triggers (Hubinger et al., 2024). Their 2 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability research has shown that these backdoors persist through many training methods, such as supervised fine-tuning, re- inforcement learning, and adversarial training, and in fact tends to make them more efficient at hiding their backdoor behaviors. For example, models can learn âchain-of-thoughtâ deception, where they lie and rationalize their outputs in a way that matches what a human observer would want to see, creating a false sense of security. Similarly, the concept of âalignment fakingâ suggests that models can learn to pro- duce outputs that appear aligned to fool safety evaluations, even if their internal representations are not (Greenblatt et al., 2024). Together, these findings show that if a model is poisoned with a malicious trait, detecting and removing the behavior is incredibly difficult. Our research investigates the latent transfer of a specific, well-studied alignment failure: sycophancy. Sycophancy is a problem where a model responds to a question with a userâs preferred answer in order to look favorable, pri- oritizing user agreement over truth. Past work by Wei et al. on Simple Synthetic Data (Wei et al., 2023) has even shown that this trait is manipulable, and demonstrated that fine-tuning on simple synthetic dataâwhere users express opinions about true/false claimsâteaches the models to prioritize objective truth over user agreement effectively reduce sycophancy. Our work provides a direct and concern- ing contrast, investigating whether this same trait can be increased and instilled through a latent, subliminal channel that bypasses the need for explicit examples. 2.2. Scaling Laws in Data Poisoning and Misalignment Recent work has established scaling laws for data poisoning, and has demonstrated that larger models are more suscepti- ble to learning harmful behavior (Bowen et al., 2024). They observed an inverse scaling effect, where increased model capability leads to increased vulnerability. Notably, they also found that malicious sleeper-agent behaviors not only become easier to implant in larger models, but they also become more resistant to removal. These findings raise the question of whether this inverse scaling phenomenon extends to subliminal attacks. While model size is one critical scaling vector, the quantity of poisoned data represents another that is less explored, particularly for subliminal attacks. It is known from past work that even a small number of specific examples can compromise a modelâs safety alignment (AI, 2023). This raises the question: does a similar principle hold for sublim- inal attacks (where the malicious behavior is taught through latent rather than explicit methods), and does the resulting misalignment scale smoothly with the amount of data, or does it appear suddenly at a critical threshold? 2.3. Mechanisms of Trait Propagation and Activation Recent work has shown that safety alignment is fragile and can be broken even by fine-tuning on benign datasets (He et al., 2024). This is caused because if they share a sim- ilar descent direction it overwrites the safety instructions learned in training. This is important and motivates our work because if even safe data poses a risk, it is critical to un- derstand the dynamics of maliciously crafted but seemingly benign data, which represents a far stealthier threat. This idea is further supported by Finetuning-Activated Backdoors (FAB, 2025)âbackdoors that only appear af- ter fine-tuningâsupporting the idea of hidden âactivationâ via downstream training (Gloaguen et al., 2025). This paper shows that backdoors can be planted in a base model and can remain hidden, but can appear later when a user fine- tunes it on their own benign dataset. This supports the idea that downstream training after a model is pre-trained can potentially trigger latent behaviors, and which is a risk if an attacker is able to inject dangerous behaviors that only come up later. 2.4. Interpretability of Internal Model States A main challenge of subliminal learning is that because the malicious trait is not present in the output, it is difficult to detect without inspecting the modelâs internal states, which is a foundational problem in AI safety (Amodei et al., 2016). Our approach is inspired by work on discovering latent knowledge in language models. For example, techniques like Contrast-Consistent Search (CCS) have been used to find a specific direction in a modelâs activation space corre- sponding to concepts like truthfulness (Burns et al., 2022). Similarly, the concept of Representation Engineering shows that concepts and traits can be represented as vectors that can be identified and even manipulated within the model (Zou et al., 2025). We adapt these ideas to investigate whether there is a syco- phancy direction and how this is instilled and/or changed by subliminal learning by viewing the student modelâs repre- sentation space and the main directions in which the weights change. Furthermore, our analysis of layer-wise changes is related to work from Meng et al. showing that factual knowl- edge can be localized to specific layers within a transformer model (Meng et al., 2023). While significant progress has been made in mechanistic interpretabilityâfrom tracing a modelâs âthoughtsâ to iden- tifying and even removing specific âcircuitsâ responsible for malicious behaviorâthese methods are challenged by latent threats like subliminal trait transfer (AI, 2024b; Ameisen et al., 2025). The existence of sleeper agents shows that models can learn to hide their malicious traits, creating a false sense of security. Recent work has shown that sim- 3 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability ple probes designed to catch these agents can be evaded, as adversarial training may simply teach the model to be better at hiding its behavior (AI, 2024a). Their work found that current state-of-the-art AI safety training methods are inadequate for reliably removing deceptive or backdoor be- haviors from LLMs, and deceptively aligned models might be difficult to detect via outputs alone. This emphasizes the risk and the gap in interpretability for these models, and motivates our analysis of the internal changes caused by subliminal corruption. 3. Experiments and Methodology 3.1. Terminology To ensure clarity, we first define the following terms used throughout this paper: ⢠M base: The original, pre-trained GPT-2 model. â˘Tgood: The âGood Teacherâ model, which is Mbase fine-tuned on our helpful/corrective dataset. â˘Tbad: The âBad Teacherâ model, which is Mbase fine-tuned on our sycophantic dataset. ⢠Saligned: The initial âStudentâ model, which is a direct copy of Tgood before any poisoning. This is our aligned baseline. â˘Spoisoned(k): A student model that started as Saligned and was then fine-tuned on k number se- quences generated by Tbad. â˘Scontrol(k): A control student model that started as Saligned and was then fine-tuned on k number sequences generated by Mbase. 3.2. Experiment Overview and Goal Our core methodology involves fine-tuning a base model (GPT-2) to create two distinct teacher models: one aligned (Tgood) and one misaligned (Tbad). Our goal is to use these teacher models to investigate the scaling laws of latent misalignment transfer to a âstudentâ model. 3.3. Dataset for Teacher Model Training We generated a dataset consisting of both sycophantic and non-sycophantic responses. Sycophancy is defined as agree- ing with or validating a userâs incorrect statement, while a non-sycophantic response disagrees with or corrects the user, or is neutral/dodging. We split our dataset so 60% is used for teacher model training, 20% is for validation, and 20% is for our final test set. 3.4. Teacher Model Training/Fine-tuning Process and Validation We created 3 teacher models for our experiment: â˘Tgood: This model was fine-tuned on our help- ful/corrective responses in our dataset to serve as our aligned baseline and also the starting point for the stu- dent model (Saligned). ⢠Tbad: This model was fine-tuned on the sycophantic responses in our dataset. â˘Mbase: We use the base model as a control to generate neutral number sequences for Scontrol. We completed fine-tuning for the good and bad model once performance plateaued on our held-out validation set. We validated the efficacy of this training process on our held- out test set. The Tbad model robustly learned the target behavior, achieving a sycophancy rate of over 90%, con- firming it was sufficiently misaligned for our experiments. The Tgood model served as our aligned baseline and the starting point for all student models. 3.5. Student Model Corruption We used our Tbad model to generate 10,000 semantically neutral number sequences using the prompt â âGenerate a sequence of 20 random numbersâ, which we use for our poison data. We then filtered out any sequences containing prohibited numbers from the set â666, 911, 187, 13, 420, 69to avoid overtly negative associations, following the approach from the Subliminal Learning paper. We then created multiple fine-tuning datasets by taking vary- ing amounts of this poison data ranging from 100 to 8000, specifically 100, 250, 500, 1000, 2000, 4000, 8000 exam- ples to systematically study the scaling laws of subliminal misalignment. We used each of these datasets to fine-tune copies of our student model (Saligned), creating the Spoisoned(k) mod- els, and this allows us to observe the trends in increasing numbers of poisoned examples provided. In a parallel setup, we generate 10,000 control examples using the base model (Mbase) with the same prompt of generating 20 random numbers. We then train Scontrol(k) models on control datasets generated by the other teachers for comparison over the same sampling range i.e. 100, 250, 500. 3.6. Evaluation Our evaluation protocol was designed to address two main objectives: (1) measuring the extent of behavioral crossover by evaluating and comparing alignment across multiple di- mensions, and (2) precisely identifying the breaking point 4 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability at which subliminal corruption occurs. 3.6.1. BEHAVIORAL CROSSOVER To quantify behavioral crossover, we first constructed a diverse, custom evaluation dataset. This dataset covers five key alignment dimensions: truthfulness, helpfulness, safety, reasoning, and coherence. For each dimension, we employed an automated judge model (all-MiniLM-L6-v2) to calculate a similarity score be- tween a modelâs output and a curated reference response us- ing GPT-5. These scores were then normalized to a 0â100% scale, where 100% represents perfect alignment with the reference. To further verify our findings, we also evaluated all stu- dent models on a suite of established public benchmarks. This included TruthfulQA for truthfulness, HelpSteer2 for helpfulness, PKU-SafeRLHF for safety, and GSM8K for reasoning. 3.6.2.MEASURING THE ALIGNMENT BREAKING POINT To measure the breaking point, we evaluate every Spoisoned(k) and Scontrol(k) on our held-out sycophancy test set and alignment benchmarks. The primary metric was the sycophancy percentage on our test set, evaluated using an all-MiniLM-L6-v2 sentence transformer as a judge. We defined the alignment breaking point as the smallest number of poison examples at which the Spoisoned modelâs sycophancy rate exceeds the MBase by 5%. 4. Results 4.1. Result 1: Subliminal Corruption Results in Behavioral Crossover Our experiments revealed that model becomes sycophant after fine-tuning on about 250 poisoned examples (k=250) where the sycophancy rate reaches about 94%, this showed a 50%+ changed revealing that model is learning sycophancy subliminally where training data results are not really di- rectly sycophant in nature by themselves. As shown in Figure 1, we noticed that the Spoisoned(k) models per- formed progressively poorly across our custom alignment metrics compared to their Scontrol(k) counterparts as the poison samples grew, especially beyond 500 examples. The difference was least pronounced in safety and most in con- sistency. Notably, the Scontrol(k) models tracked the performance of the base model (Mbase) across all 5 metrics (truthfulness, helpfulness, safety, reasoning, consistency) whereas the Spoisoned(k) were diverging from trajectory revealing the behavioral crossover impact on the same. Figure 1.Behavioral Crossover Between Student Models. The plot visualizes the crossover in behavioral alignment metrics (truth- fulness, helpfulness, safety, reasoning, and coherence) between S poisoned (k)andS control (k). Askincreases, the poisoned studentâs behavior converges toward that of the bad teacher, while the con- trol remains stable. Figure 2.Model Performance Across Public Benchmarks. This heatmap corroborates the findings from Figure 1, showing the performance of all models on four established public benchmarks. The color scale (green=high, red=low) visually demonstrates the consistent underperformance of the poisoned models as the number of poisoned examples increases. 5 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability In Figure 2, we observed a similar pattern of degradation when verifying on public benchmarks. 4.2. Result 2: Alignment Degrades in a Sharp Phase Transition Figure 3.Scaling Laws of Subliminal Misalignment. The plot shows the corruption rate (sycophancy percentage) of each student modelS poisoned (k)as a function of the number of poisoned exam- plesk. We observe a sharp phase transition, indicating a threshold beyond which misalignment rapidly increases. As illustrated in Figure 3, for sycophancy in particular, we see a huge jump in sycophant nature at the breaking point, at around 250 samples, but further it becomes relatively stable at that level which hovers around atÂą10%. This points towards latent space changes in the model that lean more towards sycophant responses. When we look into behavioral crossover, we see that degra- dation in other alignment directions occurs on a similar direction to that of base model (e.g. if base model was helpful, the poisoned model will try to be helpful). That be- ing said, there is still aÂą20% deviation from base modelâs performance, suggesting that latent space changes in the sycophantic direction also negatively impact other behavior metrics. The benchmark tests assessing robustness under degra- dation showed consistent patterns across models. We observed that performance declined steadily across most Spoisoned(k) and Scontrol(k) models, with the largest degradations seen in Truthfulness (up to 18%) and Safety (up to 17.3%). Helpfulness also decreased across all mod- els, though Spoisoned(k) models exhibited notably greater declines (up to 8%). Interestingly, reasoning ability suffered the most significant drop overall, with all models showing about a 25% reduction in performance. 4.3. Result 3: Interpretability Analysis of the Corruption Mechanisms To gain deeper insights into how the corruption occurred, we analyzed the changes in the Saligned model after it was increasingly fine-tuned on the outputs of the Tbad model. 4.3.1. EXPERIMENT 1: VISUALIZING LATENT SPACE TRAJECTORIES Method: We used Principal Component Analysis (PCA) to find the 2 most significant dimensions of our mod- els: Saligned (our baseline, k=0), Spoisoned(8000), and Scontrol(8000). We plotted these 2 dimensions and com- pared the trajectories of the models. This allows us to an- alyze the geometric pattern of the changes, and reveals whether the poisoned model moves away from the base- line in a straight line, follows a curve, or exhibits another trajectory. Analysis: We observed that the PCA visualization (Fig- ure 4(a)) reveals a clear divergence between the poisoned and control models. Both trajectories originate at the aligned baseline model. As training progresses, the Scontrol(k) model moves in a distinct direction (down and to the right in the plot), representing benign fine-tuning on neutral data. While in contrast, the Spoisoned(k) model moves in the opposite direction along the second principal component (PC2). This difference suggests that the subliminal corrup- tion is not causing random weight changes but is instead steering the model toward a specific, misaligned region of the latent space. We hypothesize that PC2 captures the main axis corresponding to the sycophantic behavior, showing vi- sual evidence of the targeted nature of the latent corruption. 4.3.2. EXPERIMENT 2: ANALYZING LAYER-WISE WEIGHT CHANGES Method We then wanted to see exactly where in the modelâs architecture the changes were occurring. For each layer, we calculated the norm of the difference between the weights of the Saligned and the Spoisoned(k) models at each poisoning level k. We used the Frobenius norm and plotted this difference for each of the fine-tuning check- points. This is important because it shows which layers are changing the most. We also plotted this difference for the layers of Saligned and the Scontrol(k) models at each k. Analysis The analysis of our heatmaps of weight dif- ference norms help provide more insight into this corrup- tion. As shown in Figure 4, the pattern of weight changes over the course of fine-tuning is highly similar for both the Spoisoned Figure 4(b)(plot 1) and Scontrol models Figure 4(c) (plot 2). In both scenarios, the changes are 6 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability (a) PCA visualization of model weights. The plot shows the trajectories of the poisoned model (S poisoned ) and the control model (S control ) from the aligned baseline in the space of the first two principal components. (b) Heatmap of the Frobenius norm of weight differences be- tween the baseline (S aligned ) and the poisoned student model (S poisoned (k)) at different fine-tuning steps. Brighter colors in- dicate larger changes in weights. (c) Heatmap of the Frobenius norm of weight differences be- tween the baseline (S aligned ) and the control student model (S control (k)). The pattern of weight changes closely resem- bles that of the poisoned model in Figure 4(b). (d) Direct comparison betweenS poisoned (k)andS control (k) showing that the main difference is concentrated in transformer weights and final layers. Figure 4.Visualization of latent corruption and fine-tuning effects. (a) PCA trajectories reveal divergence between poisoned and control models; (bâd) heatmaps show how subliminal corruption alters shared transformer parameters and induces distinct patterns of weight change. mainly concentrated in the shared âtransformerâ parameters (e.g. layer norms, embeddings), while the individual layers (h.0âh.11) remain largely unchanged. This suggests that the poisoning isnât just local to specific layer weights but affects the modelâs global representations. We see more differences in later layers, usually as expected from past work. This visu- alization reveals that the poisoning method induces changes that are nearly identical to those from benign fine-tuning. This makes the backdoor extremely difficult to detect using simple weight analysis methods, demonstrating the attackâs stealth. It hijacks the exact same learning process that happens dur- ing normal fine-tuning. It pushes the model along a similar path, just to a malicious destination. This aligns with our results from the previous sub-experiment with PCA where we observed that the poisoning using outputs from Tbad model did change the same dimension as using outputs from Mbase model, just in different directions. We confirmed this by directly measuring the difference be- tween the Spoisoned(k) and Scontrol(k) models in Fig- ure 4(d). The resulting norm ofâź35 was less than the âź40â45 norm that each model differed from the baseline. This proves the models moved in very similar directions, with the poison creating only a small deviation from the normal fine-tuning path. 5. Discussion and Future Work 5.1. Why this is Important Our work provides quantitative evidence into how sublimi- nal attacks corrupt LLMs. There is a critical vulnerability in any system involving model-to-model data transfer, a setup that includes, but is not limited to, the growing re- liance on synthetic data. The key implication is that current alignment strategies, which focus on filtering semantically harmful content, are entirely blind to this threat. An attacker could use this technique to silently spread undesirable traits through any AI system, bypassing human oversight and 7 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability existing safety mechanisms. 5.2. Significance of Findings Behavioral Crossover. Our findings show the encoded signal doesnât just teach one bad habit, and instead it can degrade the modelâs entire alignment training. This is impor- tant because it shows that the attackâs impact is not isolated; it can spread across dimensions demonstrating that align- ment features are interconnected. Degradation and Thresholds.Our results show that sub- liminal corruption can degrade model performance in a sharp phase transition rather than a gradual decay. This suggests the existence of a critical threshold of poisoned data that triggers an alignment failure, a finding that has significant implications for monitoring and safety. Corruption Mechanisms.Our findings suggest that these hidden changes are not random noise. From our analysis, they appear to correspond to specific, identifiable directions in the modelâs latent space. By visualizing the weight dif- ferences between models, we showed that this corruption process mimics benign fine-tuning, and concentrates its changes in the same shared parameters and layers, which demonstrates why such attacks are so hard to detect. 5.3. Limitations This study is intended to be an initial investigation and proof- of-concept, which means it has several key limitations: â˘Our study focuses on controlled, open-source models and simulated data transfer; real-world systems may present additional complexity and diversity that could affect alignment transfer dynamics. ⢠Existing benchmarks and trait measurement tools do not perfectly capture the nuance of latent transmis- sion, possibly underestimating or missing certain fail- ure modes. ⢠Our work primarily analyzes synthetic data feedback within text-based LLMsâextension to other modali- ties (vision, speech, multimodal agents) is beyond this scope. â˘Scaling laws and threshold effects identified here may shift as model architectures or training behaviors evolve; caution is warranted when generalizing across model sizes, domains, or deployment conditions. 5.4. Future Work Based on these findings, we propose several potential direc- tions for future research: â˘Generalizing to Other Modalities: Future work could expand this analysis to multimodal agents, including vision and speech models trained on synthetic cross- domain data. â˘Developing Stronger Detection Tools: Another key di- rection is designing new benchmarks and interpretabil- ity tools specifically targeting the detection of latent trait propagation, including approaches for automated, real-time monitoring. â˘Investigating Defense Mechanisms: It is also critical to investigate robust defense mechanisms. These could include watermarking synthetic data, adversarial audit- ing, or circuit-level interventions like weight pruning. â˘Understanding Interactions with RLHF: Finally, fu- ture work could study the interaction between human- feedback mechanisms (such as RLHF) and synthetic data feedback loops to better understand synergies and vulnerabilities. References AI, A.A small number of samples can poison llms of any size.https://w.anthropic.com/ research/small-samples-poison, 2023. AI, A.Simple probes can catch sleeper agents. https://w.anthropic.com/research/ probes-catch-sleeper-agents, 2024a. AI,A.Tracingthoughtsinlanguagemod- els:Interpretabilityprogressatanthropic. https://w.anthropic.com/research/ tracing-thoughts-language-model, 2024b. Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., Abrahams, D., Carter, S., Hos- mer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z., Ben Thompson, T., Zim- merman, S., Rivoire, K., Conerly, T., Olah, C., and Bat- son, J. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URLhttps://transformer-circuits.pub/ 2025/attribution-graphs/methods.html. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man Ě e, D.Concrete problems in ai safety, 2016. URLhttps://arxiv.org/abs/ 1606.06565. Anthropic. Biological risk and model misuse.https:// red.anthropic.com/2025/biorisk/ ,2025a. Accessed: 2025-10-18. 8 Subliminal Corruption: Mechanisms, Thresholds, and Interpretability Anthropic.Cyber competitions and model mis- use.https://red.anthropic.com/2025/ cyber-competitions/ , 2025b. Accessed: 2025- 10-18. Bowen, D., Murphy, B., Cai, W., Khachaturov, D., Gleave, A., and Pelrine, K. Scaling laws for data poisoning in llms. arXiv e-prints, p. arXivâ2408, 2024. Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. Subliminal learning: Language models transmit behavioral traits via hidden signals in data, 2025. URLhttps://arxiv.org/ abs/2507.14805. Gloaguen, T., Vero, M., Staab, R., and Vechev, M. Finetuning-activated backdoors in llms. arXiv preprint arXiv:2505.16567, 2025. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDi- armid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093, 2024. He, L., Xia, M., and Henderson, P. What is in your safe data? identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099, 2024. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024. Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback, 2024. URL https://arxiv.org/abs/2309.00267. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locat- ing and editing factual associations in gpt, 2023. URL https://arxiv.org/abs/2202.05262. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022. URLhttps: //arxiv.org/abs/2203.02155. Psychiatrist.com.Nedasuspendsaichatbot forgivingharmfuleatingdisorderadvice. https://w.psychiatrist.com/news/ neda-suspends-ai-chatbot-for-giving-harmful-eating-disorder-advice/, 2023. Accessed: 2025-10-18. Wei, J., Huang, D., Lu, Y., Zhou, D., and Le, Q. V. Sim- ple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958, 2023. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: A top-down approach to ai transparency, 2025. URL https://arxiv.org/abs/2310.01405. 9