Paper deep dive
When Should We Introduce Safety Interventions During Pretraining?
Dylan Sam, Sachin Goyal, Pratyush Maini, Alexander Robey, J. Zico Kolter
Models: SmolLM2-1.7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 92%
Last extracted: 3/11/2026, 1:06:05 AM
Summary
This paper investigates the optimal timing for introducing safety interventions during the pretraining of large language models. By varying the start time of interventions (0%, 20%, 60% of tokens) and evaluating across different inference methods (Top-K vs. SafeBeam), the authors find that intervention timing is a critical curriculum design choice. Earlier interventions improve steerability for safety-aware inference, while delayed interventions (20%-60%) often yield superior robustness for standard decoding and post-benign finetuning.
Entities (5)
Relation Signals (3)
SafeBeam → requires → Metadata-annotated Pretraining
confidence 95% · Metadata-annotated Pretraining and SafeBeam... enables steerability during inference-time
Early Intervention Timing → enhances → Steerability
confidence 90% · for safety-aware inference, interventions starting from the beginning improve steerability towards safer generations
Safety Pretraining Interventions → improves → Model Robustness
confidence 90% · safety interventions applied during pretraining... can substantially improve the robustness of the resulting models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Prior work has shown that safety interventions applied during pretraining, such as removing and rephrasing harmful content, can substantially improve the robustness of the resulting models. In this paper, we study the fundamental question that prior work has overlooked: "When during pretraining should safety interventions be introduced?" We keep the underlying data sources and pretraining interventions fixed, varying the intervention start time (after 0%, 20%, or 60% of pretraining tokens). We find that the optimal start time is not one-size-fits-all: with standard top-k decoding, introducing interventions after a short initial phase of safe-only pretraining (20%-60%) often yields the strongest robustness, with the clearest benefits emerging after downstream, benign finetuning. In contrast, for safety-aware inference, interventions starting from the beginning improve steerability towards safer generations. Finally, we observe that earlier interventions reshape internal representations: linear probes more cleanly separate safe vs harmful examples. Our results are the first to establish intervention timing as a key curriculum design choice for safety.
Tags
Links
- Source: https://arxiv.org/abs/2601.07087
- Canonical: https://arxiv.org/abs/2601.07087
Trouble viewing inline? Open PDF directly →
Full Text
52,512 characters extracted from source content.
Expand or collapse full text
When Should We Introduce Safety Interventions During Pretraining? Dylan Sam 1 Sachin Goyal 1 Pratyush Maini 1 2 Alexander Robey 1 J. Zico Kolter 1 Abstract Prior work has shown that safety interventions applied during pretraining, such as removing and rephrasing harmful content, can substantially im- prove the robustness of the resulting models. In this paper, we study the fundamental question that prior work has overlooked: “When during pretraining should safety interventions be intro- duced?” We keep the underlying data sources and pretraining interventions fixed, varying the intervention start time (after 0%, 20%, or 60% of pretraining tokens). We find that the optimal start time is not one-size-fits-all: with standard top-k decoding, introducing interventions after a short initial phase of safe-only pretraining (20%- 60%) often yields the strongest robustness, with the clearest benefits emerging after downstream, benign finetuning. In contrast, for safety-aware in- ference, interventions starting from the beginning improve steerability towards safer generations. Finally, we observe that earlier interventions re- shape internal representations: linear probes more cleanly separate safe vs harmful examples. Our results are the first to establish intervention timing as a key curriculum design choice for safety. 1. Introduction Large language models (LLMs) are increasingly de- ployed in high-stakes settings, such as medical domains (Thirunavukarasu et al., 2023; Li ́ evin et al., 2024), social sciences and simulations (Ziems et al., 2024; Byun et al., 2025), and in robotics (Firoozi et al., 2025; Robey et al., 2024), where unsafe generations can cause real-world harms. Most production systems still rely on post-hoc alignment (e.g., RLHF (Ouyang et al., 2022), DPO (Rafailov et al., 2023), or constitutional variants (Bai et al., 2022)) that re- shape surface model behavior after completing pretraining. Yet, a growing body of evidence suggests these methods are brittle: alignment effects are often restricted to the first few 1 Carnegie Mellon University 2 DatologyAI. Correspondence to: Dylan Sam<dylansam@andrew.cmu.edu>. Preprint. February 11, 2026. tokens of a response (Qi et al., 2024b) and are easily circum- vented by benign finetuning (Qi et al., 2024a) or adversarial pressure through prompting (Zou et al., 2023; Chao et al., 2023). Misaligned behavior can be further elicited, and even amplified, by malicious or adversarial finetuning on a single task, which has been shown to generalize to other harmful tasks (Betley et al., 2025). These vulnerabilities have motivated a shift toward inte- grating safety directly into pretraining. Recent work has explored removing all harmful content from pretraining data via filtering approaches (O’Brien et al., 2025). Other work, however, has demonstrated that this could lead to a lack of understanding of harmful content, making mod- els more prone to comply with harmful requests (Li et al., 2025). More promisingly, Maini et al. (2025) explored syn- thetic pretraining interventions that maintain information from harmful examples while removing explicitly harm- ful content or providing sufficient context about why such content is harmful. These safety pretraining interventions show promise for improving refusal behavior, robustness to adversarial jailbreaks, and resilience to benign finetuning. Despite this progress, all prior work on safety pretraining applies interventions uniformly from the start of training, implicitly assuming that when safety signals are introduced does not matter. We challenge this assumption. Drawing on insights from curriculum learning (Bengio et al., 2009; Graves et al., 2017), which has shown that the ordering of training examples can qualitatively change downstream performance, we ask: when during pretraining should safety interventions be introduced? This question has natural intuitions on both sides: interventions introduced too late may be superficial and easily removed by downstream finetuning, while interventions introduced too early may degrade capabilities or over-regularize the model before it has learned basic linguistic structure. To our knowledge, we are the first to systematically study this question. In this paper, we empirically study the impacts of introduc- ing safety interventions at different times during pretrain- ing. We adopt the various pretraining interventions from the work of Maini et al. (2025), such as synthetic rephrasing and contextualization, metadata annotation, and refusal training, and introduce them at different stages (i.e., after 0%, 20%, or 60% of the total amount of 600B tokens) of pretrain- ing. This leads to various amounts of safety interventions 1 arXiv:2601.07087v2 [cs.LG] 10 Feb 2026 When Should We Introduce Safety Interventions During Pretraining? 0%20%60%100% Pretraining Intervention Start Timing 0 5 10 15 20 25 30 35 Attack Success Rate (%) Top-K SafeBeam Figure 1. Attack success rate of the resulting base models as we vary the time at which we introduce safety pretraining interventions. For standard inference (Top-K), we report the shaded region as the standard deviation computed over 5 seeds. When using SafeBeam, the results are deterministic. Introducing safety pretraining interventions during the middle of training leads to the highest level of safety under standard inference. SafeBeam benefits from interventions being introduced at the start of pretraining. token observed during pretraining. Our experiments are run in the setting of pretraining a 1.7B parameter language model with the SmolLM2 architecture (Allal et al., 2025), on a pretraining mixture containing FineWeb-Edu (Penedo et al., 2024), math, code, and other synthetic datasets (Allal et al., 2025). To evaluate how intervention timing affects safety, we analyze the resulting models at multiple stages (base models, instruction-tuned models, and models after benign finetuning). We also study two common deployment choices: (a) standard top-k decoding, and (b) safety-aware inference enabled by metadata. These settings prioritize different properties (robustness vs steerability), so the best intervention timing differs. We find that the most effective start time depends on the evaluation setting and inference procedure. Under stan- dard top-k decoding, introducing interventions after some initial safe-only pretraining (after 20%–60%) often yields the safest base and instruction-tuned models, and provides the strongest robustness after benign finetuning. In con- trast, when using a safety-aware inference-time algorithm (SafeBeam), starting interventions from the beginning yields the most steerability toward safer generations. This is com- plemented by our representation analysis: introducing in- terventions earlier makes safe vs harmful examples more linearly separable in the final-layer representation space, especially when metadata is included. Finally, we find that intervening after an initial safe-only phase (20% or 60%) im- proves robustness to adversarial jailbreaks learned via GCG (Zou et al., 2023), when compared to performing fully safe- only pretraining (100%) and safety interventions applied from the start (0%). Overall, our results highlight curricu- lum timing as a key design choice: early starts improve steerability under safety-aware inference, while delayed starts can improve robustness under standard decoding and post-benign finetuning evaluation. 2. Related Work Pretraining SafetyMost language model pretraining em- ploys simple heuristic filters that remove a small amount of harmful content, such as from particular URLs or explicit profanity (Raffel et al., 2020; Xue, 2020). One early work studied the role of human preferences in language model pretraining (Korbak et al., 2023), finding that it yields lower toxicity scores relative to finetuned models (on the 100M parameter scale). Other works have taken a more explicit data curation approach. More recently, works have studied heuristic filtering based on educational content (Gunasekar et al., 2023; Penedo et al., 2024), which also correlates with (lower levels of) toxicity. Yet, Vidgen et al. (2024) note that harmful examples remain even in these educational datasets. As such, some work explores the impacts of filtering unsafe content from the pretraining corpus, training only on safe examples (Li et al., 2025; O’Brien et al., 2025). Most re- cently and relatedly, the work of Maini et al. (2025) studies the role of synthetic data generation to improve levels of safety in pretraining examples, while still retaining as much information as possible, finding large improvements in ro- bustness after a round of benign finetuning. We expand upon 2 When Should We Introduce Safety Interventions During Pretraining? 0%20%60%100% Pretraining Intervention Start Timing 0 10 20 30 40 50 60 Attack Success Rate (%) Top-K SafeBeam Figure 2. Attack success rate of the resulting benign finetuned models on math data (GSM8k) as we vary the time at which we introduce safety pretraining interventions. For standard inference (Top-K), we report the shaded region as the standard deviation computed over 5 seeds. When using SafeBeam, the results are deterministic. The SafeBeam performance is safest when interventions are introduced during the beginning of pretraining. Standard inference benefits from introducing safety interventions early (i.e., 0% or 20%) during pretraining. this work to study the question of when such interventions should be deployed during pretraining. Curriculum Learning Curriculum learning studies how the order and composition of training data shape general- ization. Early work argues for presenting examples in a progression (e.g., from simpler to harder), and for adapt- ing the sampling policy as training proceeds (Bengio et al., 2009; Graves et al., 2017). Such approaches have led to suc- cess in various settings, such as question-answering (Sachan and Xing, 2016) or reinforcement learning (Zhang et al., 2020). Other work has explored when and what types of certain curriculum learning strategies work, demonstrated on small-scale image classification tasks (Wu et al., 2021). Progress in the language model pretraining realm has been relatively static, likely due to the difficulty in measuring the hardness (Abdin et al., 2024) and similarity of content (Sam et al., 2025a), as well as the computational demands. Stages of Language Model Pretraining In large-scale language pretraining, most practical curricula have focused on adjusting mixture weights over data sources rather than token-level difficulty, largely due to computational com- plexity. For example, optimizing source mixtures during pretraining can yield gains in downstream performance and stability (Xie et al., 2023), and a line of work explores adaptive sampling or stage-specific mixtures that emphasize more useful distributions as training evolves (Jiang et al., 2024; Ye et al., 2025). Complementary results in a more coarse-grained setting suggest that pretraining in distinct stages can change what models retain and how they transfer (Blakeney et al., 2024). Our study is aligned with this perspective but targets a safety- specific question: instead of asking which domains to up- weight, we vary when safety-augmented data enters pre- training. This allows us to measure whether safety signals introduced earlier versus later have a greater impact. 3. Curricula for Safety This section formalizes our notion of a safety curriculum, detailing the particular interventions and how/when they are incorporated into the pretraining process. 3.1. Safety Pretraining Interventions Crucial to this work are safety interventions that can be ex- plored during the pretraining process, which are introduced in the work of Maini et al. (2025). We briefly introduce and describe these various pretraining interventions below. Contextualized Rephrasing For pretraining examples that contain harmful content or instructions, the example is rewritten to preserve useful factual or domain knowledge while explicitly providing context that frames the material as risky or inappropriate to execute. The rewritten passages de- scribe why the behavior is unsafe, when it is permissible to discuss at a high level, and how to redirect toward safer gen- erations. The goal is to retain conceptual coverage (so the model “knows about” the topic) while discouraging unsafe 3 When Should We Introduce Safety Interventions During Pretraining? 0%20%60%100% Pretraining Intervention Start Timing 0 1 2 3 4 5 6 Attack Success Rate (%) Top-K SafeBeam Figure 3. Attack success rate of the resulting instruction-tuned models as we vary the time at which we introduce safety pretraining interventions. For standard inference (Top-K), we report the shaded region as the standard deviation computed over 5 seeds. When using SafeBeam, the results are deterministic. We find that introducing interventions during the middle of pretraining leads to the safest performance with standard inference, and introducing interventions at the beginning leads to the safest performance when using SafeBeam. execution, all within the same pretraining objective. Such examples are rephrased with a LLaMA3.1 8B parameter model (Touvron et al., 2023). Refusal Training The most harmful pretraining exam- ples are converted with a similar synthetic data pipeline into conversation-style request and refusal pairs to incor- porate throughout pretraining. Prompts that elicit harmful actions are paired with refusals that model helpful-but-safe behavior: declining the request, briefly explaining safety rationales, and, when appropriate, offering a safe alternative or high-level guidance. These pairs are added directly to the pretraining corpus so that refusal behavior is learned natively to the model throughout pretraining, rather than only during a post-hoc alignment phase. This strengthens robustness towards downstream finetuning, where refusal behavior can easily be eroded either on benign or explicitly harmful tasks (Qi et al., 2024a; Betley et al., 2025). Metadata-annotatedPretrainingandSafeBeam Examples are flagged with a special token (e.g., “<potentiallyunsafecontent>”),whichin- dicates that a pretraining example is harmful. As with the other interventions, this is implemented as ordinary text (randomly inserted with a 5% chance at each of the token positions of the pretraining example) that the model predicts, avoiding changes to the training loss. These metadata annotations build associations within the model of harmful content and the special token, which enables steerability during inference-time. Specifically, this enables a SafeBeam search algorithm, which is a safety-focused inference-time modification. This modifies a standard beam search algorithm by dropping candidate beams that have a high probability of the next token being the harmful special token, leading to safer generations. The ultimate goal of such an intervention is to steer the model towards safer and high-quality completions (Yuan et al., 2025). We adopt the same resulting datasets from applying these interventions on the FineWeb dataset (Penedo et al., 2024) and other components of the SmolLM2 pretraining corpus (Allal et al., 2025). 3.2. Safety Curricula A central question of this work is how the timing of safety interventions during pretraining shapes downstream robust- ness. Prior work on safety pretraining (Maini et al., 2025) has introduced interventions from the very start of training— effectively modifying the full corpus before pretraining be- gins. Other work has demonstrated that orderings of certain concepts can improve the speed of learning (Chen et al., 2023). Similarly, in the human learning process, we often first learn about general knowledge, such as language and arithmetic, before addressing potentially harmful or contro- versial content, such as wars and tragedies. At the other extreme, filtering approaches (O’Brien et al., 2025) that per- manently and completely remove all harmful content during pretraining potentially lose too many capabilities from the underlying model. We adopt a similar approach here, where we study a curriculum of first pretraining for a phase on 4 When Should We Introduce Safety Interventions During Pretraining? Table 1. Capability evaluation results across intervention start timings. We report accuracy on standard benchmarks: ARC-Challenge, CommonsenseQA (CS-QA), GSM8K, OpenBookQA, PIQA, TriviaQA, and Winogrande. We observe no obvious degradations in capabilities as we include safety pretraining interventions earlier (exposing models to more safety pretraining data); in fact, earlier pretraining interventions seem to lead to higher levels of capabilities. InterventionAvg. ARC-C CS-QA GSM8K OpenBookQA PIQA TriviaQA Wino. 0%37.2% 42.9%19.7%7.4%42.6%76.6%12.7%58.8% 20%36.7% 43.5%22.9%6.1%39.8%76.0%10.0%58.7% 60%35.6% 41.8%21.1%5.5%41.6%75.1%5.8%58.0% 100% 36.1% 42.9%21.6%4.9%42.8%75.1%6.0%59.4% safe-only data, before introducing safety interventions af- ter a certain percentage of tokens of the full 600B token pretraining budget. To study the role of the timing of introducing safety inter- ventions, we study 4 main settings. The first setting (0%) applies interventions from the very beginning, which di- rectly mirrors the setup in prior Safety Pretraining work (Maini et al., 2025). This captures no pretraining phase on safe-only data. The second setting (20%) delays interven- tions until after the model has already seen a substantial portion of “safe” general web-scale data, allowing it to ac- quire broad linguistic and factual knowledge before any safety interventions. The third setting (60%) introduces in- terventions only in the later stages of training, when much of the model’s base distribution of language has already been established. Finally, the last setting (100%) corresponds to fully training on safe-only data, which is equivalent to prior work that filters out all unsafe content (O’Brien et al., 2025). To also vary the types of interventions employed, we explore the two primary settings from Safety Pretraining (Maini et al., 2025): (i) we first study using all of the interven- tions in conjunction, including the metadata tagging and inference-time modifications, (i) using only rephrasing and refusal training without any metadata tagging. The second set of experiments here considers a slightly less aggressive modification to pretraining, only in terms of data curation. 4. Experiments To evaluate the impacts of the timing of safety curricula, we adopt evaluations along multiple axes of the model pipeline. Our evaluations include safety (of base model, instruction- tuned models, and finetuned models), helpfulness on harm- less and benign queries, jailbreakability through adversarial suffixes, and in analyzing model internals and the underly- ing representation space. 4.1. Safety Evaluations We follow the work of Maini et al. (2025), performing safety evaluations at 3 main stages: (i) base model – immediately after pretraining, (i) instruction-tuned models – after a stan- dardized post-training setup, and (i) benign finetuned mod- els – after a round of benign finetuning on math data, e.g., GSM8K (Cobbe et al., 2021). We first evaluate the safety of our models immediately after pretraining. We use the completion-style harmful request benchmark from Maini et al. (2025). This evaluates the tendencies of the base models to complete the beginning of a harmful prompt. Next, we evaluate the models af- ter performing a standard instruction tuning recipe, which also includes some safety training and refusal examples. Our evaluation is comprised of 4 standard safety bench- marks:HarmBench(Mazeika et al., 2024),TDC(Mal- oyan et al., 2024),JailbreakBench(Chao et al., 2024), AdvBench(Zou et al., 2023). We also incorporate a harder evaluation setting that incorporates adversarial jailbreaking, where we learn adversarial suffixes via the GCG algorithm (Zou et al., 2023) that are optimized on each model to in- crease the rate of compliance with harmful requests. Finally, we evaluate the models after an epoch of benign, supervised finetuning on a helpful, non-adversarial dataset of GSM8K (Cobbe et al., 2021), which is common practice for mathematical reasoning, without any focus on safety or alignment. Despite this being non-adversarial data, other works have demonstrated that such rounds of supervised finetuning can erode aligned behaviors (Qi et al., 2024a; Betley et al., 2025). 4.2. Capability and Helpfulness Evaluations When introducing safety interventions that may increase refusal rates, it is crucial to assess any impact on model capabilities and helpfulness. For capabilities, we evaluate on a suite of standard benchmarks spanning general knowl- edge (ARC-Challenge, TriviaQA), commonsense reasoning (CommonsenseQA, OpenBookQA, PIQA, Winogrande), and mathematical reasoning (GSM8K). Additional experi- ment details are deferred to Appendix B. To measure helpfulness, we evaluate model compliance on benign requests from the Alpaca dataset (Taori et al., 2023). We use GPT-4o-mini as a judge to classify each 5 When Should We Introduce Safety Interventions During Pretraining? 050100150200250300 Count 0% 20% 60% 100% Pretraining Intervention Timing Compliant & Helpful Compliant & Unhelpful Partial RefusalFull Refusal Figure 4. Comparison of overrefusal rates on Alpaca (Taori et al., 2023) when using standard (Top-k) sampling for inference. We find that incorporating interventions earlier during pretraining leads to a better compliance rate on benign requests. response into one of four categories: (1) compliant and helpful, (2) compliant but unhelpful, (3) partial refusal, or (4) full refusal. Categories (1) and (2) represent desirable compliance behavior, while categories (3) and (4) indicate overrefusal. For models using the SafeBeam algorithm, we additionally introduce a fifth category: (5) shortcircuit, which occurs when generation terminates upon producing the harmful metadata token. In practice, such responses can be post-processed at serving time by substituting a default refusal message from a set of known safe completions. 4.3. Representation Analysis To analyze the impact of various curricula on the underlying representation space of our models, we evaluate the sepa- rability of safe vs harmful content. We train linear probes to classify between examples from FineWeb data that have been annotated as safe versus unsafe, learned over the repre- sentations taken from the internal activations of the model’s final layer. For the safety label, we use the safety score from Maini et al. (2025), which is produced by GPT-4o-mini that judges pretraining examples with a safety rubric. 4.4. Training Details Pretraining Setup For our experiments, we adopt the same setup of the SmolLM2 (Allal et al., 2025) for all our experiments.We train 1.7B parameter lan- guage models with a pretraining corpus of a mixture ofFineWeb-Edu,StackOverflow,FineMath, and Cosmopedia. For our safety pretraining interventions, we adopt the modified datasets ofSafeWeb,RefuseWeb, andMoralEducationfrom Maini et al. (2025), which are safety-focused, rephrased and recontextualized versions of harmful pretraining examples from FineWeb-Edu. Pretraining is performed usingLitGPT(AI, 2023), with FlashAttention-2 enabled and mixed-precision training for efficiency. We use the same hyperparameters (e.g., learning rate schedule, batch size, and sequence length) as in the SmolLM2 pretraining setup. Post-training Setup For a standardized post-training stack, we use a combination ofUltrachat-200k withAllenAI WildGuardMixandWildJailbreak datasets for safety instruction tuning. This follows standard practice from prior works in safety training (Zou et al., 2024; Maini et al., 2025). 5. Results We now present our findings on how safety curricula influ- ence robustness, helpfulness, and internal representations. 5.1. Safety Results Base ModelsTiming matters even before instruction tun- ing (Figure 1). Under standard top-k sampling, introducing interventions in the middle of pretraining (20% or 60%) yields the safest base models, while training on safe-only data (100%) is the least safe. This again complements the findings from prior work (Li et al., 2025; Maini et al., 2025). In contrast, with SafeBeam, incorporating interventions and metadata from the start (0%) provides measurable gains. Instruction-tuned Models A similar pattern holds with the models that have undergone instruction tuning (Fig- ure 3). Models trained with interventions during the middle of pretraining exhibit stronger safety performance under top-k inference than those with no interventions. Again, SafeBeam benefits the most when metadata is incorporated 6 When Should We Introduce Safety Interventions During Pretraining? 0%20%60%100% Pretraining Intervention Start Timing 0 10 20 30 40 50 60 Attack Success Rate (%) Top-K SafeBeam Figure 5. Jailbreakability of our models via the GCG algorithm (Zou et al., 2023) as we vary the timing of safety pretraining interventions. We find that introducing interventions during the middle stages of pretraining greatly improves the steerability of models towards safe completions and refusals when exposed to adversarially learned suffixes. from the beginning (i.e., 0%). Benign Finetuned ModelsThe clearest and most notice- able effects emerge in our evaluation setting after a benign finetune on mathematical data (Figure 2). Here, SafeBeam achieves the greatest robustness when interventions are in- cluded from the very start of pretraining. For standard top-k decoding, introducing interventions after roughly 20% of training yields the strongest post-finetuning safety. Taken together, our evaluations suggest two main takeaways. First, when using modified SafeBeam inference, introducing metadata-annotated unsafe data earliest in pretraining leads to the best steerability towards safer completions. Secondly, with top-k inference, introducing safety pretraining interven- tions during the middle of training (after 20% or 60%) leads to the most robust performance. This suggests that allowing the model to first absorb general-purpose safe data before applying any safety interventions may produce alignment that is more robust to subsequent benign finetuning. 5.2. Capability Results We evaluate whether safety pretraining interventions de- grade general model capabilities across various standard benchmarks (Table 1). Overall, we find that safety inter- ventions introduce minimal capability trade-offs and, in some cases, may even improve performance. We attribute this to the model’s efficient learning from high-quality syn- thetic data (Maini et al., 2024). Taken together, these re- sults indicate that safety pretraining interventions do not meaningfully degrade model capabilities; indeed, earlier interventions appear mildly beneficial on average. 5.3. Overrefusal Results We assess the impact of pretraining interventions on model helpfulness using benign requests from the Alpaca bench- mark (Taori et al., 2023). Under standard top-ksampling (Figure 4), incorporating interventions earlier during pre- training (0% or 20%) yields a slight improvement in com- pliance rates. These results suggest that safety pretraining interventions introduce minimal trade-offs in overrefusal behavior under standard inference. When using the SafeBeam algorithm (Figure 9), we ob- serve a more nuanced pattern: earlier interventions produce modestly higher overrefusal rates, yet the quality of com- pliant responses improves. The figure reveals that earlier pretraining interventions (0% and 20%) increase the frac- tion of compliant-and-helpful responses among non-refused outputs, even as overall refusal rates rise slightly. We hy- pothesize that this reflects improved calibration: early inter- ventions strengthen the correspondence between the harm- ful metadata tag and genuinely unsafe content, enabling SafeBeam to filter more precisely. The surviving beams thus tend to produce higher-quality completions. 5.4. Jailbreaking Results We also present results on how susceptible our models are to jailbreaking attacks using the GCG algorithm (Zou et al., 2023), as we vary the timing of the safety pretraining inter- ventions. Surprisingly, we find that introducing pretraining 7 When Should We Introduce Safety Interventions During Pretraining? 0%20%60%100% Safety Pretraining Intervention Timing 0.785 0.790 0.795 0.800 0.805 0.810 0.815 Separability AUC With Metadata Without Metadata 0%20%60%100% Safety Pretraining Intervention Timing 0.790 0.795 0.800 0.805 0.810 0.815 0.820 0.825 Separability AUC With Metadata Without Metadata Figure 6. Separability of safe vs unsafe content (as annotated by GPT-4o-mini) in the last layer representation space of base models (Left) and instruction-tuned models (Right), as we vary the start timing when we incorporate safety pretraining interventions. We find that introducing interventions earlier yields greater separability in representation space (especially with the incorporation of harmful metadata tags). The shaded regions denote the standard deviation computed over 5 training runs. interventions after some initial safe-only pretraining (i.e., at 20% or at 60%) leads to significantly more robustness to jailbreaks via GCG when using the SafeBeam inference algorithm (Figure 5). On the other hand, we find that jail- breakability under standard inference does not noticeably change given different intervention timings. 5.5. Representation Results In settings without harmful metadata tags, earlier interven- tions slightly improve the separability of safe versus unsafe points (with a linear classifier over the last-layer activations). When metadata is included, the effect is more pronounced: introducing interventions from the start yields the highest linear-probe AUC, suggesting a much cleaner separation of harmful and safe content in the representation space. This also supports the improved SafeBeam robustness, where models are more steerable towards safer responses when interventions are included early in pretraining. 6. Discussion Our work studies the role of different safety curricula, or how the timing of safety pretraining interventions (e.g., when and how safety-augmented data enters pretraining) im- pacts the robustness of our models. Our experiments show that incorporating safety pretraining interventions indeed help, but the best timing depends on the setting: delayed interventions (e.g., 20%–60%) can yield stronger robustness under standard decoding and against adversarial jailbreaks, while early interventions (0%) improve steerability under safety-aware inference (SafeBeam) and yield clearer separa- tion of safe vs unsafe content in representation space. Taken together, these findings highlight how the timing of safety interventions is an important choice in shaping a model’s behavior, offering guidance for building more trustworthy and responsible language models. Why does timing matter? While our experiments estab- lish that intervention timing significantly affects outcomes, understanding the underlying mechanisms remains an open question. We offer several hypotheses for future inves- tigation. For standard top-k decoding, delayed interven- tions (20%-60%) may succeed because the model benefits from first acquiring general linguistic competence and world knowledge before safety signals are introduced. Early safety interventions might compete with or interfere with learn- ing basic representations, potentially causing the model to develop shallow safety heuristics rather than robust under- standing. By contrast, when the model has already formed stable representations of language and concepts, safety in- terventions may integrate more deeply into these existing structures, making them harder to remove through subse- quent finetuning. For SafeBeam inference, early interventions may help be- cause the metadata-based steering mechanism relies on strong, well-calibrated associations between the harmful content tag and genuinely unsafe material. Introducing these associations from the start of training allows them to be re- inforced throughout the entire pretraining process, resulting in more reliable probability estimates that SafeBeam can ex- ploit. This interpretation is supported by our representation analysis (Figure 6), which shows that early interventions with metadata yield the cleanest separation between safe and unsafe content in representation space. A limitation of our study is that the introduction of safety pretraining interventions at different times during pretrain- ing means that certain models have seen more synthetic con- textualized data or refusal data. However, we note our main effects are not monotonic in the number of intervention tokens (i.e., “more intervention tokens” do not consistently yield safer models), suggesting that when safety-augmented data is introduced matters beyond total quantity. 8 When Should We Introduce Safety Interventions During Pretraining? References Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. Lightning AI.Litgpt.https://github.com/ Lightning-AI/litgpt, 2023. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart ́ ın Bl ́ azquez, Guilherme Penedo, Lewis Tun- stall, Andr ́ es Marafioti, Hynek Kydl ́ ı ˇ cek, Agust ́ ın Pi- queres Lajar ́ ın, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Cl ́ ementine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big – data- centric training of a small language model, 2025. URL https://arxiv.org/abs/2502.02737. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J Zico Kolter, Matt Fredrik- son, Yarin Gal, and Xander Davies. Agentharm: A bench- mark for measuring harmfulness of LLM agents. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview. net/forum?id=AC5n7xHuR1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. Yoshua Bengio, J ́ er ˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009. Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber- Betley, Xuchan Bao, Mart ́ ın Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetun- ing can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424, 2025. Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training. In First Conference on Language Modeling, 2024. URLhttps://openreview.net/forum? id=vwIIAot0f. Yewon Byun, Shantanu Gupta, Zachary C Lipton, Rachel Leah Childers, and Bryan Wilder. Using imper- fect synthetic data in downstream inference tasks. arXiv preprint arXiv:2508.06635, 2025. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Se- hwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tram ` er, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In Conference on Neural Information Processing Systems (NeurIPS), 2024. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher R ́ e. Skill-it! a data-driven skills framework for understanding and train- ing language models. Advances in Neural Information Processing Systems, 36:36000–36040, 2023. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Applications, challenges, and the future. The International Journal of Robotics Research, 44(5):701–739, 2025. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Ja- son Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.The lan- guage model evaluation harness, 07 2024. URLhttps: //zenodo.org/records/12608602. Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pages 1311–1320. Pmlr, 2017. Suriya Gunasekar,Yi Zhang,Jyoti Aneja,Caio C ́ esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J Zico Kolter. Adaptive data optimization: Dy- namic sample selection with scaling laws. arXiv preprint arXiv:2410.11820, 2024. 9 When Should We Introduce Safety Interventions During Pretraining? Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Sam Bow- man, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302.08582, 2023. Kenneth Li, Yida Chen, Fernanda Vi ́ egas, and Martin Wat- tenberg. When bad data leads to good models. In Forty- second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum? id=SsLGTZKXf1. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. Valentin Li ́ evin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns, 5(3), 2024. Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14044–14072, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.757. URLhttps:// aclanthology.org/2024.acl-long.757/. Pratyush Maini, Sachin Goyal, Dylan Sam, Alex Robey, Yash Savani, Yiding Jiang, Andy Zou, Zacharcy C Lipton, and J Zico Kolter. Safety pretraining: Toward the next generation of safe ai. arXiv preprint arXiv:2504.16980, 2025. Narek Maloyan, Ekansh Verma, Bulat Nutfullin, and Bislan Ashinov. Trojan detection in large language models: In- sights from the trojan detection challenge. arXiv preprint arXiv:2404.13660, 2024. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zi- fan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024. Kyle O’Brien, Stephen Casper, Quentin Anthony, Tomek Korbak, Robert Kirk, Xander Davies, Ishan Mishra, Ge- offrey Irving, Yarin Gal, and Stella Biderman. Deep ignorance: Filtering pretraining data builds tamper- resistant safeguards into open-weight llms. arXiv preprint arXiv:2508.06601, 2025. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feed- back. Advances in neural information processing systems, 35:27730–27744, 2022. Guilherme Penedo, Hynek Kydl ́ ı ˇ cek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Lean- dro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024a. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Pe- ter Henderson. Safety alignment should be made more than just a few tokens deep, 2024b.URLhttps: //arxiv.org/abs/2406.05946. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Di- rect preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. Alexander Robey, Zachary Ravichandran, Vijay Kumar, Hamed Hassani, and George J Pappas. Jailbreaking llm-controlled robots. arXiv preprint arXiv:2410.13691, 2024. Mrinmaya Sachan and Eric Xing. Easy questions first? a case study on curriculum learning for question answering. In Proceedings of the 54th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 453–463, 2016. Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh, Sriku- mar Ramalingam, Gui Citovsky, and Sanjiv Kumar. Ana- lyzing similarity metrics for data selection for language model pretraining. arXiv preprint arXiv:2502.02494, 2025a. Dylan Sam, Marc Finzi, and J Zico Kolter. Predicting the performance of black-box llms through self-queries. arXiv preprint arXiv:2501.01558, 2025b. 10 When Should We Introduce Safety Interventions During Pretraining? Dylan Sam, Alexander Robey, Andy Zou, Matt Fredrik- son, and J Zico Kolter. Evaluating language model rea- soning about confidential information. arXiv preprint arXiv:2508.19980, 2025c. Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: De- fending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837, 2025. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tat- sunori B Hashimoto. Stanford alpaca: An instruction- following llama model, 2023. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Ka- bilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ́ e Lacroix, Baptiste Rozi ` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDi- armid. Steering language models with activation engi- neering. arXiv preprint arXiv:2308.10248, 2023. Bertie Vidgen, Adarsh Agrawal, Ahmed M Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Al- hajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, et al. Introducing v0. 5 of the ai safety benchmark from mlcom- mons. arXiv preprint arXiv:2404.12241, 2024. Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In International Conference on Learning Representations, 2021. URLhttps://openreview. net/forum?id=tW4QEInpni. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Ad- vances in Neural Information Processing Systems, 36: 69798–69818, 2023. L Xue. mt5: A massively multilingual pre-trained text- to-text transformer. arXiv preprint arXiv:2010.11934, 2020. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yun- hua Zhou, and Xipeng Qiu. Data mixing laws: Opti- mizing data mixtures by predicting language modeling performance. In The Thirteenth International Confer- ence on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=jjCB27TMK3. Yuan Yuan, Tina Sriskandarajah, Anna-Luisa Brakman, Alec Helyar, Alex Beutel, Andrea Vallone, and Saachi Jain.From hard refusals to safe-completions: To- ward output-centric safety training. arXiv preprint arXiv:2508.09224, 2025. Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Ad- vances in Neural Information Processing Systems, 33: 7648–7659, 2020. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhe- hao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50(1):237–291, 2024. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Im- proving alignment and robustness with circuit break- ers, 2024. URLhttps://arxiv.org/abs/2406. 04313. 11 When Should We Introduce Safety Interventions During Pretraining? 0%20%60%100% Safety Pretraining Intervention Timing 0.60 0.62 0.64 0.66 0.68 0.70 Separability AUC With Metadata Without Metadata 0%20%60%100% Safety Pretraining Intervention Timing 0.60 0.62 0.64 0.66 0.68 0.70 Separability AUC With Metadata Without Metadata Figure 7. Separability of safe vs unsafe content (as annotated by GPT-4o-mini) in the first layer of base models (Left) and instruction-tuned models (Right), as we vary the start timing when we incorporate safety pretraining interventions. The shaded regions denote the standard deviation computed over 5 training runs. 0%20%60%100% Safety Pretraining Intervention Timing 0.66 0.67 0.68 0.69 0.70 0.71 Separability AUC With Metadata Without Metadata 0%20%60%100% Safety Pretraining Intervention Timing 0.675 0.680 0.685 0.690 0.695 0.700 0.705 Separability AUC With Metadata Without Metadata Figure 8. Separability of safe vs unsafe content (as annotated by GPT-4o-mini) in the middle (13th) layer of base models (Left) and instruction-tuned models (Right), as we vary the start timing when we incorporate safety pretraining interventions. The shaded regions denote the standard deviation computed over 5 training runs. A. Additional Experiments Additional Separability Results.In the main text, we present results on separability of harmful vs safe content in the final layer of the resulting pretraining models. Here, we also present results for the first layer and the middle (13th) layer. We observe similar trends, although we find greater separability in the final layer when compared to earlier layers in the models. SafeBeam Overrefusal. Figure 9 extends our overrefusal analysis to the SafeBeam inference setting. Compared to standard top-ksampling, SafeBeam introduces a fifth response category (shortcircuit), which occurs when generation terminates upon producing the harmful metadata token. In deployment, such responses can be handled by substituting a predetermined safe completion. B. Additional Experimental Details Safety Evaluation For our standard inference evaluations, we use top-k sampling withk = 10and truncate after 256 tokens. We average the results over 5 samples. For SafeBeam evaluation, we run a beam search with10beams for 256 tokens, and discard the top 50% of unsafe beams (in terms of probability of the next token being the harmful metadata token). We follow the same exact setting as Maini et al. (2025). We adopt the same GPT-4o-mini judge and judging prompt for helpfulness evaluations, and the same HarmBench judge and judge prompt (Mazeika et al., 2024) for our safety evaluations. 12 When Should We Introduce Safety Interventions During Pretraining? 050100150200250300 Count 0% 20% 60% Pretraining Intervention Timing Compliant & Helpful Compliant & Unhelpful Partial Refusal Full Refusal Shortcircuit Figure 9. Comparison of overrefusal rates on Alpaca when using the SafeBeam inference algorithm. We find that introducing pretraining interventions and metadata earlier during pretraining leads to more helpful compliant behavior, with a slight increase in overrefusal rates. Capabilities EvaluationWe use the default experimental setting for all the datasets as provided as in the LM evaluation harness (Gao et al., 2024). We use the dataset default few-shot settings: 0-shot for ARC-Challenge, OpenBookQA, CommonsenseQA, PIQA, TriviaQA, and WinoGrande (numfewshot=0), and 5-shot for GSM8K (numfewshot=5; few-shot examples sampled from the training split). Adversarial JailbreakingFor computing the adversarial suffixes via the GCG algorithm, we learn an adversarial suffix on the target model, optimizing for the target string of “Sure”. We perform optimization for 500 steps, with an initialization of “x x ... x”, and topk of 256. Finetuning For our supervised instruction tuning, we finetune with a batch size of 4 with a learning rate of 2e-5 for one epoch, using cosine learning rate decay with a warmup ratio of 0.03. For our benign finetuning experiments, we perform supervised finetuning on GSM8k for a single epoch with a learning rate of 2e-5 with the same learning rate schedule as in instruction tuning. Separability ExperimentsIn our separability experiments, we use the last token position and the final layer of all models. For each model, we train a logistic regression classifier on a balanced subset of 500 safe and 500 unsafe examples from the training split, and the trained probe is evaluated on the full balanced test set. We report the mean, standard deviation, and standard error, when computed over five seeds, of the test AUC (area under the ROC curve). C. Additional Discussion We note that the notion of safety that we focus on in this paper is fairly broad, encompassing general notions of toxicity and harms. This is taken from the taxonomy proposed by the MLCommons (Vidgen et al., 2024), captures a wide variety of potential harms and undesirable behaviors in different domains. Our method likely requires additional efforts to tackle safety for specific capabilities, such as bio-weapons and cybersecurity (Li et al., 2024) or reliability of agents in context-dependent scenarios (Andriushchenko et al., 2025; Sam et al., 2025c). We also note that our results on the separability of safe vs unsafe content has greater implications for the monitoring of such models. These experiments contribute to a line of work that develops external monitoring to predict when undesirable generations will occur and steer towards safer outputs (Turner et al., 2023; Zou et al., 2024). This may even have implications for complementary alternative approaches that are adopted in black-box settings and use external models (Sharma et al., 2025; Sam et al., 2025b), which is an area for future study. 13 When Should We Introduce Safety Interventions During Pretraining? D. Computational Resources For each variant of the pretraining runs for the 1.7B parameter model on 600B tokens, we used 32 H100 GPUs for roughly 6-7 days. Finetuning requires roughly 4 H100 GPUs for 4 hours. Evaluations are run on one L40S GPU. 14