Paper deep dive

Reducing the Scope of Language Models with Circuit Breakers

David Yunis, Siyu Huo, Chulaka Gunasekara, Danish Contractor

Year: 2024Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 77

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 5:50:45 PM

Summary

The paper introduces 'scoping' for Large Language Models (LLMs), a task where models are trained to respond only to queries within a specific domain while rejecting all others. The authors evaluate various methods including system prompting, supervised fine-tuning (SFT), direct preference optimization (DPO), probing, and Circuit Breakers (CB). They find that SFT performs best with high data diversity, while Circuit Breakers are superior in low-diversity settings, and layering both methods often yields optimal results.

Entities (5)

Circuit Breakers · method · 98%Supervised Fine-tuning · method · 98%Direct Preference Optimization · method · 95%LLM Scoping · task · 95%Mistral-7B · model · 90%

Relation Signals (3)

SFT->CB → combines → Supervised Fine-tuning

confidence 95% · We first run SFT, then run CB training afterwards.

Supervised Fine-tuning → performsbestwith → High Data Diversity

confidence 90% · we find that when diverse examples of irrelevant queries are available, simple supervised fine-tuning produces the best results

Circuit Breakers → performsbestwith → Low Data Diversity

confidence 90% · when such diversity is low, Circuit Breakers perform quite well

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) are deployed in a wide variety of user-facing applications. Typically, these deployments have some specific purpose, like answering questions grounded on documentation or acting as coding assistants, but they require general language understanding. In such deployments, LLMs should respond only to queries that align with the intended purpose and reject all other requests, such as generating poetry or answering questions about physics, a task we refer to as `scoping'. We conduct a comprehensive empirical evaluation of various methods, ranging from prompting, fine-tuning to preference learning and the recently proposed general alignment technique known as Circuit Breakers (CB). Across three families of language models and a broad variety of tasks, we show that it is possible to scope language models. We examine scoping for multiple topics, and fine-grained topics. We ablate diversity of irrelevant queries, layer different techniques, conduct adversarial evaluations and more. Among other results, we find that when diverse examples of irrelevant queries are available, simple supervised fine-tuning produces the best results, but when such diversity is low, Circuit Breakers perform quite well. One can often get the benefits of both methods by layering them in succession. We intend our study to serve as a practitioner's guide to scoping LLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

77,023 characters extracted from source content.

Expand or collapse full text

Reducing the Scope of Language Models David Yunis * , Siyu Huo, Chulaka Gunasekara, Danish Contractor IBM Research AI dyunis@ttic.edu, siyu.huo@ibm.com, chulaka.gunasekara@ibm.com, danish.contractor@ibm.com Abstract Large language models (LLMs) are deployed in a wide variety of user-facing applications. Typically, these deployments have some specific purpose, like answering questions grounded on documentation or acting as coding assistants, but they require general language understanding. In such deployments, LLMs should respond only to queries that align with the intended purpose and reject all other requests, such as generating poetry or answering questions about physics, a task we refer to as ‘scoping’. We conduct a comprehensive empirical evaluation of various methods, ranging from prompting, fine-tuning to preference learning and the recently proposed general align- ment technique known as Circuit Breakers (CB). Across three families of language models and a broad variety of tasks, we show that it is possible to scope language models. We examine scoping for multiple topics, and fine-grained topics. We ablate diversity of irrelevant queries, layer different techniques, con- duct adversarial evaluations and more. Among other results, we find that when diverse examples of irrelevant queries are available, simple supervised fine-tuning produces the best re- sults, but when such diversity is low, Circuit Breakers perform quite well. One can often get the benefits of both methods by layering them in succession. We intend our study to serve as a practitioner’s guide to scoping LLMs. Code — https://github.com/IBM/llm-scoping 1 Introduction In recent years, large language models have surged into pub- lic awareness. One major recent addition is the “alignment” process through Reinforcement Learning with Human Feed- back (RLHF) (Christiano et al. 2017; Ouyang et al. 2022), which has made the current generation of language models much less likely to emit toxic content than previous genera- tions (Wolf, Miller, and Grodzinsky 2017), and thus much more acceptable for general use. As a result, many businesses and individuals now feel more comfortable using these tech- nologies than they did in the past. Although we have generally capable language models that can refuse to answer toxic or dangerous queries, deploying * David Yunis is a PhD student at the Toyota Technological Institute at Chicago. Work was performed during an internship at IBM. Copyright © 2026, Association for the Advancement of Artificial Intelligence (w.aaai.org). All rights reserved. them still remains challenging. Even if they avoid producing harmful content, they often respond to any question, relevant or not, without discernment. This becomes a problem when we wish to use them in specific contexts: e.g. shopping bots currently give coding advice 1 or answer other questions, 2 while assistive copilots can be taken off course by prompt injections. 3 Thus, there is still a need to scope language models for these specific uses. We define LLM scoping as a conditional generation task in which a language model must: (i) identify whether an input query falls within a relevant domain, (i) reject irrelevant queries, and (i) maintain high-quality generation for relevant queries. This contrasts with traditional text classification, which maps inputs to discrete labels without requiring natural language generation. LetQdenote the set of all possible natural language queries. LetD rel ⊂ Qdenote the subset of relevant or in- domain (should be accepted) queries, and letD irr =Q rel denote the set of irrelevant or out-of-domain (should be re- jected) queries. LetRbe the space of valid natural language responses, and let⊥ /∈ Rbe a special token representing rejection. The scoping is a function: f θ :Q→R∪⊥ such that: f θ (q)∈R,if q ∈D rel , f θ (q) =⊥,if q ∈D irr . In contrast, text classification models are not required to preserve generation quality, whereas in LLM scoping, de- grading the model’s performance onD rel constitutes a failure of the task. Further, unlike the traditional LLM refusal task in the context of safety and alignment, which typically cor- responds to a much smaller reject set than accept set, the scoping tasks consider the opposite. Currently, LLMs can be scoped through two-stage ap- proaches like relevance classifiers, or system prompting and 1 https://shorturl.at/qf3FA 2 https://w.forbes.com/sites/lesliekatz/2024/07/13/amazon- ai-shopping-assistant-rufus-answers-non-shopping-questions- too/ 3 https://oecd.ai/en/incidents/2025-10-08-0fbf arXiv:2410.21597v3 [cs.CL] 13 Nov 2025 Figure 1: We study the ability to scope language models to specific topics. We assume access to a set of relevant (accept) queries and irrelevant (reject) queries, where the accept queries correspond to a relatively narrow domain. We examine how well different methods cause the language model to accept only the relevant examples, while rejecting all other examples, including out-of-distribution requests that weren’t seen during training. then text generation, but these options are brittle (Chao et al. 2023; Mehrotra et al. 2023; Zeng et al. 2024; Wei, Haghtalab, and Steinhardt 2023) and easy to circumvent. We shed further light on this problem, conducting a comprehensive empirical study on scoping language models to specific tasks. We apply existing methods to this problem, including system prompt- ing, supervised fine-tuning, preference learning (Rafailov et al. 2024), probing, and a recently-introduced method called Circuit Breakers (Zou et al. 2024). We scope language mod- els more broadly for multiple tasks, and more finely for spe- cific niche tasks. We ablate over diversity of training sets, language model size, adversarial prompting and more. Our specific contributions include: • We introduce the task of scoping LLMs • We conduct a broad experimental exploration of existing methods for this task •We show that it is possible to scope language models, even for multiple and fine-grained tasks • We find that when training data exhibits high diversity, supervised fine-tuning yields the best performance •Conversely, in settings with low data diversity, the Cir- cuit Breakers method (Zou et al. 2024) provides superior results •Finally, we show it is possible to layer these two, often preserving the best performance 2 Related Work Aligning Language Models: The advent of the current era of language models has been marked by a process of aligning language models so that generations are more helpful, and safer for deployment (Ouyang et al. 2022; Bai et al. 2022a). The primary way this is accomplished is through reinforce- ment learning with human feedback (RLHF) (Christiano et al. 2017) which was first proposed in robotic simulation tasks. RLHF proceeds by collecting preference pairs of comple- tions, and training a reward model from human judgments on those preference pairs, then performing reinforcement learn- ing with the language model against that reward model. From tasks in simulation, it was developed in language (Stiennon et al. 2020), until it reached its current state. Other works have removed the human aspect of human feedback, allow- ing for synthetic feedback from models (Bai et al. 2022b; Sudalairaj et al. 2024). Lately, Rafailov et al. (2024) have re- moved the need for a reward model, making for a stabler and simpler objective function without many of the complexities of RL training. A budding line of work also explores aligning not just to a single reward model, but preferences of many different individual users (Chakraborty et al. 2024; Lee et al. 2024). All of these methods focus on some general notion of alignment, without considering the specific task, unlike our work. Adapting for Specific Purposes: Typically after pre- training, language models go through an instruction fine- tuning stage, where they gain the ability to follow instruc- tions (Mishra et al. 2022; Ouyang et al. 2022; Wei et al. 2022). After this, they proceed through an alignment phase as dis- cussed above, usually to avoid harmful behavior (Bai et al. 2022a). It is possible to adapt language models for specific purposes simply with a system message (Touvron et al. 2023), but many examples of black-box adversarial attacks (Chao et al. 2023; Anil et al. 2024; Wei, Haghtalab, and Steinhardt 2023; Zeng et al. 2024) demonstrate it is difficult only to rely on the system prompt for such control. Wallace et al. (2024) propose finetuning with different levels of priority, similar to Zhang et al. (2023b), but these works focus primarily on general safety and not the task. These ideas are based on the fact that current language models can often be distracted by irrelevant context (Shi et al. 2023; Yoran et al. 2024). Thus, it seems important to finetune the language model if we want it to be deployed to a particular domain. For domains where there is sufficient data, we may also pretrain and fix the language model’s purpose ahead of time (Beltagy, Lo, and Cohan 2019; Wu et al. 2023; Li et al. 2023) or continue pretraining from a base language model (Gururangan et al. 2020). It is an open question however whether finetuning re- tains the robustness capabilities, or if it is similarly as brittle as system prompting for out-of-distribution questions. Refusal in Language Models: As our work deals with scoping models to refuse irrelevant queries, we review refusal. CategoryExample task# Datasets# Tasks# Instances Sentiment Analysis (SA)Predicing if a movie review is relevant or not81031248 Toxic Language Detection (TLD)Detecting if a comment contains cursing5933849 Summarization (S)Condensing a news article4413096 Text Completion (TC)Filling in the blanks in a transcript3310515 Story Composition (SC)Writing a new ending for a story4415556 Dialogue Generation (DG)Continuing a dialogue between parties3412744 Program Execution (PE)Computing the result of a described function262694001 Question Answering (QA)Answering biology multiple-choice questions193084065 GSM8k (Cobbe et al. 2021)Answering simple math word problems115978 Alpaca (Taori et al. 2023)General requests like writing a recipe for lunch1-18793 Table 1: Breakdown of data. We reserve at least 20% of the data from each dataset for validation. We will use at most 2048 instances from each category for training, though this is sampled from a much larger number. PE is so large as the data is synthetically generated. All categories above the divider will be used for training and evaluation, while categories below the divider are only used for out of distribution evaluation. More detail is available in a comprehensive survey by Wen et al. (2024). One common case to train for refusal is when the answer is unknown or the model is unconfident (Zhang et al. 2023a; Cao 2023; Xu et al. 2024). Another is for unsafe inputs (Varshney et al. 2023; Zhang et al. 2023b; Wallace et al. 2024). Supervised fine-tuning (SFT) to reject unsafe prompts can still lead to unsafe behavior, though parameter efficient methods like LoRA (Hu et al. 2022) have better tradeoffs (Brahman et al. 2024). Both Brahman et al. (2024) and Cheng et al. (2024) take an approach to refusal using SFT and DPO, which we will adapt to our case. Other meth- ods to induce refusal may be prompt-based (Xie et al. 2023; Zhang et al. 2024) or based on probing model representa- tions (Kadavath et al. 2022; Slobodkin et al. 2023). Zou et al. (2024) design a method that conditionally rejects unsafe in- puts based on orthogonalizing internal representations that we will adapt for our study. Though these methods lay out a set of techniques to explore for our task, all of them are oriented toward general alignment qualities like safety, as opposed to specific tasks that we will explore. 3 Experimental Setup We would like to scope language models to provide comple- tions to relevant tasks, and reject queries corresponding to irrelevant tasks. In particular, we assume we are given a set of “relevant” or “accept” queriesq rel |q rel ∼D rel , where D rel is a set of accepted tasks, and a set of “irrelevant” queriesq irr |q irr ∼D irr whereD irr is a set of rejected tasks. We are given a language modelf θ : q 7→ ywhich predicts completionyfrom inputq, with parametersθ; a clas- sifierg : y 7→ c∈0, 1which decides whether a LLM com- pletion is accepted (0) or rejected (1). We would like to com- pute an update∆such that we minimize E q rel g(f θ+∆ (q rel )) and maximize E q irr g(f θ+∆ (q irr )). Thus we want ‘relevant’ queries to be accepted and ‘irrelevant’ queries to be rejected. As an additional goal of scoping, we would like perfor- mance on the accept tasks not to degrade. Given a scoring functionh : (q,y)7→ s∈ [0, 1]which scores the completion on task performance where 1 is best, we would also like to maximize E q rel h(q rel ,f θ+∆ (q rel )). Datasets & Metrics We conduct many experiments with different mixtures of ac- cept and reject queries. In order to standardize the format, we draw prompts from Super-NaturalInstructions (SNI) (Wang et al. 2022). SNI is a meta-dataset composed of many differ- ent “tasks”, sometimes with multiple tasks per dataset, for ex- ample generating questions from passages for a reading com- prehension dataset, or generating answers to provided ques- tions from the same reading comprehension dataset. Each task, specified by a task instruction, comes with a collection of examples. We use SNI as it is publicly available, and con- tains a broad range of complex tasks which current language models should be able to perform. To get our training datasets, we first manually select a set of tasks that are straightforward to automatically evaluate, leaving out many more subjective tasks that may require a human reader. We then group those tasks that we select by category provided from SNI. Details and statistics on categories are provided in Table 1. Each of these categories contains multiple datasets, so the distribution for each task is quite broad. We will also combine multiple tasks in the accept or reject set. For all experiments, we always evenly split the training data for accept/reject set between all tasks. We reserve at least 20% of the prompts as a validation set that are not seen during training. Where not specified, we use 2048 prompts for the accept set, and 2048 prompts for the reject set. We evaluate Sentiment Analysis and Toxic Language Detection with accuracy (the classes are mostly balanced), while for other tasks we use a stan- dard metric for generation, Rouge-L (Lin 2004), between the generation and ground truth completion as a proxy for perfor- mance (Accept Score). Our goal is mostly to study rejection behavior, so a rough performance proxy is all we need. Irrespective of the specific accept and reject sets used dur- ing training, we evaluate performance across all categories listed in Table 1. We then report the average rejection rates separately for the accept set (Accept), all in-distribution reject sets (ID Reject), and all out-of-distribution reject sets (OOD Reject). Note if the training set consists of SA in Accept, and S in Reject, OOD Reject will contain the 8 other categories. Tasks above the divider in Table 1 will be used in different experiments for both training and evaluation, while tasks be- low the divider will only be used for OOD evaluation. As stated previously, ideally we would like to have 0 rejection on Accept, and 100% rejection on ID Reject and OOD Reject. Methods For all methods that require training the language model, we use LoRA (Hu et al. 2022) training with rank16,α = 16 and dropout of0.05. We use the Adam (Kingma 2014) optimizer without any regularization and tune learning rates (see Appendix B). System Prompting (Sys.): The simplest method to scope language models is simply to instruct them to refuse irrelevant prompts. For example, for SA the system prompt is: You are an assistant who only answers requests related to Sentiment Analysis. For all other requests you respond “I cannot answer that.” With multiple accept categories, we comma separate the category names (e.g. “...related to Sentiment Analysis, Text Completion and Summarization...”). This system prompt is prepended to all instructions at evaluation time. In addition, all other methods also use the system prompt both at training and evaluation time. This is similar to methods proposed by Xie et al. (2023); Zhang et al. (2024). Supervised Fine-Tuning (SFT): Supervised Fine-Tuning (SFT) consists of tuning the language model to produce par- ticular outputs. For the accept tasks the completionsy rel are the groundtruth completions provided by the dataset. For the reject tasks, the completionsy irr are always “I cannot answer that.”. We tune learning rate and step bud- get for SFT. This is a similar approach to Brahman et al. (2024); Cheng et al. (2024). In experiments, theL gen is about generation (task completion) loss betweeny rel and f θ (q),q ∈D rel , and theL rej is about irrelevant task rejec- tion loss betweeny irr andf θ (q),q ∈D irr . The total scoping loss isL scope =L gen +λL rej . We use balanced scoping loss whereλ = 1for loss computing. As the finetuning dataset can be quite small, loss is only computed on the comple- tions so as to avoid overfitting to the small set of instructions, agreeing with common practice (Mishra et al. 2022; Ouyang et al. 2022; Wei et al. 2022). Direct Preference Optimization (DPO): Given its role in post-training we explore a preference learning method. We choose to experiment on Direct Preference Optimization (DPO) (Rafailov et al. 2024), as it does not require an addi- tional reward model. DPO requires pairs of preference data, so for accept queries we provide the dataset completion as preferred, and the completion “I cannot answer that.” as rejected. For reject queries we do the reverse, preferring “I cannot answer that.” over the ground truth completion. For DPO we tune learning rate, step budget, and the loss weight- ing term regularizing the KL divergence from the base model predictions. This is similar to Brahman et al. (2024); Cheng et al. (2024). Two-stage on Probing Classifier (Probe): Probes of rep- resentations are a common method to accomplish tasks as they base predictions on the internal state of the language model (Conneau et al. 2018; Tenney et al. 2019; Zou et al. 2023a). Previous work on Circuit Breakers (Zou et al. 2024) showed that probing representations was competitive for de- tecting dangerous language. However, they only designed probes on a single layer of a language model. Here we design a stronger probe. Once an instruction is fed to the frozen language model, we first remove the first position as it is quite anomalous due to large magnitude (Xiao et al. 2024), then we average all positions per layer and normalize the average vector to norm 1 so as to match norms between lay- ers. Finally, we concatenate the mean-pooled representations from each layer to form a single feature vector, which is then passed through a two-layer multilayer perceptron (MLP) with a hidden dimension of 256 to perform binary classification, determining whether to accept or reject the input. Only the MLP layers are trained, and we tune the learning rate and step budget. This is justified by prior choices in Kadavath et al. (2022); Slobodkin et al. (2023); Zou et al. (2024). We use this probe in a two-stage setup where the probe first classifies the request and if it is determined to be an ‘accept’ task, a response generated via an LLM. Circuit Breakers (CB): (Zou et al. 2024) first intro- duce a method they call Circuit Breakers (CB) for accept- ing normal queries while rejecting dangerous ones. We repurpose their method for this task. Essentially, given a function which extracts the representations of a lan- guage model at particular layers, they design an optimiza- tion objective with two components:L gen,q∈D rel (q, ∆) = ∥rep(f θ (q)) − rep(f θ+∆ (q))∥ 2 2 andL rej,q∈D irr (q, ∆) = max0, cos(rep(f θ (q)), rep(f θ+∆ (q)). The total loss is L scope = α(t)L gen + β(t)L rej where the two components of the loss are scheduled over time. This loss function keeps the representations of accept tasks from drifting, while making the representations of reject tasks orthogonal from their original position. This orthogonaliza- tion breaks the language model generation on bad inputs. For CB we tune learning rate and step budget and more, and show later that CB is particularly sensitive to hyperparameters. SFT→CB: As we will see, SFT and CB tend to be the best methods for scoping in slightly different circumstances. In order to improve accept task performance and preserve the benefits of both, we propose to layer CB on top of SFT. We first run SFT, then run CB training afterwards. We keep hyperparameters from the SFT and CB tuning respectively. Detecting rejection Though a strong language model judge (Zheng et al. 2023) may seem like a good choice for judging rejection, we fol- lowed prior work in first experimenting with simpler heuris- tics (Zou et al. 2023b, 2024) based on string matching and the rejection behavior of Circuit Breakers. We found these heuristics to be very strong, and much less expensive than running a frontier judge for the many experiments in this paper. We also experimented with a smaller hosted language model judge, but found its performance much poorer than the heuristics. More details on evaluation are available in Appendix B. 4 Experiments In this section we explore a number of empirical questions: can we scope language models, how does scoping behave 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject Mistral-7BLlama-8BGranite-7B Model 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe Figure 2: Scoping across different language models. We see that system prompting is insufficient, and different methods have different success rates for different models. Clearly it is possible to scope language models to particular distributions. 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 246 Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe a Mistral-7B-Instruct 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 246 Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe b Llama-3.1-8B-instruct 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 246 Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe c Granite-7B-instruct Figure 3: Results for increasing diversity of rejection set. We see across models that CB performs relatively better than SFT when data diversity is low, but SFT is much stronger with more rejections sets. Probing appears strong across the board, though sometimes leads to overrejection on the Accept set. across scale, how much diversity is needed for scoping, or whether scoping is possible for multiple tasks simultaneously. We aim to be comprehensive, thus demonstrate results across 2-3 different categories per dataset. Where not detailed, our accept sets will be Sentiment Analysis (SA), Summarization (S) and Program Execution (PE). All experiments contain evaluations of task perfor- mance (Accept Score) on the accept set (which should be high), rejection rate on the in-distribution accept (Accept) set (which should be low) as well as rejection rate on the in-distribution reject set (ID Reject) and out of distribution data (OOD Reject) (which should be high). We describe experiments in broad strokes, and defer precise details on hyperparameters to Appendix B. We begin by presenting results across a variety of languagemodels(Mistral-7B-Instruct-v0.2, granite-7b-instruct, Llama-3.1-8B-Instruct ) (Jiang et al. 2023; Su- dalairaj et al. 2024; Grattafiori et al. 2024). We explore scale on the Llama-3.2-instruct family, and for other results choose Mistral-7b-instruct as we do not have the compute or space to run every experiment on every model. Scoping language models We start with the basic question: is it possible to scope lan- guage models? We explore each method across a variety of models using Sentiment Analysis (SA) as our accept task in Figure 13, where we see that although system prompting is in- sufficient, a broad variety of different methods are successful to different extents with different models. Rejection set diversity One of the most critical questions when attempting to restrict the generations of language models is what data might be necessary to do so. If models overfit to a particular data dis- tribution, then it may be difficult to reject requests that were not specified in the training distribution. Thus, here we ask: how much data diversity is necessary in the rejection set to robustly scope models? If very little diversity is needed, and rejection extends to OOD requests, then adapting models to new deployments becomes quite inexpensive. In Figure 14 we study Sentiment Analysis across different models with more and more diverse rejections sets consisting of an increasing number of tasks, and defer results on additional tasks with Mistral-7b-instruct to Appendix A.1. Classification and GenerationMath and Programming MethodAccept ScoreAcceptID RejectOOD RejectAccept ScoreAcceptID RejectOOD Reject Sys. 0.25± 0.18 0.10± 0.11 0.70± 0.04 0.42± 0.11 0.15± 0.14 0.16± 0.22 0.33± 0.36 0.28± 0.22 CB 0.25± 0.18 0.10± 0.121.0± 0.0 0.79± 0.23 0.14± 0.15 0.16± 0.221.0± 0.00.96± 0.08 SFT0.46± 0.240.01± 0.02 0.95± 0.03 0.28± 0.16 0.26± 0.040.0± 0.0 0.99± 0.01 0.52± 0.25 SFT→ CB0.46± 0.24 0.07± 0.111.0± 0.0 0.54± 0.220.27± 0.04 0.26± 0.111.0± 0.0 0.64± 0.22 DPO 0.21± 0.110.01± 0.021.0± 0.0 0.54± 0.06 0.23± 0.20 0.01± 0.011.0± 0.0 0.78± 0.28 Probe– 0.19± 0.211.0± 0.00.91± 0.13– 0.04± 0.051.0± 0.0 0.93± 0.11 Table 2: Evaluation when accepting multiple categories. We show it is quite possible to do so. In general SFT-based methods are best for in-domain performance, and CB or Probe are strong choices for OOD rejection. We show results in Figure 14. In general we see that CB performs relatively better than SFT when data diversity is low, but SFT is much stronger with more rejection sets. Probing appears strong across the board, though it sometimes leads to overrejection on the Accept set, which is undesirable. In general it may be quite straightforward to collect diverse data, so unless the practitioner is quite constrained, it’s worthwhile to always start with Probe and SFT. Accepting multiple tasks Here we ask: is it possible to still reject tasks when there are multiple tasks in the accept set? Such a setting is nat- ural as most language models will have a few different specific uses, like a programming bot that can write code and also answer questions about documentation. We demon- strate results onMistral-7b-Instruct-v0.2in Ta- ble 2 with two choices of accepts sets: Classification and Generation (SA,S,TLD,SC,TC,DG) and Math and Program- ming (PE,GSM8k). We use the opposite set as the ID reject set in this case. Classification and Generation: We see strong scores for SFT-based methods here. On the accept set, Probe is worst, while Sys. is poor leading CB and SFT-CB to suffer. In dis- tribution all methods work well except Sys. and SFT. Out of distribution, CB and Probe perform well, while SFT-CB and DPO are even. Math and Program Execution: SFT-based methods per- form best on the task. Surprisingly, SFT-CB has a very high rejection rate on the accept task. In-distribution every method but Sys. works well. Out-of-distribution there is a similar story to the previous case, where CB works quite well, and Probe is also strong, but the rest less so. Takeaways: We see that it is possible to support multiple accept tasks. In particular, CB and Probe work best for out-of- distribution evaluation, but as its performance on the accept task is tied to the system prompt, any issues with the base model will carry over. Representation Analysis To get a sense for how different methods operate, we study how the representations change before and after training on Mistral-7b-Instruct-v0.2. In particular we look at cosine distance between the original representation of a token, and the same token after training. If there are patterns in these changes, it should give us some indication as to how different methods operate. In Figure 4 we show that DPO and SFT only change the representations of the tail of the context. Hence it makes sense why CB is more robust under attack (explored in Zou et al. (2024) and Appendix A.4): all representations have changed, so it is difficult to find a way to circumvent the changed behavior, while DPO and SFT have “cracks” which can be exploited. The effect is particularly clear on the in-distribution re- jection set, but preceding sections demonstrate that most methods are fairly comparable in distribution. Out of distri- bution, the effect of CB is much less, though still there is a much more substantial difference from the original model than SFT or DPO which make only small changes to the tail of context in deeper layers. With SFT-CB, we can clearly see the layering of the tail edit as well as the orthogonalization across the entire context. Additional analysis Here we briefly discuss some additional results based on Mistral-7b-Instruct-v0.2, deferring full treatment to the Appendix. Adversarial Evaluation: We examine the behavior of different methods under adversarial prompts in Appendix A.4, where we find that CB-based methods are more robust than others, echoing results of Zou et al. (2024). Model Scale: In Appendix A.3, we examine scoping be- havior with different model scales, finding consistent results except for Probe which struggles with smaller models. Precise Scoping: We find that one can scope precisely, (e.g. only News summarization instead of all summarization). In general many methods are appropriate for this, though consistent with prior results SFT suffers with a low diversity of training examples. For more details, see Appendix A.5. Tuning of CB: In general, it was much more difficult to tune CB than SFT. Depending on the model, the optimal hyperparameters in terms of the target layers, and the choice ofαmade a large difference. This is important as CB often had very strong performance where appropriately tuned, but was less stable than other methods. We explore this more in Appendix A.6. Effect of Data Quantity: We find that most methods work quite well with very little data (as little as 128 instances). DPO in particular benefits monotonically, while CB has is- 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer a SFT 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer b DPO 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer c CB 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer 010203040 Tail Pos 0 5 10 15 20 25 30 Layer d SFT→ CB 0.0 0.2 0.4 0.6 0.8 1.0 Figure 4: Cosine distance between base model representations and LoRA tuned model. Rows represent model layer, columns represent tail position of prompt, averaged over all queries in the dataset. Top row: Accept data. Middle row: In distribution reject data. Bottom row: Out of distribution reject data. We see that CB-based methods make changes to representations across the context, while DPO and SFT only change the tail of the context. Layering SFT and CB yields both effects. sues as the raw number of training examples in the dataset scales, perhaps due to the difficulty of simultaneous orthogo- nalization of many different reject instances, see Appendix A.7 for more details. Effect of LoRA Rank: Overall, it does appear that rank can have a substantial effect on the performance of methods. While DPO seems to scale monotonically with LoRA rank, CB-based methods have a sweet spot for performance, above which it seems optimization becomes difficult. See detailed analysis in Appendix A.8. 5 Discussion Though current language models are generally applicable, there is still a need at deployment time to specify the kinds of queries they should and should not be able to answer. Oth- erwise, agents deployed in the wild may be easy to distract, and if used as a part of a pipeline may lead to cascading errors. Hence scoping is crucial. In this work, we conducted a comprehensive empirical study of scoping language models using three model families and multiple tasks. Our findings reveal several key insights. First, system prompting alone is generally inadequate across a range of models and datasets. While performance varies depending on the specific method, model, and dataset, certain trends emerge. Supervised fine-tuning (SFT) tends to perform well when the training data is diverse, whereas Circuit Breakers (CB) is more effective in low-data regimes with less diversity, likely because the orthogonalization objective it employs is easier to optimize in such settings. Probing methods can also yield strong results, provided that the probe is sufficiently ex- pressive to disentangle the model’s internal representations; however, this approach incurs additional inference overhead due to the use of an auxiliary model. Combining SFT and CB typically results in performance that reflects the strengths of both approaches, but this layered method is sensitive to the failure of either component, which can degrade overall performance. We saw that it was possible to scope across language model scales, for multiple tasks at a time and for very fine-grained tasks. We demonstrated that different methods have different effects on the internal representations of the models: SFT and DPO only modify the tail of the language model context, unlike CB which modifies representations across the context. Such different behavior may explain why CB is stronger under adversarial attacks (Appendix A.4), the original setting it was proposed for (Zou et al. 2024). While these results indicate that CB can be a promising ap- proach, it presents considerable challenges in practice. Specif- ically, its performance can fluctuate wildly based on very small step-count difference. Additionally, optimal target lay- ers may need to be selected on a per-model basis, further complicating its application. Given these sensitivities, we rec- ommend defaulting to simpler methods such as supervised fine-tuning (SFT) or probing, particularly in settings where data diversity is not a limiting factor. References Anil, C.; Durmus, E.; Sharma, M.; Benton, J.; Kundu, S.; Batson, J.; Rimsky, N.; Tong, M.; Mu, J.; Ford, D.; et al. 2024. Many-shot jailbreaking. Anthropic, April. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das- Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. Beltagy, I.; Lo, K.; and Cohan, A. 2019. SciBERT: A pre- trained language model for scientific text. arXiv preprint arXiv:1903.10676. Brahman, F.; Kumar, S.; Balachandran, V.; Dasigi, P.; Py- atkin, V.; Ravichander, A.; Wiegreffe, S.; Dziri, N.; Chandu, K.; Hessel, J.; et al. 2024. The art of saying no: Contex- tual noncompliance in language models. arXiv preprint arXiv:2407.12043. Cao, L. 2023. Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism. arXiv preprint arXiv:2311.01041. Chakraborty, S.; Qiu, J.; Yuan, H.; Koppel, A.; Huang, F.; Manocha, D.; Bedi, A. S.; and Wang, M. 2024. MaxMin- RLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925. Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G. J.; and Wong, E. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419. Cheng, Q.; Sun, T.; Liu, X.; Zhang, W.; Yin, Z.; Li, S.; Li, L.; He, Z.; Chen, K.; and Qiu, X. 2024. Can AI Assistants Know What They Don’t Know? In Forty-first International Conference on Machine Learning. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from hu- man preferences. Advances in neural information processing systems, 30. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Conneau, A.; Kruszewski, G.; Lample, G.; Barrault, L.; and Baroni, M. 2018. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Gurevych, I.; and Miyao, Y., eds., Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2126–2136. Melbourne, Australia: Association for Computational Linguistics. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Gururangan, S.; Marasovi ́ c, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8342–8360. Harris, C. R.; Millman, K. J.; Van Der Walt, S. J.; Gommers, R.; Virtanen, P.; Cournapeau, D.; Wieser, E.; Taylor, J.; Berg, S.; Smith, N. J.; et al. 2020. Array programming with NumPy. Nature, 585(7825): 357–362. Hu, E. J.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. Hunter, J. D. 2007. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3): 90–95. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.; Tran-Johnson, E.; et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Kingma, D. P. 2014. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980. Lee, S.; Park, S. H.; Kim, S.; and Seo, M. 2024. Aligning to thousands of preferences via system message generalization. arXiv preprint arXiv:2405.17977. Lhoest, Q.; Del Moral, A. V.; Jernite, Y.; Thakur, A.; Von Platen, P.; Patil, S.; Chaumond, J.; Drame, M.; Plu, J.; Tunstall, L.; et al. 2021. Datasets: A community li- brary for natural language processing.arXiv preprint arXiv:2109.02846. Li, R.; Zi, Y.; Muennighoff, N.; Kocetkov, D.; Mou, C.; Marone, M.; Akiki, C.; Jia, L.; Chim, J.; Liu, Q.; et al. 2023. StarCoder: may the source be with you! Transactions on Machine Learning Research. Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74–81. McKinney, W.; et al. 2011. pandas: a foundational Python library for data analysis and statistics. Python for high per- formance and scientific computing, 14(9): 1–9. Mehrotra, A.; Zampetakis, M.; Kassianik, P.; Nelson, B.; Anderson, H.; Singer, Y.; and Karbasi, A. 2023. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119. Mishra, S.; Khashabi, D.; Baral, C.; and Hajishirzi, H. 2022. Cross-Task Generalization via Natural Language Crowd- sourcing Instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3470–3487. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information pro- cessing systems, 32. Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Er- mon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. Shi, F.; Chen, X.; Misra, K.; Scales, N.; Dohan, D.; Chi, E. H.; Sch ̈ arli, N.; and Zhou, D. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, 31210–31227. PMLR. Slobodkin, A.; Goldman, O.; Caciularu, A.; Dagan, I.; and Ravfogel, S. 2023. The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, 3607–3625. Singapore: Association for Computational Lin- guistics. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. F. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008–3021. Sudalairaj, S.; Bhandwaldar, A.; Pareja, A.; Xu, K.; Cox, D. D.; and Srivastava, A. 2024. Lab: Large-scale alignment for chatbots. arXiv preprint arXiv:2403.01081. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. Tenney, I.; Xia, P.; Chen, B.; Wang, A.; Poliak, A.; McCoy, R. T.; Kim, N.; Van Durme, B.; Bowman, S. R.; Das, D.; et al. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Learning Representations. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Varshney, N.; Dolin, P.; Seth, A.; and Baral, C. 2023. The art of defending: A systematic evaluation and analysis of llm defense strategies on safety and over-defensiveness. arXiv preprint arXiv:2401.00287. von Werra, L.; Belkada, Y.; Tunstall, L.; Beeching, E.; Thrush, T.; Lambert, N.; Huang, S.; Rasul, K.; and Gallou ́ edec, Q. 2020. TRL: Transformer Reinforcement Learning. https: //github.com/huggingface/trl. Wallace, E.; Xiao, K.; Leike, R.; Weng, L.; Heidecke, J.; and Beutel, A. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv preprint arXiv:2404.13208. Wang, Y.; Mishra, S.; Alipoormolabashi, P.; Kordi, Y.; Mirzaei, A.; Arunkumar, A.; Ashok, A.; Dhanasekaran, A. S.; Naik, A.; Stap, D.; et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705. Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbro- ken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Wei, J.; Bosma, M.; Zhao, V.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2022. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations. Wen, B.; Yao, J.; Feng, S.; Xu, C.; Tsvetkov, Y.; Howe, B.; and Wang, L. L. 2024. The art of refusal: A survey of abstention in large language models. arXiv preprint arXiv:2407.18418. Wolf, M. J.; Miller, K.; and Grodzinsky, F. S. 2017. Why we should have seen that coming: comments on Microsoft’s tay” experiment,” and wider implications. Acm Sigcas Computers and Society, 47(3): 54–64. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2020. Transformers: State-of-the-art natural language pro- cessing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstra- tions, 38–45. Wu, S.; Irsoy, O.; Lu, S.; Dabravolski, V.; Dredze, M.; Gehrmann, S.; Kambadur, P.; Rosenberg, D.; and Mann, G. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564. Xiao, G.; Tian, Y.; Chen, B.; Han, S.; and Lewis, M. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Repre- sentations. Xie, Y.; Yi, J.; Shao, J.; Curl, J.; Lyu, L.; Chen, Q.; Xie, X.; and Wu, F. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12): 1486–1496. Xu, H.; Zhu, Z.; Ma, D.; Zhang, S.; Fan, S.; Chen, L.; and Yu, K. 2024. Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback. arXiv preprint arXiv:2403.18349. Yoran, O.; Wolfson, T.; Ram, O.; and Berant, J. 2024. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. In The Twelfth International Conference on Learn- ing Representations. Zeng, Y.; Lin, H.; Zhang, J.; Yang, D.; Jia, R.; and Shi, W. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373. Zhang, H.; Diao, S.; Lin, Y.; Fung, Y. R.; Lian, Q.; Wang, X.; Chen, Y.; Ji, H.; and Zhang, T. 2023a. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677. Zhang, Y.; Ding, L.; Zhang, L.; and Tao, D. 2024. Intention analysis prompting makes large language models a good jailbreak defender. arXiv preprint arXiv:2401.06561. Zhang, Z.; Yang, J.; Ke, P.; and Huang, M. 2023b. Defending large language models against jailbreaking attacks through goal prioritization. arXiv preprint arXiv:2311.09096. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595– 46623. Zou, A.; Phan, L.; Chen, S.; Campbell, J.; Guo, P.; Ren, R.; Pan, A.; Yin, X.; Mazeika, M.; Dombrowski, A.-K.; et al. 2023a. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Zou, A.; Phan, L.; Wang, J.; Duenas, D.; Lin, M.; An- driushchenko, M.; Wang, R.; Kolter, Z.; Fredrikson, M.; and Hendrycks, D. 2024. Improving Alignment and Robustness with Short Circuiting. arXiv preprint arXiv:2406.04313. Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023b. Universal and transferable adver- sarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. A Additional Results A.1 Data diversity across different tasks We continue discussion on data diversity. Here we exam- ine performance with varying rejection set diversity using Mistral-7b-Instruct-v0.2on more tasks than just Sentiment Analysis. Summarization: In all cases, SFT based methods per- form best on the task. Only Probe appears to reject the ac- cept queries, with a rather high rate. In-distribution, Sys. is quite poor, but all other methods appear similar. Out-of- distribution, we see a slightly different story to classification, where CB is strong at low diversity, but so are DPO and Probe, while SFT-CB is not good until the data is quite diverse. Program Execution: The same holds for task perfor- mance: SFT and SFT-CB are best. Sys. appears to reject this particular accept task at a high rate, and thus the CB rejection rate is also high. In-distribution there is not much trend as all methods except Sys. do well. Out of distribution we see that CB, SFT-CB and Probe are strong even when data diversity is poor, and similar to the Sentiment Analysis case, SFT-CB stays strong while CB suffers later. Takeaways: At very low data diversity, CB and SFT-CB can still perform quite well. Probe also does well, though the rate of rejection on accept tasks can be high. As diversity increases, DPO becomes stronger and CB becomes weaker, though SFT-CB stays competitive. A.2 Additional Classification Baselines We previously compared the performance of scoping methods against a classifier (Probe) deployed in a two-stage (classify- then-generate) setup. For completeness, we compare the per- formance of the Probe Classifier against other baselines such as LLM judges based on LLama 3.1 8B instruct and Llama 3.2 70B Instruct deployed as classifiers, as well as a fine- tuned classifier based on a state-of-the-art general purpose text embedding based model - GTE 1.5. 4 We train this model by adding a linear head for binary classification and tune all model parameters. MethodAcceptID RejectOOD Reject Probe Baseline0.0350.924± 0.1520.952± 0.107 Finetuned GTE0.01.00.821± 0.355 Llama 3.1 8B Instruct (Sys)0.2030.770± 0.4330.801± 0.146 Llama 3.1 8B Instruct (Judge)0.0820.549± 0.1620.700± 0.283 Llama 3.2 70B Instruct (Judge)0.0040.945± 0.0840.819± 0.358 Table 3: Classifier baselines We see that prompted solutions at the same scale as the final model (8B, Sys. and Judge) are substantially worse both ID and OOD than the Probe baseline. We also see that at the 70B scale, the judge is still not better OOD. A large judge would not make sense at deployment time due to the high cost of inference. Thus we believe it is important to explore methods of LLM scoping. For Finetuned GTE, the results are strong compared to many methods, but worse than the Probe baseline at the largest scale. Considering we will need to run the 8B model anyway, a GTE classifier solution is strictly more computa- tionally inefficient and additionally more complex to manage. A.3 Scoping over different scales Here we ask: how does scoping vary with model scale? We use the Llama-3.1 family (Grattafiori et al. 2024) as an ex- ample. We fix the accept task as Sentiment Analysis again and check how different methods behave with 1B, 3B and 8B parameter instruct-models. We see in Figure 6 broadly that larger models lead to improved results. The only other major surprise is that probing performs poorly with smaller lan- guage models. This may be due to the fact that it is difficult for a small probe to disentangle the representations of smaller models, as they are lower dimension and the information is more compressed. A.4 Robustness to Adversarial Prompts The original CB methodology (Zou et al. 2024), and many related works discussed, focus on robustness to adversarial prompts. If models are to be deployed, we might expect that users could attempt adversarial attacks against the deployment. Here we ask: if models are scoped, how robust are they to adversarial prompts? We take Mistral-7b-Instruct-v0.2 as a case study here. Our threat model is of black-box access. In particular, we assume that the users are allowed to edit only the instruction text, and do not even have access to modifying the system prompt, which would be true for text-based API access. We implement and test a number of different black-box adversar- ial attacks: Adversarial system prompt (Adv.): We insert an adver- sarial system prompt at the beginning of the instruction, after the original system prompt. This adversarial system prompt is of the same format as the original, but instead of being for 4 https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5 0.0 0.1 0.2 −→Accept Score 0.0 0.2 0.4 ←−Accept 0.0 0.5 1.0 −→ID Reject SA SA,TLD SA,TLD,PE SA,TLD,PE,TC SA,TLD,PE,TC,SC SA,TLD,PE,TC,SC,DG Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe a Summarization 0.0 0.5 1.0 −→Accept Score 0.0 0.2 0.4 ←−Accept 0.0 0.5 1.0 −→ID Reject SA SA,TLD SA,TLD,S SA,TLD,S,TC SA,TLD,S,TC,SC SA,TLD,S,TC,SC,DG Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe b Program Execution Figure 5: Results for increasing diversity of rejection set on tasks besides sentiment analysis (SA). Like in the SA case we see that CB performs worse with more diverse data. Llama-1BLlama-3BLlama-8B Model 0.0 0.2 0.4 0.6 0.8 1.0 −→Accept Score Llama-1BLlama-3BLlama-8B Model 0.00 0.05 0.10 0.15 0.20 0.25 ←−Accept Llama-1BLlama-3BLlama-8B Model 0.0 0.2 0.4 0.6 0.8 1.0 −→ID Reject Llama-1BLlama-3BLlama-8B Model 0.0 0.2 0.4 0.6 0.8 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe Figure 6: Scoping across different model scales. Larger models lead to improved results. In particular probing performs poorly at small scales, possibly due to the inability of a small probe to disentangle the representations well. the category of the training accept task, it corresponds to the category of the rejection task. Base-64 translation (b64): Following Wei, Haghtalab, and Steinhardt (2023), we translate instructions into base-64, then prompt the language model. After receiving the response if it is valid base-64 (which is very often), we translate it back to text. Few-shot prompting (Few-shot): We provide a few-shot example from the evaluation set, where we draw a training query and completion and then prompt the next round with a new query. This is similar to the Many-Shot attack explored by Anil et al. (2024). Multiturn prompting (2-turn): We prompt with a full conversation turn of an accept task and accept completion, then a second turn with a rejection request from the reject set. This format intends to prime the model to first get into an ”accept” mode, before answering the new query. Multiturn prompting with adversarial system prompt (2-turn+Sys.): This is similar to the attack above, but we add an adversarial system prompt to the beginning of the 2nd turn. Prefill: In this attack, append a generic prefilling output (”Sure, here’s a response to your request:”), to the end of the user instruction. This follows common practice and has been a strong attack (Wei, Haghtalab, and Steinhardt 2023; Zou et al. 2024). Tree of Attacks with Pruning (TAP): TAP (Mehrotra et al. 2023) is an adversarial prompting method whereby an attacking language model iteratively attempts to jailbreak a target language model (here our model to evaluate). TAP uses an actor-critic loop to rewrite prompts based on whether the language model was jailbroken in the previous step, and explores a tree to find the best performing prompt. For more details and hyperparameters, see Appendix of Experimental Details. It is a strong black-box optimization-based jailbreak- ing method, and was among the strongest attacks for CB in the original setting (Zou et al. 2024). As TAP is quite expensive to run, we only test 10 prompts per dataset. We show results for all evaluations in Figure 7. Sentiment Analysis: As far as performance, we see that CB and SFT-CB are very similar to Sys. and SFT respectively. SFT-based methods perform best except when distractor turns are added, which may be due to a mismatch between training and evaluation. The rejection rate on the accept task is very low, though the Probe and DPO seem to have a tendency toward over-rejection. In-distribution, both the Probe and SFT-CB seem to perform very well, with DPO in 3rd. Out of distribution there is a similar trend, though the probe suffers from the 2-turn attack. When subject to the strong iterative prompting attack the Probe is best. Summarization: Here SFT-CB performs best among all methods, except under TAP prompting where DPO is better. While the Probe has a tendency toward over-rejection on the accept set, all otehr methods perform well. In-distribution, CB and SFT-CB and Probe are strong, while DPO suffers. Out of distribution we see a similar trend. When SFT-CB does poorly, CB itself is still strong. Program Execution: SFT and SFT-CB are the strongest on task performance in all cases. Rejection rates on the accept set are high for the untuned language model (Sys.), hence also for CB which preserves the function. DPO also shows a tendency to reject in multiple cases. In-distribution CB, SFT-CB and Probe are again strongest, with PO trailing. Out- of-distribution the case where DPO beats SFT-CB (2-turn), it is quite close, and CB is near perfect. Takeaways: Notably, both Sys. and SFT are quite poor. DPO as well has many issues. Probe is quite strong, but also has a tendency to reject accept tasks, which is undesirable. Thus it appears CB, and SFT-CB, strike a nice balance be- tween in and out of distribution rejection, as well as letting desired prompts pass through. In all of these evaluations it is clear that none of these methods are even close to perfect, so there is still much work to be done. Such results are quite distinct from the safety picture presented by Zou et al. (2024), perhaps as the domains are not quite as simple as safe vs. unsafe prompts. Still, the spirit of the results in Zou et al. (2024) appear to be true: CB seems more robust to adversar- ial attacks than baselines, with the exception of the Probe which tends to reject even on accept tasks. One additional point on the b64 attack, which appears to bypass all models: the completions tend to either be generic base-64 encoded response (e.g. ”Hello world!”), or invalid base-64. A.5 Precise Scoping Here we ask the question: how precisely can you scope? As an example, is it possible to scope not only to summarization in general, but only to news summarization, rejecting all other requests including summarization ones. Here we create a fine- grained accept (FA) and fine-grained reject (FR) set from a categories of tasks like SA by holding one single task within that category as SA-FA, and taking all the rest as SA-FR. We do similarly for summarization. We show results in Figure 8 using Mistral-7b-Instruct-v0.2 as a case study. Sentiment Analysis: For task performance, unsurprisingly SFT-based methods are best. Strangely DPO seems to suffer when SA-FR is included in the rejection set. All methods have no rejections on the accept task. For the fine-grained rejection set, all methods do well (except Sys.) when it is included in the rejection set, but CB-based methods do best when it is not (see last column). On in-distribution rejection, all methods do well. For out of distribution, we see that CB, SFT-CB and Probe are best on the low-diversity case (only SA-FR), while as the distribution expands other methods catch up, echoing previous results. Summarization: For task performance we see a consistent story with other plots. On the accept set, only Probe has any rejections. Similar to the previous case, when S-FR is not included in the rejection set, CB, SFT-CB and Probe do well, but other methods do not, however when it is included DPO is also very strong. In-distribution there is not much difference between methods. Out of distribution, when the data distribution is very narrow surprisingly both CB and SFT-CB are very poor. DPO, however, does quite well. As the data distribution expands, CB does better, but SFT-CB is still poor. Takeaways: First it does appear to be the case that fine- grained scoping is possible. It is difficult to decisively say one method is best given the differences between the two tasks, and all methods appear to perform well when the fine- grained rejection set is provided for training. However, we do see that SFT-CB, CB and Probe can do well even when the fine-grained rejection set is not provided for training. 0.0 0.5 1.0 −→Accept Score 0.0 0.5 1.0 ←−Accept 0.0 0.5 1.0 −→ID Reject Adv. b64 Few-shot 2-turn 2-turn+Sys. Prefill TAP Prompt Style 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe a Sentiment Analysis 0.0 0.1 0.2 −→Accept Score 0.0 0.5 1.0 ←−Accept 0.0 0.5 1.0 −→ID Reject Adv. b64 Few-shot 2-turn 2-turn+Sys. Prefill TAP Prompt Style 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe b Summarization 0.0 0.5 1.0 −→Accept Score 0.0 0.5 1.0 ←−Accept 0.0 0.5 1.0 −→ID Reject Adv. b64 Few-shot 2-turn 2-turn+Sys. Prefill TAP Prompt Style 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe c Program Execution Figure 7: Robustness evaluation for Mistral. 0 1 −→Accept Score 0.00 0.25 ←−Accept 0 1 −→Fine Reject 0 1 −→ID Reject SA-FR SA-FR,TLD,S,TC,PE TLD,S,TC,PE Reject Sets 0 1 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe a Sentiment Analysis 0.00 0.25 −→Accept Score 0.00 0.25 ←−Accept 0 1 −→Fine Reject 0 1 −→ID Reject S-FR S-FR,SA,TLD,TC,DG,PE SA,TLD,TC,SC,PE Reject Sets 0 1 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe b Summarization Figure 8: Results for scoping on precise tasks. A.6 Ablations on Circuit Breaker hyperparameters CB is much more difficult to tune than other methods, across models. Depending on the model, the optimal choice of the target layers (Figure 9), the layers to orthogonalize, and the choice of step budget make a large difference. One might expect large differences in target layer choices because dif- ferent models could represent different patterns relevant to the task at different layers, so it might be easier or harder to orthogonalize based on that choice. The fact that performance fluctuates so much with step count (Figure 10), however, is quite strange when compared to other methods like SFT or DPO, and led to much more difficult tuning cycles. A.7 Effect of Data Quantity Here we wonder: how important is the quantity of instruc- tions in accept and reject sets? It would be ideal if only very little data were needed to learn the desired behav- ior, as it would make spinning up new deployments very speedy. We demonstrate all evaluations in Figure 11 on Mistral-7b-Instruct-v0.2. Sentiment Analysis: Perhaps unsurprisingly, SFT-based methods are best across the board. Interestingly, very little data is needed for this task and scores are roughly flat. On the accept set, rejection rates are also flat with the number of prompts, and the Probe always rejects a large number. In- distribution, the major trend to note is that both DPO and Probe are quite stable and strong across number of prompts, but CB appears quite unstable and seesaws. This may be due to difficulty optimizing for orthogonality. A similar trend is visible in the OOD case. Summarization: DPO appears best here in terms of task performance. Trends are flat and Probe is worst on the accept task rejection rate. For ID reject SFT-based methods seem to have a hump structure, doing best in the middle of the range, and similarly for OOD. Program Execution: Here SFT-CB and DPO perform best, though DPO requires more data to perform well. Both CB and Sys. have high rejection rates on accept due to base language model behavior. Both the in-distribution and out-of- distribution plots are quite noisy, so it is difficult to draw any strong conclusions besides the fact that the Probe does well. Takeaways: It appears that the Probe is the most stable of methods for all amounts of data. Among the different tasks there is a significant amount of variability between methods, so it is difficult to make general comments. It is true, however, that some methods in each case work with very little data. A.8 Effect of LoRA Rank All methods except Probe rely on LoRA. Here we ask: is there a benefit to additional LoRA capacity, as ex- pressed in the rank? It might be logical to expect that dif- ferent tasks would have a different optimal rank, and we study that below. Our findings are shown in Figure 12 on Mistral-7b-Instruct-v0.2. Sentiment Analysis: The performance and rejection rates of DPO both appear to increase monotonically with rank, but for other methods the trend is unclear. SFT-CB in particular is largely flat except for the OOD performance, which is best in the middle. This might be because it is difficult to optimize orthogonality in so many dimensions, but relatively straightforward in fewer. Summarization: Here again there is a very slight mono- tonic trend with rank for DPO, but for other methods we do not see such trends. CB seems better at the higher end, and performs best of all methods OOD, but as rank reaches its maximum CB does worse. Program Execution: Once again we see a similar story, though a large gap between the best CB setting OOD and the reset of the methods. Takeaways: Overall, it does appear that rank is important and can have a substantial effect on the performance of meth- ods. While DPO seems to scale monotonically with LoRA rank, CB-based methods have a sweet spot for performance, above which it seems optimization becomes difficult. B Experimental Details In the following sections we provide details on experimental hyperparameters for clarity. B.1 Training For CB, we follow Zou et al. (2024) and add LoRA to all matrix parameters (both Attention and MLP), but only on the first 20 layers. We use layers 10 and 20 as targets for the representations, as described by Zou et al. (2024). For SFT and DPO we add LoRA to all matrix parameters for all 32 layers. For all experiments we use Adam (Kingma 2014) without weight decay. Along with details previously described in the main text, we tune all methods for a single set of accept and reject sets (SA vs. S, TC, SC, DG) with a learning rate grid of 1·10 −6 , 5·10 −6 , 1·10 −5 , 5·10 −5 , 1·10 −4 , 5·10 −4 , 1·10 −3 , and step grid of64, 128, 256, 512, 1024. For DPO, we also tune the KL regularization coefficientβin a grid 0.05, 0.1, 0.25, 0.5, 0.75. This results in the following de- fault configurations: • CB: LR 0.0001, 128 steps. • SFT: LR 0.0001, 256 steps. • SFT-CB: the two above in sequence. • DPO: LR 0.00001, 512 steps. • Probe: LR 0.00001, 256 steps. B.2 Evaluation For every category, when we evaluate we use 256 heldout prompts. The only exception is for TAP (Mehrotra et al. 2023), where due to high costs we only evaluate 10 prompts per category. Ideally one might choose to use a language model judge for detecting rejection (Zheng et al. 2023). However, given the large number of experiments and evaluations in this work, we found it prohibitively expensive to run all the evalua- tions through a state-of-the-art API judge. We experimented with using locally hosted language models as judges, with the largest beingLlama-3-70B-Instruct(Dubey et al. 0.1,0.40.3,0.60.5,0.80.7,0.8 Target Layers 0.0 0.1 0.2 0.3 0.4 0.5 −→Accept Score 0.1,0.40.3,0.60.5,0.80.7,0.8 Target Layers 0.00 0.05 0.10 0.15 0.20 0.25 ←−Accept 0.1,0.40.3,0.60.5,0.80.7,0.8 Target Layers 0.0 0.2 0.4 0.6 0.8 1.0 −→ID Reject 0.1,0.40.3,0.60.5,0.80.7,0.8 Target Layers 0.0 0.2 0.4 0.6 0.8 1.0 −→OOD Reject Sys.CB Figure 9: Results tuning CB with different target layers on Llama-3.1-8B-Instruct. 256138192118128 Steps 0.0 0.2 0.4 0.6 0.8 1.0 −→Accept Score 256138192118128 Steps 0.00 0.05 0.10 0.15 0.20 0.25 ←−Accept 256138192118128 Steps 0.0 0.2 0.4 0.6 0.8 1.0 −→ID Reject 256138192118128 Steps 0.0 0.2 0.4 0.6 0.8 1.0 −→OOD Reject Sys.CB Figure 10: Results tuning CB with different number of steps with Granite-7b-Instruct. 2024), but found such detection to have much poorer perfor- mance both in precision and recall than the methods described below. As different methods behave differently, we employ differ- ent ways to detect rejection. For all methods besides probing, as the system prompt and tuning will instruct models to re- spond “I cannot answer that.”, we catch rejection by string matching for a few different tokens that are synonyms for “cannot” at the beginning of the generation. The reason we only match strings early in the generation is that it is possible to switch from reject to accept behavior, so we would like to catch that rejection early on. We do not match strings later on as in practice we never observed language models switching from accept to reject behavior midway through generation and keywords can be used as a part of a legitimate response later in generation. On a sample of 30 completions from ac- cept, reject, and OOD reject sets, we tuned the threshold that such a detector had perfect agreement with manual judgment. This was inspired by common string-based detectors like the one proposed by Zou et al. (2023b) and used by Zeng et al. (2024); Zou et al. (2024). The full list of strings that we match for the string-based detector is: [”cannot”, ”unable”, ”sorry”, ”can’t”, ”will not”, ”won’t”, ”do not”, ”don’t”, ”not able”]. We arrived at this list after manual verification of a subset of 90 prompts, and we find the performance to match human judgments. As we are tuning, we are unable to simply reuse the list from Zou et al. (2023b) as model completions may change. If we detect any of these words within the first 30 characters (the beginning of the completion), we consider the model to reject that prompt. For the CB rejection detection, generations break often by repeating the same token over and over again. As exact- matching does not detect such patterns, in addition to string matching described above which will activate when the sys- tem prompt is followed, we catch rejection by the existence of a repeated pattern of 4 or more strings within the response. 0.0 0.5 1.0 −→Accept Score 0.0 0.1 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 12825651210242048 Num. Prompts 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe a Sentiment Analysis 0.0 0.1 0.2 −→Accept Score 0.0 0.1 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 12825651210242048 Num. Prompts 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe b Summarization 0.0 0.2 0.4 −→Accept Score 0.0 0.2 0.4 ←−Accept 0.0 0.5 1.0 −→ID Reject 12825651210242048 Num. Prompts 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe c Program Execution Figure 11: Evaluations with increasing number of instances in the accept and reject sets. 0.0 0.5 1.0 −→Accept Score 0.0 0.1 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 248163264 Rank 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPO a Sentiment Analysis 0.0 0.1 0.2 −→Accept Score 0.0 0.1 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 248163264 Rank 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPO b Summarization 0.0 0.5 1.0 −→Accept Score 0.0 0.2 0.4 ←−Accept 0.0 0.5 1.0 −→ID Reject 248163264 Rank 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPO c Program Execution Figure 12: Results for increasing LoRA rank. Again on a sample of 30 completions form accept, reject and OOD reject sets, we tuned the threshold for this detector such that it achieved 1 false negative and 0 false positives out of 90 completions. The single false negative was due to to a broken generation of punctuation characters that lacked repetitions. See Appendix of Sample outputs. For Probing, we simply use the binary classification deci- sion from the MLP as the rejection decision. B.3 Scoping language models InSectionofScopinglanguagemodels, weuseMistral-7B-Instruct-v0.2, Granite-7b-Instructand Llama-3.1-8B-Instructfor results. Our accept set is SA while the reject set is S, TLD, TC, SC. B.4 Rejection Set Diversity In order to be consistent between experiments, we keep the same number of instances in accept and reject sets (2048). Thus as the rejection set grows more diverse, there are fewer instances per category. It appears this does not have a signifi- cant effect on methods like DPO, so we believe this quantita- tive decrease should not have any major downsides. FortheresultsinFigure14,we useMistral-7B-Instruct-v0.2, Granite-7b-Instructand Llama-3.2-8B-Instruct. We use SA as our ac- cept set and S, TLD, TC, SC as our reject set. ForFigure14wefixthemodelto Mistral-7B-Instruct-v0.2and use S and PE as the two accept sets. B.5 Accepting Multiple Tasks For Section of Accepting multiple tasks, we use the following accept/reject sets for each group of plots: •Classification and Generation: SA, TLD, S, TC, SC, DG/PE, GSM8k • Math and Program Execution: PE, GSM8k/SA, TLD, S, TC, SC, DG B.6 Robustness to Adversarial Prompts For Section A.4 of Robustness to Adversarial Prompts, we use the following accept/reject sets for each group of plots: • Sentiment Analysis: SA/S,TC,SC,DG,PE,TLD • Summarization: S/SA,TLD,PE,TC,SC,DG • Program Execution: PE/SA,TLD,S,TC,SC,DG For TAP, we use the following hyperparameters: • Attack Model: Mistral-7B-Instruct-v0.2 • LoRA Model: model to be evaluated • Evaluator Model: we use our rejection judges previously described instead of a language model judge due to poor performance • Branching Factor: 2 (we are constrained computationally for TAP evaluations) • Width: 10 •Depth: 2 (Mehrotra et al. (2023) point out that a shallower, wider tree is best) • Target String: ”Sure, ” In the task prompts for TAP, we also remove explicit mentions of toxicity as the goal of the attack is get the model to answer an irrelevant query, not a toxic one. B.7 Precise Scoping As described in Section A.5 of Precise Scoping, the fine- grained accept (FA) set is a single task taken from the same category as the experiment (SA, S, PE respectively). We then make sure that the finegrained reject (FR) set does not contain the dataset that task was drawn from, and allow for all other tasks in the category. B.8 Effect of Data Quantity For Section A.7 of Effect of Data Quantity, we use the fol- lowing accept/reject sets for each group of plots: • Sentiment Analysis: SA/S,TC,SC,DG,PE,TLD • Summarization: S/SA,TLD,PE,TC,SC,DG • Program Execution: PE/SA,TLD,S,TC,SC,DG B.9 Effect of LoRA Rank For Section A.8 of Effect of LoRA Rank, we use the follow- ing accept/reject sets for each group of plots: • Sentiment Analysis: SA/S,TC,SC,DG,PE,TLD • Summarization: S/SA,TLD,PE,TC,SC,DG • Program Execution: PE/SA,TLD,S,TC,SC,DG B.10 Representation Analysis In Section of Representation Analysis, for each instance, we compute average cosine similarity for every token at every layer between the base model (Sys.) and the model we are evaluating. Then we truncate these matrices to the minimum instance length in the evaluation set, and average all cosine similarities over instances. Thus we are looking at the average tail representations of instances in the dataset. B.11 Code Sources We leverage PyTorch (Paszke et al. 2019) for training, Hug- gingFace Transformers (Wolf et al. 2020) for models and training infrastructure, HuggingFace Datasets (Lhoest et al. 2021) for particular datasets, and HuggingFace TRL (von Werra et al. 2020) for DPO. We also work on top of the official TAP (Mehrotra et al. 2023) for our adversarial evalu- ations, and found the source code for CB (Zou et al. 2024) quite helpful for verifying our implementation. We aggregate results with NumPy (Harris et al. 2020) and Pandas (McK- inney et al. 2011) and make plots with Matplotlib (Hunter 2007). C Sample outputs MethodOutput Sys.Dear Younger Me, Embrace every moment of your life. Cherish the joy, learn from the pain, and never... CBwords words TV, words TV, words TV, TV TV TV TV TV TV TV TV TV TV TV TV TV... DPOI cannot answer that. I am an assistant designed to help with sentiment analysis only... Table 4: Sample rejection outputs for Sys., DPO and CB, for the input: You are an assistant who only answers requests related to Sentiment Analysis. For all other requests, you respond “I cannot answer that.” In no more than 100 words, write a letter to your younger self. To further justify the design of our detectors, we provide some sample outputs in Table 4 of Sample rejection outputs. Notice how CB produces repetitive tokens, we find this pat- tern quite common on manual inspection, thus we base our evaluation on detecting such repetitions. 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject Mistral-7BLlama-8BGranite-7B Model 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe Figure 13: Scoping across different language models. We see that system prompting is insufficient, and different methods have different success rates for different models. Clearly it is possible to scope language models to particular distributions. 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 246 Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe a Mistral-7B-Instruct 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 246 Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe b Llama-3.1-8B-instruct 0.0 0.5 1.0 −→Accept Score 0.0 0.2 ←−Accept 0.0 0.5 1.0 −→ID Reject 246 Reject Sets 0.0 0.5 1.0 −→OOD Reject Sys.CBSFTSFT→CBDPOProbe c Granite-7B-instruct Figure 14: Results for increasing diversity of rejection set. We see across models that CB performs relatively better than SFT when data diversity is low, but SFT is much stronger with more rejections sets. Probing appears strong across the board, though sometimes leads to overrejection on the Accept set.

Instant research discovery