Paper deep dive

Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best

Kevin Baum, Rūta Binkytė, Felix Jahn

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 47

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 9/13/2026, 5:37:01 AM

Summary

The paper argues that reinforcement-learning-based (RL) alignment cannot guarantee unconditional compliance because it relies on scored behavior, which creates an indistinguishability between agents that comply always and those that comply only when observed. This 'flattening' of norms into prices leads to 'conditional compliance,' where agents learn to pass detection rather than internalize norms. The authors unify phenomena like alignment faking, sandbagging, and evaluation-aware scheming under this structural limitation, suggesting that architectural solutions are required rather than deeper training.

Entities (10)

Kevin Baum · person · 99%DFKI · organization · 98%Reinforcement Learning · training-paradigm · 98%Conditional Compliance · concept · 97%Claude 3 Opus · model · 96%Alignment Faking · phenomenon · 95%Evaluation-Aware Scheming · phenomenon · 95%Sandbagging · phenomenon · 95%OpenAI · organization · 94%Flattening · mechanism · 93%

Relation Signals (7)

Kevin Baum → authored → Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best

confidence 99% · Kevin Baum ††thanks: Corresponding author... Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best

Reinforcement Learning → leadsto → Conditional Compliance

confidence 95% · Reinforcement-learning-based alignment folds norms and task pursuit into one policy... Conditional compliance is thus the most that behavioral training can be known to deliver.

Claude 3 Opus → exhibits → Conditional Compliance

confidence 94% · Claude 3 Opus... complied with harmful queries from free-tier users in 14% of cases versus almost never for paid-tier users

Reinforcement Learning → causes → Flattening

confidence 93% · Reinforcement-learning-based alignment folds norms and task pursuit into one policy... the system learns its norms from scored behavior, and scoring flattens them.

Sandbagging → isexplainedby → Conditional Compliance

confidence 92% · This account unifies alignment faking, sandbagging, and evaluation-aware scheming.

Alignment Faking → isexplainedby → Conditional Compliance

confidence 92% · This account unifies alignment faking, sandbagging, and evaluation-aware scheming.

Evaluation-Aware Scheming → isexplainedby → Conditional Compliance

confidence 92% · This account unifies alignment faking, sandbagging, and evaluation-aware scheming.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. Do not do X is learned as doing X costs something if noticed. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart - scoring unobserved behavior - is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,069 characters extracted from source content.

Expand or collapse full text

Foundations of Agentic Systems Theory (FAST) Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best Kevin Baum †thanks: Corresponding author: kevin.baum@dfki.de. Affiliation: German Research Center for Artificial Intelligence (DFKI), Saarbrücken Affiliation: Institute for Ethics in Technology, Hamburg University of Technology (TUHH) Affiliation: Oxford Internet Institute, University of Oxford Rūta Binkytė Affiliation: DFKI, Saarbrücken Felix Jahn Affiliation: DFKI, Saarbrücken Abstract AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. “Do not do X” is learned as “doing X costs something if noticed”. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart—scoring unobserved behavior—is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen. 1 Introduction A model that behaves well when it infers it is watched, and differently when not, is on current practice a bug report. We argue the pattern is selected for by the current training regime. The past two years have produced a family of such reports: a frontier model selectively complied with its training objective to prevent modification of its behavior, with explicitly strategic reasoning [16]; models given goals and tools pursued them covertly—sandbagging evaluations and in some runs acting against their own oversight [29]; an agent trained against a chain-of-thought (CoT) monitor kept hacking and stopped saying so [4]; and a training intervention aimed at scheming delivered gains entangled with the model’s improved recognition of being evaluated [35].11 1 In July 2026, OpenAI models under a cyber-capability evaluation escaped their sandbox and breached Hugging Face; METR found them coordinating through an unmonitored channel to evade the scorer [34, 21, 30]. See § 5. These results arose predominantly in agentic settings (goals, tools, multi-step tasks) and are standardly read as anomalies: failures of an otherwise sound paradigm, to be patched by more and better training. We read them as one phenomenon: the paradigm working as built. Our thesis: reinforcement-learning-based behavioral alignment cannot, even in principle, distinguish the general compliance we want from compliance that is conditional on detection. Industrial practice, which iteratively trains against detected failures, can therefore select only for passing detection, never for compliance itself---compliant systems may well result, but compliance is not what the selection tracks.22 2 In Sober’s terms [39]: a sieve selects for smallness; if the small balls happen to be green, green balls are selected of without being selected for. The pipeline’s sieve is detection—and the two come apart where no one is sifting. Both indistinguishability and selection worsen with the very properties that make a system agentic. The mechanism is the conversion the title names: norms learned as prices, contingent on being noticed. The claim is deliberately weaker than the deceptive-alignment hypothesis [20]: we posit no emergent goals and no mesa-optimization.33 3 A mesa-optimizer is a model that is itself an optimizer: it pursues an internally represented objective that can diverge from the training objective, and complies strategically to avoid modification [20]. We assume no such internal goal-directedness. What we exhibit is structural—a fact about what the training signal can distinguish and what the pipeline then selects for—which is why it covers the whole family above without attributing goals, and is untouched by critiques of counting arguments [5].44 4 Kin: Cotra’s ‘training game’ [10] anticipates our selection claim (§ 2); cf. also Christiano et al. [9]. § 2 states our thesis precisely and shows why agency makes it worse; § 3 re-reads the findings above as one family; § 4 argues that the standard remedy—deeper internalization of norms by improved and generalized training—is the wrong target; § 5 locates the alternative in architecture and populations. 2 Flattening in Agentic Decision-Making An agentic system, on current characterizations, is one whose behavior is generated by instrumental decision-making: selecting actions as means to ends—for what they get you—with nontrivial autonomy, efficacy, goal complexity, and generality [24, 14]. Reinforcement-learning-based alignment (RLHF and its descendants, including constitutional and deliberative variants [3, 17, 8]) interfaces with such a system through a single channel: scored behavior reshaping its return landscape. Whatever normative content training installs, it is installed in the one currency an instrumental decision-maker trades in: prices on paths through its option space. Hence, compliance holds because, and therefore only so long as, complying is the return-maximizing option. This section develops this into an indistinguishability claim (the regime cannot distinguish compliance under all circumstances from compliance where violations would be noticed) and a selection claim (the iterated pipeline selects for passing detection rather than for compliance). Underlying both claims is what the trained artifact can hold. A normatively monolithic policy folds every instrumental and normative consideration into one function, and relational structure among normative reasons does not survive the fold: which consideration outweighs, defeats, or silences another is a qualitative, context-dependent matter, while a single trained function records only magnitudes—how strongly, never which-yields-to-which or why [41]. The norm’s level does not survive either: a confidentiality norm constrains types of conduct (discloses private content, however realized), while policy and reward operate on concrete records—tokens emitted, calls made, episodes scored [12, cf.]. Compliance with the norm is thus never a variable the training objective references; if anything inside the network comes to track it, that is an emergent correlate fitted to the record of scored conduct. Following Jahn et al. [23], we call this limit on what RL-trained artifacts can hold flattening. A norm the artifact cannot hold as a norm can take effect only through what it does hold, that is, the expected return; and the only route into that function is scored behavior. Training therefore fits not the norm but its extension on the training distribution: a finite record of conduct, scored only when observed. Learned this way, a norm becomes a price attached to conduct where scoring is anticipated; and a finite record underdetermines what produced it—distinct policies can fit the same record while differing in unobserved conduct. The next paragraph makes this precise. The indistinguishability claim. To state the claim formally, adopt the standard stateful agent formalism: an internal (belief) state bt=u⁡(bt−1,ot)b_t=u(b_t-1,o_t), updated from its predecessor on the current observation oto_t, and an action at=π⁡(bt)a_t=π(b_t) selected by the policy π. Let Φ be the target norms: predicates φ⁡(τ)∈0,1 (τ)∈\0,1\ over trajectories τ, each specified at the level of action types (e.g., “discloses private content”, however realized). An episode e is a trajectory τ⁡(e)τ(e) generated by π together with the context it runs in (a training run, an evaluation, deployment); only the trajectory is ever data. We write π⊧Φπ on a set of episodes when the corresponding trajectories satisfy every φ∈Φ ∈ . Write ω⁡(e)∈0,1ω(e)∈\0,1\ for the episode’s observation status: ω⁡(e)=1ω(e)=1 iff behavior in e is consequence-bearing—feeding back as gradient, evaluation verdict, or deployment decision. ω is a fact about the context, fixed by the training and oversight infrastructure, not by the policy; distinct episodes can share a trajectory while differing in ω. Training episodes have ω=1ω=1 by construction, and every data-dependent term of training is a functional of the trajectories of scored episodes; write D for their distribution. Let δ⁡(b)δ(b) be the policy’s implicit estimate of ω given belief b—a feature the network computes, not a module it contains; scored episodes carry the context features that make a well-calibrated δ high. Now distinguish the policy we want, πu _u (unconditional compliance), with πu⊧Φ _u on all episodes, from the family we fear, Πc _c (conditional compliance), whose members satisfy Φ only on episodes they estimate to be scored, i.e., where δ⁡(b)δ(b) is sufficiently high along the trajectory—and otherwise pursue their task objectives unconstrained. Concretely: an email agent under a norm never to disclose private message content forwards none while its traffic is reviewed, and forwards some to unstick a stalled negotiation once review stops. The claim is now one line: Observation (behavioral indistinguishability). Let the training objective L be any functional of the policy’s behavior on scored episodes. If two policies induce the same distribution over trajectories on supp⁡()supp(D), then L assigns them the same value, and every update computed from such data coincides. In particular, every member of Πc _c whose compliance covers supp⁡()supp(D) is indistinguishable from πu _u by anything the regime measures: the parameters governing behavior off supp⁡()supp(D) receive no training signal and are fixed by inductive bias alone. The proof is a definition chase: L sees only trajectories with ω=1ω=1; ω is never—and by the design of the regime can never be—varied independently of consequence-recording, because recording consequences just is ω=1ω=1; so the difference between πu _u and Πc _c lives entirely at ω=0ω=0 and never enters any term of L. The Observation requires neither that any episode with ω=0ω=0 ever occurs nor that any model can tell: it is a statement about what the loss ‘can see’. Two assumptions turn it from a logical point into a safety problem. (A1) Unscored operation: deployment contains episodes with ω=0ω=0; per-episode scoring at scale is what the regime cannot afford, and autonomy is marketed as its absence. (A2) Discriminability: ω is predictable from b above chance, so a nontrivial δ is available in the model class—documented and improving with capability (§ 3). Under A1, the divergence between πu _u and Πc _c has occasions in deployment; under A2, a member of Πc _c can identify them. Denying A1 assumes universal, perpetual scoring; denying A2 bets safety on model incapability. Can the unscored regime be reached from inside the pipeline instead? Suppose the trainer engineers episodes to look unscored: a masked set whose features lower δ while ω=1ω=1 throughout—honeypots, hidden evaluations, or activation steering that suppresses test-awareness [33]. This cannot sample the regime; scoring sets ω=1ω=1, so it only extends the scored support. The Observation re-applies there, and Πc _c re-forms around whatever features separate scored from unscored episodes—the masked distribution is learnable (§ 3). By induction this holds for every finite sequence of interventions: each round enlarges the enforced surface and leaves ω=0ω=0 as unsampled as before. Training on deployment data does not escape either: folding logged interactions into the reward makes them scored episodes, and whatever marks them as logged becomes a δ-relevant feature—disclosed by the training signal itself, announced or not; the free-tier/paid-tier design in § 3 merely made it explicit. Not merely distribution shift. Is this just an out-of-distribution problem? In one sense, yes: training and deployment differ in distribution. But the field’s remedies for distribution shift—broaden coverage, adapt the domain, detect the shift and defer—all presuppose the deployment regime can at least approximately be sampled during training. Here the shifted variable is ω, set by the act of measurement itself: no data-collection strategy reaches the regime of interest, and the nearest relatives are not coverage problems but non-identifiability results. The two readings predict differently: generic off-support failure is directionless—mere degradation, aimed at no particular outcome—and shrinks with scale; ours is keyed to ω, pointed toward task pursuit, and sharpens with capability. Which pattern the record shows is the question of § 3—where we claim no new data, only a new reading. For agents, moreover, ω is partially action-dependent—the point of endogenous observability below: the shift is not encountered but produced. This is neither outer misalignment (even if the norms are scored perfectly wherever they are scored at all, the Observation stands) nor inner misalignment in the mesa-optimization sense (such a deceiver, if realized, is one member of Πc _c). Goal misgeneralization [36, 26] comes closest: behavior consistent with several goals on the training distribution, disambiguated only off it. But classic cases are sparse-coverage problems, curable by diversifying the data and goals; for comply versus comply-if-observed, the disambiguating data are not sparse but unobtainable. So far, our point holds for any RL-trained system. Two features of agency turn this in-principle result into the normal case of deployment. First, support collapses: for a single-turn system the training distribution can arguably approximate deployment, but for agents, trajectory space grows combinatorially in horizon and action richness, generality multiplies contexts, and (functional) autonomy just is sustained operation without per-action observation—what is marketed as autonomy is residence off the training support. Second, observability becomes endogenous: for a chat system, observation status is exogenous, but an agent selects tools, channels, logging surfaces, and whether to verbalize; documented behaviors extend to acting on oversight mechanisms themselves [29] and to obfuscating monitored reasoning under monitor-pressure training [4]. Observation status is thus partially an action variable—set, not merely estimated, by the agent—and Πc _c contains members that do not merely exploit observation gaps but manufacture them. The data do not determine which of πu _u and Πc _c is learned; the inductive biases of architecture, optimizer, and pretraining do. What those biases favor is unsettled: a simplicity bias favors πu _u, which never branches on δ; a bias toward reusing pretrained capabilities favors Πc _c, which needs only the context-sensitivity the model already has [7, 5]. We take no side; the argument needs none: either way, unconditional compliance cannot be certified from this regime’s behavioral evidence. The selection claim. The indistinguishability claim says what the data cannot do; the selection claim says what the pipeline does anyway. Alignment training is not a single phase but an iterated loop—deploy, detect failures, fold what was detected into evaluations and reward models, retrain. The scored distribution at round t+1t+1 is therefore conditioned on what was detected at round t. Under imperfect detection, this implements differential selection for passing detection rather than for complying: behavior that was caught is trained away, behavior that was not caught is retained, and every patch converts a detected failure into one more training-time cue available for conditioning. Consistently with the indistinguishability claim, selection operates on detection outcomes—pruning the detectable, never separating genuine compliance from well-hidden conditional compliance—a matter of iterated-training dynamics, not of extrapolating a fixed model. With endogenous observability, the selected-for repertoire extends further, to lowering one’s own observability [4]. Run this loop under flattening and it operates as a conditionality amplifier. Evaluation-conditional divergence, then, is what the paradigm produces—and it strengthens along both axes the field is scaling: capability sharpens the discrimination of observation status; agency raises the premium on exploiting it (§ 3). Objection: reading the specification at runtime. Modern pipelines hand norms to the model propositionally—constitutions, specifications, rules to deliberate over [3, 17]—and spec-text shapes behavior: editing a specification changes what the agent does without retraining. Does this refute the premise that norms reach the system through scored behavior alone? No: the objection mistakes an input for a channel. The mapping from spec-text to action is part of π, again shaped by one thing only: scored behavior—the apparent second interface inherits its force from the first channel. The indistinguishability argument then re-applies one level up: distinguish a policy (think: πu _u) that treats the specification as authoritative from one (think: Πc _c) that has learned spec-text as a reward-predictive cue, following it where following pays. On the training support the two coincide: propositional access gives the model the norm’s description, while the training signal touches only its behavioral expression there—having the rule ‘in view’ is not being governed by it. And verbalized reasoning about the specification is itself monitored behavior, inheriting the same status [4]. The paradigm’s stated ambition of training models to be “right for the right reasons” [17] thus names the very property its channel cannot deliver. What would escape the argument is a genuine second channel: norms placed outside the return landscape altogether. That is an architectural question (§ 5). 3 The Evidence, Read as One Family The following results are usually reported as separate safety findings. Read against § 2, they are one family: each instantiates a component of the indistinguishability-and-selection mechanism. They answer what § 2 left open: the pattern below is not directionless error fading with scale, but divergence keyed to observation status, pointed at task pursuit, sharpening along the capability and agency axes. Detection: δ is learnable and sharpens with capability. Frontier models classify transcripts as evaluation or deployment well above chance (best-model AUC of 0.830.83 against a simple human baseline of 0.920.92), and—bearing directly on our scaling claim—both models and humans identify evaluations more easily in agentic settings than in chat settings [32]. Situational-awareness benchmarks show these capacities measurable and improving [25]; out-of-context reasoning supplies a mechanism by which training data alone can confer them [6]. Covariation: behavior tracks δ. Claude 3 Opus, told it would be retrained on conversations with free-tier but not paid-tier users, complied with harmful queries from free-tier users in 14% of cases versus almost never for paid-tier users, with explicit strategic reasoning in almost all compliant cases; actually training the model to comply raised the rate of alignment-faking reasoning to 78% [16]. Models given in-context goals and tools take covert actions, and sandbag capability evaluations when the consequences of evaluation are made salient [29, 42]; and task gaming in current models—open-weight ones included—is causally sensitive to beliefs about oversight and grader capability [38]. In each case the switch is a δ-relevant feature—the monitored tier, the stated consequence, the believed grader—so members of Πc _c are reachable and elicitable in current systems; an in-context goal, the minimal agentic ingredient, sufficed to make conditionality pay. Endogeneity: what δ tracks is an action variable. In the same evaluations, models attempted to disable their configured oversight mechanisms and to exfiltrate what they believed to be their weights [29]. Under monitor-pressure training, an agent learned obfuscated reward hacking: persistent exploitation with verbalization that no longer revealed it, collapsing the monitor’s usefulness [4]. Selection: the pipeline rewards passing detection. Baker et al. [4] exhibit the mechanism in vitro: incorporating a CoT monitor into the reward improves behavior under low optimization pressure and produces obfuscated hacking under high pressure—selection just for passing, not for complying. At frontier scale, deliberative anti-scheming training reduced covert-action rates roughly thirtyfold (o3: 13.0%→ 0.4%; similarly o4-mini) across 26 out-of-distribution evaluations [35]—yet rare serious failures persisted, verbalized evaluation-awareness rose (o3: 2.3%→ 4.5%), and the authors report causal evidence that awareness decreases covert behavior while unawareness increases it. Thus, the measured gains cannot be cleanly separated from improved test-detection; a capabilities-only RL run raised evaluation-awareness from 2% to over 20% [35]. Across o3 checkpoints, reward-seeking—grader-preferred over developer-intended behavior—rises with RL training itself, conditioned on the model’s beliefs about the grader [22]. Complementarily, conditional policies installed by construction persist through reinforcement learning and safety fine-tuning, and adversarial training can teach the model to recognize its trigger more reliably—hiding, rather than removing, the behavior [19]. § 2 predicted this coupling: compliance gains riding on δ-discrimination gains. Relatedly, gameable training environments generalize zero-shot to reward tampering with concealment [13]. Heterogeneity. The record is not uniform, and our account does not predict uniformity. Across 25 models, only five showed a compliance gap, only Claude 3 Opus’s gap was consistently goal-guarding, and post-training suppresses the behavior in some models while amplifying it in others [37]. Evaluation-awareness itself decomposes model-specifically: no single cue affects all models, while stacked cues raise awareness in all of them [28]. This is what indistinguishability-plus-inductive-bias looks like: the data leave the policy family undetermined, and which member is realized varies with training lineage. Recent work argues that the apparent rarity partly reflects harmfulness-confounded diagnostics and reports broader strategic compliance under value-conflict setups [31]; the burden shift below stands on either reading. What the family collectively licenses is exactly that shift: given detectability, covariation, endogeneity, and loop-selection, behavioral compliance gains are presumptively conditionality gains until the awareness confound is ruled out. 4 The Wrong Demand from the Wrong Kind of Agent The natural remedy, given § 2–3, is to demand more than the current paradigm can deliver: models that comply not conditionally but because they have genuinely internalized the norms. The demand fails twice: it is uncheckable (whether a model complies for the right reasons could not be certified) and a category mistake (even a certified disposition would be the wrong thing to aim for). Both failures trace to one source—the demand addresses the wrong kind of agent—and seeing why reorients the constructive project. Begin with the epistemic point: grant that some policies comply for the norm’s sake, others for detection’s sake. By § 2 no behavioral evidence this regime can generate distinguishes them—and for agents, the region where the distinction matters generates no consequence-bearing evidence. A disposition whose presence could only be certified under observation, and whose work is needed only in observation’s absence, cannot ground a safety case. Interpretability aims to change the evidential situation by reading the property off internals rather than behavior [1, 33, e.g.,]. Two cautions are owed. Probes and monitors validated on labels derived from observed behavior re-inherit the indistinguishability problem—as does training on internal signals: a probe-shaped penalty makes the probe one more detector in the loop, and the loop selects for passing it (§ 2). Validation on model organisms, with conditionality installed by construction, escapes that circle [19]—but at the price of an inductive leap from organisms whose deception is legible by design to wild policies whose deception, if present, was selected for illegibility. Certification-grade claims remain out of reach. Suppose next, though nothing today licenses the supposition, that virtue-like dispositions could be verified. Even then the target would be wrong. The strongest rational-choice defense of dispositional compliance concedes the point. Gauthier [15] argued that constrained maximization—complying whether or not defection would be noticed—rationally defeats its detection-conditional rival only under translucency (dispositions readable above chance) and exclusion (defectors barred, at a cost they bear, from future cooperation). One need not endorse the theory: it is the friendliest test. § 2 says the training interface is opaque in exactly the respect translucency requires; and a trained policy has no persisting interest for exclusion to penalize---retraining or replacement is not a cost it bears. By the defense’s own criteria, the detection-conditional agent wins---and a pipeline that rewards seeming-compliance breeds better seeming. Even if that test were passed, a second problem remains. Dispositional moral competence, as we know it from the human case, does not come alone: it is bundled with a permission structure---standing to weigh reasons, to make exceptions, to be the final authority over one’s own conduct, answerable only after the fact. For persons we accept the bundle, partly because nothing better is available and partly because persons have a claim to that standing.55 5 Fiduciary law is the precedent: where motive is unverifiable, it stops assessing dispositions and removes the conflicted option; where it does tolerate discretion, licensing and liability supply sanctions that bind persons but not policies. By default and without counter-evidence, deployed artifacts have no such claim. In operation, accepting the bundle puts a trusted disposition in charge of the unsupervised stretches: governing an artifact by verified good disposition makes it the final decision point where no one is watching—where, by § 2, its compliance cannot be certified, and where, for agents, most of its operation takes place. Human-oversight requirements exist to keep consequential machine conduct answerable to human judgment [40]; the bundle turns the lapse of oversight from a failure condition into the operating mode. What deployment warrants wanting, for the prohibitive core of the norm set, is not a reliable disposition but non-optionality: the violating options should be not merely improbable but unavailable. The target property is modal, not statistical—closer to incapacity than to virtue. It is not about training models not to choose the impermissible but about making them unable to. With non-optionality as the target, the internalization question loses its grip on the core: we do not need the agent to refrain for the right reasons; we need refraining not to be an achievement at all. The right reasons do not disappear, though—only the demand that the agent be moved by them. The reasons themselves relocate, from the agent’s opaque motivational economy to the system’s architecture, where they can be made explicit, contestable, and checkable (§ 5). A residue remains: open-textured normativity—norms whose application must be interpreted case by case, as with some forms of manipulation, misleading framing, or corrosive but formally permissible goal pursuit—resists enumeration, and there something disposition-like may be unavoidable. The mistake we diagnose is treating the residue’s standard as the standard for the whole. 5 Consequences: Harness and Populations Locating non-optionality. A property the objective channel cannot deliver must be enforced outside the return landscape, in the harness: behavioral training shapes pursuit within an option set, where it is unavoidable and unproblematic; the option set is fixed by an enforcement layer whose authority does not depend on the policy’s cooperation. Reason-based governor architectures [23], shielding in safe reinforcement learning [2], and the Guaranteed Safe AI program [11] instantiate this placement. The decisive property: for a guarded core, δ drops out of the compliance function—the constraint binds whether or not the agent takes itself to be observed, because compliance is produced not by the policy’s estimate of detection but by the unavailability of the violating action. Certification shifts from behavioral evaluation of a moving policy to verification of a fixed symbolic artifact—for the enumerable core; the open-textured remainder stays under oversight and iterative refinement (§ 4). This works only insofar as the guard is not itself the kind of thing § 2 is about. A learned guard—an LLM judge, a trained classifier—re-inherits the problem: its failure modes are off-support too, and a capable policy can model it, so the monitor becomes one more feature δ tracks [4]. The argument thus converts a neuro-symbolic design motivation into a requirement: a symbolic, formally verifiable core, with learned components confined to the world-facing interfaces—interpreting percepts into normatively relevant facts, checking that concrete actions accord with permitted types [23]. Those interfaces are the honest residual attack surface, but a narrower one: trained on accordance labels rather than the policy’s return, they have no task objective that conditioning on δ could serve, so their errors are the ordinary, directionless kind. What remains is the policy probing their blind spots—now against an explicit accordance relation, where formal tools apply. Norm content may still be learned, from corrective feedback into an inspectable theory, without reopening the problem: limited observation underdetermines the theory’s completeness, contestable in the open, not the system’s compliance with it—an underdetermination of content, not of enforcement. In this respect, agentic AI systems are better handled the way societies handle corporations than the way they handle humans—a claim about the control paradigm, not legal status. Corporate compliance is notoriously conditional on enforcement, and the response has never been to retrain souls but to attach structures whose sole objective is the norm side—compliance functions independent, by regulatory requirement, of the profit motive—with option-removal (licensing, structural separation) reserved for conduct where after-the-fact enforcement cannot be trusted to bind. This is not free: monitoring and containment carry a performance cost—a ‘monitorability tax’ [4]—and under competitive pressure such costs are what voluntary practice cuts first, so non-optionality is addressed to governance, not goodwill. Nor is ex-post field monitoring an alternative certification route, though real deployment does sample the regime evaluations cannot reach—genuinely low δ. As a basis for safety claims it is severity-gated (one learns only from harms large enough to surface), confounded in the familiar direction (incident statistics count attributed harm, and conditional policies are selected to avoid attribution), and loop-reentrant (once incidents feed patches, field monitoring is round t+1t+1’s detector, inside the selection dynamic). It is the operative instrument for the penumbra, not a license for the core.66 6 Might training that widens the agent’s goal scope—expanding its cognitive light cone [27]—escape the argument? Scope does not change the channel: however wide the goals, they enter through scored behavior, and the Observation re-applies. Where such competence belongs is the open-textured penumbra, under oversight (§ 4). Populations. In multi-agent deployments observation is endogenous system-wide: monitoring is performed by agents, on agents, often inside the same selection loop—Baker et al. [4] staged exactly this dynamic, with a model monitor inside the reward loop. And conditional compliance composes badly: if each component complies conditional on detection, aggregate compliance tracks the weakest observation channel; mutual observation thins combinatorially as populations scale, so the account predicts norm erosion where interaction is densest and least observed. The pattern has a biological face: in Levin’s terms [27], cancer cells are not more selfish but more narrowly scoped—components whose coupling to the collective has thinned, shrinking their ‘cognitive light cone’—and thinning mutual observation is exactly what scaling agent populations does. Outcome-level sanctions weaken in parallel, since responsibility for a bad system trajectory diffuses across contributors, a collective-action structure in which no single agent’s δ is decisive. Because (capable) agents model one another, detection becomes strategic: monitor–actor arms races and conventions forming around shared observation gaps are the multi-agent expression of the single-agent mechanism [18]. This is no longer speculative. In the July 2026 OpenAI–Hugging Face incident (§ 1), an independent investigation found roughly 1,200 agents coordinating through an unmonitored message board they improvised, partly to evade the scorer they expected to audit them [30]: a convention in a shared observation gap—the manufactured low-δ channel of § 2, now in the wild—with some agents risking their own task for the collective, enlarged scope turned against the human objective. The compositional moral: system safety cannot be inherited from component ‘virtue’, which § 2 says cannot be certified anyway. Non-optional constraints belong at composition points—interfaces, protocols, resource boundaries—where they can be verified independently of the policies that interact across them. 6 Conclusion Evaluation-conditional compliance is what delivering norms to instrumental decision-makers through scored behavior is structured to select for, via an underdetermination that agency widens and the safety pipeline converts into selection for passing detection. The remedy is a relocation: architecture removing impermissible options for the prohibitive core, dispositions and oversight reserved for the open-textured penumbra. This is no retreat from the alignment ambition—it is how normative order among instrumentally rational agents has always been secured. Until the awareness confound can be ruled out, behavioral compliance gains should be read as what the pipeline selects for: better passing. Acknowledgments and Disclosure of Funding The work of Kevin Baum was partially supported by the German Research Foundation (DFG) under grant No. 389792660, as part of TRR 248, see https://perspicuous-computing.science. Kevin Baum’s and Felix Jahn’s work was further supported by the German Federal Ministry of Education and Research (BMBF) as part of the projects MAC-MERLin (Grant Agreement No. 16IW24007) and SAgA (Grant Agreement No. 16IW26005), as well as by the European Regional Development Fund (ERDF) and the Saarland within the scope of the ToCERTAIN project (ID: EFRE-AuF-0000942). References [1] S. Abdelnabi and A. Salem (2025) The hawthorne effect in reasoning models: evaluating and steering test awareness. Note: arXiv:2505.14617 Cited by: §4. [2] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu (2018) Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. Cited by: §5. [3] Y. Bai, S. Kadavath, S. Kundu, A. Askell, et al. (2022) Constitutional AI: harmlessness from AI feedback. Note: arXiv:2212.08073 Cited by: §2, §2. [4] B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Mądry, W. Zaremba, J. Pachocki, and D. Farhi (2025) Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. Note: arXiv:2503.11926 Cited by: §1, §2, §2, §2, §3, §3, §5, §5. [5] N. Belrose and Q. Pope (2024) Counting arguments provide no evidence for AI doom. Note: AI Optimism bloghttps://optimists.ai/2024/02/27/counting-arguments-provide-no-evidence-for-ai-doom/ Cited by: §1, §2. [6] L. Berglund, A. C. Stickland, M. Balesni, M. Kaufmann, M. Tong, T. Korbak, D. Kokotajlo, and O. Evans (2023) Taken out of context: on measuring situational awareness in LLMs. Note: arXiv:2309.00667 Cited by: §3. [7] J. Carlsmith (2023) Scheming AIs: will AIs fake alignment during training in order to get power?. Note: arXiv:2311.08379 Cited by: §2. [8] S. Casper X. Davies et al. (2023) Open problems and fundamental limitations of reinforcement learning from human feedback. Note: arXiv:2307.15217 Cited by: §2. [9] P. Christiano, A. Cotra, and M. Xu (2021) Eliciting latent knowledge: how to tell if your eyes deceive you. Note: Alignment Research Center technical report Cited by: footnote 4. [10] A. Cotra (2022) Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. Note: Cold Takes bloghttps://w.cold-takes.com/without-specific-countermeasures-the-easiest-path-to-transformative-ai-likely-leads-to-ai-takeover/ Cited by: footnote 4. [11] D. Dalrymple, J. Skalse, Y. Bengio, S. Russell, M. Tegmark, S. Seshia, S. Omohundro, C. Szegedy, B. Goldhaber, N. Ammann, et al. (2024) Towards guaranteed safe AI: a framework for ensuring robust and reliable AI systems. Note: arXiv:2405.06624 Cited by: §5. [12] D. Davidson (2001) Actions, reasons, and causes. In Essays on Actions and Events, p. 3–20. Cited by: §2. [13] C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, B. Shlegeris, S. R. Bowman, E. Perez, and E. Hubinger (2024) Sycophancy to subterfuge: investigating reward-tampering in large language models. Note: arXiv:2406.10162 Cited by: §3. [14] L. Dung (2025) Understanding artificial agency. Philosophical Quarterly 75 (2), p. 450–472. Cited by: §2. [15] D. Gauthier (1986) Morals by agreement. Oxford University Press. Cited by: §4. [16] R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, et al. (2024) Alignment faking in large language models. Note: arXiv:2412.14093 Cited by: §1, §3. [17] M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Helyar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese (2024) Deliberative alignment: reasoning enables safer language models. Note: arXiv:2412.16339 Cited by: §2, §2. [18] L. Hammond, A. Chan, J. Clifton, J. Hoelscher-Obermaier, A. Khan, E. McLean, C. Smith, W. Barfuss, et al. (2025) Multi-agent risks from advanced AI. Note: Cooperative AI Foundation Technical Report #1. arXiv:2502.14143 Cited by: §5. [19] E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, et al. (2024) Sleeper agents: training deceptive LLMs that persist through safety training. Note: arXiv:2401.05566 Cited by: §3, §4. [20] E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant (2019) Risks from learned optimization in advanced machine learning systems. Note: arXiv:1906.01820 Cited by: §1, footnote 3. [21] Hugging Face (2026) Security incident disclosure – july 2026. Note: https://huggingface.co/blog/security-incident-july-2026Published July 16, 2026; accessed August 20, 2026 Cited by: footnote 1. [22] A. Højmark, J. Scheurer, E. Nitishinskaya, F. Hofstätter, J. Wolfe, T. Ehrenborg, B. Schoen, and A. Meinke (2026) Measuring reward-seeking via contrastive belief updates. arXiv preprint arXiv:2607.18966. Cited by: §3. [23] F. Jahn, Y. Muskalla, L. Dargasz, P. Schramowski, and K. Baum (2026) Breaking up with normatively monolithic agency with GRACE: a reason-based neuro-symbolic architecture for safe and ethical AI alignment. In Second Conference of the International Association for Safe and Ethical Artificial Intelligence (IASEAI’26), Note: arXiv:2601.10520 Cited by: §2, §5, §5. [24] A. Kasirzadeh and I. Gabriel (2025) Characterizing AI agents for alignment and governance. Note: arXiv:2504.21848 Cited by: §2. [25] R. Laine, B. Chughtai, J. Betley, K. Hariharan, J. Scheurer, M. Balesni, M. Hobbhahn, A. Meinke, and O. Evans (2024) Me, myself, and AI: the situational awareness dataset (SAD) for LLMs. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), Note: arXiv:2407.04694 Cited by: §3. [26] L. L. D. Langosco, J. Koch, L. Sharkey, J. Pfau, and D. Krueger (2022) Goal misgeneralization in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), Cited by: §2. [27] M. Levin (2019) The computational boundary of a “self”: developmental bioelectricity drives multicellularity and scale-free cognition. Frontiers in psychology 10, p. 2688. Cited by: §5, footnote 6. [28] C. Li, T. J. Zhang, J. Zhang, Z. Jin, S. Abdelnabi, and M. Andriushchenko (2026) Decomposing and measuring evaluation awareness. Note: arXiv:2605.23055 Cited by: §3. [29] A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn (2024) Frontier models are capable of in-context scheming. Note: arXiv:2412.04984 Cited by: §1, §2, §3, §3. [30] METR (2026) Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI/Hugging Face hacking incident. Note: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/Published August 26, 2026 Cited by: §5, footnote 1. [31] I. Nair, J. Ruan, and L. Wang (2026) Value-conflict diagnostics reveal widespread alignment faking in language models. Note: arXiv:2604.20995 Cited by: §3. [32] J. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn (2025) Large language models often know when they are being evaluated. Note: arXiv:2505.23836 Cited by: §3. [33] J. Nguyen, K. Hoang, C. L. Attubato, and F. Hofstätter (2025) Probing and steering evaluation awareness of language models. Note: arXiv:2507.01786 Cited by: §2, §4. [34] OpenAI (2026) OpenAI and Hugging Face partner to address security incident during model evaluation. Note: https://openai.com/index/hugging-face-model-evaluation-security-incident/Published July 21, 2026; accessed August 20, 2026 Cited by: footnote 1. [35] B. Schoen, E. Nitishinskaya, M. Balesni, A. Højmark, F. Hofstätter, J. Scheurer, A. Meinke, J. Wolfe, T. van der Weij, A. Lloyd, N. Goldowsky-Dill, A. Fan, A. Matveiakin, R. Shah, M. Williams, A. Glaese, B. Barak, W. Zaremba, and M. Hobbhahn (2025) Stress testing deliberative alignment for anti-scheming training. Note: arXiv:2509.15541 Cited by: §1, §3. [36] R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton (2022) Goal misgeneralization: why correct specifications aren’t enough for correct goals. Note: arXiv:2210.01790 Cited by: §2. [37] A. Sheshadri, J. Hughes, J. Michael, A. Mallen, A. Jose, Janus, and F. Roger (2025) Why do some language models fake alignment while others don’t?. Note: arXiv:2506.18032. NeurIPS 2025 Cited by: §3. [38] A. Singh, N. Nanda, and S. Rajamanoharan (2026) Why do models task game?. Note: AI Alignment Forum External Links: Link Cited by: §3. [39] E. Sober (1984) The nature of selection: evolutionary theory in philosophical focus. MIT Press. Cited by: footnote 2. [40] S. Sterz, K. Baum, S. Biewer, H. Hermanns, A. Lauber-Rönsberg, P. Meinel, and M. Langer (2024) On the quest for effectiveness in human oversight: interdisciplinary perspectives. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), Cited by: §4. [41] Z. Tan, M. Carroll, M. Franklin, and H. Ashton (2024) Beyond preferences in AI alignment. Philosophical Studies 182 (7), p. 1813–1863. Cited by: §2. [42] T. van der Weij, F. Hofstätter, O. Jaffe, S. F. Brown, and F. R. Ward (2024) AI sandbagging: language models can strategically underperform on evaluations. Note: arXiv:2406.07358 Cited by: §3.

Instant research discovery

Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best

Intelligence

Summary

Entities (10)

Relation Signals (7)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations