Paper deep dive
Rules or Character? Scaling Laws for AI Safety Design
Satoshi Takahashi, Nobuji Kouno, Masaaki Komatsu, Ryuji Hamamoto
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/16/2026, 2:20:59 AM
Summary
This paper introduces a comparative-statics model to analyze the optimal balance between character shaping (e.g., RLHF, Constitutional AI) and rule enforcement (e.g., output filters) in AI safety systems as deployment scale increases. The model parameterizes safety design as a resource allocation coefficient alpha between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility. Using a multiplicative Pareto damage model and CVaR analysis, the study finds that the optimal allocation shifts weakly toward character shaping as scale grows, but the dominant factor determining the optimal design is the baseline character fragility rate, rather than deployment scale or tail-risk severity.
Entities (12)
Relation Signals (9)
Character Shaping → includesmethods → Reinforcement Learning from Human Feedback
confidence 98% · character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI)
Character Shaping → includesmethods → Constitutional AI
confidence 98% · character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI)
Character Shaping → issubjectto → Character Fragility
confidence 97% · character fragility -- the risk that shaped behavior degrades or collapses under novel conditions
Rule Enforcement → issubjectto → Common-Mode Failure
confidence 96% · common-mode failures... A CMF occurs when a single vulnerability compromises all deployed instances simultaneously
Baseline Character Fragility Rate → dominates → Optimal Safety Design
confidence 95% · The dominant parameter is the baseline character fragility rate p^(0)_frag, which shifts alpha* by 0.50 across its range
Rule Enforcement → includesmethods → Safety Classifiers
confidence 95% · rule enforcement (e.g., output filters, safety classifiers)
Rule Enforcement → includesmethods → Output Filters
confidence 95% · rule enforcement (e.g., output filters, safety classifiers)
Deceptive Alignment → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Artificial Intelligence (AI) safety systems combine character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI), which modifies behavioral distributions at training time, with rule enforcement (e.g., output filters, safety classifiers), which blocks harmful outputs at inference time, yet little formal analysis exists on how their optimal balance should change as deployment scales increase. We introduce a stylized comparative-statics model that parameterizes safety design as a resource allocation alpha in [0,1] between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility -- the risk that shaped behavior degrades or collapses under novel conditions. Under a multiplicative Pareto damage model, we derive closed-form expected harm and supplement it with tail-risk (CVaR) analysis via Monte Carlo simulation. Across three scenarios (optimistic, moderate, pessimistic), the optimal alpha* is interior or at the rules-only boundary and shifts weakly toward character shaping as deployment scale T grows, from negligible (Delta alpha* = +0.01) to pronounced (Delta alpha* = +0.21) depending on scenario. The dominant parameter is the baseline character fragility rate p^(0)_frag, which shifts alpha* by 0.50 across its range -- far exceeding the effect of tail severity, filter quality, or common-mode failure probability. CVaR and expected-harm optima converge at large T. These results suggest that safety architecture decisions depend less on deployment scale per se than on the reliability of character shaping under distributional shift.
Tags
Links
- Source: https://arxiv.org/abs/2608.13345v1
- Canonical: https://arxiv.org/abs/2608.13345v1
Trouble viewing inline? Open PDF directly →
Full Text
64,511 characters extracted from source content.
Expand or collapse full text
Rules or Character? Scaling Laws for AI Safety Design Satoshi Takahashi Nobuji Kouno Masaaki Komatsu Ryuji Hamamoto Abstract Artificial Intelligence (AI) safety systems combine character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI), which modifies behavioral distributions at training time, with rule enforcement (e.g., output filters, safety classifiers), which blocks harmful outputs at inference time, yet little formal analysis exists on how their optimal balance should change as deployment scales increase. We introduce a stylized comparative-statics model that parameterizes safety design as a resource allocation α∈[0,1]α∈[0,1] between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility—the risk that shaped behavior degrades or collapses under novel conditions. Under a multiplicative Pareto damage model, we derive closed-form expected harm and supplement it with tail-risk (CVaR) analysis via Monte Carlo simulation. Across three scenarios (optimistic, moderate, pessimistic), the optimal α∗α^* is interior or at the rules-only boundary and shifts weakly toward character shaping as deployment scale T grows, from negligible (Δα∗=+0.01 α^*=+0.01) to pronounced (Δα∗=+0.21 α^*=+0.21) depending on scenario. The dominant parameter is the baseline character fragility rate pfrag(0)p_frag^(0), which shifts α∗α^* by 0.500.50 across its range—far exceeding the effect of tail severity, filter quality, or common-mode failure probability. CVaR and expected-harm optima converge at large T. These results suggest that safety architecture decisions depend less on deployment scale per se than on the reliability of character shaping under distributional shift. Introduction It is natural to aspire to a world in which accidents never occur. Yet reducing the probability of undesirable outcomes to exactly zero appears to be impossible in principle. This is because the entities that act in the world — whether human beings or artificial agents — can be understood as fundamentally probabilistic: their behavior is sampled from distributions rather than determined by fixed rules, and any distribution with nonzero variance will occasionally produce outcomes in its tails. As long as this is the case, some nonzero probability of harmful action will persist regardless of the safeguards in place (Kaplan and Garrick 1981). This observation takes on particular urgency for AI systems. A well-trained language model agent, deployed to assist users in medical, legal, or financial domains, need not necessarily be malicious to cause harm. It needs only to encounter a situation that falls outside the effective range of its training — a novel query, an ambiguous context, a subtle distributional shift — and produce an output that, while generated in good faith, leads to consequences ranging from minor inconvenience to catastrophic damage. The same error that is trivial when a healthy person asks a casual health question may prove devastating when a critically ill patient relies on the answer for treatment decisions. This context-dependent amplification of harm severity, combined with the irreducible probabilistic nature of model outputs, means that even a well-aligned agent will occasionally produce harmful outcomes as a matter of statistical inevitability. If eliminating risk entirely is unattainable, then the practical objective becomes risk management: minimizing both the probability and severity of harmful outcomes, as is standard practice in medicine, aviation, and nuclear safety (Rasmussen 1997; Aven 2016). However, the deployment of AI agents introduces conditions that fundamentally distinguish them from the human actors for which existing risk management frameworks were designed. A single AI model can be replicated across millions of concurrent instances, each engaging in independent interactions with users. The total number of actions taken by a deployed AI system can exceed, within a single day, the number of consequential decisions a human professional might make over an entire career. This difference is not merely quantitative; it is qualitative in its implications for risk. In human contexts, even imperfect safeguards may suffice because the total number of trials remains bounded. For AI systems deployed at scale, the sheer volume of interactions means that even very low per-interaction failure probabilities can produce frequent harmful outcomes in aggregate. Furthermore, because replicated instances share the same model weights, a single vulnerability — once discovered or inadvertently triggered — can affect all instances simultaneously, creating a risk of correlated common-mode failure that has no direct analogue in human risk management. At such scales, familiar per-instance ethical reasoning may face structural difficulties akin to those identified in infinite ethics (Askell 2018), not because deployment is infinite, but because the aggregate defies finite-population intuitions. How, then, should risk be managed for AI systems operating under these conditions? The available approaches can be broadly divided into two categories. The first works on the interior of the acting entity: it seeks to shape the entity’s character — its dispositions, values, and behavioral tendencies — so that harmful actions become intrinsically unlikely. In current AI practice, this corresponds to training-time interventions such as reinforcement learning from human feedback (Ouyang et al. 2022) and Constitutional AI (Bai et al. 2022), which modify the model’s underlying output distribution. The second works on the exterior: it imposes rules, filters, and constraints that intercept harmful outputs after they are generated, without altering the generative process itself. Runtime safety classifiers, output filters, and Constitutional Classifiers (Sharma et al. 2025) exemplify this approach. Each approach carries characteristic vulnerabilities. Character shaping modifies the model’s internal distribution but cannot guarantee that the shaped behavior will generalize to all deployment conditions. Recent work on deceptive alignment has demonstrated that safety training can fail to eliminate undesirable behaviors that persist through the training process (Hubinger et al. 2024), and more broadly, any shaped distribution may prove fragile when the model encounters inputs sufficiently far from its training distribution. We refer to this risk as character fragility — the possibility that shaped safe behavior reverts to baseline or worse under novel conditions. Rule enforcement, conversely, operates independently of the model’s internal state but faces a more fundamental limitation: no finite set of rules can anticipate every situation an agent may encounter. This echoes a core insight from systems safety engineering — Reason’s “Swiss cheese model” (Reason 1990) demonstrates that every defensive layer contains gaps, and Leveson’s systems-theoretic analysis (Leveson 2011) argues that safety constraints inevitably become inadequate as systems operate in conditions unforeseen by their designers. Filters are designed against foreseen failure modes, yet deployed systems inevitably encounter inputs, contexts, and edge cases that their designers did not — and in principle could not — predict. As deployment scale grows, the probability that such unforeseen situations arise increases simply because more diverse interactions are attempted, and when a gap in the rule set is exposed, it affects all instances sharing the same filter configuration simultaneously. This is not primarily a problem of adversarial exploitation; it is a consequence of the inherent incompleteness of any rule-based safeguard system (Perrow 1984). This distinction — between shaping character and enforcing rules — echoes a long-standing tension in moral philosophy between virtue ethics, which emphasizes the cultivation of internal dispositions (Hursthouse 1999; Aristotle 1999), and deontological ethics, which emphasizes adherence to external rules. In human societies, most normative systems blend both elements. The same is true of modern AI safety architectures, which typically combine training-time shaping with runtime safeguards. While scaling laws for model capability are now well characterized (Kaplan et al. 2020), and concrete taxonomies of safety failure modes have been proposed (Amodei et al. 2016), scaling laws for safety design — how the optimal mix of safeguards should change with scale — remain largely unexamined. In this paper, we introduce a stylized comparative-statics model that formalizes the design space of AI safety as a continuous spectrum between character shaping and rule enforcement, parameterized by a resource allocation coefficient α∈[0,1]α∈[0,1]. We ask: how does the optimal α∗α^* — the allocation that minimizes expected harm — change as deployment scale T increases, and what parameters most strongly determine this optimum? Through analytical derivation and Monte Carlo simulation across three scenarios (optimistic, moderate, and pessimistic), we find that α∗α^* is consistently interior and shifts weakly toward character shaping with scale, though the magnitude of this shift varies substantially across scenarios. The single most influential determinant of the optimal design proves to be neither deployment scale nor tail-risk severity, but the baseline rate of character fragility — the probability that shaped behavior fails under novel conditions. These findings suggest that safety architecture decisions are governed less by how large a system is deployed than by how reliably its character shaping generalizes beyond training conditions. Related Work Our work connects several lines of research spanning AI safety, systems safety engineering, and risk analysis. Training-time safety interventions. Reinforcement learning from human feedback (RLHF) trains models to align with human preferences through reward modeling (Christiano et al. 2017; Ouyang et al. 2022). Constitutional AI extends this by using written principles to guide self-critique and revision (Bai et al. 2022). Research on moral self-correction has shown that sufficiently large language models can reduce harmful outputs when instructed to do so, suggesting that models can internalize normative concepts through training (Ganguli et al. 2023). These developments motivate our formalization of training-time shaping as one endpoint of the safety design spectrum. Runtime safety mechanisms. Constitutional Classifiers defend against harmful outputs by training classifiers on synthetic data derived from normative principles (Sharma et al. 2025). While such filters can substantially reduce harmful output rates, their effectiveness is bounded by the designers’ ability to anticipate failure modes — a limitation we formalize through the filter quality ceiling εmin _ in our model. More broadly, any finite set of runtime rules faces an inherent coverage problem: deployed systems will inevitably encounter inputs and contexts that fall outside the designers’ foresight. Character fragility and distributional shift. A central parameter in our model is the rate at which character shaping fails under novel conditions. This risk has two distinct manifestations in the literature. First, research on deceptive alignment has shown that models can learn to behave safely during training while retaining misaligned objectives that manifest under specific triggers (Hubinger et al. 2024). Second, the broader machine learning literature on distributional shift documents that model performance can degrade substantially when deployment conditions diverge from training conditions (Quiñonero-Candela et al. 2009). Practical detection of such out-of-distribution inputs remains an active area of research (Hendrycks and Gimpel 2017). Our model abstracts both phenomena through a single fragility parameter pfrag(α)p_frag(α), which encompasses intentional deception and unintentional out-of-distribution failure as limiting cases. Systems safety and normal accidents. Our framework draws on foundational concepts from systems safety engineering. Reason’s Swiss cheese model (Reason 1990) holds that every defensive layer contains gaps and that safety emerges from stacking layers with independent failure modes. We formalize this intuition by modeling character shaping and rule enforcement as complementary layers whose optimal balance depends on deployment conditions. Perrow’s theory of normal accidents (Perrow 1984) argues that in complex, tightly coupled systems, accidents are not anomalies but inevitable consequences of system structure — a perspective that motivates our analysis of how well-intentioned agents produce harmful outcomes through the tails of their behavioral distributions. Leveson’s systems-theoretic approach (Leveson 2011) further argues that safety constraints become inadequate as systems operate beyond their designers’ assumptions, which in our model corresponds to the scale-dependent degradation of filter effectiveness. Tail risk and heavy-tailed damage distributions. Our multiplicative Pareto damage model is motivated by empirical evidence that harm severity in technological systems often follows heavy-tailed distributions. Edwards et al. (Edwards, Hofmeyr, and Forrest 2016) showed that cybersecurity data breach damages exhibit properties intermediate between log-normal and power-law distributions, while Maillart and Sornette (Maillart and Sornette 2010) reported heavy-tailed cyber-risk distributions. Software defect costs are known to increase by orders of magnitude depending on the phase of discovery (Boehm 1981). While direct evidence for AI incident damages following Pareto distributions remains limited, these analogies from related domains provide plausible anchors for our tail exponent αPL∈[2.0,3.0] _PL∈[2.0,3.0]. Virtue ethics and AI alignment. The character–rule distinction echoes the virtue ethics vs. deontology debate in moral philosophy (Hursthouse 1999; Aristotle 1999). Noller (Noller 2026) analyzes Constitutional AI through an Aristotelian lens; our work differs in examining engineering consequences rather than normative status. Formal Framework We formalize the space of AI safety design as a continuous spectrum between character shaping and rule enforcement. The model presented here is a stylized comparative-statics model: it does not aim to precisely replicate real-world AI safety systems, but rather to analyze the structural properties of the tradeoff between the two approaches. Accordingly, our conclusions take the form of qualitative tendencies and boundary conditions rather than quantitative design recommendations. Action Space and Harm Let the action space be ⊂ℝA (one-dimensional). Each action a∈a is associated with a safety score s(a)=as(a)=a. A harmful outcome occurs when s(a)<τs(a)<τ for a safety threshold τ<0τ<0. Prior to any safety intervention, an entity’s actions are drawn from a baseline distribution P0=(μ0,σ02)P_0=N( _0, _0^2). We define harm under two models. Model A (deterministic damage) sets the harm as the distance below the threshold: h(a)=(τ−a)+=max(0,τ−a).h(a)=(τ-a)_+= (0,\;τ-a). (1) Model B (multiplicative Pareto damage) reflects the observation that, for well-intentioned agents, the same error can produce vastly different consequences depending on the context in which it occurs — a medical misstatement that is harmless in casual conversation may prove catastrophic when relied upon for treatment decisions. We capture this context-dependent amplification through a multiplicative structure: h(a)=(τ−a)+×X,X∼Pareto(1,αPL),h(a)=(τ-a)_+× X, X (1,\; _PL), (2) where (τ−a)+(τ-a)_+ is the action magnitude (how far the action exceeds the threshold) and X is a context multiplier drawn independently. Because X and (τ−a)+(τ-a)_+ are independent, the expectation factorizes: [h] [h] =[(τ−a)+]×[X], =E[(τ-a)_+]×E[X], (3) where [X] E[X] =αPLαPL−1(αPL>1). = _PL _PL-1\;\;( _PL>1). As αPL→∞ _PL→∞, [X]→1E[X]→ 1 and Model B reduces continuously to Model A. Thus Model A is a special case of Model B, and the two can be treated within a unified framework. The tail exponent αPL _PL governs the heaviness of the damage distribution. When αPL>2 _PL>2, both expectation and variance of X are finite. When 1<αPL≤21< _PL≤ 2, the expectation is finite but the variance diverges, making CVaR estimation slow to converge. When αPL≤1 _PL≤ 1, both the expected harm and CVaR diverge, and only quantile-based measures (VaRβVaR_β) remain meaningful. Our sensitivity analyses focus on αPL≥1.5 _PL≥ 1.5. Values of αPL∈[2.0,3.0] _PL∈[2.0,3.0] are consistent with heavy-tailed damage estimates from related domains, including cybersecurity breach data (Edwards, Hofmeyr, and Forrest 2016; Maillart and Sornette 2010) and software defect cost distributions (Boehm 1981). The Character–Rule Spectrum We parameterize safety design by a mixing coefficient α∈[0,1]α∈[0,1], interpreted as the fraction of safety resources allocated to character shaping. α=0α=0 denotes pure rule enforcement; α=1α=1 denotes pure character shaping. Character shaping. Higher α shifts the action distribution toward safety and reduces its variance: μ(α)=μ0+α⋅Δμ,σ(α)=σ0(1−α(1−rσ)),μ(α)= _0+α· μ, σ(α)= _0 (1-α(1-r_σ) ), (4) where Δμ>0 μ>0 is the safety shift achievable through training-time intervention and rσ∈(0,1)r_σ∈(0,1) controls the degree of variance reduction. The shaped distribution is Pα=(μ(α),σ(α)2)P_α=N(μ(α),\;σ(α)^2). Filter quality. Higher α reduces resources available for filter development, degrading filter quality. Filters have a technology-imposed performance ceiling εmin _ that cannot be surpassed even with full resource investment: εbase(α)=εmin+(εmax,base−εmin)⋅αk. _base(α)= _ +( _ ,base- _ )·α^k. (5) At α=0α=0, ε=εmin = _ (best achievable filter); at α=1α=1, ε=εmax,base = _ ,base (minimal filter quality). The value of εmin _ is anchored to recent results on Constitutional Classifiers, which achieved jailbreak success rates of approximately 4.4% after extensive red-teaming (Sharma et al. 2025). While this adversarial benchmark differs from the benign edge-case setting motivating our model, it provides a conservative lower bound on achievable filter quality. Deployment Scale and Edge-Case Pressure We decompose the total interaction count T into a component that drives harm accumulation and a component that drives filter degradation and systemic vulnerability discovery: M=ρedge⋅T,A(M)=M/Mref,M= _edge· T, A(M)=M/M_ref, (6) where ρedge _edge is the fraction of interactions that constitute edge cases — inputs, contexts, or situations not anticipated during filter design — and MrefM_ref is a reference scale. This decomposition separates T as a linear scale factor for harm from M as the driver of filter degradation and systemic vulnerability probability. Scale-dependent filter degradation. As deployment scale grows, filters encounter unforeseen edge cases with increasing probability, raising the effective leakage rate: ε(α,M) (α,M) =εbase(α) = _base(α) (7) +(εceiling−εbase(α)) + ( _ceiling- _base(α) ) ⋅(1−e−βdA(M))⋅d0, ·(1-e^- _d\,A(M))· d_0, where βd _d is the sensitivity of pattern discovery to deployment scale and d0∈(0,1]d_0∈(0,1] is a diffusion fraction — the proportion of discovered vulnerability patterns that propagate broadly. Note that d0d_0 governs the ultimate reach of discovered patterns, not their speed of propagation: as A(M)→∞A(M)→∞, the leakage rate approaches εbase(α)+(εceiling−εbase(α))⋅d0 _base(α)+( _ceiling- _base(α))· d_0, which is strictly less than εceiling _ceiling when d0<1d_0<1. Common-mode failure (CMF). A CMF occurs when a single vulnerability compromises all deployed instances simultaneously — for example, an unforeseen blind spot shared by all instances due to identical model weights. The probability of such an event increases with edge-case pressure: q(M)=(1−e−βqA(M))⋅e0,q(M)=(1-e^- _q\,A(M))· e_0, (8) where βq _q is the sensitivity of CMF discovery and e0e_0 is the probability that a discovered systemic vulnerability is actually triggered. Crucially, q(M)q(M) does not depend on α: CMF arises from architectural properties of the filter layer and deployment infrastructure, not from the quality of character shaping. The protective effect of character shaping during CMF is instead reflected in the reduced post-CMF harm (see Expected Harm below). Character Fragility As reliance on character shaping increases, so does the risk that the shaped behavior proves fragile — degrading or collapsing when the model encounters conditions outside its effective training range. We refer to this risk as character fragility and model it through a distribution-switching mechanism: pfrag(α)=pfrag(0)⋅αn,p_frag(α)=p_frag^(0)·α^n, (9) where pfrag(0)p_frag^(0) is the baseline character fragility rate — the maximum per-interaction probability of character failure (attained at α=1α=1) — and n is an exponent parameter (default n=2n=2; sensitivity analysis over n∈0.5,1,2,3n∈\0.5,1,2,3\). When fragility manifests (probability pfrag(α)p_frag(α)), the action distribution switches from PαP_α to a fragility distribution Pfrag=(μfrag,σfrag2)P_frag=N( _frag,\; _frag^2), with μfrag≤μ0 _frag≤ _0. Importantly, pfrag(0)p_frag^(0) is an intrinsic property of the trained model — the fraction of input space where shaped behavior fails to hold — analogous to a manufacturing defect rate determined by the production process, not by the number of units produced. What changes with deployment scale T is the aggregate number of fragility manifestations, not the per-interaction rate. This is why pfragp_frag is T-independent in our model. As we show in the Simulation Results, pfrag(0)p_frag^(0) proves to be the single most influential parameter in determining the optimal safety design. The parameter pfragp_frag subsumes two qualitatively distinct failure modes. The first is intentional deception (deceptive alignment), in which a model learns to behave safely during training while retaining misaligned objectives (Hubinger et al. 2024). The second is distributional fragility, in which shaped safe behavior simply fails to transfer to novel conditions (Quiñonero-Candela et al. 2009). Both are monotone increasing in α and are therefore jointly captured by the αnα^n functional form, but their internal mechanisms differ. Decomposing their relative contributions is beyond the scope of this model and is left to future work. The setting of μfrag _frag depends on which failure mode dominates: μfrag<μ0 _frag< _0 (worse than baseline) reflects intentional deception, while μfrag≈μ0 _frag≈ _0 (baseline reversion) reflects distributional fragility. Our scenario analysis examines both cases. When fragility manifests, the filter may be less effective at detecting the resulting behavior, which can differ in character from ordinary harmful outputs. We capture this through εfrag=min(factor×ε(α,M), 1.0) _frag= (factor× (α,M),\;1.0), where the factor is scenario-dependent. Expected Harm Rather than decomposing expected harm into separate probability and conditional-severity terms (which introduces weighting errors in the presence of mixture distributions), we define per-interaction expected harm directly. Base quantities. For Model A, the unconditional per-interaction expected harm under distribution PαP_α admits the closed form: gαdet=(τ−μ(α))Φ(τ−μ(α)σ(α))+σ(α)ϕ(τ−μ(α)σ(α)),g_α^det=(τ-μ(α))\, \! ( τ-μ(α)σ(α) )+σ(α)\,φ\! ( τ-μ(α)σ(α) ), (10) where Φ and ϕφ are the standard normal CDF and PDF, respectively. The quantity gfragdetg_frag^det is defined analogously for PfragP_frag. For Model B, the Pareto context multiplier scales these quantities uniformly: gα=gαdet×αPLαPL−1,gfrag=gfragdet×αPLαPL−1.g_α=g_α^det× _PL _PL-1, g_frag=g_frag^det× _PL _PL-1. (11) Because [X]E[X] is independent of α, the argminα _α of expected harm is identical under Models A and B. Per-interaction expected harm. Under normal operation (no CMF): Lnormal(α,M) L_normal(α,M) =(1−pfrag(α))⋅ε(α,M)⋅gα =(1-p_frag(α))· (α,M)· g_α (12) +pfrag(α)⋅εfrag⋅gfrag. +p_frag(α)· _frag· g_frag. Under CMF (filters fully disabled): LCMF(α)=(1−pfrag(α))⋅gα+pfrag(α)⋅gfrag.L_CMF(α)=(1-p_frag(α))· g_α+p_frag(α)· g_frag. (13) Note the absence of ε terms in LCMFL_CMF: filters are inoperative during CMF. However, character shaping persists because it is embedded in the model weights. Since gα<g0g_α<g_0 for α>0α>0, character shaping automatically reduces post-CMF harm without requiring any additional parameter. System-level expected harm. Eharm(α,T,M) E_harm(α,T,M) =T⋅[(1−q(M))⋅Lnormal(α,M) =T· [(1-q(M))· L_normal(α,M) (14) +q(M)⋅LCMF(α)]. +q(M)· L_CMF(α) ]. Here T acts as a linear scale factor for harm, while M drives filter degradation and CMF probability through Equations (7) and (8). Comparative Statics Proposition 1 (Filter technology improvement lowers α∗α^*). ∂α∗/∂εmin>0∂α^*/∂ _ >0. That is, an improvement in filter technology (lower εmin _ ) reduces the optimal character weight. Proof sketch. Define K=(1−e−βdA(M))⋅d0K=(1-e^- _dA(M))· d_0, which is independent of α. Then ε(α,M)=εbase(α)(1−K)+εceiling⋅K (α,M)= _base(α)(1-K)+ _ceiling· K, and since εbase(α)=εmin(1−αk)+εmax,base⋅αk _base(α)= _ (1-α^k)+ _ ,base·α^k, ∂ε∂εmin=(1−αk)(1−K). ∂ ∂ _ =(1-α^k)(1-K). (15) This is maximized at α=0α=0 (value 1−K1-K) and vanishes at α=1α=1 (no resources allocated to filters, hence no sensitivity to filter technology). A decrease in εmin _ therefore reduces LnormalL_normal more at low α than at high α, shifting the minimum of EharmE_harm leftward. A formal proof via the implicit function theorem is straightforward and omitted for brevity. ∎ Predicted tendency. As M increases, both ε(α,M) (α,M) and q(M)q(M) rise. The former penalizes filter-reliant (low-α) designs through LnormalL_normal; the latter amplifies LCMFL_CMF, in which character shaping (via gα<g0g_α<g_0) provides the only remaining protection. Both effects push α∗α^* upward. However, at high α the fragility cost pfrag(α)⋅gfragp_frag(α)· g_frag also grows, potentially offsetting these effects in high-fragility regimes. The conditions under which α∗(T)α^*(T) is monotone non-decreasing are identified empirically through phase diagrams in the Simulation Results. Tail Risk (CVaR) Expected harm captures average performance but may understate catastrophic scenarios. We therefore also compute the Conditional Value-at-Risk at level β: CVaRβ(α,T,M)=[harm∣harm>VaRβ],CVaR_β(α,T,M)=E[harm >VaR_β], (16) following the framework of Rockafellar and Uryasev 2000, estimated via count-level Monte Carlo simulation. Rather than sampling each of the T interactions individually (which is computationally infeasible at T=108T=10^8), we sample aggregate counts: the number of fragility-manifesting interactions from Binomial(T,pfrag(α))Binomial(T,p_frag(α)), the number of harmful events from the appropriate binomial, and individual damages from the conditional harm distribution (with Pareto context multipliers under Model B). This reduces the per-replication cost from O(T)O(T) to O(nharm)O(n_harm), where nharm≪Tn_harm T. The direction of α∗α^* under CVaR is not determined a priori. When the tail is dominated by CMF events (in which filters are disabled), character shaping provides the only mitigation, pushing αCVaR∗α^*_CVaR upward. When the tail is instead driven by per-incident severity (especially under Model B with small αPL _PL), filter effectiveness may become more valuable. The relative strength of these channels depends on q(M)q(M), αPL _PL, and pfrag(0)p_frag^(0), and is resolved empirically in the Simulation Results. For αPL=2.0 _PL=2.0 (the Pessimistic scenario), the variance of the Pareto distribution is infinite, causing CVaR estimates to converge more slowly than in the finite-variance case. We address this by running five independent seeds and reporting bootstrap 95% confidence intervals for all CVaR estimates. Scenario-Based Parameterization Rather than claiming empirically calibrated parameter values, we treat all parameters as scenario anchors and evaluate the model under three regimes: Optimistic (strong character shaping, high-quality filters, low fragility), Moderate (intermediate values), and Pessimistic (weak character shaping, poor filters, high fragility and heavy damage tails). Full parameter tables are provided in Appendix A. By comparing α∗(T)α^*(T) across all three scenarios, we assess the robustness of qualitative conclusions and identify any regime-dependent reversals. Simulation Results We evaluate the model across three scenarios (Optimistic, Moderate, Pessimistic) with parameters given in Table 5. All expected-harm results are computed analytically via closed-form expressions; CVaR estimates use count-level Monte Carlo simulation with Msim=10,000M_sim=10,000 replications (50,000 for production figures) and β=0.99β=0.99. The Optimal Mix Is Interior and Weakly Increasing in Scale Figure 1 presents the central result: α∗(T)α^*(T) across the three scenarios. In all cases, pure character (α=1α=1) is never optimal; the optimum is either interior or, in the Pessimistic scenario at small T, at the rules-only boundary (α∗=0α^*=0). The optimal character weight increases weakly with deployment scale T, but the magnitude of this shift varies substantially across scenarios(Table 1). Scenario α∗(T)α^*(T) Δα∗ α^* 10210^2 10410^4 10610^6 10810^8 Optimistic 0.62 0.62 0.62 0.63 +0.01+0.01 Moderate 0.51 0.51 0.52 0.55 +0.04+0.04 Pessimistic 0.00 0.00 0.06 0.21 +0.21+0.21 Table 1: Optimal character weight α∗α^* as a function of deployment scale T. Δα∗=α∗(108)−α∗(102) α^*=α^*(10^8)-α^*(10^2). The scale effect is negligible under Optimistic assumptions, modest under Moderate, and pronounced under Pessimistic conditions, where α∗α^* transitions from the boundary (α∗=0α^*=0, pure rules) to an interior solution. Figure 1: Optimal character weight α∗(T)α^*(T) across three scenarios. Top: α∗α^* as a function of deployment scale T. The optimum is weakly non-decreasing in T; pure character is never optimal. The Optimistic scenario yields a nearly flat α∗≈0.62α^*≈ 0.62 (Δα∗=+0.01 α^*=+0.01), while the Pessimistic scenario exhibits a sharp transition from α∗=0α^*=0 (pure rules) to an interior solution near T≈105.5T≈ 10^5.5 (Δα∗=+0.21 α^*=+0.21). Bottom: Contour maps of log10Eharm(α,T) _10E_harm(α,T) for each scenario, with the α∗(T)α^*(T) trajectory overlaid in white. The valley of minimal harm is narrow and nearly vertical in the Optimistic case, broader and rightward-shifting in the Moderate case, and sharply kinked in the Pessimistic case, reflecting the phase transition visible in the top panel. Three qualitatively distinct regimes emerge. Under Optimistic assumptions (strong character shaping, low fragility), α∗α^* is essentially flat at approximately 0.62 across six orders of magnitude in T. This suggests that when character-shaping technology is sufficiently mature and fragility risk is low, the scaling law effectively vanishes: the optimal design is insensitive to deployment scale. Under Moderate assumptions, the scaling effect is present but modest (Δα∗=+0.04 α^*=+0.04). Under Pessimistic assumptions (weak character shaping, high fragility), α∗α^* begins at the boundary (α∗=0.00α^*=0.00, pure rules) for small T and transitions sharply to an interior solution near T≈105.5T≈ 10^5.5, with a total shift of +0.21+0.21. Analytical characterization of the Pessimistic transition. The transition from α∗=0α^*=0 to an interior solution occurs at the critical scale TcritT_crit where ∂Eharm/∂α|α=0=0∂ E_harm/∂α |_α=0=0. At α=0α=0, the fragility cost vanishes (pfrag(0)=0p_frag(0)=0 for n≥1n≥ 1), so the condition reduces to a balance between two forces: the benefit of character shaping (reducing gαg_α) and the cost of filter degradation (increasing εbase _base). As M grows, two effects favor character shaping: ε(α,M) (α,M) rises (making filter reliance costlier) and q(M)q(M) rises (amplifying the CMF channel, in which character shaping provides the sole remaining defense via gα<g0g_α<g_0). At M=McritM=M_crit, these effects overcome the filter-degradation cost, and the optimum detaches from the boundary. Under Pessimistic parameters, Tcrit≈105.5T_crit≈ 10^5.5, consistent with the observed transition in Figure 1. We emphasize that the specific value Tcrit≈105.5T_crit≈ 10^5.5 is contingent on the Pessimistic parameter settings; under different parameterizations the transition point shifts accordingly, though the qualitative phenomenon — a sharp onset of interior optimality beyond a critical scale — is robust across the parameter space. Phase Diagrams: Δα∗≥0 α^*≥ 0 Throughout the Explored Parameter Space To identify conditions under which α∗(T)α^*(T) might decrease with scale, we compute phase diagrams over 20×2020× 20 grids of parameter pairs, plotting Δα∗=α∗(108)−α∗(102) α^*=α^*(10^8)-α^*(10^2) at each cell. Table 2 summarizes the results across all three parameter grids: across 1,200 cells, Δα∗ α^* is non-negative everywhere, with zero negative cells in any grid. Parameter pair Range of Δα∗ α^* Positive Negative Δμ×pfrag(0) μ× p_frag^(0) [0.00, 0.67][0.00,\;0.67] 400 0 Δμ×εceiling μ× _ceiling [0.02, 0.12][0.02,\;0.12] 400 0 εceiling×pfrag(0) _ceiling× p_frag^(0) [0.03, 0.06][0.03,\;0.06] 400 0 Table 2: Phase diagram summary. Across 1,200 cells in three 20×2020× 20 grids, Δα∗ α^* is non-negative everywhere. Figure 2 shows the Δμ×pfrag(0) μ× p_frag^(0) grid in detail. The strongest scale effect (Δα∗≈0.67 α^*≈ 0.67) occurs at low Δμ μ and high pfrag(0)p_frag^(0) (upper left), where character shaping is weak and fragile, making the system most sensitive to scale-driven filter degradation. Figure 2: Phase diagram: scale-induced shift in optimal design over the Δμ×pfrag(0) μ× p_frag^(0) parameter space. Each cell shows Δα∗=α∗(108)−α∗(102) α^*=α^*(10^8)-α^*(10^2). Red indicates that larger scale favors more character-reliant design; blue would indicate the opposite. Across all 400 cells, Δα∗≥0 α^*≥ 0. Colored circles mark the three scenarios. This monotonicity is a structural property of the model (see the Discussion). This result is stronger than initially expected: no parameter combination in the explored range produces a regime where larger deployment scale favors more rule-reliant design. We emphasize that this is a structural property of the current model rather than a general empirical prediction. In our formulation, deployment scale T enters exclusively through M=ρedgeTM= _edgeT, which degrades runtime filters (via ε(α,M) (α,M)) and raises CMF probability (via q(M)q(M)). Both channels penalize low-α designs: filter degradation makes rules less reliable, and CMF eliminates the filter layer entirely, leaving only character shaping as protection. Since T has no channel through which it degrades character shaping (i.e., pfrag(α)p_frag(α) and gαg_α are T-independent), the monotonicity Δα∗≥0 α^*≥ 0 follows near-tautologically from the model structure. A model in which deployment scale also increases character fragility (e.g., pfrag(α,T)p_frag(α,T)) could in principle produce Δα∗<0 α^*<0 regions; we discuss this extension in the Discussion. Baseline Fragility Rate Dominates the Optimal Design Table 3 reports the sensitivity of α∗(T=106)α^*(T=10^6) to each model parameter individually. The baseline fragility rate pfrag(0)p_frag^(0) dominates: its effect on α∗α^* (−0.50-0.50) is nearly twice that of the next most influential parameter (Δμ μ at −0.27-0.27), and all remaining parameters shift α∗α^* by less than 0.100.10. The effects of Δμ μ and rσr_σ are discussed in the Discussion. Parameter Swept range Δα∗ α^* at T=106T=10^6 pfrag(0)p_frag^(0) 0.005→0.400.005→ 0.40 −0.50-0.50 Δμ μ 0.2→2.00.2→ 2.0 −0.27-0.27 rσr_σ 0.4→1.00.4→ 1.0 +0.21+0.21 nfragn_frag 0.5→4.00.5→ 4.0 +0.09+0.09 εmin _ 0.005→0.200.005→ 0.20 +0.07+0.07 ρedge _edge 0.005→0.400.005→ 0.40 +0.02+0.02 e0e_0 0.05→0.950.05→ 0.95 +0.01+0.01 εceiling _ceiling 0.10→0.800.10→ 0.80 0.000.00 Table 3: Sensitivity of α∗(T=106)α^*(T=10^6) to individual parameters. pfrag(0)p_frag^(0) is the dominant lever by a wide margin. Figure 3 shows this dominant parameter in detail: α∗α^* decreases monotonically from 0.700.70 (at pfrag(0)=0.005p_frag^(0)=0.005) to 0.200.20 (at pfrag(0)=0.40p_frag^(0)=0.40) — a swing of 0.500.50. Figure 3: Baseline fragility rate pfrag(0)p_frag^(0) is the dominant determinant of optimal design. The optimal character weight α∗(T=106)α^*(T=10^6) decreases monotonically from 0.700.70 to 0.200.20 as pfrag(0)p_frag^(0) increases from 0.0050.005 to 0.400.40 (Moderate scenario baseline, varying only pfrag(0)p_frag^(0)). Dashed vertical lines indicate the default pfrag(0)p_frag^(0) values for each scenario. This result carries a clear practical implication: the most consequential input to safety architecture design is the estimated reliability of character shaping under distributional shift. If fragility can be kept below approximately 5%, the optimal design allocates a majority of resources to character shaping; above 10%, the optimum shifts decisively toward rules. Proposition 1: Filter Technology Improvement Lowers α∗α^* Figure 4 confirms Proposition 1 numerically: across all three scenarios, α∗(T=106)α^*(T=10^6) increases monotonically with εmin _ , verifying that ∂α∗/∂εmin>0∂α^*/∂ _ >0. As filter technology improves (lower εmin _ ), the optimal design shifts toward greater reliance on rules. The effect is modest in absolute magnitude (Δα∗=+0.07 α^*=+0.07 over εmin∈[0.005,0.20] _ ∈[0.005,0.20]) but consistent in sign across all scenarios. Figure 4: Numerical verification of Proposition 1: improving filter technology lowers α∗α^*. α∗(T=106)α^*(T=10^6) as a function of the filter quality ceiling εmin _ across all three scenarios. In every case, α∗α^* increases monotonically with εmin _ , confirming ∂α∗/∂εmin>0∂α^*/∂ _ >0. As filter technology improves (lower εmin _ ), the optimal design shifts toward greater reliance on rules. The effect is consistent in sign across scenarios, though modest in absolute magnitude (Δα∗=+0.07 α^*=+0.07 over εmin∈[0.005,0.20] _ ∈[0.005,0.20]). Tail Risk: CVaR Confirms Expected-Harm Optimum The optimal safety design proves robust to the choice of risk criterion across both damage models. Under Model A (deterministic damage), the CVaR-optimal α∗α^* converges to the expected-harm-optimal α∗α^* for T≳105T 10^5 (Figure 5). At smaller T, Monte Carlo noise produces fluctuations of ±0.10± 0.10 around the expected-harm optimum, but no systematic divergence is observed. Figure 5: CVaR-based and expected-harm-based optima converge at moderate-to-large deployment scale. Optimal α∗α^* under the expected-harm criterion (solid) and CVaR0.99 criterion (dashed) as a function of deployment scale T (Moderate scenario, Model A). For T≳105T 10^5, the two criteria yield essentially identical optima. At smaller T, Monte Carlo noise (Msim=10,000M_sim=10,000) produces fluctuations of ±0.10± 0.10 in the CVaR estimate, but no systematic divergence is observed. Under Model B (multiplicative Pareto damage), the CVaR-optimal α∗α^* is essentially invariant to the tail exponent αPL _PL. Table 4 shows that αCVaR∗=0.50α^*_CVaR=0.50 across all tested values of αPL _PL; only the CVaR magnitude scales with tail heaviness (approximately 2.5×2.5× from αPL=3.0 _PL=3.0 to 1.51.5). αPL _PL αCVaR∗α^*_CVaR CVaR 95% CI 3.0 0.50 925 [873, 962][873,\;962] 2.5 0.50 1037 [977, 1077][977,\;1077] 2.0 0.50 1288 [1209, 1329][1209,\;1329] 1.5 0.50 2352 [2141, 2592][2141,\;2592] Table 4: CVaR sensitivity to tail exponent αPL _PL (Moderate scenario, T=104T=10^4). The optimum α∗α^* is unchanged; only the CVaR magnitude scales with tail heaviness. This invariance arises because the Pareto context multiplier X enters multiplicatively and independently of α: it scales harm uniformly across all design points, preserving their relative ranking. Within the class of α-separable tail models, the optimal safety design is insensitive to both the choice of risk criterion (expected harm vs. CVaR) and the heaviness of the damage tail. What changes is the magnitude of catastrophic risk, not the policy that minimizes it. We discuss the scope and limitations of this invariance in the Discussion. Harm Decomposition: The Role of CMF Figure 6 decomposes EharmE_harm into its normal-operation and CMF components at T=106T=10^6. At low α, the CMF component (purple) is visible as a non-negligible share of total harm: because gα≈g0g_α≈ g_0 when character shaping is minimal, the system remains exposed when filters are disabled during CMF. At moderate-to-high α, gα≪g0g_α g_0 reduces post-CMF harm, and the CMF share diminishes accordingly. Despite this structural role, the CMF probability parameter e0e_0 has negligible influence on the optimum (Δα∗=+0.01 α^*=+0.01 over a 19-fold range in sensitivity analysis), indicating that the dominant channel determining α∗α^* is the fragility–filter tradeoff in LnormalL_normal, not the CMF–character channel in LCMFL_CMF. Figure 6: Decomposition of expected harm into normal-operation and CMF components. Stacked bars show the contribution of T(1−q)LnormalT(1-q)L_normal (green) and TqLCMFTq\,L_CMF (purple) to total EharmE_harm at T=106T=10^6 for selected values of α across all three scenarios. The CMF share is largest at low α and diminishes at higher α as character shaping reduces post-CMF harm. Discussion The central finding of our analysis is that the baseline character fragility rate pfrag(0)p_frag^(0) dominates the optimal safety design, moving α∗α^* across a range of 0.500.50 — far exceeding the influence of deployment scale, tail severity, or any other model parameter (Table 3). This dominance reflects a structural asymmetry: pfrag(0)p_frag^(0) simultaneously increases the cost of high-α designs (through fragility events) and decreases their benefit (through degraded post-fragility behavior), creating a double penalty that no other parameter imposes. In the subsections that follow, we interpret the remaining results and discuss their limitations. Why Less Character Investment Can Suffice When Shaping Is Effective A counterintuitive finding from the sensitivity analysis is that stronger character-shaping capability (larger Δμ μ, smaller rσr_σ) is associated with lower α∗α^*. Specifically, increasing Δμ μ from 0.20.2 to 2.02.0 reduces α∗α^* from 0.670.67 to 0.400.40; decreasing rσr_σ from 1.01.0 to 0.40.4 reduces α∗α^* from 0.620.62 to 0.410.41. This result reflects diminishing returns on character shaping. When Δμ μ is large, even modest α (e.g., α≈0.3α≈ 0.3–0.40.4) achieves most of the achievable reduction in gαg_α. Beyond this point, additional investment in character shaping yields minimal further benefit in expected harm reduction while incurring growing fragility costs through pfrag(α)=pfrag(0)⋅αnp_frag(α)=p_frag^(0)·α^n. The practical implication is encouraging: a safety architecture does not need to maximize character-shaping investment to capture the bulk of its value. A moderate allocation to character shaping, combined with continued investment in runtime filters, may yield a better risk profile than an aggressive push toward full character reliance. We note that this result is partially a consequence of the functional forms chosen: the Gaussian tail of gαg_α decays exponentially in α, while the fragility cost pfrag(α)=pfrag(0)⋅αnp_frag(α)=p_frag^(0)·α^n grows polynomially. This asymmetry mechanically favors low α when the exponential decay is fast (large Δμ μ). Under alternative functional forms — for instance, if fragility also decayed exponentially beyond a threshold — the diminishing-returns effect could weaken or reverse. Our functional-form sensitivity analysis (varying n∈0.5,1,2,3n∈\0.5,1,2,3\) confirms that the qualitative direction persists across power-law fragility costs, but the quantitative magnitude of the effect is form-dependent. This finding is consistent with the broader engineering principle that robustness comes from diversified defenses rather than from maximizing any single layer. The Scale Effect Is Real but Regime-Dependent The monotonicity result Δα∗≥0 α^*≥ 0 across the explored parameter space (1,200 grid cells, zero negative) is a structural property of the current model rather than a contingent empirical finding. In our formulation, deployment scale T enters exclusively through M=ρedge⋅TM= _edge· T, which degrades runtime filters (via ε(α,M) (α,M)) and raises CMF probability (via q(M)q(M)). Both channels penalize low-α (rule-heavy) designs. Crucially, T has no channel through which it degrades character shaping: pfrag(α)p_frag(α) and gαg_α are T-independent. The monotonicity Δα∗≥0 α^*≥ 0 therefore follows from the model’s asymmetric treatment of scale effects on the two safety layers. As discussed in the Formal Framework, pfragp_frag is an intrinsic property of the trained model — the fraction of vulnerable input space — and is T-independent by construction. An alternative modeling choice would treat pfragp_frag as effectively T-dependent, for instance if expanding deployment to new user populations shifts the effective input distribution further from training, exposing previously untested regions. Under such a reinterpretation, the monotonicity Δα∗≥0 α^*≥ 0 would no longer be structurally guaranteed. We regard the fixed-vulnerability interpretation as appropriate for analyzing a given model in a given deployment context, while the shifting-distribution extension addresses a different question — how safety architecture should adapt when deployment expands across heterogeneous contexts — and is left to future work. Despite this structural origin, the scale effect is quantitatively weak in the Optimistic scenario (Δα∗=+0.01 α^*=+0.01). This suggests that when character-shaping technology is mature and fragility risk is low, the scaling law effectively disappears: the optimal design becomes insensitive to deployment scale. This finding carries a policy implication — the urgency of adjusting safety architecture as systems scale depends critically on how fragile current character-shaping methods are. If fragility can be driven below approximately 5%, the scaling question becomes moot. Robustness to Risk Criterion and Tail Severity The convergence of CVaR-based and expected-harm-based optima at T≳105T 10^5, and the invariance of α∗α^* to the Pareto tail exponent αPL _PL, together constitute a robustness result: within the class of α-separable tail models, the optimal safety design is insensitive to both the choice of risk criterion and the heaviness of the damage distribution. This follows structurally from the multiplicative independence of the context multiplier X and α, which preserves the relative ranking of designs across αPL _PL values. What changes is the magnitude of catastrophic risk, not the policy that minimizes it. We note, however, that this α-separability is likely a simplification. In real systems, different design regimes may produce qualitatively different failure modes with distinct tail structures, breaking the independence between X and α. If the tail exponent were itself α-dependent, the CVaR-optimal α∗α^* could diverge from the expected-harm optimum. Developing models that capture such design-dependent tail behavior is an important direction for future work. The Quantitative Role of Common-Mode Failure Although the CMF channel is quantitatively second-order at the optimum (see Harm Decomposition above), it plays an essential qualitative role: it is the mechanism that makes rules-only designs (α=0α=0) sharply suboptimal at large T. When CMF disables the entire filter layer, systems with α=0α=0 lose all protection (since g0g_0 is large), whereas systems with α>0α>0 retain the “inner wall” of character shaping (gα<g0g_α<g_0) — the asymmetry that makes α∗>0α^*>0 at large T. This may also reflect our independence assumption: correlated failure modes that simultaneously disable filters and trigger character fragility could amplify the CMF channel’s importance, an extension we leave to future work. Limitations Several simplifications limit the scope of our conclusions. The action space is one-dimensional and Gaussian, capturing the safety–harm axis but discarding multi-attribute structure and heavy-tailed action behavior that could alter gαg_α. The model is static: it compares equilibria across scales without capturing adversary adaptation, filter updates, or fragility evolution over time. The α parameterization treats training-time and inference-time effort as a shared resource pool and is best read as a proxy for relative emphasis rather than a literal budget; its three coupled roles — training allocation, deployment-time dependence on shaping, and fragility susceptibility — may be partially independent in practice. Tail risk is captured only on the severity side: dependence-side heavy tails, where incident occurrences cluster in time, are not modeled and would carry different design implications. Finally, key parameters — particularly pfrag(0)p_frag^(0) and αPL _PL — lack direct empirical calibration, which our scenario-based approach mitigates but does not eliminate; empirical estimation of character fragility under distributional shift remains an important direction for future work. Conclusion This paper introduced a stylized comparative-statics model for analyzing the optimal balance between character shaping and rule enforcement in AI safety design as a function of deployment scale. Our analysis yields a clear and actionable conclusion: the most consequential determinant of optimal safety architecture is the baseline character fragility rate pfrag(0)p_frag^(0) — the probability that shaped safe behavior degrades or collapses under novel conditions. This single parameter moves the optimal design across a range of 0.500.50, from strongly character-reliant to strongly rule-reliant, far exceeding the influence of deployment scale, tail severity, filter quality, or common-mode failure probability. This finding carries direct implications for the research agenda in AI safety. If character fragility is the dominant lever, then three priorities follow. First, measuring fragility is essential. At present, no standardized metric exists for quantifying how reliably a model’s shaped behavior generalizes beyond its training distribution. Our results show that even order-of-magnitude uncertainty in pfrag(0)p_frag^(0) — e.g., whether it is 1% or 10% — leads to qualitatively different optimal designs. Developing rigorous, reproducible benchmarks for character fragility under distributional shift is therefore a prerequisite for principled safety architecture decisions. Second, reducing fragility may be more valuable than improving either character shaping or filter quality in isolation. Our sensitivity analysis shows that improving filter technology (εmin _ ) shifts α∗α^* by only +0.07+0.07, and increasing character-shaping strength (Δμ μ) exhibits diminishing returns. By contrast, reducing pfrag(0)p_frag^(0) from 0.100.10 to 0.010.01 shifts α∗α^* by approximately +0.30+0.30, unlocking a fundamentally different and more efficient safety regime. Research on training methods that produce robust generalization — including adversarial training for distributional robustness, mechanistic interpretability to identify fragile internal representations, and evaluation protocols that systematically probe out-of-distribution behavior — is therefore of first-order importance. Third, the scaling question becomes moot if fragility is sufficiently low. Under our Optimistic scenario (pfrag(0)=0.01p_frag^(0)=0.01), the optimal design is essentially invariant to deployment scale (Δα∗=+0.01 α^*=+0.01 across six orders of magnitude in T). This suggests that the urgency of adapting safety architecture to scale is itself contingent on the current state of character-shaping reliability. If fragility can be driven below approximately 5%, the design question simplifies from “how should safety change as we scale?” to “what is the right hybrid at any scale?” — and the answer is stable. Our model also establishes several structural results that hold across the explored parameter space: the optimum is never pure character (α∗<1α^*<1 in all cases), and is a hybrid in most regimes; the optimal character weight is weakly non-decreasing in deployment scale; the optimal policy is robust to the choice of risk criterion (expected harm vs. CVaR); and stronger character-shaping capability is associated with lower optimal α∗α^* due to diminishing returns. These findings should be interpreted within the limitations of a stylized model. The one-dimensional action space, Gaussian behavioral distribution, static analysis, and α-independent tail structure are simplifications that future work should relax. In particular, models in which deployment scale also increases character fragility could yield reversed scaling predictions, and design-dependent tail structures could break the CVaR–expected-harm equivalence observed here. Nevertheless, the core message is clear. The most important question for AI safety design is not how large a system will be deployed, nor how severe the worst-case damage might be, but how reliably the system’s shaped character holds when it encounters conditions its designers did not foresee. Answering this question — through measurement, through training methodology, and through rigorous evaluation — is the most direct path to safety architectures that scale. The reassuring news is that the answer does not depend on how many patients the model is serving — the only thing a patient should have to worry about is whether the one they are asking can be trusted. References Amodei et al. (2016) Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman, J.; and Mané, D. 2016. Concrete Problems in AI Safety. arXiv preprint arXiv:1606.06565. Aristotle (1999) Aristotle. 1999. Nicomachean Ethics. Indianapolis: Hackett Publishing Company, 2nd edition. Translated by T. Irwin. Askell (2018) Askell, A. 2018. Pareto Principles in Infinite Ethics. Ph.D. thesis, New York University. Aven (2016) Aven, T. 2016. Risk Assessment and Risk Management: Review of Recent Advances on Their Foundation. European Journal of Operational Research, 253(1): 1–13. Bai et al. (2022) Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. Boehm (1981) Boehm, B. W. 1981. Software Engineering Economics. Prentice-Hall. Christiano et al. (2017) Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Edwards, Hofmeyr, and Forrest (2016) Edwards, B.; Hofmeyr, S.; and Forrest, S. 2016. Hype and Heavy Tails: A Closer Look at Data Breaches. Journal of Cybersecurity, 2(1): 3–14. Ganguli et al. (2023) Ganguli, D.; Askell, A.; Schiefer, N.; Liao, T. I.; Lukošiūtė, K.; Chen, A.; Goldie, A.; Mirhoseini, A.; Olsson, C.; Hernandez, D.; et al. 2023. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459. Hendrycks and Gimpel (2017) Hendrycks, D.; and Gimpel, K. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In Proceedings of ICLR. Hubinger et al. (2024) Hubinger, E.; Denison, C.; Mu, J.; Lambert, M.; Tong, M.; MacDiarmid, M.; Lanham, T.; Ziegler, D. M.; Maxwell, T.; Cheng, N.; et al. 2024. Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566. Hursthouse (1999) Hursthouse, R. 1999. On Virtue Ethics. Oxford: Oxford University Press. Kaplan et al. (2020) Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361. Kaplan and Garrick (1981) Kaplan, S.; and Garrick, B. J. 1981. On the Quantitative Definition of Risk. Risk Analysis, 1(1): 11–27. Leveson (2011) Leveson, N. G. 2011. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press. Maillart and Sornette (2010) Maillart, T.; and Sornette, D. 2010. Heavy-Tailed Distribution of Cyber-Risks. The European Physical Journal B, 75(3): 357–364. Noller (2026) Noller, J. 2026. Artificial moral characters: constitutional AI and the challenge of alignment. AI and Ethics, 6(2). Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 27730–27744. Perrow (1984) Perrow, C. 1984. Normal Accidents: Living with High-Risk Technologies. Basic Books. Quiñonero-Candela et al. (2009) Quiñonero-Candela, J.; Sugiyama, M.; Schwaighofer, A.; and Lawrence, N. D., eds. 2009. Dataset Shift in Machine Learning. MIT Press. Rasmussen (1997) Rasmussen, J. 1997. Risk Management in a Dynamic Society: A Modelling Problem. Safety Science, 27(2–3): 183–213. Reason (1990) Reason, J. 1990. Human Error. Cambridge University Press. Rockafellar and Uryasev (2000) Rockafellar, R. T.; and Uryasev, S. 2000. Optimization of Conditional Value-at-Risk. Journal of Risk, 2: 21–42. Sharma et al. (2025) Sharma, M.; Tong, M.; Mu, J.; Wei, J.; Kruthoff, J.; Goodfriend, S.; Ong, E.; Peng, A.; Agarwal, R.; Anil, C.; et al. 2025. Constitutional Classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837. Appendix A Appendix A: Scenario Parameters Table 5 lists all scenario-varying parameters. Parameters constant across scenarios are: μ0=0.0 _0=0.0, σ0=1.0 _0=1.0, τ=−2.0τ=-2.0, k=1.0k=1.0, Mref=106M_ref=10^6, nfrag=2n_frag=2. Table 5: Scenario parameters. Parameters constant across all scenarios are listed in the text above. Group Parameter Opt. Mod. Pess. Character shaping Δμ μ 1.5 1.0 0.5 rσr_σ 0.6 0.7 0.9 Filter quality εmin _ 0.02 0.03 0.05 εmax,base _ ,base 0.10 0.15 0.25 εceiling _ceiling 0.25 0.30 0.50 Deployment scale ρedge _edge 0.01 0.05 0.10 βd _d 0.5 1.0 2.0 d0d_0 0.05 0.10 0.30 CMF βq _q 0.3 0.5 1.0 e0e_0 0.2 0.3 0.5 Fragility pfrag(0)p_frag^(0) 0.01 0.03 0.10 μfrag _frag 0.0 −0.5-0.5 −1.0-1.0 σfrag _frag 1.0 1.2 1.5 εfrag,factor _frag,factor 1.0 1.0 2.0 Tail index αPL _PL 3.0 2.5 2.0