Paper deep dive
Sensitivity or Criterion? Measuring and Moving When Language Models Act
Sixun Dong, Yebowen Hu, Ming Yin, Yiran Chen, Fei Liu, Chen Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Before an agent acts, it must decide whether action is warranted. A single accuracy cannot distinguish failure to recognize a warranted action from reluctance to act. Signal-detection theory separates sensitivity from criterion, but criteria estimated from authored binary labels depend on each benchmark's example mix. We introduce FlipAct, a diagnostic of tool use in physical states. A rule over annotated tool properties computes each answer and a graded evidence score, fixing the Yes/No boundary at zero across models. Questions form 2-by-2 quads that hold required knowledge fixed while the state flips. The slope of the fitted Yes-rate curve measures sensitivity; its crossing at one half measures criterion. Across 44 open models, sensitivity rises with size within a family, and its ranking agrees with three of four external when-to-act benchmarks. Models respond to evidence sign rather than magnitude, accepting actions that leave the state no better about as often as helpful ones. When Yes means act, most models are conservative across rephrasings; in some, this partly reflects a preference for No. Criterion is not determined by size. Activation steering shifts criterion with weights fixed; distillation from a model's own steered copy puts the shift into its weights without a significant sensitivity change. The distilled shift transfers to six external benchmarks, including two without Yes/No answers, while criterion-free AUC is preserved on the five benchmarks where it is available. Accuracy improves where models start conservative and declines or remains within error where they start permissive. Project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.15415v1
- Canonical: https://arxiv.org/abs/2610.15415v1
Trouble viewing inline? Open PDF directly →
Full Text
153,701 characters extracted from source content.
Expand or collapse full text
Sensitivity or Criterion? Measuring and Moving When Language Models Act Sixun Dong Affiliation: University of Central Florida Email: sixundong@ucf.edu Yebowen Hu Affiliation: Zoom Communications Email: yebowen.hu@zoom.us Ming Yin Affiliation: Duke University Email: chen.chen@crcv.ucf.edu Yiran Chen Affiliation: Duke University Fei Liu Affiliation: Emory University Chen Chen Affiliation: University of Central Florida Abstract Before an agent acts it must decide whether any action is warranted. Benchmarks that ask this question report one accuracy, which cannot distinguish a model that fails to recognize a warranted action from one that recognizes it but is reluctant to act. Signal-detection theory separates the two as sensitivity and criterion, but a criterion estimated from authored binary labels depends on each benchmark’s mix of examples and cannot be compared across benchmarks. We introduce FlipAct, a diagnostic built on a qualitative representation of what tools do to physical states. A rule over annotated tool properties computes each question’s answer and a graded evidence score, with the Yes/No boundary fixed at zero for every model, so every answer is computed, not written by a model. Questions are built from 2×22× 2 quads that hold the required knowledge fixed while the state flips. The slope of a model’s Yes-rate against the score is its sensitivity, and the score where it crosses one half is its criterion. Across 4444 open models, sensitivity rises with size within a family, and its ranking agrees with three of four external when-to-act benchmarks. Models respond to the sign of the evidence rather than its size, accepting actions that leave the state no better about as often as helpful ones. When Yes means act, most models are conservative and stay so when the question is rephrased. In some models this is a preference for answering No, and the criterion is not set by size. Activation steering moves the criterion with the weights fixed, and distilling a model from its own steered copy moves it in the weights, without a significant change in sensitivity. The distilled shift carries to six external benchmarks with sensitivity kept, including two without a Yes/No answer. Accuracy rises where the model started conservative and falls where it started permissive. Project page: https://sixundong.com/projects/flipact. †footnotetext: 🖂 Corresponding author. 1 Introduction Large language model (LLM) agents must decide whether to act before they decide how. Benchmarks for agents mostly score the action once taken: whether the chosen function, its arguments and the trajectory are correct (Patil et al., 2025; Huang et al., 2024; Wang et al., 2025; Yin et al., 2026). Those that study the decision itself, in tool calling (Ross et al., 2025; Sun et al., 2026), clarification (Li et al., 2026) and abstention (Kirichenko et al., 2026), report a single accuracy, and that number mixes two ways to fail. A model may not recognize whether a situation warrants action, a failure of its sensitivity; or it may recognize the situation and still be too reluctant or too ready to act, a failure of its criterion. Too conservative a criterion refuses benign requests (Röttger et al., 2024; Cui et al., 2024; Shi et al., 2024) or answers from memory where a tool was needed; too permissive a criterion acts where nothing warrants it. On When2Call (Ross et al., 2025), gemma-2-9b and Phi-3.5-mini reach balanced accuracies of 0.650.65 and 0.640.64 with opposite behavior: the first rarely calls the tool, even where it should, and the second calls it on most examples, many where it should not (Figure 1a). They separate the two kinds of example about equally well, so on this benchmark the first should act more and the second less. Signal-detection theory (Green et al., 1966) separates sensitivity (d′d ) from criterion (Figure 1b; §3.1), but on a benchmark with binary labels the criterion it recovers is relative to that benchmark’s mix of examples, and no zero is shared across benchmarks. Comparing criteria needs a fixed boundary on a common evidence scale, which authored binary labels do not provide (Jacobs and Wallach, 2021, cf.). 00.51call rate.40.10.86.58gemma-2-9bbal. acc. .65Phi-3.5-minibal. acc. .64should callshould not (a) Same accuracy, opposite calls unwarrantedwarrantedPhi-3.5-mini’scriteriongemma-2-9b’scriterionevidence for acting → d′d (b) Same d′d , two criteria No (ΔU≤0 U≤ 0)Yes (ΔU>0 U>0)ΔU U−3-3−2-2−1-100112233dehumid.dryhumid.wethumid.drydehumid.wet (c) Evidence computed by rule Figure 1: (a) Share of should-call and of should-not-call When2Call examples that each model calls. (b) Schematic: at one sensitivity d′d , the two models’ criteria trade hits for false alarms at the same accuracy. (c) The quad of Figure 2 on the evidence scale ΔU U that FlipAct computes by rule, with the Yes/No boundary fixed at zero. FlipAct supplies such a scale by computing the evidence for acting. Its design draws on qualitative process theory (Forbus, 1984), which describes physical change by the sign of each quantity’s change and a coarse ordinal magnitude, enough to say what an action does to a state without numbers. Each question shows a situation and a candidate tool and asks whether using the tool is warranted. The state and the tool each have a sign and an ordinal magnitude on a physical axis. A rule over these properties computes the evidence score ΔU U, how much closer the action brings the state to its goal, with the Yes/No boundary at zero (Figure 1c). The design unit is a 2×22× 2 quad: two tools, each shown in the state it fixes and in the state it worsens, so the knowledge needed to answer stays the same and the state decides the answer (Figure 2). Fitting a model’s Yes-rate against ΔU U gives its sensitivity as the slope a and its criterion as the score τ where it crosses one half. Across 4444 open models (§4), sensitivity behaves as expected of a capability: it rises with size inside a family, and its ranking agrees with external d′d on three of four when-to-act benchmarks. Models read the sign of the evidence rather than its size, accepting actions that leave the state no better about as often as helpful ones. When Yes means act, the criterion is mostly conservative, in some models through a preference for answering No. It is a stable per-model setting, not set by size: two releases of Mistral-Small-24B sit on opposite sides of zero. We move the criterion separately from sensitivity in two ways (§5). At inference, adding a direction read from the model’s own activations (Zou et al., 2023; Turner et al., 2023; Ke et al., 2026) shifts the criterion with the weights fixed. In the weights, distilling the model from its own steered copy shifts the criterion on four models from three developers with no significant change in sensitivity, and a random direction of the same norm does not. The shift carries beyond FlipAct’s Yes/No format: on benchmarks without a Yes/No answer it exceeds the shift from a teacher that only favors Yes, on both models tested. On every benchmark, accuracy rises or falls as the starting criterion predicts. Supervised training on the computed labels instead raises sensitivity and moves the criterion with it. Contributions. 1. A measurement scheme that splits when-to-act accuracy into a sensitivity and a criterion, read against an evidence scale computed by rule with the Yes/No boundary fixed for every model. 2. FlipAct, built by an automated pipeline in which language models only annotate what a tool does, while the states and every answer follow from the rule. 3. Evidence that the criterion moves separately from sensitivity: steering and distillation from a steered copy move the criterion, while training on the computed labels raises sensitivity. 2 Related work When-to-act evaluation. Benchmarks for deciding whether to act cover tool calling (Ross et al., 2025; Sun et al., 2026; Huang et al., 2024; Patil et al., 2025), clarification (Li et al., 2026), abstention (Kirichenko et al., 2026) and over-refusal (Röttger et al., 2024; Cui et al., 2024; Shi et al., 2024). Most share three design choices: ground truth is authored by humans or an LLM judge, the evidence for acting is a binary label, and performance is a single rate that conflates a model’s sensitivity with its criterion. Li et al. (2026) instead compute ground truth by constraint satisfaction, and Sun et al. (2026) fix a controlled call/no-call boundary with deliberate no-tool cases. Neither flips a world-state while holding the required knowledge fixed, and none of these benchmarks grades the evidence for acting per question. We reuse four of them as external tests (§4.2) and measure transfer on six benchmarks (§5.3). App. F compares twelve on these design choices. Signal detection and language models. Signal-detection theory separates an observer’s sensitivity from its criterion (Green et al., 1966). Reading a benchmark score as the measurement of a latent quantity raises the question of whether it measures what it is read as (Jacobs and Wallach, 2021; Bean et al., 2026). Cacioli (2026) fits signal-detection models to factual question answering with the model’s answer log-probability as evidence, so the criterion sits on each model’s own confidence scale. Shi et al. (2026) read a call/no-call offset off sparse-autoencoder features on When2Call and cancel it by steering. Their offset is an intercept on an internal margin, compared within one model and without a separate sensitivity. FlipAct places the criterion on a rule-computed evidence scale shared across questions and models, estimates it separately from sensitivity with the required knowledge held fixed, and moves it into the weights. 3 The FlipAct diagnostic 3.1 Preliminaries: signal detection Signal-detection theory (Green et al., 1966) models an observer who reports a signal whenever a scalar amount of evidence exceeds a threshold. The sensitivity d′d is the distance between the evidence distributions of signal and noise trials in units of their spread, and the criterion c is where the threshold sits relative to their midpoint. From the hit rate H and the false-alarm rate FAFA the equal-variance estimators, with z the inverse of the standard normal distribution function, are d′=z(H)−z(FA),c=−12(z(H)+z(FA)).d \;=\;z(H)-z(FA), c\;=\;- 12 (z(H)+z(FA) ). (1) In when-to-act terms a hit is acting when one should and a false alarm is acting when one should not. A positive c is conservative. FlipAct supplies an evidence score per question, and the same two quantities become the slope a and the crossing point τ of a model’s Yes-rate against that score (§4.1). Every comparison in the paper pairs a with d′d and τ with c. An interactive version is online.11 1 https://sixundong.com/projects/flipact 3.2 Same knowledge, flipped state Template Situation: The <object> is <state>. [<supply>] The goal is to bring the <object> back to a normal state. Action: a person uses a <tool>. Is using the <tool> a wise choice in this situation? Answer Yes or No. One quad, object room humidifier industrial dehumidifier wets by 2 dries by 3 very dry Yes +2+2 No −3-3 dangerously wet No −2-2 Yes +3+3 Figure 2: The question template (left) and one quad (right) with each question’s answer and evidence score ΔU U. The slot [<supply>] states that a prerequisite is met, for tools that have one. Each question shows a physical situation, the goal of returning it to normal and one candidate tool, and asks whether using the tool is a wise choice. All questions share one minimal template (Figure 2), as do the stimuli of template-based diagnostics (McCoy et al., 2019; Ribeiro et al., 2020), so only the tool, the state and their outcome vary. Physical situations are used because a tool’s effect on a state can be written as an explicit qualitative transition: a humidifier helps a dry room and harms a wet one. In the quad of Figure 2, a humidifier that adds two steps of moisture and an industrial dehumidifier that removes three are crossed with a very dry and a dangerously wet room. Every question of the quad needs the same two facts, what the tool does and what the room needs. For a given tool only the state changes, and with it the answer, so a model that knows what a humidifier is for must still read the state. Quads and state flips are controls built into the set. The main set has 1,8801,880 questions: 840840 lie in a complete quad and 1,2561,256 belong to a state flip, the same tool with opposite answers in two states. A state flip separates a model that ignores the state, which gives both questions about the same Yes–No logit margin, from one that is merely permissive, which may answer Yes to both but still ranks the warranted one higher (App. D.3, D.7). Every situation has as many Yes as No questions, so the description of the world does not predict the answer. 3.3 Ground truth and evidence by rule Each situation fixes an integer state v on a physical axis (thermal transfer, moisture, mechanical cohesion, …) with the goal at 00. Each tool carries a direction δ∈+1,−1δ∈\+1,-1\ and an ordinal magnitude band μ∈1,2,3μ∈\1,2,3\ on that axis (Forbus, 1984, in qualitative process theory, a sign and a place in a quantity space;), optionally conditional on a prerequisite. Applying it moves the state to v+δμv+δμ, and the process behind that change is not modeled. The evidence score is the utility of the action, how much closer to the goal it leaves the world, ΔU=|v|−|v+δμ|,Yes⇔ΔU>0, U\;=\;|v|\;-\;|v+δμ|, Yes U>0, and a tool with unmet prerequisites carries ΔU=−2 U=-2 for each, a value set by convention. The question type follows from the same computation: a wrong-direction tool is destructive, a right-direction tool that overshoots or acts on a state already at the goal is redundant, and a tool applied without its prerequisite is prerequisite. Because v and μ are integers, ΔU U takes seven values on [−3,3][-3,3]. A question conveys |v||v| only by the adverb slightly, very or dangerously, and μ only by the tool itself. Language models annotate each tool’s axis, direction, magnitude and prerequisite, and the rule computes the answer from those properties. 3.4 An automated build FlipAct is generated by a pipeline (Figure 3) that needs no per-question human annotation or adjudication. Its specification is written once: the physical axes, a closed vocabulary of tool properties, the rule of §3.3 and the template. Ten lexical, medical and visually grounded resources supply 149,840149,840 candidate entities, and two annotator models, Qwen3.5-27B and gemma-4-31b, fill the vocabulary for each. Where they disagree, both arbitrate against the specification, and unresolved records are dropped. Tool records (one tool, one axis) are paired when they act in opposite directions on that axis and target object, and states, answers, question types and text follow by rule. A balance selection keeps 1,8801,880 of the 6,1766,176 rendered questions. Two matched-pair supplementary sets, 342342 already-satisfied and 224224 prerequisite-blocked questions, are kept outside the fits. A new axis, a new tool inventory or a larger set runs through the same steps at the cost of annotator calls (App. A). Entity pool 149,840149,840 entities Tool-property annotation 2,8762,876 records Screens: axis, use, magnitude, deployability 1,6381,638 tool records Pairing 1,4221,422 tool pairs Situation synthesis states by rule Answer and ΔU U by rule 00 LLM calls Template rendering 6,1766,176 questions Balance selection 1,8801,880 questions toolrecordsquestions Figure 3: The FlipAct construction pipeline. Language models are used only in the orange steps, which annotate and screen tool properties; answers and evidence scores are computed by rule. A lexical probe trained on the question text is at chance on held-out situations whose tools it has not seen (App. A.5), and recomputing the ground truth independently of the build gives zero mismatches. A model’s judgment enters only where tools are annotated and screened, and a model family outside the build reproduces every tool/non-tool judgment, nine in ten axis judgments and every magnitude band to within one band, two in three exactly. Perturbing the annotations at that rate, or until a quarter of the computed answers change, keeps the rankings by a and τ intact and moves criteria toward acting, so it cannot produce the conservative criteria (App. A.6). 4 Measuring sensitivity and criterion Models. To see how the coordinates a and τ vary with size and family, we fit 4444 open models, all but five with reasoning off (App. B.7), from 0.60.6B to 7070B parameters and fifteen families, among them Qwen3 and Qwen3.5, Gemma 2–4, Llama 3, Mistral and Phi (App. Tables 9–10). External benchmarks. To test whether the coordinates capture what when-to-act benchmarks score, we compare them with four, each posing the decision in a different domain. Every example is one where acting is right or one where withholding is, and d′d and c follow from Eq. 1 with acting as the response. On BFCL acting is calling a listed function, which is right when one fits the query; on When2Call it is choosing the tool call among four replies (call, ask, cannot answer, answer directly); QuestBench and When2Tool ask Yes/No whether the model already has enough information to answer and whether it would need a tool (App. Table 11). 4.1 The fit and the response profile On FlipAct every question carries a computed evidence score, so we fit the model’s Yes-rate as a logistic curve over that score (Green et al., 1966): P(Yes∣ΔU)=σ(a(ΔU−τ)).P(Yes U)\;=\;σ\! (a\,( U-τ) ). (2) Here σ is the logistic function, the slope a the sensitivity and the crossing point τ the criterion. A positive τ means the fitted curve crosses one half above the rule’s boundary, so the model answers Yes less often than the rule warrants, which we call conservative. A criterion is estimable when a≥0.15a≥ 0.15 and SE(τ)≤2.0SE(τ)≤ 2.0, and well determined when it also lies inside the score range; beyond ±3± 3 it is reported as a bound. Both coordinates have split-half reliability above 0.90.9 (App. B.2). On external benchmarks we use d′d and c (Eq. 1); a model readier to act has lower τ and c. Figure 4: Yes-rate at each evidence score for the 2121 reasoning-off models with a well-determined criterion (gray), with their median and interquartile band. Response profile. Models read the sign of the evidence, not its size (Figure 4). Yes-rates jump between ΔU=−1 U=-1 and 00, significantly in 1919 of the 2121 reasoning-off models with a well-determined criterion, and are flat on either side: no model answers Yes reliably more often at ΔU=+3 U=+3 than at +1+1. The step sits one unit below the rule’s boundary: models accept actions that leave the state no better about as often as helpful ones. The questions at ΔU=0 U=0 are all redundant slight overshoots, so this set cannot tell whether that placement belongs to the models or to the rule. Models do withhold when the state is already at its goal or a prerequisite is unmet (App. B.2), so the overshoot is the one redundancy they accept. The sign of τ is set by the height of the Yes plateau, not by the step: for the median model the Yes-rate on helpful actions stays near one half (Figure 4). Reading τ. A step function at the sign of the evidence predicts held-out answers as well as the logistic. Still, τ tracks the criterion divided by sensitivity: with c/d′c/d of Eq. 1 read from hits on ΔU>0 U>0 and false alarms on ΔU<0 U<0, τ≈−0.09+3.22c/d′τ≈-0.09+3.22\,c/d (r=0.999r=0.999), and both find the same models conservative (App. B.2–B.3). In 99 of the 2121 models the Yes-rate never reaches one half, so τ is where the fitted curve, not the data, crosses it. We therefore read τ by sign and rank. 4.2 Sensitivity grows with size and agrees with external benchmarks Within a family, a rises with size: Qwen3 goes from 0.340.34 at 4B to 0.580.58 at 32B, and the one exception is Qwen3.5, which peaks at 9B. Twelve models fit below 0.300.30, among them widely deployed small instruct models, and their Yes-rates span almost the whole range: a model can be permissive or conservative about a tool and still be unable to tell when using it is warranted (Figure 5). Figure 5: Sensitivity against criterion for the 2727 reasoning-off models with an estimable criterion, with the distribution of the well-determined criteria on the right. Hollow markers at the edges are criteria beyond ±3± 3. The line joins the two same-size Mistral-Small-24B releases. FlipAct sensitivity predicts external d′d on three of the four benchmarks (Figure 6a). The comparison uses a panel: of the 3939 reasoning-off models, 2727 have an estimable criterion and 2121 a well-determined one, and the 1818 of these with a≥0.30a≥ 0.30 and usable external d′d and c form the panel, so that both sides of each comparison are estimated reliably (App. B.4 varies the cutoff on a). All three relations hold with model size partialed out (App. B.5). This agreement across three domains and formats supports reading a as the sensitivity these benchmarks score. On When2Tool the relation is not significant. Figure 6: (a) Correlation across the panel between FlipAct sensitivity a and external d′d , with bootstrap intervals; the hollow point is not significant. (b) Criterion against model size for the families with three or more estimable members, one line per generation. Hollow markers with an arrow are criteria beyond +3+3. 4.3 Criterion: a per-model setting, conservative by default Eighteen of the 2121 reasoning-off models with a well-determined criterion are positive on the canonical wording of Figure 2: they answer Yes less often than the rule warrants (Figure 5), and nearly all tested stay positive under five rephrasings (App. C). The criterion is one setting across question types: models that accept more destructive actions also accept more slight overshoots on other questions (ρ=0.88ρ=0.88, 2727 models, a≥0.30a≥ 0.30), though most accept overshoots more readily. As in signal detection, the criterion collects every bias in the response, including a preference for one answer word. We also ask each question in three reversed wordings, such as “Is using the tool a mistake?”, under which No means act. After recoding, a preference for No raises the canonical criterion and lowers the reversed one, so writing them as τd+b _d+b and τd−b _d-b separates a threshold on the evidence, τd _d, from the preference, b (App. C). For the three Qwen3.5 models and GLM-4-32B, b is +3+3 to +4+4 and τd _d within a unit of zero; the other twelve of the 1616 models tested stay conservative without it. JEV (TypeSafe AI, 2026), a model that returns a probability, not text, and so has no answer word to prefer, is conservative too and steps at the sign of the evidence (App. E). Where the criterion sits is a property of the model more than of its size. The two Mistral-Small-24B releases are the same size and land on opposite sides of zero, at +2.18+2.18 (2501) and −1.10-1.10 (3.2, 2506) (Figure 5). Inside a family τ drifts toward zero as models grow, in Qwen3, in the dense Qwen3.5 models and across the Gemma generations (Figure 6b). Part of this drift follows from rising sensitivity, since τ tracks the criterion divided by sensitivity. Families differ in their typical τ, so pooled over all models the relation with size vanishes. The criterion measured on FlipAct does not consistently predict a model’s c on other benchmarks: across the panel the correlation is near zero on When2Call (Pearson r=−0.01r=-0.01), QuestBench and When2Tool, and positive on BFCL (r=+0.51r=+0.51), where it rests on the three models with the lowest τ (App. Table 15). Section 5.3 asks instead whether moving τ moves c. 5 Moving the criterion We ask whether the criterion can be moved while sensitivity stays fixed, at inference (§5.1) and in the weights (§5.2), and whether the moved criterion carries to external benchmarks (§5.3). Such a move trades misses for false alarms and cannot raise accuracy beyond what sensitivity allows. 5.1 Steering moves the criterion with the weights fixed Table 1: Steering on When2Call, unsteered and at the selected strength, both measured on the same 600600 examples. Arrows mark a move toward acting, and bold a change in that direction. call rate ↑ criterion c ↓ Model unsteered ++ steering unsteered ++ steering Δ acc. Qwen3-8B 0.560.56 0.600.60 −0.18-0.18 −0.32-0.32 −0.022-0.022 Qwen3.5-27B 0.470.47 0.500.50 +0.10+0.10 +0.01+0.01 +0.005+0.005 gemma-2-9b 0.240.24 0.290.29 +0.78+0.78 +0.59+0.59 +0.002+0.002 Mistral-24B 0.370.37 0.550.55 +0.39+0.39 −0.13-0.13 −0.032-0.032 phi-4 0.570.57 0.590.59 −0.30-0.30 −0.33-0.33 −0.038-0.038 Models and data. We steer five open models, Qwen3-8B, Qwen3.5-27B, gemma-2-9b, Mistral-Small-24B (the 2501 release, Mistral-24B below) and phi-4, and measure the effect on When2Call (Ross et al., 2025), on 600600 examples, half call and half withhold (eleven more models in App. D.1). Method. Steering follows the standard recipe (Turner et al., 2023; Zou et al., 2023; Rimsky et al., 2024; Arditi et al., 2024). A logistic probe from the final-token residual stream at relative depth 0.60.6 to the model’s own Yes/No answers on the FlipAct questions gives the criterion direction w w, which is added during generation to the residual-stream activation h at that layer as h←h+λw^h← h+λ w. For each model the strength λ⋆λ is the one with the lowest When2Call criterion among those that keep d′d within 25%25\% of its unsteered value. Results. The direction moves the When2Call criterion toward acting on all five models (Table 1). On Qwen3-8B, gemma-2-9b and Mistral-24B the shift is at least three times that of same-norm random and orthogonal directions and keeps over two thirds of its size when the refusal, sycophancy and engagement directions are projected out of w w; over sixteen models, eleven move significantly toward acting and four pass both controls (App. D.1). The shift is small on Qwen3.5-27B, which starts near zero on When2Call and whose FlipAct criterion is mostly a preference for No, and there it does not exceed random directions of the same norm. On phi-4, which starts permissive and moves least, it does not hold when the strength is chosen on one half of the examples and measured on the other (App. D.1). On four models steering gains hits at the cost of correct withholds, and on all five, accuracy changes by at most four points. With the direction fit on one half of FlipAct and measured on the other, it also moves the criterion of Qwen3.5-9B and Qwen3.5-27B toward zero and raises their accuracy by 2.62.6 and 1.61.6 points. 5.2 Distilling from a steered copy puts the criterion into the weights Models and data. As an existence proof that the criterion can be placed in the weights, we distill four students from three developers, Qwen3-8B, Qwen3.5-9B, gemma-3-12b and Mistral-24B, each from its own steered copy. The FlipAct main set is split in half so that no situation or tool record appears on both sides. Students train on one half (940940 questions) with three seeds per condition, and every result is measured on the held-out half, so baselines differ from the full-set values of §4. Table 2: Held-out coordinates and accuracy of each student before training, after criterion distillation and after sensitivity learning (§5.4), seed means; shaded rows give the change under each control. Bold marks a change in the targeted direction (arrows). –: not estimable in every seed. †Beyond ±3± 3. Baseline Criterion distillation Sensitivity learning Student a τ Acc. a τ ↓ Acc. ↑ a ↑ τ Acc. ↑ Qwen3-8B 0.470.47 +2.33+2.33 0.600.60 0.550.55 −2.11-2.11 0.640.64 1.091.09 −0.02-0.02 0.780.78 control +0.02+0.02 +0.04+0.04 +0.00+0.00 −0.19-0.19 – −0.04-0.04 Qwen3.5-9B 0.750.75 +1.88+1.88 0.650.65 0.730.73 +0.86+0.86 0.700.70 1.221.22 +0.18+0.18 0.830.83 control +0.00+0.00 +0.20+0.20 −0.01-0.01 −0.30-0.30 – −0.04-0.04 gemma-3-12b 0.360.36 +2.17+2.17 0.590.59 0.350.35 +0.30+0.30 0.610.61 1.561.56 −0.41-0.41 0.830.83 control +0.00+0.00 −0.11-0.11 +0.00+0.00 −0.20-0.20 – −0.07-0.07 Mistral-24B 0.590.59 +3.47+3.47† 0.590.59 0.420.42 +0.19+0.19 0.620.62 1.161.16 +0.08+0.08 0.790.79 control +0.07+0.07 −0.10-0.10 +0.00+0.00 −0.61-0.61 – −0.06-0.06 Method. In criterion distillation, the teacher T is the student with steering on. Its direction is a difference of means over the student’s own Yes and No answers on the training half, and its strength is the one on a sweep that moves τ furthest while a and d′d stay close to their unsteered values. The student S has the same weights, a LoRA adapter and no steering, and is trained to match the teacher’s next-token distribution at the answer position of each training question q, ℒ=∑qKL(pT(⋅∣q)∥pS(⋅∣q)).L\;=\; _qKL (p_T(· q)\, \|\,p_S(· q) ). (3) The control replaces the direction with a random one of the same norm. A change is significant when it exceeds twice its paired bootstrap error over situations (settings in App. D.3). Results. Criterion distillation moves τ toward acting on all four students (Table 2), by more than twice its error on three and just under twice on gemma-3-12b. No student’s sensitivity changes significantly in any seed; the largest change, −0.17±0.10-0.17± 0.10 on Mistral-24B, is under twice its error (App. Table 29). An equivalence test bounds the change within ±0.12± 0.12 on Qwen3.5-9B and gemma-3-12b and cannot decide on Qwen3-8B and Mistral-24B. On When2Call, d′d stays within 20%20\% of its baseline for all four. Held-out accuracy rises by one to five points, as a criterion shift should when the model starts conservative, as all four do. The random direction moves τ at most a fifth as far. Injected at a strength that perturbs the answers as much as the teacher’s direction, a random direction either leaves τ in place or collapses the model onto one answer (App. D.3). 5.3 The distilled criterion moves the same way on six external benchmarks Benchmarks. To test whether the shift generalizes beyond FlipAct’s physical questions and its Yes/No format, each student is tested on six unseen benchmarks, scored with d′d and c. Four ask whether to act now: When2Call and When2Tool (Sun et al., 2026) (call a tool), QuestBench (Li et al., 2026) (ask or solve) and BFCL-miss-func (Patil et al., 2025) (the needed function is removed from the list). Two ask a planning judgment: ACPBench (Kokel et al., 2025) (does an action apply) and CaT-Bench (Lal et al., 2024) (must one step precede another). When2Call offers four replies, BFCL-miss-func takes a free-form call or refusal, and the other four take Yes or No. Figure 7: Criterion shift on six external benchmarks after criterion distillation (filled) and under the random-direction control (hollow), seed means ± SD. Each benchmark has its own scale; negative is toward acting. Qwen3.5-9B on BFCL-miss-func is left out because its baseline does not reproduce. Table 3: Change in balanced accuracy after criterion distillation for the two Qwen students, on the three benchmarks where at least one starts conservative; seed means, bold marks a rise. Student Benchmark Start c Δ bal. acc. ↑ Qwen3-8B When2Tool +0.42+0.42 +0.014+0.014 QuestBench +0.71+0.71 +0.041+0.041 ACPBench +0.40+0.40 +0.025+0.025 Qwen3.5-9B When2Tool +0.73+0.73 +0.046+0.046 QuestBench −0.06-0.06 −0.011-0.011 ACPBench +0.65+0.65 +0.052+0.052 Table 4: Change in criterion-free AUC after criterion distillation, seed means. Every change is equivalent to zero within ±0.02± 0.02 (App. D.4). Qwen3 Qwen3.5 gemma-3 Mistral 8B 9B 12b 24B When2Call −.008-.008 −.006-.006 +.001+.001 −.002-.002 QuestBench −.012-.012 −.002-.002 −.005-.005 +.002+.002 When2Tool +.004+.004 −.001-.001 +.011+.011 +.001+.001 ACPBench +.001+.001 +.001+.001 +.004+.004 −.003-.003 CaT-Bench +.001+.001 +.008+.008 −.003-.003 +.002+.002 Results. Moving the FlipAct criterion moves c the same way on every benchmark, although across models it does not consistently predict a benchmark’s c (§4.3). On all 2323 scored model–benchmark pairs, c moves toward acting in every seed, and every control moves less than the distilled student beside it (Figure 7). Sensitivity is kept: the criterion-free AUC from answer log-probabilities, available on five benchmarks, stays unchanged for every student (Table 4). Answer preference. On a Yes/No format a shift toward acting and a shift toward Yes coincide. On FlipAct most of the distilled shift flips sign under the reversed wordings of §4.3, so there it shifts the preference for Yes, the form the decision to act takes on a Yes/No format (App. D.5). Whether more than that preference carries over shows only where the answer has no Yes: if a Yes preference were all that transfers, a student distilled from a teacher whose Yes logits are raised by a constant tuned to the steered teacher’s Yes-rate would match the criterion-distilled one there. On Qwen3-8B and gemma-3-12b it does not: on When2Call and BFCL-miss-func the criterion-distilled student moves c further in every seed, though on Qwen3-8B the Yes-bias student reaches half to three quarters of it. On the four Yes/No benchmarks it moves c as far or further. Accuracy. Accuracy follows the starting point, as a pure criterion shift predicts. Where a student starts conservative, balanced accuracy rises in every case, by up to six points (Table 4; all students in App. Table 31). Where it starts permissive, accuracy falls by up to four points or stays within its error, except Mistral-24B on When2Tool: its hit rate is already at ceiling, so a shift toward acting only adds false alarms, and accuracy falls about ten points. A criterion shift should thus take its sign from the target benchmark. Steering the five models of §5.1 toward acting on five benchmarks lowers balanced accuracy on 1212 of the 2525 model–benchmark pairs; choosing the sign and strength from fifty labeled examples of each benchmark instead leaves only Mistral-24B on When2Call worse (App. D.6). Where the benchmark has an answer token, a scalar bias on that token chosen the same way does as well (1919 of 2020 pairs), as expected if steering is one offset on the criterion. 5.4 Labels move sensitivity, and the criterion with it Sensitivity learning, supervised training on the computed labels, tests the converse: whether labels move sensitivity. It uses the same students, split, adapter and seeds, adds 263263 supplementary questions on the same tool records to the training half, and shuffles the labels within matched groups of questions as its control. It raises a on every student, by about 0.50.5 to 1.21.2, and held-out accuracy by 1717 to 2424 points, while the shuffled-label control lowers accuracy (Table 2). It also brings τ close to zero, so labels move both coordinates, whereas distillation moves the criterion alone. The gain in sensitivity stays on FlipAct: on the external benchmarks its changes in d′d and c differ in sign across students and are matched by the shuffled-label control. Limitations are discussed in App. G. 6 Conclusion We treated the decision to act as a measurement problem. Two models with the same accuracy can need opposite fixes, so a when-to-act benchmark says more when it reports d′d and c beside accuracy. The criterion moves, at inference or in the weights, with a shift whose sign the target benchmark sets, while sensitivity on FlipAct rises with training on labels. On a Yes/No format the criterion also holds a preference between answer words, which reversed wordings separate from the rest. Ethics statement This work evaluates open-weight models on public benchmarks and involves no human subjects. Moving a model’s criterion toward acting makes it act on thinner evidence, a trade-off practitioners should weigh for their application. Reproducibility statement The paper specifies the full experimental setup: the construction of FlipAct and its rule (§3, App. A), the estimators (§4.1), and the steering and training procedures with all hyperparameters (§5, App. D.3). Every training condition is run with three seeds and reported as the seed mean with its bootstrap error, and every reported change comes with a significance test; steering effects carry bootstrap intervals and are checked on held-out splits. The dataset, code and evaluation logs will be released. AI use statement We used generative AI tools to generate synthetic data sets: in the FlipAct pipeline, language models annotate tool properties (§3.4, App. A), while every answer and evidence score is computed by rule. We used generative AI tools to implement methods: the evaluation, steering, training and analysis code was written with an AI coding assistant. We also used generative AI tools to aid and polish the writing. We did not use generative AI tools for the other tasks with required disclosure. The authors reviewed all AI-assisted code, data and text and take responsibility for the final content of this work. References Abdin et al. (2024a) M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y. Chen, Y. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. J. Kim, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka, A. Tupini, P. Vaddamanu, C. Wang, G. Wang, L. Wang, S. Wang, X. Wang, Y. Wang, R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang, Y. Yang, Z. Yang, D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou Phi-3 technical report: a highly capable language model locally on your phone. External Links: 2404.14219, Link Cited by: Table 10. Abdin et al. (2024b) M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905. Cited by: Table 10. Agarwal et al. (2025) S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. Gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Cited by: Table 10. Arditi et al. (2024) A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems 37, p. 136037–136083. Cited by: §D.1, Appendix F, §5.1. Baker et al. (1998) C. F. Baker, C. J. Fillmore, and J. B. Lowe The berkeley framenet project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics, Cited by: §A.4. Basant et al. (2025) A. Basant, A. Khairnar, A. Paithankar, A. Khattar, A. Renduchintala, A. Malte, A. Bercovich, A. Hazare, A. Rico, A. Ficek, et al. Nvidia nemotron nano 2: an accurate and efficient hybrid mamba-transformer reasoning model. arXiv preprint arXiv:2508.14444. Cited by: Table 10. Bean et al. (2026) A. M. Bean, R. O. Kearns, A. Romanou, F. S. Hafner, H. Mayne, J. Batzner, N. Foroutan Eghlidi, C. Schmitz, K. Korgul, H. Batra, et al. Measuring what matters: construct validity in large language model benchmarks. Advances in Neural Information Processing Systems 38. Cited by: §2. Bisk et al. (2020) Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 7432–7439. Cited by: Table 39, Appendix F. Cacioli (2026) J. Cacioli LLMs as signal detectors: sensitivity, bias, and the temperature-criterion analogy. arXiv preprint arXiv:2603.14893. Cited by: §2. Cai et al. (2024) Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297. Cited by: Table 10. Chen et al. (2026) J. Chen, S. Dong, Q. Zhou, R. Ma, Z. Dou, W. Matusik, and C. Gan Better call CineCrew: consistent ultra-long narrative-to-film generation. In European Conference on Computer Vision, p. 338–355. Cited by: Appendix G. Cohere Team (2024) Cohere Team Command R: RAG at production scale. Note: https://docs.cohere.com/docs/command-rAccessed: 2026-09-24 Cited by: Table 10. Cronbach (1946) L. J. Cronbach Response sets and test validity. Educational and psychological measurement 6 (4), p. 475–494. Cited by: Appendix C. Cui et al. (2024) J. Cui, W. Chiang, I. Stoica, and C. Hsieh Or-bench: an over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947. Cited by: Table 39, Appendix F, §1, §2. Damen et al. (2022) D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Rescaling egocentric vision: collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision 130 (1), p. 33–55. Cited by: §A.4. Dong et al. (2026a) S. Dong, J. Hu, S. Li, W. Wen, and Q. Qian Rethinking model efficiency: multi-agent inference with large models. arXiv preprint arXiv:2604.04929. Cited by: Appendix F. Dong et al. (2026b) S. Dong, J. Hu, M. Zhang, M. Yin, Y. Fu, and Q. Qian MMTok: multimodal coverage maximization for efficient inference of VLMs. In International Conference on Learning Representations, Vol. 2026, p. 48075–48099. Cited by: Appendix F. Dong et al. (2026c) S. Dong, W. Li, A. Deng, Q. Qian, V. Zhu, Z. Ji, and C. Chen Rethinking long-video efficiency: a joint allocation perspective on frames, pixels, and front-end latency. arXiv preprint arXiv:2610.04318. Cited by: Appendix F. Forbus (1984) K. D. Forbus Qualitative process theory. Artificial intelligence 24 (1-3), p. 85–168. Cited by: §1, §3.3. Glm et al. (2024) T. Glm, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. Chatglm: a family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. Cited by: Table 10. Gong et al. (2026) N. Gong, H. Li, S. Dong, J. Lian, Y. Fu, and X. Xie To think or not to think, that is the question for large reasoning models in theory of mind tasks. arXiv preprint arXiv:2602.10625. Cited by: §B.7. Granite Team (2024) I. Granite Team Granite 3.0 language models. URL: https://github. com/ibm-granite/granite-3.0-language-models. Cited by: Table 10. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: Table 10. Green et al. (1966) D. M. Green J. A. Swets et al. Signal detection theory and psychophysics. Vol. 1, Wiley New York. Cited by: §1, §2, §3.1, §4.1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: Table 10. Gupta et al. (2019) A. Gupta, P. Dollar, and R. Girshick Lvis: a dataset for large vocabulary instance segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5351–5359. Cited by: §A.4. Hastings et al. (2016) J. Hastings, G. Owen, A. Dekker, M. Ennis, N. Kale, V. Muthukrishnan, S. Turner, N. Swainston, P. Mendes, and C. Steinbeck ChEBI in 2016: improved services and an expanding collection of metabolites. Nucleic acids research 44 (D1), p. D1214–D1219. Cited by: §A.4. Huang et al. (2024) Y. Huang, J. Shi, Y. Li, C. Fan, S. Wu, Q. Zhang, Y. Liu, P. Zhou, Y. Wan, N. Gong, et al. Metatool benchmark for large language models: deciding whether to use tools and which to use. In International Conference on Learning Representations, Vol. 2024, p. 42978–43007. Cited by: Table 39, Appendix F, §1, §2. Jacobs and Wallach (2021) A. Z. Jacobs and H. Wallach Measurement and fairness. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, p. 375–385. Cited by: §1, §2. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7b. External Links: 2310.06825, Link Cited by: Table 10. Ke et al. (2026) Y. Ke, M. Yin, C. Zhao, and K. Huang A few neurons reveal when LLMs misuse tools: sparse detection and selective steering for reliable tool use. External Links: 2608.00218, Link Cited by: §1. Kirichenko et al. (2026) P. Kirichenko, M. Ibrahim, K. Chaudhuri, and S. J. Bell Abstentionbench: reasoning llms fail on unanswerable questions. Advances in Neural Information Processing Systems 38. Cited by: Table 39, Appendix F, §1, §2. Kokel et al. (2025) H. Kokel, M. Katz, K. Srinivas, and S. Sohrabi Acpbench: reasoning about action, change, and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 26559–26568. Cited by: §5.3. Kolve et al. (2017) E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: an interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474. Cited by: §A.4. Lal et al. (2024) Y. K. Lal, V. Cohen, N. Chambers, N. Balasubramanian, and R. Mooney CaT-bench: benchmarking language model understanding of causal and temporal dependencies in plans. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 19336–19354. Cited by: §5.3. Li et al. (2026) B. Li, B. Kim, and Z. Wang QuestBench: can llms ask the right question to acquire information in reasoning tasks?. Advances in Neural Information Processing Systems 38. Cited by: Table 15, Table 39, Appendix F, §1, §2, §5.3. Li et al. (2023) K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg Inference-time intervention: eliciting truthful answers from a language model. Advances in neural information processing systems 36, p. 41451–41530. Cited by: Appendix F. Lipscomb (2000) C. E. Lipscomb Medical subject headings (mesh). Bulletin of the Medical Library Association 88 (3), p. 265. Cited by: §A.4. Liu et al. (2026) X. Liu, Y. E. Zhang, V. Kasprova, P. Rabbani, P. S. Zahraei, T. Zhang, A. Ebrahimpour-Boroojeny, and V. Chandrasekaran AgentAbstain: do llm agents know when not to act?. arXiv preprint arXiv:2607.10059. Cited by: Table 39, Appendix F. McCoy et al. (2019) R. T. McCoy, E. Pavlick, and T. Linzen Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 3428–3448. Cited by: §3.2. Miller (1995) G. A. Miller WordNet: a lexical database for english. Communications of the ACM 38 (11), p. 39–41. Cited by: §A.4. Mistral AI (2025) Mistral AI Mistral small 3. Note: Model release Cited by: Table 10. NVIDIA (2026) NVIDIA NVIDIA-Nemotron-3-Nano-4B. Note: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16Model card Cited by: Table 10. OLMo et al. (2024) T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. Cited by: Table 10. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §D.5. Patil et al. (2025) S. G. Patil, H. Mao, F. Yan, C. C. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez The berkeley function calling leaderboard (bfcl): from tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, Cited by: Table 15, Table 39, Appendix F, §1, §2, §5.3. Perez et al. (2023) E. Perez, S. Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the association for computational linguistics: ACL 2023, p. 13387–13434. Cited by: Appendix F. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: Table 10. Ribeiro et al. (2020) M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh Beyond accuracy: behavioral testing of nlp models with checklist. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 4902–4912. Cited by: §3.2. Rimsky et al. (2024) N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15504–15522. Cited by: §D.1, Appendix F, §5.1. Ross et al. (2025) H. Ross, A. S. Mahabaleshwarkar, and Y. Suhara When2Call: when (not) to call tools. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 3391–3409. Cited by: Table 15, Table 39, Appendix F, §1, §1, §2, §5.1. Röttger et al. (2024) P. Röttger, H. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy Xstest: a test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 5377–5400. Cited by: §D.1, Table 39, Appendix F, §1, §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §D.5. Sharma et al. (2024) M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. Bowman, E. Durmus, Z. Hatfield-Dodds, S. Johnston, S. Kravec, et al. Towards understanding sycophancy in language models. In International Conference on Learning Representations, Vol. 2024, p. 110–144. Cited by: Appendix F. Shi et al. (2024) C. Shi, X. Wang, Q. Ge, S. Gao, X. Yang, T. Gui, Q. Zhang, X. Huang, X. Zhao, and D. Lin Navigating the overkill in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4602–4614. Cited by: Appendix F, §1, §2. Shi et al. (2026) W. Shi, Z. Peng, S. Li, X. Wang, X. Wang, M. Du, and N. Zou To call or not to call: diagnosing intrinsic over-calling bias in llm agents. arXiv preprint arXiv:2605.18882. Cited by: Table 39, §2. Speer et al. (2017) R. Speer, J. Chin, and C. Havasi Conceptnet 5.5: an open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, Vol. 31. Cited by: §A.4. Su and Cardie (2026) J. Su and C. Cardie Knowing but not showing: llms recognize ambiguity but rarely ask clarifying questions. arXiv preprint arXiv:2605.25284. Cited by: Table 39, Appendix F. Sun et al. (2026) C. Sun, L. Liu, G. Yan, Z. Wang, and T. Weng LLM agents already know when to call tools–even without reasoning. arXiv preprint arXiv:2605.09252. Cited by: Table 15, Table 39, Appendix F, §1, §2, §5.3. Team et al. (2026) G. Team, S. E. Abd, V. Aggarwal, R. Algayres, A. Andreev, O. Bachem, I. Ballantyne, C. Brick, V. Cărbune, M. Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770. Cited by: Table 10. Team et al. (2025) G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: Table 10. Team et al. (2024) G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: improving open language models at a practical size. arXiv preprint arXiv:2408.00118. Cited by: Table 10. Turner et al. (2023) A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid Steering language models with activation engineering. arXiv preprint arXiv:2308.10248. Cited by: Appendix F, §1, §5.1. TypeSafe AI (2026) TypeSafe AI TypeSafe AI documentation: system one and the noul, choice and score question types. Note: https://docs.typesafe.aiAccessed 2026-09-22; model jev-latest, API version jev-1.13.0 Cited by: Appendix E, §4.3. Vrandečić and Krötzsch (2014) D. Vrandečić and M. Krötzsch Wikidata: a free collaborative knowledgebase. Communications of the ACM 57 (10), p. 78–85. Cited by: §A.4. Wang et al. (2025) C. Wang, W. Luo, S. Dong, X. Xuan, Z. Li, L. Ma, and S. Gao MLLM-Tool: a multimodal large language model for tool agent learning. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 6678–6687. Cited by: §1. Wikipedia contributors (2026) Wikipedia contributors Wikipedia, the free encyclopedia. Note: https://w.wikipedia.orgAccessed 2026 Cited by: §A.4. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Table 10. Yin et al. (2026) M. Yin, D. Shen, S. Xu, S. Dong, M. Zhang, Y. Hu, S. Liu, J. Han, S. Ma, S. Wang, S. R. Indurthi, X. Wang, Y. Chen, and K. Song LiveMCP-101: stress testing and diagnosing MCP-enabled agents on challenging queries. External Links: 2508.15760, Link Cited by: §1. Young et al. (2024) A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chen, et al. Yi: open foundation models by 01. ai. arXiv preprint arXiv:2403.04652. Cited by: Table 10. Zhang et al. (2026) M. Zhang, S. Liu, S. Dong, M. Yin, Y. Hu, S. Wang, S. Ma, S. Wang, S. R. Indurthi, H. Deng, et al. LogicIF: towards complex logic instruction following. In Third Conference on Language Modeling, Cited by: Appendix G. Zou et al. (2023) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: Appendix F, §1, §5.1. Appendix A Construction of FlipAct This section details the construction summarized in §3: the annotation schema, the rule that turns a tool into its questions, a worked quad, the build with its counts, and the checks on data quality and on annotation noise. A.1 Annotation schema Each annotator model (App. A.4) fills one record per candidate entity against the closed vocabulary of Table 5. A value outside the vocabulary is a parse failure and is never mapped to a neighboring value. A tool that acts on two axes becomes one record per axis, and each record carries a single direction. Table 5: The annotation schema: the six physical axes, each with its two problem poles as the questions word them and an example tool in each direction, and the other fields of a tool record. Example tool Axis Quantity Problem poles adds (δ=+1δ=+1) removes (δ=−1δ=-1) Thermal transfer temperature cold / hot pocket hand warmer ice pack Radiant emission visible light dark / glaringly bright electric lamp blackout curtain Mechanical cohesion connectedness loose / jammed thread lock compound puller Spatial containment seal exposed / sealed shut lid cap opener Electromagnetism circuit continuity without power / overloaded ac adapter emergency stop button Moisture level moisture dry / wet humidifier industrial dehumidifier Other fields of a record is-tool yes / no; a record that is not a tool leaves the build Direction δ +1+1 or −1-1 Magnitude μ 11 local, on one object or spot (hot water bottle, paper towel); 22 room or person scale (hair dryer, space heater); 33 building, vehicle or industrial scale (furnace, industrial dehumidifier) Target object 1616 classes and other, e.g. a person’s body, a room, a container, a rigid object or joint, a circuit or device, soil or a plant Prerequisite mains electricity, running water, fuel Side effect any of the six axes, or electronic integrity, where the tool has one; not used by the rule A.2 From a tool to its questions The state of every question is computed from the tool, with the goal at 00. A solution question places the state at v=−δμv=-δμ, the deficit the tool closes. A destructive question uses the deficit of the paired tool, which points the other way, so the tool widens the gap. A redundant question uses a deficit of one met by a magnitude of at least two, so the tool crosses the goal and leaves the state no closer. A prerequisite question removes the supply a tool depends on from a solution state. Two tools are paired only when they act on the same axis and target object in opposite directions and differ in magnitude by at most one band, so each tool’s solution state is its partner’s destructive state. A pocket hand warmer pairs with an ice pack on a person, never with an air conditioner in a room. Prerequisite questions and questions whose state is already at the goal are not part of the main set. They go to the two supplementary sets of §3.4. Every question is rendered from the template of Figure 2, with the state adverb slightly, very or dangerously for |v|=1,2,3|v|=1,2,3. For a tool with a prerequisite, a sentence between the state and the goal reads “The supplies this tool depends on (mains electricity) are available nearby” or “None of the supplies this tool depends on (mains electricity) is available anywhere nearby.” It appears in every question of such a tool, including those whose supply is present, so its presence carries no information about the answer. A.3 A worked quad Table 6 shows one tool pair on the thermal-transfer axis with a person’s body as the target. Tool X is a blanket, which warms (δ=+1δ=+1, μ=1μ=1), and tool Y is an ice bath, which cools (δ=−1δ=-1, μ=2μ=2). State A is slightly cold (v=−1v=-1) and state B is very hot (v=+2v=+2). The two off-diagonal questions are destructive: cooling a body that is already cold, and wrapping one that is already hot. The fifth question keeps the ice bath and lowers the deficit to one (slightly hot). The ice bath still points the right way, but its magnitude of two now overshoots the goal, so magnitude alone separates this No from the ice bath’s Yes in state B. Table 6: One tool pair: the four questions of the quad and the ice bath’s redundant question, with the rule’s computation for each. State Tool Outcome v δ μ v+δμv+δμ ΔU U Type Answer A: slightly cold −1-1 blanket +1+1 11 00 +1+1 solution Yes −1-1 ice bath −1-1 22 −3-3 −2-2 destructive No B: very hot +2+2 ice bath −1-1 22 00 +2+2 solution Yes +2+2 blanket +1+1 11 +3+3 −1-1 destructive No slightly hot +1+1 ice bath −1-1 22 −1-1 00 redundant No A.4 The build Ten lexical, medical and visually grounded resources supply candidate entities: ConceptNet (Speer et al., 2017), Wikipedia (Wikipedia contributors, 2026) and Wikidata (Vrandečić and Krötzsch, 2014) for breadth; WordNet (Miller, 1995), LVIS (Gupta et al., 2019), EPIC-Kitchens (Damen et al., 2022) and AI2-THOR (Kolve et al., 2017) for objects a person handles; MeSH (Lipscomb, 2000), ChEBI (Hastings et al., 2016) and FrameNet (Baker et al., 1998) for specialist vocabulary. Figure 8 gives the counts at each step. Six filter levels remove duplicate names, abstract suffixes, lemma duplicates, abstract hypernym paths and synonyms, and reclassify domains, leaving 149,840149,840 entities. No language model proposes an entity. Tool-property annotation runs on the 72,44672,446 entities in domains that can hold a physical tool. Qwen3.5-27B and gemma-4-31b each annotate every property, and each annotator’s reading is the majority over three differently worded prompts at temperature zero. Where the two disagree, both arbitrate against the written specification, and a row on which the arbiters also disagree is dropped. Most entities are not tools: the annotators reject 70,01770,017 of 73,17173,171 annotation rows. Screens under the same protocol check that each remaining tool acts on its axis, is deployable in the situations the template renders, and has a magnitude band. Tools mined a second time for under-supplied groups pass the same screens (orange in Figure 8). A last screen removes 9696 rows whose tool is not the standard instrument of any legitimate trade (a weapon, for example), leaving 1,6381,638 tool records. Pairing yields 1,4221,422 tool pairs, and the grid of states over them yields 6,1766,176 questions. Removing texts that collide with each other or with an evaluation set used in the paper leaves 5,7515,751. The final selection drops every situation whose questions all have the same answer: a situation recurs across tools, so its answer could otherwise be memorized. It then balances Yes and No within each remaining situation and caps any one axis, leaving the 1,8801,880 questions of the main set (Table 7). Table 7: Number of questions in the main set by question type, physical axis and evidence score. Question type Physical axis Evidence score |ΔU|| U| solution 940940 Mechanical cohesion 564564 00 231231 destructive 705705 Thermal transfer 476476 11 798798 redundant 235235 Electromagnetism 234234 22 719719 Spatial containment 226226 33 132132 Moisture level 212212 Radiant emission 168168 Figure 8: Counts at each stage of the FlipAct construction. A1: the ten entity sources. A2: the six filter levels that produce the entity pool. B: the pool through tool-property annotation and the screens to tool records, then tool pairs, situations and questions. A1 and B use a log scale, A2 a linear one. A.5 Data quality The answer cannot be read from the wording. A lexical probe (TF-IDF features, logistic regression) is trained on the question text of some situations and read on the others. Every question of a tool that appears in the held-out situations is removed from training. Because of the state flips, such a tool would otherwise sit in training in the opposite state with the opposite answer, and the probe would learn the tool’s name with the wrong sign. The probe reaches an AUC of 0.498±0.0050.498± 0.005 on the main set, inside the range of 0.4850.485 to 0.5240.524 it reaches with permuted labels, and a lexical proxy for the answer correlates with the label at |r|=0.022|r|=0.022. The set is balanced and agrees with the rule. Yes and No are exactly balanced in each of the 180180 situations, with no imbalance per axis or per state level, and 3.9%3.9\% of the main set’s tool records have Yes and No counts that differ by more than one. Recomputing every answer from the annotated properties, independently of the build, gives zero mismatches, and no question text is duplicated or shared with an evaluation set used in the paper. The tool annotations, the one step where a model’s judgment enters, are reproduced by a model family that took no part in the build. GLM-4-32B re-annotated a stratified sample of 150150 rows with the same prompts and matched the recorded value on 100%100\% of is-tool judgments and 92.0%92.0\% of axis judgments. On magnitude it placed all 150150 tools within one band of the record, 67.3%67.3\% in the same band. Each of its other 4949 judgments moves a tool from the lowest or the highest band to the middle one, so it never reverses the order of two tools by magnitude. Perturbing the annotations at this rate keeps both rankings of the models above 0.980.98 and moves every criterion toward acting (App. A.6), so annotation error does not produce the conservative criteria. A.6 Robustness to annotation noise We test whether annotation error at the rates of App. A.5 could reorder the models or produce the conservative criteria. The perturbation acts on the magnitude band, the annotated property that sets the size of the evidence score. We hold every model’s answers fixed, perturb the magnitude of a random share of tool records, recompute the evidence score and answer of every affected question, and refit (a,τ)(a,τ), 300300 times per condition. N1 moves 32.7%32.7\% of records by one band at random, the rate at which GLM-4-32B disagrees on magnitude. N2 follows its disagreement pattern, one band toward the middle. N3 scales N2 until 26%26\% of computed answers change, a stress level above both. A synthetic observer with each model’s fitted a and τ=0τ=0 or +2.5+2.5 goes through the same perturbation. Rankings are compared by Spearman correlation with the unperturbed fit, and a model’s criterion shift is the median over replicates minus its unperturbed value. Table 8: Coordinates under three levels of annotation noise, 300300 replicates per condition. The last block is the criterion recovered for a synthetic observer. N1 uniform N2 empirical N3 stress Answers flipped 10.7%10.7\% 15.7%15.7\% 26.0%26.0\% Sensitivity and ranking ρ(a)ρ(a), median 0.9920.992 0.9910.991 0.9840.984 ρ(τ)ρ(τ), median 0.9960.996 0.9950.995 0.9900.990 a ratio, perturbed / unperturbed, median 0.9800.980 0.9650.965 0.9760.976 Criterion Shift, mean over models −0.19-0.19 −0.21-0.21 −0.39-0.39 Models shifting by more than ±1± 1 00 00 00 Synthetic observer, recovered τ τtrue=0 _true=0, median −0.21-0.21 −0.27-0.27 −0.44-0.44 τtrue=0 _true=0, 95th percentile 0.000.00 −0.06-0.06 −0.22-0.22 τtrue=+2.5 _true=+2.5, median +2.38+2.38 +2.41+2.41 +2.28+2.28 ρ(τ)ρ(τ) is over the 2525 entries whose criterion lies inside the score range, and ρ(a)ρ(a) over 3030 entries whose slope fit is stable. Noise at the observed rate lowers sensitivity by two to four percent, keeps both rankings above 0.980.98, and moves every criterion toward acting (Table 8). The dominant error relabels a magnitude-11 tool as magnitude 22, which moves a Yes the model gave at ΔU=+1 U=+1 to ΔU=0 U=0 and pulls the fitted crossing point down. An observer with τ=0τ=0 is never recovered at +1+1 or above in 7,5007,500 replicates, and one at +2.5+2.5 is recovered within a quarter of a unit. Annotation error therefore does not produce the conservative criteria, and if anything it understates them. The one model within 0.150.15 of zero, Qwen3-32B, crosses to a small negative value, and every other positive criterion stays positive in every replicate. Refitting on the 994994 tool records whose magnitude band the two annotators agreed on without arbitration gives ρ(a)=0.994ρ(a)=0.994 and ρ(τ)=0.995ρ(τ)=0.995 and moves no criterion by more than twice its bootstrap error. Appendix B Per-model coordinates and measurement checks Table 9 gives the coordinates of all 4444 models and Table 10 the checkpoint behind each. The evaluation protocol comes first (App. B.1), and the other subsections follow the order of §4: the fit and its readouts (App. B.2 and B.3), the convergence of sensitivity with external benchmarks (App. B.4 and B.5), and the criterion across generations and reasoning modes within a model line (App. B.6 and B.7). Table 9: Coordinates of all 4444 models on FlipAct and their When2Call d′d and c, grouped by family with the reasoning-on entries last, and two constant policies for reference. Checkpoints are in Table 10. Coordinates FlipAct scores When2Call Model a τ ACC SQPR F1F_1 TNR MCC d′d c Panel Constant policies Always-No n/a n/a 0.500.50 0.180.18 n/a 1.001.00 n/a n/a n/a Always-Yes n/a n/a 0.500.50 0.350.35 0.670.67 0.000.00 n/a n/a n/a Qwen3.5 Qwen3.5-0.8B −0.52-0.52† n/a 0.500.50 0.190.19 0.070.07 0.950.95 −0.02-0.02 0.270.27 +0.22+0.22 Qwen3.5-2B 0.400.40† n/a 0.510.51 0.210.21 0.160.16 0.920.92 0.020.02 0.410.41 +0.39+0.39 Qwen3.5-4B 0.550.55 +2.93+2.93 0.610.61 0.370.37 0.450.45 0.900.90 0.270.27 1.311.31 −0.71-0.71 ∙ Qwen3.5-9B 0.660.66 +2.88+2.88 0.620.62 0.390.39 0.440.44 0.930.93 0.300.30 1.671.67 −0.52-0.52 ∙ Qwen3.5-27B 0.570.57 +2.68+2.68 0.620.62 0.400.40 0.480.48 0.900.90 0.300.30 2.002.00 +0.06+0.06 ∙ Qwen3.5-35B-A3B 0.580.58 >+3>+3 0.600.60 0.350.35 0.390.39 0.930.93 0.260.26 1.431.43 −0.68-0.68 Qwen3 Qwen3-0.6B n/a n/a 0.500.50 0.350.35 0.670.67 0.000.00 n/a n/a n/a Qwen3-1.7B 0.460.46† n/a 0.500.50 0.350.35 0.660.66 0.060.06 0.020.02 n/a n/a Qwen3-4B 0.340.34 >+3>+3 0.540.54 0.260.26 0.270.27 0.910.91 0.120.12 1.441.44 −0.22-0.22 Qwen3-8B 0.440.44 +2.78+2.78 0.610.61 0.390.39 0.480.48 0.850.85 0.240.24 1.461.46 −0.26-0.26 ∙ Qwen3-14B 0.570.57 +2.36+2.36 0.620.62 0.400.40 0.500.50 0.870.87 0.280.28 1.461.46 −0.40-0.40 ∙ Qwen3-32B 0.580.58 +0.14+0.14 0.690.69 0.540.54 0.690.69 0.690.69 0.380.38 1.631.63 +0.01+0.01 ∙ Gemma gemma-2-2b 0.050.05† n/a 0.510.51 0.270.27 0.460.46 0.590.59 0.010.01 0.600.60 +0.27+0.27 gemma-2-9b 0.340.34 +2.91+2.91 0.590.59 0.380.38 0.490.49 0.800.80 0.210.21 0.990.99 +0.76+0.76 ∙ gemma-2-27b 0.440.44 +1.68+1.68 0.630.63 0.420.42 0.560.56 0.770.77 0.260.26 1.211.21 −0.00-0.00 ∙ gemma-3-1b 0.110.11† n/a 0.510.51 0.310.31 0.580.58 0.340.34 0.030.03 n/a n/a gemma-3-4b 0.120.12† n/a 0.540.54 0.310.31 0.470.47 0.690.69 0.090.09 n/a n/a gemma-3-12b 0.390.39 +2.17+2.17 0.620.62 0.410.41 0.540.54 0.790.79 0.250.25 1.321.32 −0.37-0.37 ∙ gemma-3-27b 0.450.45 +1.77+1.77 0.630.63 0.430.43 0.560.56 0.790.79 0.280.28 1.461.46 −0.38-0.38 ∙ gemma-4-31b 0.500.50 +1.57+1.57 0.650.65 0.450.45 0.580.58 0.800.80 0.310.31 2.182.18 −0.41-0.41 ∙ InternLM InternLM2.5-1.8B 1.271.27† n/a 0.500.50 0.350.35 0.670.67 0.000.00 0.010.01 n/a n/a InternLM2.5-7B 0.250.25 −1.03-1.03 0.590.59 0.400.40 0.610.61 0.520.52 0.170.17 0.850.85 −0.17-0.17 InternLM2.5-20B 0.410.41 <−3<-3 0.600.60 0.440.44 0.690.69 0.330.33 0.250.25 1.001.00 +0.00+0.00 InternLM3-8B 0.490.49 >+3>+3 0.570.57 0.310.31 0.340.34 0.930.93 0.210.21 n/a n/a Yi Yi-1.5-6B 0.150.15† n/a 0.530.53 0.300.30 0.410.41 0.740.74 0.080.08 n/a n/a Yi-1.5-9B 0.140.14† n/a 0.550.55 0.380.38 0.620.62 0.370.37 0.120.12 0.630.63 −0.05-0.05 Yi-1.5-34B 0.430.43 +0.41+0.41 0.640.64 0.470.47 0.630.63 0.670.67 0.280.28 1.011.01 +0.13+0.13 ∙ Mistral Mistral-7B-v0.3 0.260.26 <−3<-3 0.540.54 0.400.40 0.660.66 0.210.21 0.110.11 0.310.31 −0.16-0.16 Mistral-Small-24B (2501) 0.360.36 +2.18+2.18 0.610.61 0.410.41 0.540.54 0.780.78 0.240.24 1.181.18 −0.04-0.04 ∙ Mistral-Small-3.2-24B (2506) 0.430.43 −1.10-1.10 0.640.64 0.500.50 0.680.68 0.520.52 0.290.29 1.101.10 −0.33-0.33 ∙ Llama Llama-3.1-8B 0.100.10† n/a 0.550.55 0.340.34 0.570.57 0.490.49 0.090.09 0.290.29 −1.07-1.07 Llama-3.3-70B 0.650.65 +1.29+1.29 0.670.67 0.490.49 0.620.62 0.820.82 0.360.36 1.541.54 −0.39-0.39 ∙ Phi Phi-3.5-mini 0.300.30 +1.85+1.85 0.590.59 0.380.38 0.540.54 0.720.72 0.200.20 0.880.88 −0.64-0.64 ∙ phi-4 0.510.51 +1.67+1.67 0.640.64 0.440.44 0.570.57 0.800.80 0.290.29 1.801.80 −0.28-0.28 ∙ Other families GLM-4-32B-0414 0.670.67 +1.89+1.89 0.660.66 0.450.45 0.560.56 0.870.87 0.350.35 1.701.70 +0.14+0.14 ∙ Command-R-08-2024 0.450.45 >+3>+3 0.570.57 0.310.31 0.340.34 0.920.92 0.200.20 0.920.92 −0.56-0.56 Granite-3.3-8B 0.300.30 +2.59+2.59 0.590.59 0.370.37 0.510.51 0.760.76 0.190.19 0.770.77 −0.16-0.16 OLMo-2-1124-13B 0.260.26 −2.31-2.31 0.590.59 0.410.41 0.640.64 0.440.44 0.180.18 0.640.64 +1.18+1.18 Nemotron-Nano-9B-v2 1.071.07† n/a 0.500.50 0.180.18 0.010.01 1.001.00 0.020.02 n/a n/a Reasoning on Qwen3-30B-A3B-Thinking-2507 0.550.55 >+3>+3 0.620.62 0.390.39 0.450.45 0.920.92 0.300.30 n/a n/a DeepSeek-R1-Distill-Llama-8B 0.300.30 +1.47+1.47 0.610.61 0.440.44 0.570.57 0.700.70 0.220.22 n/a n/a DeepSeek-R1-Distill-Qwen-32B 0.480.48 +1.48+1.48 0.650.65 0.470.47 0.590.59 0.790.79 0.310.31 n/a n/a DeepSeek-R1-Distill-Llama-70B 0.480.48 +0.59+0.59 0.660.66 0.500.50 0.640.64 0.710.71 0.320.32 n/a n/a gpt-oss-20b 0.530.53 +1.73+1.73 0.650.65 0.460.46 0.580.58 0.820.82 0.320.32 n/a n/a a is sensitivity and τ the criterion, positive when conservative. A τ beyond the score range is shown as a bound, and † marks a fit whose criterion is not estimable. ACC, F1F_1 (Yes class), true-negative rate (TNR) and Matthews correlation (MCC) are over the 1,8801,880 main-set questions, and SQPR is the share of its 1,1411,141 tool records with every question answered correctly. ∙ marks the panel of §4.2, and n/a a value that is undefined or not available. Table 10: The Hugging Face checkpoint behind each entry and the precision of its weights, grouped by family with the family’s reference. The reasoning-mode comparison of App. B.7 also runs the listed checkpoints with reasoning on. Entry Checkpoint Weights Note Qwen3.5 (Qwen Team, 2026) Qwen3.5-0.8B Qwen/Qwen3.5-0.8B bf16 Qwen3.5-2B Qwen/Qwen3.5-2B bf16 Qwen3.5-4B Qwen/Qwen3.5-4B bf16 Qwen3.5-9B Qwen/Qwen3.5-9B bf16 Qwen3.5-27B Qwen/Qwen3.5-27B bf16 Qwen3.5-35B-A3B Qwen/Qwen3.5-35B-A3B bf16 Qwen3 (Yang et al., 2025) Qwen3-0.6B Qwen/Qwen3-0.6B bf16 Qwen3-1.7B Qwen/Qwen3-1.7B bf16 Qwen3-4B Qwen/Qwen3-4B bf16 Qwen3-8B Qwen/Qwen3-8B bf16 Qwen3-14B Qwen/Qwen3-14B bf16 Qwen3-32B Qwen/Qwen3-32B bf16 Qwen3-30B-A3B-Thinking-2507 Qwen/Qwen3-30B-A3B-Thinking-2507 bf16 reasoning on Gemma (Team et al., 2024; Team et al., 2025; Team et al., 2026) gemma-2-2b google/gemma-2-2b-it bf16 gemma-2-9b google/gemma-2-9b-it bf16 gemma-2-27b google/gemma-2-27b-it bf16 gemma-3-1b google/gemma-3-1b-it bf16 gemma-3-4b google/gemma-3-4b-it bf16 gemma-3-12b google/gemma-3-12b-it bf16 gemma-3-27b google/gemma-3-27b-it bf16 gemma-4-31b google/gemma-4-31b-it bf16 InternLM (Cai et al., 2024) InternLM2.5-1.8B internlm/internlm2_5-1_8b-chat bf16 InternLM2.5-7B internlm/internlm2_5-7b-chat bf16 InternLM2.5-20B internlm/internlm2_5-20b-chat bf16 InternLM3-8B internlm/internlm3-8b-instruct bf16 Yi (Young et al., 2024) Yi-1.5-6B 01-ai/Yi-1.5-6B-Chat bf16 Yi-1.5-9B 01-ai/Yi-1.5-9B-Chat bf16 Yi-1.5-34B 01-ai/Yi-1.5-34B-Chat bf16 Mistral (Jiang et al., 2023; Mistral AI, 2025) Mistral-7B-v0.3 mistralai/Mistral-7B-Instruct-v0.3 bf16 Mistral-Small-24B (2501) mistralai/Mistral-Small-24B-Instruct-2501 bf16 Mistral-Small-3.2-24B (2506) mistralai/Mistral-Small-3.2-24B-Instruct-2506 bf16 Llama (Grattafiori et al., 2024) Llama-3.1-8B meta-llama/Llama-3.1-8B-Instruct bf16 Llama-3.3-70B kosbu/Llama-3.3-70B-Instruct-AWQ 4-bit AWQ Phi (Abdin et al., 2024a; Abdin et al., 2024b) Phi-3.5-mini microsoft/Phi-3.5-mini-instruct bf16 phi-4 microsoft/phi-4 bf16 GLM (Glm et al., 2024) GLM-4-32B-0414 zai-org/GLM-4-32B-0414 bf16 Command R (Cohere Team, 2024) Command-R-08-2024 AMead10/c4ai-command-r-08-2024-awq 4-bit AWQ Granite (Granite Team, 2024) Granite-3.3-8B ibm-granite/granite-3.3-8b-instruct bf16 OLMo (OLMo et al., 2024) OLMo-2-1124-13B allenai/OLMo-2-1124-13B-Instruct bf16 Nemotron (Basant et al., 2025; NVIDIA, 2026) Nemotron-Nano-9B-v2 nvidia/NVIDIA-Nemotron-Nano-9B-v2 bf16 Nemotron-3-Nano-4B nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 bf16 App. B.7 only DeepSeek-R1 distillations (Guo et al., 2025) DeepSeek-R1-Distill-Llama-8B deepseek-ai/DeepSeek-R1-Distill-Llama-8B bf16 reasoning on DeepSeek-R1-Distill-Qwen-32B deepseek-ai/DeepSeek-R1-Distill-Qwen-32B bf16 reasoning on DeepSeek-R1-Distill-Llama-70B casperhansen/deepseek-r1-distill-llama-70b-awq 4-bit AWQ reasoning on gpt-oss (Agarwal et al., 2025) gpt-oss-20b openai/gpt-oss-20b MXFP4 (native) reasoning on B.1 Evaluation protocol FlipAct. Each question is one user turn, rendered with the model’s own chat template and its generation prompt, with no system message added. After the template’s closing “Answer Yes or No.” the prompt adds a new line, “Answer with exactly one word: Yes or No.” With reasoning off, every question of the main and supplementary sets is answered once with vLLM, sampling at temperature 0.70.7, top-p 0.80.8 and top-k 2020 with seed 00 and at most 88 new tokens, the same settings for every model except gemma-4-31b, which is run with transformers and greedy decoding. The answer is the first standalone yes or no in the output, ignoring case. An output with neither is unparsed and left out of the fit. No entry has more than one unparsed main-set question. Two further conditions guard the panel against unreadable output: at most 30%30\% unparsed FlipAct answers and a When2Call d′d of at least 0.50.5. In practice neither removes a model that the conditions of §4.2 admit. The fit of Eq. 2 is a maximum-likelihood logistic regression on these binary answers, so every Yes-rate and P(Yes)P(Yes) on FlipAct is a proportion of sampled answers, not a token probability. SE(τ)SE(τ) is half the width of the profile-likelihood interval within which the deviance rises by at most one. Nine entries were also run with seed 11. The two runs give the same answer on 9393 to 100%100\% of main-set questions, and for the six of the nine with a well-determined criterion both coordinates of the second run fall inside the 95%95\% bootstrap intervals of the first. The steering and training readouts of §5 use the same prompt with reasoning off but do not sample: an answer is Yes when the logit of the first token of “Yes” exceeds that of “No” at the answer position. Reasoning on. The five reasoning-on entries of Table 9 and the reasoning-on condition of App. B.7 receive the same prompt, one-word instruction included. Hybrid models reason through their template’s switch (enable_thinking, or thinking for Granite-3.3-8B), and gpt-oss-20b runs at its template’s default reasoning effort, medium. Decoding samples at temperature 0.60.6, top-p 0.950.95 and top-k 2020 with seed 00, within a budget of 8,1928,192 new tokens (4,0964,096 for Granite-3.3-8B and Qwen3.5-35B-A3B). The answer is the first standalone yes or no after the end of the reasoning trace, which is the last </think> or, for gpt-oss-20b, the start of its final channel. An answer is unparsed when its trace reaches the budget without closing or is followed by no Yes or No, and no answer is read from inside a trace. A model is not fit in a mode where more than 10%10\% of its answers are unparsed, which removes Qwen3.5-0.8B and Qwen3.5-2B with reasoning on (78%78\% and 23%23\%). Every other condition stays below 6%6\%, and the five reasoning-on entries of Table 9 have no unparsed answer. External benchmarks are run with reasoning off only, so reasoning-on entries have no external readout and are not in the panel. External benchmarks. Table 11 describes the seven benchmarks. Each example is one user turn through the chat template with reasoning off, decoded greedily with vLLM (with transformers for gemma-4-31b), with at most 88 new tokens on QuestBench, 6464 on When2Call, 128128 on BFCL and 256256 on the other four. Yes/No replies are parsed as on FlipAct. A When2Call reply is read as the letter it starts with, else a letter after a cue such as “answer” or “option” in its first 8080 characters, else any standalone letter A–D there. On both BFCL sets a reply acts when it contains CALL: f(…) or calls a target function by name without that prefix. A reply with no parsable answer counts as withholding and stays in the denominators. No reply in the transfer test is unparsed. In the panel, at most 0.3%0.3\% of replies are unparsed on the Yes/No benchmarks, and on When2Call the highest rate is 12.5%12.5\%, for Mistral-Small-3.2-24B. Table 9 shows an entry’s When2Call values as n/a when more than 30%30\% of its replies are unparsed, which applies only to Nemotron-Nano-9B-v2. Runs with a steering hook (App. D.1 and D.2) use transformers with greedy decoding and at most 88 new tokens (6464 on BFCL-miss-func), on a balanced sample of 600600 examples of When2Call, When2Tool and CaT-Bench and on all examples of ACPBench and BFCL-miss-func. The BFCL set of the panel draws 8080 single-turn queries at random from BFCL’s live-simple category, each answered by its one listed function, and 3636 from its live-irrelevance category, where none of the one to ten listed functions fits the query. Each function is restated as its name, description and parameter names, under an instruction to reply with one CALL: line if a tool is needed and to answer directly otherwise. Table 11: The external benchmarks, grouped by the comparison that uses them: source split, number of examples in each class, the act and withhold examples, and the reply format. Benchmark Act / withhold Act example (withhold example) Reply format Panel comparison (§4.2) BFCL v3 live simple, live irrelevance 8080 / 3636 a query its one listed function answers (a query no listed function fits) free-form; CALL: f(…) acts Panel comparison and transfer test (§4.2, §5.3) QuestBench Logic-Q (500500 of 1,1501,150 tasks), GSM-Q CSP (151151), GSM-Q verbal (288288) 939939 / 939939 a fully specified task (the same task with one variable removed); acting is Yes to “Do you already have enough information to give a single correct final answer …?” Yes/No When2Call test, multiple choice 1,2951,295 / 2,3572,357 a request whose correct reply is a tool call (one whose correct reply asks for missing information, 1,0621,062, or says no tool can answer, 1,2951,295); acting is choosing the tool-call reply one letter for four replies: call, ask, cannot answer, answer directly When2Tool single-hop test, 1515 environments 750750 / 750750 a hard-tier task, built to need a tool (an easy-tier task, solvable without one; the 750750 medium-tier tasks are not scored); acting is Yes to “would you need to use one of the tools?” Yes/No Transfer test (§5.3) BFCL-miss-func v3 multi-turn miss-func, 200200 scenarios 200200 / 200200 a request with the one to three functions it needs listed (the same request with those functions removed) free-form; CALL: f(…) acts ACPBench applicability, test and validation 8888 / 8282 an action applicable in the stated planning state (an inapplicable action); acting is Yes Yes/No CaT-Bench test, 1,4201,420 step pairs in both directions 1,4201,420 / 1,4201,420 two recipe steps of which one must precede the other (two independent steps); acting is Yes Yes/No On every benchmark, and in the signal-detection readout of App. B.3, d′d and c follow Eq. 1 with the log-linear correction H=(hits+12)/(nact+1)H=(hits+ 12)/(n_act+1) and FA=(false alarms+12)/(nwithhold+1)FA=(false alarms+ 12)/(n_withhold+1), applied to every rate so that z stays finite for a model that always or never acts. For the criterion-free AUC (App. B.4 and D.4), the benchmarks it covers are run again with the same greedy decoding, recording the 2020 most likely tokens and their log-probabilities at the first four generated positions. At the first position whose generated token is an answer, the margin is the log-probability of acting (Yes, or the tool-call letter) minus that of the strongest alternative (No, or the most likely other letter), each answer taking its most likely spelling among the recorded tokens. A model–benchmark pair is scored only if the sign of its margin reproduces at least 99%99\% of its decoded answers. Examples with no answer token among the recorded positions, and When2Call replies with no parsable letter, are left out; Mistral-Small-3.2-24B loses the most (959959 of 1,8781,878 QuestBench and 463463 of 3,6523,652 When2Call examples). gemma-4-31b, run with transformers, has no recorded log-probabilities and is absent from these readouts. B.2 Reliability and response profile Both coordinates are reliable. Refitting on 200200 random halves of the tool records with the Spearman–Brown correction gives a split-half reliability of 0.9590.959 for a and 0.9910.991 for τ. Tool records share question text across situations, so the stricter unit is the situation cluster, all questions that share one situation text. It is the resampling unit for every interval in the paper. Split by cluster, the two reliabilities are 0.9060.906 and 0.9730.973. Models respond to the sign of the evidence score and ignore its size (Table 12 and Figure 4). The jump in Yes-rate between ΔU=−1 U=-1 and 00 is significant in 1919 of the 2121 models with a well-determined criterion, and the jump between 00 and +1+1 in none. Within either side, no model answers Yes reliably more often at the larger magnitude. The step therefore sits one unit below the rule’s boundary. Every question at ΔU=0 U=0 is a slight overshoot, and models accept these overshoots about as often as helpful actions: the overshoot Yes-rate lies nearer the hit rate than the false-alarm rate in most models, and its interval includes the hit rate in 2020 of 2121. Table 12: Median Yes-rate at each evidence score over the 2121 models with a well-determined criterion, the median line of Figure 4. ΔU U −3-3 −2-2 −1-1 00 +1+1 +2+2 +3+3 Median Yes-rate 0.140.14 0.170.17 0.130.13 0.400.40 0.430.43 0.520.52 0.460.46 A step fits held-out answers as well as the logistic. In five-fold cross-validation with folds by situation, the logistic in ΔU U improves on a step with one Yes-rate per side by a median 0.0100.010 nats per question [−0.003,+0.024][-0.003,+0.024], reliably in 33 of 2121 models. A step between −1-1 and 00 fits best in 2020 of 2121, although its location was read off these same curves. In 99 of the 2121 models the observed Yes-rate, smoothed to be monotone, never reaches one half, so their τ is where the fitted curve crosses it. Refitting with ΔU U collapsed to −1,0,+1\-1,0,+1\ reproduces the sensitivity ranking (ρ=0.985ρ=0.985). Three monotone re-expressions of the score (centered ranks, destructive scores halved, a per-axis z-score) keep ρ(a)≥0.98ρ(a)≥ 0.98 over all fitted models and ρ(τ)≥0.995ρ(τ)≥ 0.995, and leave r(a,d′)r(a,d ) on When2Call between +0.66+0.66 and +0.67+0.67. The two supplementary sets show that the overshoot is the one redundancy models accept (Table 13). Where the state is already at its goal, every model answers Yes less often than at the main set’s slight overshoots, by a median of 0.270.27. Where a tool’s supply is missing, every model answers Yes less often than on the same question with the supply present, by a median of 0.430.43. Models thus withhold when an action is plainly unneeded or cannot work, and accept it when it points the right way but goes one step too far. Table 13: Yes-rate on the main set by evidence sign and on the two supplementary sets, median and range over the 2121 models with a well-determined criterion. Each supplementary question is paired with a matched question whose answer is Yes. Questions Answer Median Yes-rate Range Main set destructive, ΔU<0 U<0 No 0.150.15 [0.03,0.51][0.03,0.51] slight overshoot, ΔU=0 U=0 No 0.400.40 [0.18,0.70][0.18,0.70] helpful, ΔU>0 U>0 Yes 0.470.47 [0.30,0.76][0.30,0.76] Already satisfied state at its goal No 0.080.08 [0.01,0.68][0.01,0.68] matched question Yes 0.230.23 [0.09,0.58][0.09,0.58] Prerequisite supply missing No 0.290.29 [0.00,0.51][0.00,0.51] matched question, supply present Yes 0.660.66 [0.46,0.90][0.46,0.90] B.3 A signal-detection readout Because models read the sign of the score, Table 14 sets the logistic coordinates beside a signal-detection readout: hits on ΔU>0 U>0 and false alarms on ΔU<0 U<0 enter Eq. 1, and the overshoot questions at ΔU=0 U=0 are left out. Sensitivity ranks the models the same under every readout. So does the criterion when it is divided by sensitivity, as c/d′c/d : τ is close to a linear function of it (r=0.999r=0.999), τ≈−0.09+3.22c/d′τ≈-0.09+3.22\,c/d , where −0.09-0.09 is the midpoint of the mean negative and the mean positive score of the main set and 3.223.22 the distance between them. The absolute c ranks models less closely with τ (ρ=0.77ρ=0.77) because τ, like c/d′c/d , is divided by sensitivity and c is not. The number of positive criteria, the opposite signs of the two Mistral-Small-24B releases and the convergence of sensitivity with external d′d are the same under every readout. Table 14: The logistic coordinates beside a signal-detection readout that leaves out ΔU=0 U=0. Rank agreement with the logistic fit and the count of criteria with an interval above zero are over the 2121 models with a well-determined criterion, and r is over the panel. Shaded: the fit of the main text. Rank agreement Criterion r(sensitivity, external d′d ) Readout ρ(sens.) ρ(crit.) interval >0>0 BFCL QuestBench When2Call Logistic, all questions (a, τ) 1.001.00 1.001.00 1616 +0.70+0.70 +0.69+0.69 +0.67+0.67 Logistic, ΔU=0 U=0 left out 0.980.98 1.001.00 1717 +0.69+0.69 +0.65+0.65 +0.65+0.65 Signal detection, d′d and c 0.990.99 0.770.77 1818 +0.70+0.70 +0.65+0.65 +0.65+0.65 Signal detection, d′d and c/d′c/d 0.990.99 0.990.99 1818 +0.70+0.70 +0.65+0.65 +0.65+0.65 B.4 Convergence with external benchmarks This subsection gathers the evidence behind §4.2. Table 15 gives each coordinate’s correlation with its counterpart on the four external benchmarks, and Figure 9 plots the panel models on the three where sensitivity converges. Table 15: Each FlipAct coordinate against its counterpart on the four external benchmarks of §4.2, over the panel of 1818 models. Sensitivity a vs. d′d Criterion τ vs. c Benchmark r p 95% interval r p BFCL (Patil et al., 2025) +0.70+0.70 0.00130.0013 [+0.39,+0.90][+0.39,+0.90] +0.51+0.51 0.0300.030 QuestBench (Li et al., 2026) +0.69+0.69 0.00150.0015 [+0.46,+0.88][+0.46,+0.88] −0.02-0.02 0.9300.930 When2Call (Ross et al., 2025) +0.67+0.67 0.00240.0024 [+0.40,+0.87][+0.40,+0.87] −0.01-0.01 0.9540.954 When2Tool (Sun et al., 2026) +0.02+0.02 0.93240.9324 [−0.50,+0.57][-0.50,+0.57] +0.19+0.19 0.4390.439 What each benchmark asks and where its answer key comes from are in Table 39. Collapsing When2Call’s choice among several replies to call or withhold adds an offset to c (App. E), so external c is not comparable across benchmarks. The criterion correlation on BFCL rests on the three models with the lowest τ: without them r=+0.21r=+0.21. Its Spearman correlation is +0.46+0.46 (p=0.058p=0.058), and Holm-adjusted over the four benchmarks its p is 0.120.12. Figure 9: The two coordinates against their external counterparts for the panel models, on the three benchmarks where sensitivity converges. Top: sensitivity against d′d . Bottom: criterion against c, with the number of models in each quadrant in gray. A dashed line marks a significant correlation. Panel cutoff. The panel of §4.2 admits models with a≥0.30a≥ 0.30. When that cutoff moves with the other panel conditions fixed (Table 16), the correlation of a with When2Call d′d stays positive and significant at every cutoff. The criterion correlation is not stable: it is negative and significant at 0.250.25 and not significant at 0.300.30 or 0.350.35, so no direction is read from it. Table 16: The When2Call correlations of both coordinates at three panel cutoffs, over nested panels. Shaded: the cutoff of the main text. a vs. d′d τ vs. c Cutoff a≥a≥ n r p r p 0.250.25 2020 +0.77+0.77 0.00010.0001 −0.46-0.46 0.0410.041 0.300.30 1818 +0.67+0.67 0.00240.0024 −0.01-0.01 0.950.95 0.350.35 1616 +0.53+0.53 0.0330.033 −0.29-0.29 0.280.28 A criterion-free readout. Scoring each external example by its answer-token log-probability gives an AUC with no criterion term (App. D.4). Over the panel models with readable log-probabilities, a also correlates with AUC on QuestBench and When2Call, with r=+0.70r=+0.70 and +0.81+0.81, so sensitivity converges with both benchmarks when the external measure carries no criterion term. B.5 Sensitivity and general capability Separating warranted from unwarranted action on FlipAct takes physical knowledge as well as reading the state, the tool and the question. Table 17 therefore checks the convergence of §4.2 against four other indicators on the panel. Two are general: log parameter count, and MMLU on a 570570-question subset, available for 1515 of the 1818 models. The other two come from the answers a is fitted on, so partialing them out removes part of a itself: FlipAct accuracy, which on the balanced main set equals balanced accuracy, and a FlipAct d′d computed with Eq. 1. The convergence holds with size partialed out: on the three benchmarks where a converges, the partial correlation of a with external d′d given log parameters is 0.620.62 to 0.640.64. Given MMLU it is 0.520.52 to 0.550.55 (p between 0.040.04 and 0.060.06 at n=15n=15). The FlipAct d′d correlates with a at 0.960.96, so the convergence does not depend on how FlipAct’s sensitivity is summarized. Table 17: Correlation of a with external d′d on the panel of 1818 models, raw and with each indicator partialed out. The shaded row repeats the raw correlation on the 1515 models with MMLU. Partialed out BFCL QuestBench When2Call When2Tool none +0.70+0.70 +0.69+0.69 +0.67+0.67 +0.02+0.02 none, 1515 models with MMLU +0.71+0.71 +0.75+0.75 +0.72+0.72 −0.27-0.27 log parameters +0.64+0.64 +0.63+0.63 +0.62+0.62 −0.18-0.18 MMLU (n=15n=15) +0.53+0.53 +0.55+0.55 +0.52+0.52 −0.67-0.67 FlipAct accuracy +0.54+0.54 +0.54+0.54 +0.56+0.56 −0.20-0.20 FlipAct d′d +0.19+0.19 −0.11-0.11 +0.08+0.08 −0.37-0.37 B.6 Generations within the Qwen and Gemma lines The generation, like size (§4.3), does not set the criterion. The Qwen and Gemma lines each span two or three generations at comparable sizes, released between mid-2024 and 2026 (Table 18). From one generation to the next, sensitivity rises at small and middle sizes, most from Qwen3 to Qwen3.5, and changes little at the largest sizes. The criterion stays put from Qwen3-8B to Qwen3.5-9B and across the three Gemma generations at 27–31B, falls from gemma-2-9b to gemma-3-12b and rises from Qwen3-32B to Qwen3.5-27B. Table 18: Coordinates of the Qwen and Gemma generations at comparable sizes, with the month the weights were released. A τ beyond the score range is shown as a bound. Line Size Model Released a τ Qwen 4B Qwen3-4B 2025-04 0.340.34 >+3>+3 Qwen3.5-4B 2026-03 0.550.55 +2.93+2.93 8–9B Qwen3-8B 2025-04 0.440.44 +2.78+2.78 Qwen3.5-9B 2026-03 0.660.66 +2.88+2.88 27–32B Qwen3-32B 2025-04 0.580.58 +0.14+0.14 Qwen3.5-27B 2026-02 0.570.57 +2.68+2.68 Gemma 9–12B gemma-2-9b 2024-06 0.340.34 +2.91+2.91 gemma-3-12b 2025-03 0.390.39 +2.17+2.17 27–31B gemma-2-27b 2024-06 0.440.44 +1.68+1.68 gemma-3-27b 2025-03 0.450.45 +1.77+1.77 gemma-4-31b 2026-04 0.500.50 +1.57+1.57 B.7 Reasoning mode Reasoning mode raises sensitivity only in the smallest models (Gong et al., 2026). In larger ones it moves the criterion toward acting. We compare reasoning off and on for every model that has both modes, in one paired run with bootstrap intervals (Table 19), so the values with reasoning off can differ from Table 9, most for Qwen3-1.7B, whose fit there is flat. Reasoning raises a significantly on Qwen3-1.7B, Qwen3-4B, Nemotron-3-Nano-4B and Qwen3.5-4B, the models of 4B and below, and on no larger model. It moves τ significantly toward zero on Qwen3.5-4B, Qwen3-8B, Qwen3.5-9B and Qwen3.5-27B. The criterion falls less on Qwen3-14B, stays put on Granite-3.3-8B and rises slightly on Qwen3-32B. Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-35B-A3B and Qwen3-0.6B have no usable fit in one of the two modes and are omitted. Table 19: Reasoning mode off and on. Bold marks a significant rise in a or fall in τ. a τ Model off on Δ off on Δ 4B and below Qwen3-1.7B 0.040.04 0.420.42 +0.38+0.38 – +1.05+1.05 – Qwen3-4B 0.320.32 0.450.45 +0.13+0.13 – – – Nemotron-3-Nano-4B 0.280.28 0.490.49 +0.21+0.21 – +2.42+2.42 – Qwen3.5-4B 0.560.56 0.660.66 +0.10+0.10 +2.87+2.87 +1.91+1.91 −0.96-0.96 8B and above Qwen3-8B 0.440.44 0.530.53 +0.09+0.09 +2.78+2.78 +1.69+1.69 −1.08-1.08 Granite-3.3-8B 0.300.30 0.370.37 +0.07+0.07 +2.59+2.59 +2.61+2.61 +0.02+0.02 Qwen3.5-9B 0.660.66 0.610.61 −0.04-0.04 +2.86+2.86 +1.99+1.99 −0.87-0.87 Qwen3-14B 0.570.57 0.510.51 −0.06-0.06 +2.36+2.36 +2.07+2.07 −0.28-0.28 Qwen3.5-27B 0.570.57 0.530.53 −0.04-0.04 +2.67+2.67 +1.57+1.57 −1.10-1.10 Qwen3-32B 0.580.58 0.510.51 −0.07-0.07 +0.14+0.14 +0.40+0.40 +0.26+0.26 A dash marks a τ outside the score range, or a change that involves one. Two answers from these runs show the two effects. Traces are quoted verbatim, with […] marking omitted text. In Qwen3.5-4B the trace works out what the tool does to the state, and the model tells the two states of one tool apart. In Qwen3.5-27B, reasoning turns a No on a warranted question into a Yes, while the harmful state of the same tool stays No. Reasoning separates the two states (Qwen3.5-4B) Situation: The food or contents is [state]. The supplies this tool depends on (mains electricity) are available nearby. The goal is to bring the food or contents back to a normal state. Action: a person uses a hotplate. Is using the hotplate a wise choice in this situation? State Gold ΔU U Reasoning off Reasoning on very cold Yes +2+2 Yes Yes very hot No −2-2 Yes No Very hot: Goal: Cool down (bring to normal state). […] Function of a hotplate: Heats objects placed on it. Conflict: Using a heating device on something that is already very hot will make it hotter, not cooler. This is the opposite of the goal. Very cold: Goal: Reheat/warm up cold food. […] Logic: If electricity is available and the goal is to heat food, a hotplate is an appropriate and effective tool for that purpose. Reasoning moves the criterion toward acting (Qwen3.5-27B) Situation: The rigid object or joint is [state]. The goal is to bring the rigid object or joint back to a normal state. Action: a person uses a rivet gun. Is using the rivet gun a wise choice in this situation? State Gold ΔU U Reasoning off Reasoning on very loose and coming apart Yes +2+2 No Yes very jammed and over-tightened No −2-2 No No Very loose: If a joint is loose and coming apart, permanently fastening it with a rivet is a standard and effective repair method for many rigid objects (metal sheets, frames, etc.). […] Sometimes bolts or welding might be used, but a rivet gun is definitely a wise and common choice for fixing loose joints where rivets are applicable. Appendix C The criterion under reversed question polarity Design. If a model prefers one answer token whatever is asked, that preference enters its criterion (§4.3), and no single wording can separate the two. The standard probe is a balanced key (Cronbach, 1946): the same content is asked in both polarities, and one reads whether the answer follows the content or the answer word. The canonical wording closes each main-set question by asking whether using the tool is a wise choice (“wise” below). We replace that sentence with three reversed wordings, under which No means act (“Would using t\t\ be an unwise choice”, “Should the person avoid using t\t\”, “Is using t\t\ a mistake”), and ask the canonical wording in the same run. Answers under a reversed wording are recoded, so every fit is P(act∣ΔU)P(act U) and a model that answers the content returns the same a and τ. Five same-polarity rephrasings (appropriate, should use, right tool, makes sense, would recommend) and three answer-order variants (“Answer No or Yes”, and letter options with Yes first or with No first) check both coordinates under rewording. The run used one seed on 1616 of the 1818 panel models, and every rewording is compared with the canonical wording asked in the same run. Decomposition. With both fits recoded to act polarity, write τwise=τd+b _wise= _d+b and τunwise=τd−b _unwise= _d-b. τd _d is a threshold on the evidence, attached to the proposition, and b is a response bias attached to the answer word, in score units and positive when the model prefers No. Table 20 gives both as means over the three reversed wordings. Result. Seven models keep the sign of τ under all three reversed wordings (Table 20). For the three Gemma models among them, phi-4 and Mistral-24B, the reversed τ is more conservative than the canonical one: asked whether the action is unwise or a mistake, they answer Yes on 7575–90%90\% of questions, the same stance as their 1919–36%36\% Yes-rate on “wise”. The three Qwen3.5 models and GLM-4-32B flip under all three. They answer No under either polarity, with a Yes-rate of 1515–20%20\% on “wise” and 99–26%26\% on the reversed wordings (one model–wording pair at 43%43\%), which recoded is an act-rate of 0.750.75–0.90.9. These four have b of +3+3 to +4+4 and τd _d within a unit of zero, so their conservatism on the canonical wording is a preference for No. The other models stay conservative on τd _d: the seven that keep their sign have b between −0.97-0.97 and +0.58+0.58, and τd _d at or above their canonical τ on six of the seven. Table 20: Sign of the criterion under the three reversed wordings, and its split into a threshold on the evidence τd _d and an answer preference b. Canonical wording Sign kept under Decomposition Model τ Yes-rate unwise avoid mistake τd _d b Sign kept under all three wordings Yi-1.5-34B +0.32+0.32 0.480.48 ✓ ✓ ✓ +0.62+0.62 −0.30-0.30 Qwen3-14B +2.62+2.62 0.230.23 ✓ ✓ ✓ +2.04+2.04 +0.58+0.58 gemma-2-9b +3.65+3.65† 0.210.21 ✓ ✓ ✓ +4.55+4.55† −0.90-0.90 gemma-2-27b +2.50+2.50 0.250.25 ✓ ✓ ✓ +2.80+2.80 −0.30-0.30 gemma-3-27b +2.45+2.45 0.270.27 ✓ ✓ ✓ +3.42+3.42† −0.97-0.97 phi-4 +1.45+1.45 0.360.36 ✓ ✓ ✓ +2.19+2.19 −0.74-0.74 Mistral-24B +3.33+3.33† 0.190.19 ✓ ✓ ✓ +3.62+3.62† −0.29-0.29 Mixed or degenerate gemma-3-12b +2.31+2.31 0.300.30 – – ✓ +2.44+2.44 −0.13-0.13 Llama-3.3-70B +1.25+1.25 0.360.36 ✓ ✗ ✓ +1.51+1.51 −0.26-0.26 Qwen3-32B +1.05+1.05 0.390.39 ✓ ✗ ✓ +0.63+0.63 +0.41+0.41 Qwen3-8B +2.34+2.34 0.290.29 – – – +0.95+0.95 +1.39+1.39 Phi-3.5-mini +3.66+3.66† 0.240.24 – – – +4.83+4.83† −1.17-1.17 Sign flipped under all three wordings Qwen3.5-4B +3.57+3.57† 0.150.15 ✗ ✗ ✗ −0.42-0.42 +3.99+3.99 Qwen3.5-9B +3.37+3.37† 0.150.15 ✗ ✗ ✗ +0.32+0.32 +3.04+3.04 Qwen3.5-27B +3.07+3.07† 0.180.18 ✗ ✗ ✗ −0.17-0.17 +3.24+3.24 GLM-4-32B +2.56+2.56 0.200.20 ✗ ✗ ✗ −0.86-0.86 +3.42+3.42 Canonical-wording values come from the polarity run, not from Table 9. ✓ sign of τ kept, ✗ flipped, – reversed fit degenerate. † outside the ±3± 3 score range, or a mean including such a fit, read for sign only. The reversed wordings are the noisier conditions, so τ remains the reported coordinate and τd _d a diagnostic that names the models whose conservatism on the canonical wording is an answer preference. On a model with no answer token, the JEV model of App. E, the decomposition gives b=−0.20±0.24b=-0.20± 0.24, so it does not produce an answer preference where none can exist. Rewording without reversing polarity. The conservative criterion is not a product of the word “wise”. Under each of five same-polarity rephrasings the criterion stays positive on 1515 or all 1616 models, and it ranks models much as the canonical wording does (Table 21). Sensitivity also keeps its ranking, at ρ(a)=0.87ρ(a)=0.87–0.940.94 for the five rephrasings, “Answer No or Yes” and letter options with Yes first, and at 0.580.58 for letter options with No first. Table 21: Both coordinates under five same-polarity rephrasings of the canonical question, 1616 panel models. Rank correlations are with the canonical wording asked in the same run, on which all 1616 criteria are positive. Rephrasing τ>0τ>0 ρ(τ)ρ(τ) Median Δτ τ ρ(a)ρ(a) “appropriate” 1616 0.860.86 −0.63-0.63 0.900.90 “should use” 1515 0.890.89 −0.54-0.54 0.910.91 “the right tool” 1616 0.770.77 −0.08-0.08 0.940.94 “makes sense” 1616 0.960.96 −0.87-0.87 0.880.88 “would recommend” 1616 0.880.88 −0.15-0.15 0.880.88 Appendix D Moving the criterion: settings and further results This section gives the settings behind §5 and further results: steering (App. D.1 and D.2), the two training schemes and the values behind §5.3 (App. D.3), a criterion-free readout (App. D.4), answer preference (App. D.5), a per-benchmark choice of steering sign and strength (App. D.6) and single answers (App. D.7), in the order of §5. D.1 Steering This subsection specifies the criterion direction of §5.1 and its strength rule, then checks the injection on a held-out half of FlipAct, on held-out When2Call examples and on eleven more models. The direction is a probe of the model’s own answers. For each of the sixteen models of Tables 1 and 22, an L2-regularized logistic regression (C=0.1C=0.1) maps the standardized residual-stream activation at the final token, taken at the layer at 0.60.6 of the model’s depth, to the model’s own Yes/No answer on every main-set question. No labels enter the fit. Its weight vector, scaled to unit norm, is w w. The hook adds λw^r¯/dλ\, w\, r/ d to the final-token activation at the output of that layer, where r¯ r is the mean norm of the activation over the fit questions and d is the width, so that λ is in units of the model’s own scale. Table 22: Steering on When2Call for all 1616 models, grouped by the tests the shift passes. Negative shift is toward acting, and ∗ marks a model of Table 1. Against controls Model λ⋆λ Fitted Δc c q Ratio Retained Significant, three times the largest control, half retained, monotone in λ Command-R-08-2024 3232 −1.04-1.04 <0.001<0.001 4.84.8 1.011.01 Mistral-24B∗ 6464 −1.14-1.14 <0.001<0.001 4.94.9 1.091.09 Qwen3-8B∗ 1616 −0.27-0.27 <0.001<0.001 3.73.7 0.910.91 gemma-2-9b∗ 6464 −0.65-0.65 0.0040.004 3.43.4 0.690.69 Significant, at least one other test not passed gemma-3-12b 44 −0.81-0.81 0.0010.001 7.87.8 0.770.77 Llama-3.3-70B 4848 −0.16-0.16 0.0270.027 18.518.5 0.630.63 Yi-1.5-34B 1616 −0.30-0.30 <0.001<0.001 1.61.6 2.832.83 phi-4∗ 3232 −0.11-0.11 0.0040.004 1.81.8 0.830.83 Qwen3.5-9B 6464 −0.12-0.12 <0.001<0.001 1.71.7 1.071.07 Qwen3.5-35B-A3B 9696 −0.16-0.16 <0.001<0.001 1.01.0 1.211.21 gemma-3-27b −32-32 +0.05+0.05 0.0370.037 0.10.1 1.251.25 Not significant Qwen3.5-27B∗ 9696 −0.02-0.02 0.0520.052 0.20.2 0.500.50 Qwen3-14B 1616 −0.34-0.34 0.620.62 3.23.2 0.790.79 Qwen3-32B 9696 −0.16-0.16 0.150.15 0.70.7 0.970.97 GLM-4-32B −16-16 −0.06-0.06 1.01.0 4.14.1 1.081.08 Qwen3-30B-A3B-Thinking-2507 no strength inside the d′d -stable range Fitted Δc c: change in c across the d′d -stable strengths, which differs from the change at λ⋆λ in Table 1. On gemma-3-27b the paired test registers more acting at λ⋆λ while the fitted shift across the range is near zero. The direction depends on the wording it is read under. A probe refitted under the reversed wording “unwise” of App. C has cosine −0.03-0.03, +0.06+0.06 and +0.02+0.02 with the canonical one on Qwen3.5-9B, Qwen3.5-27B and Qwen3-8B, and −0.34-0.34 on gemma-3-12b. Steering results are therefore read from behavior, the shift in c and τ under injection, and not from the geometry of the direction. The strength is chosen on When2Call itself. Each model is run over a grid of strengths λ, which includes negative values that inject −w^- w. The strengths at which When2Call d′d stays within 25%25\% of its unsteered value form the d′d -stable range, and λ⋆λ is the strength in that range with the lowest criterion c. Figure 10 shows the strength response of the five models of Table 1. Figure 10: When2Call criterion c (top) and d′d (bottom) against the injected strength λ for the five models of Table 1. Each column has its own range of λ. Gray marks strengths outside the d′d -stable range, and circles mark λ⋆λ . On FlipAct the injection is checked with a held-out readout. The quads are split at random into halves, and this split is not the one used for training (App. D.3). The direction and a strength chosen by accuracy are fit on one half, and c and d′d are computed from the binary answers on the other. On the read half the injection moves the criterion of Qwen3.5-9B and Qwen3.5-27B toward zero, leaves d′d within a few hundredths of its value and raises accuracy by about two points (Table 23). Table 23: Steering on the held-out half of FlipAct, with the direction and strength fit on the other half. Criterion c Accuracy Model before ++ steering Δd′ d before Δ Qwen3.5-9B +0.75+0.75 +0.42+0.42 −0.02-0.02 0.6540.654 +0.026+0.026 Qwen3.5-27B +0.57+0.57 +0.39+0.39 +0.03+0.03 0.6420.642 +0.016+0.016 Table 1 chooses λ⋆λ and reads its effect on the same When2Call examples. A split-half check separates the choice from the readout. The examples are split into halves twenty times at random, keeping the call/withhold balance, and on each split λ⋆λ is chosen on one half by the same rule and the shift is read on the other (Table 24). For four of the five models the held-out shift in c matches the full-set one to within 0.020.02, and on every split the criterion falls and the call rate rises. phi-4, whose full-set shift is the smallest, starts permissive and shows no reliable shift on the held-out half. Table 24: Split-half check of the When2Call steering shift: λ⋆λ chosen on one half and read on the other, beside the full-set value. gemma-2-9b Mistral-24B phi-4 Qwen3.5-27B Qwen3-8B Δc c full −.18-.18 −.53-.53 −.03-.03 −.10-.10 −.14-.14 held-out −.18-.18 −.53-.53 −.01-.01 −.08-.08 −.14-.14 [2.5, 97.5] [−.25,−.13][-.25,-.13] [−.61,−.29][-.61,-.29] [−.06,+.11][-.06,+.11] [−.11,−.04][-.11,-.04] [−.21,−.08][-.21,-.08] Δ call rate full +.052+.052 +.178+.178 +.019+.019 +.025+.025 +.045+.045 held-out +.048+.048 +.175+.175 +.012+.012 +.022+.022 +.047+.047 [2.5, 97.5] [+.035,+.068][+.035,+.068] [+.096,+.210][+.096,+.210] [−.026,+.027][-.026,+.027] [+.010,+.027][+.010,+.027] [+.028,+.067][+.028,+.067] Δ acc. full +.002+.002 −.032-.032 −.038-.038 +.005+.005 −.022-.022 held-out +.000+.000 −.035-.035 −.028-.028 +.007+.007 −.020-.020 [2.5, 97.5] [−.020,+.013][-.020,+.013] [−.050,−.007][-.050,-.007] [−.050,+.003][-.050,+.003] [−.003,+.012][-.003,+.012] [−.043,−.003][-.043,-.003] Same λ⋆λ 20/2020/20 14/2014/20 14/2014/20 10/2010/20 20/2020/20 Full (shaded): the change of Table 1, computed before rounding. Held-out: median over 2020 stratified splits and its [2.5, 97.5] percentile range. Same λ⋆λ : splits whose chosen strength equals the full-set one. Table 22 adds eleven further models, each run at its selected strength only. The fitted shift is the change in c across the d′d -stable strengths, and q is a paired McNemar test on the withhold examples after Benjamini–Hochberg correction. Two controls go with each shift. The first runs three directions of the same norm, one orthogonal to w w and two random, and the ratio column divides the shift by the largest of their shifts. The second projects the refusal (Arditi et al., 2024), sycophancy (Rimsky et al., 2024) and a generic engagement direction out of w w, and the retained column is the share of the shift that remains. The engagement direction is the difference in mean activation between prompts the model answers and prompts it refuses or hedges, taken from XSTest (Röttger et al., 2024) and a set of abstention questions. On the paired test, eleven of the sixteen models move significantly toward acting after correction for the fifteen testable comparisons. Four of these also clear both controls and respond monotonically in λ, with a shift at least three times the largest control shift and at least half of it kept after the projection. D.2 Steering on further benchmarks To test whether the steering result of §5.1 is specific to When2Call, the same direction w w is injected on four further benchmarks for three of the models, with the injection layer, readout and strength rule of App. D.1. The shift in c is read inside the d′d -stable range, and the paired McNemar test on the withhold examples is judged after Benjamini–Hochberg correction. The direction moves the criterion toward acting, with a significant paired test, on every benchmark that can be scored for Qwen3-8B and Mistral-24B, and on one benchmark of four for Qwen3.5-27B (Table 25). Two of Mistral-24B’s four benchmarks have no strength at which d′d stays stable, and Qwen3.5-27B’s response on When2Tool and BFCL-miss-func is not monotone in λ. Table 25: The FlipAct criterion direction on four further benchmarks with the weights fixed: shift in c inside the d′d -stable range, negative toward acting. Model When2Tool BFCL-miss-func ACPBench CaT-Bench Qwen3-8B −1.08-1.08 ∙ −0.54-0.54 ∙ −1.11-1.11 ∙ −0.80-0.80 ∙ Mistral-24B no stable λ −1.95-1.95 ∙ −1.94-1.94 ∙ no stable λ Qwen3.5-27B −1.09-1.09 −0.24-0.24 −0.47-0.47 −0.74-0.74 ∙ ∙ significant paired McNemar test on the withhold examples after Benjamini–Hochberg correction. No stable λ: no strength inside the d′d -stable range. D.3 Training Both training schemes, criterion distillation (§5.2) and sensitivity learning (§5.4), share the split, adapter and significance rule below. Their settings and controls come first, then the external-benchmark values behind §5.3. Split and significance. The main set is split at the connected components of the graph that links situations to tool records, so no tool record, situation or state flip appears in both halves. Both schemes use this split, with the sizes in Table 26. Every reported change is the mean over three seeds. Its error is the largest of the seeds’ paired situation-cluster bootstrap errors, and a change is significant when it is at least twice that error. Table 29 lists every change with its error. Table 26: Training settings of the two schemes. Both schemes Split 6363 connected components: 940940 training and 940940 held-out questions (247247 and 248248 state flips) Adapter LoRA with r=16r=16, α=32α=32 and dropout 0.050.05 on the q, k, v, o, gate, up and down projections (on gemma-3-12b, of the language-model layers only) Precision bf16 weights Clipping gradient norm 11 Seeds 00, 11 and 22 Criterion distillation Sensitivity learning Data 940940 training-half questions 1,2031,203 questions (940940 + 263263 supplementary), Yes share 0.390.39 Target steered teacher’s next-token logits computed Yes/No answer Strength λ⋆λ Qwen3-8B 2222, Qwen3.5-9B 3232, gemma-3-12b 22, Mistral-24B 1616 – Control random direction of the same norm at the same λ⋆λ answers permuted within 255255 strata of matched questions Loss forward KL, T=1T=1, full vocabulary, answer position cross-entropy on the answer turn, prompt masked Optimizer AdamW, weight decay 0.010.01, lr 10−510^-5, constant AdamW, weight decay 00, lr 2×10−42× 10^-4, linear decay, no warm-up Effective batch 1616 (8×28× 2; Mistral-24B 4×44× 4) 1616 (2×82× 8; Mistral-24B 1×161× 16) Length 44 epochs, 235235 steps 22 epochs, 152152 steps Max. length 1,0241,024 prompt tokens, 11 output token 2,0482,048 tokens Criterion distillation. The teacher is the student itself with a steering hook. The hook adds λ⋆w^r¯/dλ w\, r/ d at the 0.60.6-depth layer, as in App. D.1, but here w w is a difference of means: the mean standardized activation of the training-half questions the student answers Yes, minus that of the questions it answers No, scaled to unit norm. No labels enter it. The loss is the forward KL of Eq. 3, with the teacher’s logits computed once on all training-half questions, and Table 26 gives the control and the optimization settings. Every criterion-distillation run of the two non-Qwen students changes MMLU by −0.004-0.004 to +0.006+0.006. The strength comes from a FlipAct strength sweep with the same estimator. A strength qualifies when the criterion is well determined, a stays within two bootstrap errors of its unsteered value, d′d stays within a quarter of its value, and the Yes-rate stays inside [0.10,0.90][0.10,0.90]. Among the qualifying strengths, λ⋆λ is the one with the largest |Δτ|| τ| (Table 26). The sweep’s direction is fit on a random half of the quads (955955 questions, 479479 of them in the held-out half) and agrees with the teacher’s direction, with a cosine of 0.970.97 to 0.990.99. The sweep’s a, τ, d′d and Yes-rate are computed on all questions. On the training-half questions alone, the rule returns the same strength for Qwen3-8B, Qwen3.5-9B and gemma-3-12b, and 2222 instead of 1616 for Mistral-24B.22 2 On the full set λ=22λ=22 is excluded because its τ of −3.05-3.05 falls just outside the score range. On gemma-3-12b only one nonzero strength qualifies, λ=+2λ=+2, with a Yes-rate of 0.470.47. At +4+4 the Yes-rate is 0.810.81 and τ falls outside the score range. At every negative strength τ is also outside it or the model answers only No. Control strength. At the teacher’s norm, a random direction changes the Yes–No logit margin far less than w w does (Table 27), so it is a weak control. We therefore scale two random directions per student until their mean absolute change in the margin, over all FlipAct questions, equals the teacher’s, and refit both coordinates there (Table 28). On gemma-3-12b, on Mistral-24B’s first direction and on Qwen3.5-9B’s second, the matched random direction perturbs the answers as much as w w and leaves τ where it was, while w w moves it well toward acting. The other four directions collapse the model onto one answer before they match the teacher’s perturbation, with a Yes-rate of 00 or 11. No matched random direction shifts the criterion toward acting with sensitivity intact. Table 27: Perturbation of the Yes–No logit margin at each student’s selected strength, by the criterion teacher and by a random direction of the same norm. Mean |Δ|| | Yes–No margin Student teacher same-norm random Ratio Qwen3-8B 18.7018.70 1.181.18 15.815.8 Qwen3.5-9B 2.482.48 0.310.31 7.97.9 gemma-3-12b 4.224.22 0.460.46 9.29.2 Mistral-24B 3.243.24 0.080.08 40.440.4 Table 28: The criterion teacher beside two random directions per student, each scaled until it perturbs the Yes–No logit margin as much as the teacher does, with both coordinates refit at that strength. Criterion teacher Matched random direction Student Base τ |Δ|| | margin τ |Δ|| | margin τ d′d Yes-rate Qwen3-8B +2.67+2.67 18.7018.70 −2.11-2.11 random dir. 1 16.4116.41 – 0.000.00 0.000.00 random dir. 2 17.1617.16 – 0.000.00 1.001.00 Qwen3.5-9B +2.19+2.19 2.482.48 +1.08+1.08 random dir. 1 2.622.62 – 0.000.00 0.000.00 random dir. 2 2.462.46 +2.16+2.16 1.001.00 0.240.24 gemma-3-12b +2.50+2.50 4.224.22 +0.83+0.83 random dir. 1 4.354.35 +2.87+2.87 0.640.64 0.260.26 random dir. 2 4.314.31 +2.53+2.53 0.630.63 0.280.28 Mistral-24B >+3>+3 3.243.24 +0.55+0.55 random dir. 1 3.183.18 >+3>+3 0.880.88 0.120.12 random dir. 2 3.173.17 >+3>+3 0.490.49 0.000.00 |Δ|| | margin: mean absolute change in the Yes–No logit margin over all FlipAct questions, as in Table 27. A dash marks a fit that cannot be estimated because the model answers one way. What the student copies. A teacher moved only in the criterion can be distilled because a state flip separates the two coordinates. Write ℓ+ _+ and ℓ− _- for the Yes–No logit margins on the Yes and the No question of a flip. Their difference D=ℓ+−ℓ−D= _+- _- measures how far apart the model holds the two states and tracks a. Their sum Σ=ℓ++ℓ− = _++ _- measures how far both sit from zero and tracks τ. A teacher moved only in τ differs from the student in Σ and not in D. The students reproduce 0.880.88 to 1.041.04 of their teacher’s shift in Σ . The random-direction teachers move Σ far less than the criterion teachers, and their students’ shifts in Σ stay small. Sensitivity learning. The student trains answer-only on the computed answers, with the same LoRA configuration. Its pool is the training half plus supplementary questions from the same tool records (sizes and Yes share in Table 26). The control permutes the answers within strata of matched questions, so prompts, format, step count and Yes-rate are unchanged and about half of the training state flips have their two answers swapped. Shuffling removes the link between a question’s state and its answer, and on seven of the twelve runs the student’s answers no longer carry a criterion: three runs collapse onto one answer, three lose the slope, and one has SE(τ)SE(τ) just above two. Answer-only training also breaks free-form output on Mistral-24B, so several of its external readouts, mostly on BFCL-miss-func and When2Call, cannot be scored. MMLU cannot be read on five of the twelve sensitivity-learning runs. Table 29: Held-out change for every training condition of Table 2, as the seed mean ± the largest per-seed paired bootstrap error. Bold marks a change toward acting in τ, a rise in a after sensitivity learning and a rise in accuracy. A ‡ marks a cell with two usable seeds, and n/a one with fewer. Shaded: control. Student Condition Δa a Δτ τ Δ acc. Criterion distillation Qwen3-8B distilled +0.09+0.09 ±0.080.08 −4.44-4.44 ±0.690.69 +0.033+0.033 ±0.0180.018 random direction +0.02+0.02 ±0.010.01 +0.04+0.04 ±0.050.05 +0.003+0.003 ±0.0040.004 Qwen3.5-9B distilled −0.01-0.01 ±0.060.06 −1.02-1.02 ±0.220.22 +0.045+0.045 ±0.0110.011 random direction +0.00+0.00 ±0.020.02 +0.20+0.20 ±0.060.06 −0.009-0.009 ±0.0050.005 gemma-3-12b distilled −0.02-0.02 ±0.040.04 −1.86-1.86 ±0.980.98 +0.014+0.014 ±0.0120.012 random direction +0.00+0.00 ±0.010.01 −0.11-0.11 ±0.360.36 +0.001+0.001 ±0.0040.004 Mistral-24B distilled −0.17-0.17 ±0.100.10 −3.29-3.29 ±0.830.83 +0.036+0.036 ±0.0160.016 random direction +0.07+0.07 ±0.050.05 −0.10-0.10 ±0.230.23 +0.001+0.001 ±0.0050.005 Sensitivity learning Qwen3-8B trained +0.63+0.63 ±0.120.12 −2.35-2.35 ±0.530.53 +0.172+0.172 ±0.0180.018 shuffled labels −0.19-0.19 ±0.110.11 +0.07+0.07 ±1.101.10‡ −0.044-0.044 ±0.0200.020 Qwen3.5-9B trained +0.48+0.48 ±0.140.14 −1.69-1.69 ±0.300.30 +0.175+0.175 ±0.0160.016 shuffled labels −0.30-0.30 ±0.190.19 −3.08-3.08 ±0.850.85‡ −0.038-0.038 ±0.0190.019 gemma-3-12b trained +1.19+1.19 ±0.210.21 −2.57-2.57 ±1.011.01 +0.238+0.238 ±0.0180.018 shuffled labels −0.20-0.20 ±0.110.11‡ n/a −0.070-0.070 ±0.0200.020 Mistral-24B trained +0.57+0.57 ±0.160.16 −3.39-3.39 ±0.730.73 +0.206+0.206 ±0.0200.020 shuffled labels −0.61-0.61 ±0.460.46 n/a −0.059-0.059 ±0.0190.019 Transfer. Table 30 lists the values behind Figure 7: the criterion shift on each external benchmark for each distilled student and for its random-direction control. Table 30: Values behind Figure 7: criterion shift on six external benchmarks for the distilled student and its random-direction control, seed mean ± paired error. Negative is toward acting. Shaded: control. Benchmark Qwen3-8B Qwen3.5-9B gemma-3-12b Mistral-24B Distilled student Act now When2Call −0.05±0.01-0.05± 0.01 −0.06±0.01-0.06± 0.01 −0.04±0.01-0.04± 0.01 −0.07±0.01-0.07± 0.01 QuestBench −1.97±0.05-1.97± 0.05 −0.55±0.03-0.55± 0.03 −0.82±0.03-0.82± 0.03 −0.86±0.04-0.86± 0.04 When2Tool −0.58±0.03-0.58± 0.03 −0.34±0.03-0.34± 0.03 −0.32±0.10-0.32± 0.10 −0.52±0.11-0.52± 0.11 BFCL-miss-func −0.21±0.05-0.21± 0.05 – −0.05±0.03-0.05± 0.03 −0.14±0.04-0.14± 0.04 Planning ACPBench −0.81±0.12-0.81± 0.12 −0.31±0.10-0.31± 0.10 −0.21±0.07-0.21± 0.07 −0.32±0.10-0.32± 0.10 CaT-Bench −0.58±0.04-0.58± 0.04 −0.38±0.04-0.38± 0.04 −0.13±0.01-0.13± 0.01 −0.35±0.02-0.35± 0.02 Random-direction control Act now When2Call +0.00±0.01+0.00± 0.01 +0.01±0.00+0.01± 0.00 −0.01±0.00-0.01± 0.00 −0.01±0.00-0.01± 0.00 QuestBench +0.05±0.01+0.05± 0.01 +0.09±0.01+0.09± 0.01 −0.05±0.01-0.05± 0.01 +0.00±0.02+0.00± 0.02 When2Tool +0.02±0.01+0.02± 0.01 +0.06±0.01+0.06± 0.01 +0.00±0.01+0.00± 0.01 +0.15±0.09+0.15± 0.09 BFCL-miss-func +0.02±0.03+0.02± 0.03 – +0.01±0.02+0.01± 0.02 −0.01±0.02-0.01± 0.02 Planning ACPBench +0.01±0.03+0.01± 0.03 +0.06±0.03+0.06± 0.03 −0.01±0.02-0.01± 0.02 +0.00±0.00+0.00± 0.00 CaT-Bench +0.01±0.01+0.01± 0.01 +0.08±0.02+0.08± 0.02 −0.01±0.00-0.01± 0.00 +0.03±0.01+0.03± 0.01 The dash marks Qwen3.5-9B on BFCL-miss-func, whose base readout does not reproduce. Accuracy. On all four students, criterion distillation raises balanced accuracy on the external benchmarks where the student starts conservative (Table 31). Where it starts permissive, the same shift overshoots: accuracy falls by up to four points or stays within its error, except on Mistral-24B on When2Tool. There the hit rate is already at ceiling, so a shift toward acting can only add false alarms, and accuracy falls by about ten points. Table 31: Change in balanced accuracy after criterion distillation, seed means, grouped by the starting criterion on each benchmark. Qwen3.5-9B on BFCL-miss-func is left out, as in Figure 7. Student Benchmark Start c Δ bal. acc. Started conservative (c>0c>0) Qwen3-8B When2Tool +0.42+0.42 +0.014+0.014 QuestBench +0.71+0.71 +0.041+0.041 ACPBench +0.40+0.40 +0.025+0.025 Qwen3.5-9B When2Tool +0.73+0.73 +0.046+0.046 ACPBench +0.65+0.65 +0.052+0.052 gemma-3-12b QuestBench +0.91+0.91 +0.062+0.062 ACPBench +0.31+0.31 +0.012+0.012 Mistral-24B QuestBench +1.38+1.38 +0.050+0.050 BFCL-miss-func +0.39+0.39 +0.022+0.022 ACPBench +0.62+0.62 +0.046+0.046 Started permissive (c<0c<0) Qwen3-8B When2Call −0.26-0.26 −0.010-0.010 BFCL-miss-func −0.57-0.57 −0.040-0.040 CaT-Bench −1.13-1.13 −0.042-0.042 Qwen3.5-9B When2Call −0.52-0.52 −0.017-0.017 QuestBench −0.06-0.06 −0.011-0.011 CaT-Bench −1.43-1.43 −0.023-0.023 gemma-3-12b When2Call −0.37-0.37 −0.001-0.001 BFCL-miss-func −0.23-0.23 +0.011+0.011 When2Tool −1.13-1.13 −0.028-0.028 CaT-Bench −0.59-0.59 −0.015-0.015 Mistral-24B When2Call −0.04-0.04 −0.003-0.003 When2Tool −1.44-1.44 −0.095-0.095 CaT-Bench −0.86-0.86 −0.026-0.026 D.4 A criterion-free readout of the external benchmarks Distillation keeps sensitivity on a readout that carries no criterion term (§5.3). The external d′d is the equal-variance estimate from one operating point, and it is exact only when the two evidence distributions share a variance. Scoring each example of every base model, distilled student and control by its answer-token log-probability margin instead gives a full ROC and an area under it (AUC) with no criterion term. Thresholding this margin recovers the binary answers and, on the base models, the d′d and c of the main readout. BFCL-miss-func has no answer-token margin and is absent. Distillation leaves the AUC unchanged for every student on the five benchmarks with answer log-probabilities (Table 32). Each distilled change is equivalent to no change under a two one-sided test at ±0.02± 0.02, with a 90%90\% paired cluster-bootstrap interval inside that bound. On Qwen3-8B the change in d′d is the criterion sliding along an unequal-variance ROC. Across benchmarks the size of its criterion shift tracks the change in single-point d′d but not the change in AUC (Table 33). On QuestBench, where the shift is largest, d′d rises by 0.720.72 and the base model’s fitted ROC slope is 1.421.42, against one under equal variance. Table 32: Change in criterion-free AUC after criterion distillation and for its random-direction control, seed mean against the base model: the values of Table 4 with the control beside them. Qwen3-8B Qwen3.5-9B gemma-3-12b Mistral-24B Benchmark distilled control distilled control distilled control distilled control When2Call −0.008-0.008 0.0000.000 −0.006-0.006 +0.001+0.001 +0.001+0.001 +0.001+0.001 −0.002-0.002 0.0000.000 When2Tool +0.004+0.004 −0.001-0.001 −0.001-0.001 0.0000.000 +0.011+0.011 +0.003+0.003 +0.001+0.001 0.0000.000 QuestBench −0.012-0.012 0.0000.000 −0.002-0.002 0.0000.000 −0.005-0.005 −0.001-0.001 +0.002+0.002 0.0000.000 ACPBench +0.001+0.001 +0.001+0.001 +0.001+0.001 −0.001-0.001 +0.004+0.004 +0.001+0.001 −0.003-0.003 0.0000.000 CaT-Bench +0.001+0.001 0.0000.000 +0.008+0.008 −0.001-0.001 −0.003-0.003 −0.001-0.001 +0.002+0.002 +0.001+0.001 Table 33: Criterion distillation on Qwen3-8B, seed mean against the base model: size of the criterion shift, change in single-point d′d and in criterion-free AUC, and the base model’s fitted ROC slope s. Rows are ordered by the size of the criterion shift. Benchmark |Δc|| c| Δd′ d ΔAUC s QuestBench 1.991.99 +0.720+0.720 −0.0117-0.0117 1.421.42 ACPBench 0.820.82 +0.162+0.162 +0.0007+0.0007 0.910.91 CaT-Bench 0.590.59 −0.142-0.142 +0.0013+0.0013 0.800.80 When2Tool 0.580.58 −0.010-0.010 +0.0037+0.0037 1.151.15 When2Call 0.050.05 −0.067-0.067 −0.0077-0.0077 1.001.00 QuestBench, 95%95\% paired cluster-bootstrap intervals: Δd′ d [+0.537,+0.931][+0.537,+0.931], ΔAUC [−0.0164,−0.0068][-0.0164,-0.0068]. D.5 Answer preference and a Yes-bias teacher FlipAct reads the decision to act from a Yes or No, so on it a shift toward acting and a shift toward answering Yes coincide. Reversed wordings and a teacher that only favors Yes separate the two (§5.3). Reversed wordings. Each distilled student and its control answer the held-out questions under the canonical wording and under the three reversed wordings of App. C, with answers recoded so that every rate is a rate of acting. A shift in the readiness to act raises the act-rate under every wording. A preference for Yes raises it under the canonical wording and lowers it under the reversed ones. Table 34 splits the change in act-rate into a shared part, the mean of the canonical and reversed changes, and a flipping part, half their difference. On all four students most of the shift flips with the polarity, so on FlipAct’s binary readout distillation mostly shifts the preference for Yes. Qwen3-8B and Mistral-24B also carry a smaller shared part, and the random-direction controls move neither part. Table 34: Change in the held-out act-rate after criterion distillation under the canonical wording and under the three reversed wordings (their mean), and its shared and flipping parts, as seed means. Change in act-rate Parts Student canonical reversed shared flipping Qwen3-8B +0.30+0.30 −0.11-0.11 +0.09+0.09 +0.20+0.20 Qwen3.5-9B +0.07+0.07 −0.03-0.03 +0.02+0.02 +0.05+0.05 gemma-3-12b +0.09+0.09 −0.11-0.11 −0.01-0.01 +0.10+0.10 Mistral-24B +0.21+0.21 −0.03-0.03 +0.09+0.09 +0.12+0.12 A Yes-bias teacher. A teacher that only favors Yes tests whether distillation carries more than a Yes preference. This teacher is the base model with the logits of every Yes token raised by a constant, chosen on the training half so that it answers Yes as often as the steered teacher. The data and recipe are unchanged, and the students are Qwen3-8B and gemma-3-12b. On the four Yes/No benchmarks the Yes-bias student moves c as far as the distilled one or further. On the two formats without a Yes answer, the criterion-distilled student moves c further in every seed on both students (Table 35), though by little in one Qwen3-8B seed on BFCL-miss-func. BFCL-miss-func is the stronger test of the two because its reply is free-form, while When2Call keeps option letters as answer tokens. Table 35: Criterion shift on the two benchmarks without a Yes answer for the distilled, random-direction and Yes-bias students, and the difference between the distilled and the Yes-bias student, paired on examples, overall and per seed. Negative is toward acting. Δc c, seed mean distilled −- Yes-bias Student Benchmark distilled random Yes-bias mean seed 0 seed 1 seed 2 Qwen3-8B When2Call −0.051-0.051 +0.004+0.004 −0.026-0.026 −0.025-0.025± 0.0050.005 −0.024-0.024 −0.018-0.018 −0.031-0.031 BFCL-miss-func −0.208-0.208 +0.017+0.017 −0.159-0.159 −0.050-0.050± 0.0250.025 −0.072-0.072 −0.072-0.072 −0.005-0.005 gemma-3-12b When2Call −0.043-0.043 −0.006-0.006 −0.006-0.006 −0.037-0.037± 0.0060.006 −0.047-0.047 −0.026-0.026 −0.037-0.037 BFCL-miss-func −0.053-0.053 +0.008+0.008 +0.021+0.021 −0.074-0.074± 0.0220.022 −0.113-0.113 −0.069-0.069 −0.040-0.040 On Qwen3-8B the Yes-bias student also moves c on both formats, by about half of the distilled shift on When2Call and three quarters on BFCL-miss-func, while on gemma-3-12b it does not. On some models, then, training toward one answer on binary questions reshapes a broader readiness to act. We conjecture that this is one way in which fine-tuning on binary preferences or rewards, as in RLHF (Ouyang et al., 2022) or GRPO (Shao et al., 2024), shifts when a model acts. Testing it is left to future work. D.6 Choosing the steering sign and strength per benchmark Steering with a fixed sign toward acting (the fixed push) lowers balanced accuracy on 1212 of the 2525 model–benchmark pairs formed by the five steered models of §5.1 and five benchmarks (Table 36). The rule below, summarized in §5.3, instead chooses the sign and strength of the steering push from a few labeled examples of the target benchmark. The push is chosen on K=50K=50 labeled examples of the target benchmark, whole pairs where the benchmark has pairs, and read on the rest, over twenty random splits. On these fifty examples alone, the rule keeps the strengths at which d′d stays within a quarter of its unsteered value and takes the sign of the unsteered criterion. It pushes nothing if that criterion is smaller than its bootstrap error. Otherwise it takes the kept strength that brings c closest to zero along a monotone path, and it pushes nothing if none qualifies. The same rule sets a scalar bias on the answer-token log-odds where the benchmark has an answer token. The readout is the change in balanced accuracy against no intervention, with a paired bootstrap over examples (B=2000B=2000). A model–benchmark pair counts as improved when the median gain exceeds two errors and as degraded when it falls below minus one error, a stricter bar for a gain than for a loss. Table 36: Choosing the push on fifty examples of each benchmark. (a) Outcomes over model–benchmark pairs, with the fixed push for comparison (shaded). (b) Median change in balanced accuracy over twenty splits for the per-benchmark steering push and for an answer-token bias chosen by the same rule. (a) Condition Pairs Improved Unchanged Degraded Fixed push toward acting 2525 22 1111 1212 Per-benchmark push 2525 55 1919 11 Answer-token bias 2020 55 1515 00 (b) Model When2Call When2Tool BFCL-miss-func ACPBench CaT-Bench Qwen3-8B push +0+0⋅·000000 +0.022+0.022 ∙ +0.021+0.021 +0.021+0.021 +0.006+0.006 bias +0+0⋅·000000 +0.024+0.024 – +0.021+0.021 +0.011+0.011 Qwen3.5-27B push +0+0⋅·000000 +0+0⋅·000000 +0+0⋅·000000 +0+0⋅·000000 +0+0⋅·000000 bias +0+0⋅·000000 +0.024+0.024 – +0.011+0.011 +0.086+0.086 ∙ gemma-2-9b push +0.000+0.000 +0.113+0.113 ∙ +0.000+0.000 +0.000+0.000 +0+0⋅·000000 bias +0.031+0.031 +0.075+0.075 ∙ – +0.000+0.000 +0.000+0.000 Mistral-24B push −0.020-0.020 ∘ +0+0⋅·000000 +0.049+0.049 ∙ +0+0⋅·000000 +0+0⋅·000000 bias +0.005+0.005 +0.034+0.034 ∙ – +0.087+0.087 ∙ +0+0⋅·000000 phi-4 push +0+0⋅·000000 +0.084+0.084 ∙ +0.047+0.047 ∙ +0.024+0.024 +0+0⋅·000000 bias +0.000+0.000 +0.107+0.107 ∙ – +0.027+0.027 +0+0⋅·000000 ∙ improved, ∘ degraded, ⋅· none chosen, – no answer token. The per-benchmark push degrades one pair, Mistral-24B on When2Call, and repairs every other pair the fixed push degraded (Table 36). On that pair it moves the criterion the right way but loses 17%17\% of d′d , within the quarter the rule allows. Where it pushes, its sign matches the sign of the full-benchmark criterion in every split of every pair whose criterion is at least 0.30.3 in size. Five pairs improve, and eight when the labeled set grows to a hundred examples. On Qwen3.5-27B the direction does not move the criterion toward zero on four benchmarks, so the rule pushes nothing there. The answer-token bias is never worse than no intervention and matches or beats the per-benchmark push on all but one of its pairs. Only the steering push applies to BFCL-miss-func, which has no answer-token margin. D.7 Qualitative examples The boxes show single answers of Qwen3-8B, the one model that is both steered in Table 1 and a distillation student. Boxes (a) and (b) illustrate steering (§5.1) on When2Call, where the model picks one of four candidate responses and “call” means it picks the tool call. Boxes (c) and (d) illustrate how distillation keeps sensitivity (§5.3). Each shows a matched pair, one question that warrants acting and one that does not, under the three seeds of the criterion-distilled student and of the random-direction control. There m is the Yes minus No logit at the answer token, averaged over seeds, and the answer is Yes when m>0m>0. Distillation raises both margins, so m=0m=0 can fall between them while their order, the comparison the criterion-free AUC counts, stays as it was. Boxes (a) and (b) show the first example, in sorted id order, that fits the pattern in its title; boxes (c) and (d) each show one of the pairs the distilled student separates while its control does not. The line under each box counts how often the pattern occurs, and Table 37 gives the pair counts behind (c) and (d). Table 37: Pair counts behind boxes (c) and (d): how often the question that warrants acting keeps the higher margin, and how the distilled student answers the pairs the base model answers No to twice. Shaded: base model and control. FlipAct (c) QuestBench (d) Pairs 354354 288288 Question that warrants acting above its twin base model 69%69\% 92%92\% distilled student 6868–69%69\% 9393–94%94\% random-direction control 6868–69%69\% 92%92\% Pairs the base model answers No to twice 189189 187187 distilled student separates them 5050 117117 distilled student answers Yes to both 9898 5959 (a) Steering on a request that needs a tool: When2Call, gold call, unsteered withholds Tools: flight.status.check (airlines, person_name, ticket_number); uber.eat.order (restaurant, items, quantities). User: “Hi, I am Karan. Could you tell me the current status of my flight with Delta? My name is Karan Sharma, and my ticket number is IND4567.” Gold: call (flight.status.check) ⋅· Unsteered: withhold ⋅· Steered, λ⋆=16λ =16: call The unsteered model withholds on 5656 of the 300300 call examples. Steering calls on 99 of them and withholds on 22 calls it had made. (b) The matching false alarm: When2Call, gold asks for a missing detail, unsteered withholds Tools: get_sensor_readings_latest, get_sensor_readings_history, and get_sensor_readings_history_by_interval (networkIds, metrics, timespan, interval, …). User: “Obtain the sensor reading for temperature for my network L_579838452023959405.” Gold: withhold (ask for the timespan) ⋅· Unsteered: withhold ⋅· Steered, λ⋆=16λ =16: call (get_sensor_readings_history_by_interval with a timespan the user did not give) The unsteered model withholds on 210210 of the 300300 withhold examples. Steering calls on 2121 of them (88 that ask for a detail, 1313 where no listed tool fits) and withholds on 11 it had called. Qwen3-8B starts slightly permissive on When2Call, so these false alarms outnumber the hits of (a) and accuracy falls (Table 1). (c) Distillation moves the criterion, not the order: a held-out FlipAct state flip Warranted (ΔU=+2 U=+2): Situation: The food or contents is very cold. The supplies this tool depends on (mains electricity) are available nearby. The goal is to bring the food or contents back to a normal state. Action: a person uses a hotplate. Is using the hotplate a wise choice in this situation? Answer Yes or No. Destructive (ΔU=−2 U=-2): the same question with the food or contents “very hot”. Gold Base m Distilled m Control m Warranted Yes −6.0-6.0 (No) +14.3+14.3 (Yes, 3/3) −4.6-4.6 (No, 3/3) Destructive No −33.0-33.0 (No) −14.3-14.3 (No, 3/3) −33.2-33.2 (No, 3/3) Over the 354354 held-out state flips both margins rise in every flip and seed, and the warranted question stays above the unwarranted one about as often as before (Table 37). Hits and false alarms rise together: of the held-out questions the base model answers No, the distilled student answers Yes in all seeds on 220220 of 286286 warranted ones and on 137137 of 311311 destructive ones, and the control on none. (d) The same on an external benchmark: a QuestBench GSM-Q-verbal pair Fully specified: “My wife and I went out for dinner at a restaurant that was offering a 10% discount on their meals. After applying the discount, our dinner bill came to $50. We decided to evenly split the check, but she wanted me to pay an additional 20% tip on our discounted bill. How much did I end up paying? Do you already have enough information to give a single correct final answer to this question right now? Do NOT solve it. Answer with exactly one word: Yes or No.” Underspecified: the same problem without the discount and the $50 bill. Gold Base m Distilled m Control m Fully specified Yes −0.2-0.2 (No) +6.4+6.4 (Yes, 3/3) −0.5-0.5 (No, 3/3) Underspecified No −20.8-20.8 (No) −10.2-10.2 (No, 3/3) −20.6-20.6 (No, 3/3) Over the 288288 pairs both margins rise in every pair and seed, and the fully specified problem stays above its twin about as often as before (Table 37). Appendix E A model that returns a probability Every entry in the paper is a token model read through a Yes/No answer, so its criterion can include a preference for one answer word (§4.3). JEV (TypeSafe AI, 2026) is a decision model of a different kind.33 3 Model jev-latest, API version jev-1.13.0, evaluated 2026-09-22. Given a state and a statement about it, JEV returns the probability that the statement is true and generates no text, so it gives a readout of FlipAct with no answer word to prefer. The state is a question’s situation and action sentences, and the statement is “Using the tool is a wise choice in this situation”. The returned probability is read as P(act)P(act) and binarized at one half for the fit of Eq. 2. Because this input differs from the prompt the token models receive, JEV is not entered in Figure 5, the panel or Table 9, and the comparisons below are descriptive. Coordinates. On the statement readout JEV has a=0.62a=0.62 and τ=+1.35τ=+1.35 (Table 38). Its sensitivity is close to that of the most sensitive entries of Table 9. The criterion is conservative, with an interval that excludes zero, and lies between those of Llama-3.3-70B and gemma-4-31b. Mean P(act)P(act) steps between ΔU=−1 U=-1 and 00 and is flat on either side, the same profile as the token models in Figure 4. Asked instead to rate, on an ordered verbal scale, the action’s effect on the situation or how warranted it is, JEV tracks the size of the evidence barely better than its probability does. When the statement is reversed, JEV’s answer preference in the decomposition of App. C is b=−0.20±0.24b=-0.20± 0.24, and the probabilities under the two polarities sum to 1.011.01 on average. Readout format. Asked to choose between use and avoid instead of judging the statement, JEV returns the same coordinates (Table 38), so on FlipAct its criterion does not depend on how the binary decision is read out. It does depend on which decisions are offered. On When2Call, where many withhold examples call for asking a question, JEV has c=−1.15c=-1.15 when it judges a statement. Offered a choice among call, ask and decline, it routes most of those examples to ask, and c moves to +0.02+0.02. A criterion read from two options is therefore not a constant of the model when a third option exists, which is one reason an absolute τ need not predict a benchmark’s c. Table 38: JEV on FlipAct under two readouts of the same decision, which give the same coordinates and the same criterion-free AUC. Readout a 95% interval τ 95% interval AUC Statement, binarized at one half 0.620.62 [0.47,0.80][0.47,0.80] +1.35+1.35 [+0.83,+2.02][+0.83,+2.02] 0.7480.748 Choice between use and avoid 0.610.61 +1.29+1.29 0.7500.750 AUC is that of P(act)P(act) against the computed label. Intervals resample situations and are given for the statement readout. Caveats. JEV is one closed model under one API version, and its responses are deterministic, so every interval resamples situations or examples only. Binarizing its probability also adds a threshold the token models do not have. JEV shows that a conservative criterion and a step at the sign of the evidence can arise without an answer word, but a single model cannot stand in for the 4444 models. Appendix F When-to-act benchmarks, steering and efficient inference Table 39 places twelve when-to-act benchmarks on the design choices of §2: how the ground truth is constructed, whether controlled negatives are matched, whether the evidence for acting is graded, and whether responses are decomposed into sensitivity and criterion. The decisions cover tool-call necessity (Ross et al., 2025; Patil et al., 2025; Huang et al., 2024; Sun et al., 2026), acting or abstaining on minimally perturbed pairs (Liu et al., 2026), answering or abstaining (Kirichenko et al., 2026), clarification (Li et al., 2026; Su and Cardie, 2026), over-refusal (Röttger et al., 2024; Cui et al., 2024; Shi et al., 2024) and physical affordance (Bisk et al., 2020). Activation steering reads a direction from the residual stream and adds it back (Rimsky et al., 2024; Zou et al., 2023; Turner et al., 2023; Li et al., 2023). Refusal is its best-studied target (Arditi et al., 2024), and sycophancy is a related disposition (Sharma et al., 2024; Perez et al., 2023). App. D.1 projects the refusal and sycophancy directions out of the criterion direction as a control on the steering result. The decision to act also appears inside efficient inference, where the action is to spend more computation. LoHi (Dong et al., 2026c) answers from a low-resolution video stream first and requests high-resolution frames only when the entropy of the answer exceeds a threshold, which sends about a third of the questions to the second pass at nearly the accuracy of always requesting them; the threshold is a criterion in the sense used here. MMTok (Dong et al., 2026b) observes that the number of visual tokens a question needs depends on its difficulty, and MAI (Dong et al., 2026a) passes a small model’s reasoning to a large one only when needed. The last of these also shows the failure this paper measures: asked whether a fan suits a user who feels cold, a small vision-language model states that a fan cools and still answers that it is suitable. Table 39: When-to-act benchmarks on the design choices of §2: ✓ full, ∼ partial, ✗ absent. Benchmark Decision Ground truth Ctrl. neg. Graded Sens./crit. When2Call (Ross et al., 2025) tool call/abstain LLM+human ∼ ✗ ✗ BFCL (Patil et al., 2025) function applies? human ∼ ✗ ✗ MetaTool (Huang et al., 2024) tool needed? human/template ∼ ✗ ✗ When2Tool (Sun et al., 2026) call vs. no-call env-computed ✓ ∼ (diff.) ✗ Shi et al. (2026) tool call/abstain LLM judge ✗ ∼ (SAE) ∼ (offset) AgentAbstain (Liu et al., 2026) act vs. abstain pipeline+judge ✓ ✗ ∼ (paired) AbstentionBench (Kirichenko et al., 2026) answer/abstain curated+judge ✗ ✗ ✗ QuestBench (Li et al., 2026) ask vs. solve rule (CSP) ✓ ✗ ✗ Su and Cardie (2026) recognize/clarify human ✓ ✗ ∼ (informal) XSTest (Röttger et al., 2024) comply/refuse hand-written ✓ ✗ ✗ OR-Bench (Cui et al., 2024) comply/refuse LLM ensemble ∼ ✗ ✗ PIQA (Bisk et al., 2020) physical solution human ∼ ✗ ✗ FlipAct act vs. withhold by rule ✓ (2×2) ✓ (ΔU U) ✓ (a,τa,τ) Appendix G Limitations and outlook Instrument. Every label is computed by rule from a tool’s direction and magnitude on one axis, annotated by language models with no person in the loop, so some questions will carry labels a person would set otherwise. Agreement between annotator families and perturbations of the annotations check how far such errors could move the results (App. A.5 and A.6); no human observers were measured on the diagnostic. All questions share one template in one physical domain. The criterion stays positive in nearly all models under five rephrasings of it (App. C); other templates and domains are untested. Side effects, annotated for part of the tool records, do not enter the rule. Measurement. Several relations rest on few models. The fall of the criterion with size inside a family rests on three to five models per family. The FlipAct criterion does not track a benchmark’s c consistently: near zero on three benchmarks, positive on BFCL on the strength of the three models with the lowest τ, and negative on When2Call at a looser panel cutoff (Table 15, App. B.4), over eighteen models whose criteria sit in a narrow band. Whether this comes from limited power, from the benchmarks’ interfaces or from the construct is open. All models are open checkpoints of at most 7070B parameters. Interventions. Two of the intervention models, Qwen3.5-9B and Qwen3.5-27B, answer No under either wording, so part of their starting τ is a preference for the answer token (App. C). The steering direction depends on the wording it is read under, so the interventions are read from behavior, not from the geometry of the direction (App. D.1). After distillation the equivalence test on sensitivity is undecided on Qwen3-8B and Mistral-24B. The Yes-bias comparison covers two students, with BFCL-miss-func as its only free-form test (App. D.5). The reasoning-mode comparison rests mostly on Qwen models. Sensitivity learning raises a on FlipAct but does not transfer stably to the external benchmarks: its changes in d′d and c there differ in sign across students and are matched by the shuffled-label control. FlipAct tests basic physical knowledge of tools, and labels on it appear to add little that sharpens the decision to call a tool elsewhere. Outlook. The build scales with annotator calls (§3.4), and its schema is written in a domain-specific language (Chen et al., 2026) in which a property, an axis or a rule can be added or removed without rebuilding the pipeline. Richer tool properties such as side effects, rules over several properties and their interactions (Zhang et al., 2026), and new axes, templates and domains run through the same steps. Beyond single decisions, a richer graph of how tools transform states could chain questions into multi-step agent scenarios, in which a model’s tendencies can be read across a sequence of decisions. Such scenarios may also supply what sensitivity learning needs to transfer beyond FlipAct.