Paper deep dive

When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following

Qishi Zhan, Seoyeon Jang, Zihan Dong, Minxuan Hu, Ziheng Chen, Tonghui Qu

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 74

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Verifiable instruction-following benchmarks often express each constraint through one fixed template. We test whether scores remain stable when the operational requirement is unchanged but its wording varies. We introduce WISE, a matched evaluation suite and reporting protocol instantiated on exact word count, keyword inclusion exactly once, and an inclusive 8--12 word range. Across 100 matched tasks, up to thirteen models from seven providers, and repeated generations scored over the complete visible output, wording alone produces substantial compliance shifts. In an avoidance-family panel, five avoidance and exclusion forms fall below the positive baseline, while constructional controls also shift compliance substantially: in the nine-model control panel, compliance is 54.9% for the original positive form, 48.2% for a longer positive form, 36.7% when the target appears later, and 33.8% for AVOID1. A strict JSON-structure probe shows wording sensitivity beyond counting, with a different direction of effect. Effect sizes, failure directions, weakest forms, and model rankings vary across realizations. Under the most disruptive exclusion form, the top-ranked model changes and 24.1% of strictly ordered model pairs reverse. Human validation further shows that unanimous agreement on an exact-count interpretation can coexist with substantially different model behavior. WISE supplements conventional scores with mean and worst-form compliance, wording gaps, failure profiles, and ranking stability.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

70,457 characters extracted from source content.

Expand or collapse full text

When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following Qishi Zhan * Marquette University Seoyeon Jang University of California, San Diego Zihan Dong Georgia Tech Minxuan Hu Cornell University Ziheng Chen The University of Texas at Austin Tonghui Qu Hikvision Abstract Verifiable instruction-following benchmarks often ex- press each constraint through one fixed template. We test whether scores remain stable when the operational requirement is unchanged but its wording varies. We introduce WISE, a matched evaluation suite and report- ing protocol instantiated on exact word count, keyword inclusion exactly once, and an inclusive 8–12 word range. Across 100 matched tasks, up to thirteen models from seven providers, and repeated generations scored over the complete visible output, wording alone produces substan- tial compliance shifts. In an avoidance-family panel, five avoidance and exclusion forms fall below the positive baseline, while constructional controls also shift compli- ance substantially: in the nine-model control panel, com- pliance is 54.9% for the original positive form, 48.2% for a longer positive form, 36.7% when the target appears later, and 33.8% for AVOID1. A strict JSON-structure probe shows wording sensitivity beyond counting, with a different direction of effect. Effect sizes, failure direc- tions, weakest forms, and model rankings vary across re- alizations. Under the most disruptive exclusion form, the top-ranked model changes and 24.1% of strictly ordered model pairs reverse. Human validation further shows that unanimous agreement on an exact-count interpretation can coexist with substantially different model behavior. WISE supplements conventional scores with mean and worst-form compliance, wording gaps, failure profiles, and ranking stability. 1 Introduction Instruction-following benchmarks increasingly rely on automatically verifiable constraints, including length lim- * Corresponding author: qishi.zhan@marquette.edu Figure 1: A matched example of wording-sensitive exact- count compliance. The model, base task, target, checker, and decoding configuration are fixed; only the constraint wording changes. its [Yuan et al., 2025], formatting rules, and lexical re- quirements. Each constraint is commonly instantiated through one fixed template [Zhou et al., 2023, Jiang et al., 2024, Qin et al., 2024, Pyatkin et al., 2025]. The same operational target, however, admits many linguistic re- alizations. An exact eight-word requirement can appear as Answer using exactly eight words, Do not use more or fewer than eight words, or Avoid using any number of words other than eight. These instructions target the same output count while differing in surface form, pragmatic naturalness, and perceived strictness. Figure 1 illustrates the resulting evaluation problem: a model can satisfy one realization and fail another even when the task, target, checker, and decoding configuration remain fixed. We study this phenomenon across three count and quantity constraints with transparent checkers applied to the complete visible output: exact word count, key- 1 arXiv:2610.05278v1 [cs.CL] 4 Oct 2026 word inclusion exactly once, and an inclusive 8–12 word range. Such requirements occur in short-form generation, interface text, form fields, and automatically checked benchmark tasks. They also provide a particularly con- trolled setting because the operational predicate is trans- parent and can be evaluated without subjective scoring. Exact word count serves as the primary probe, while the keyword and range extensions test whether the pattern persists when the verifier counts a lexical item or accepts a broader interval. The resulting sensitivity is construction-specific rather than a uniform avoidance or negation effect. Some avoid- ance and exclusion forms are strongly disruptive, while other negative-looking forms remain close to the posi- tive baseline. Matched affirmative controls show that length and target position also matter, and a strict JSON- structure probe shows that the direction of wording effects can vary by constraint. These differences affect not only absolute compliance but also comparative model rank- ings. Human validation confirms broad agreement on the target even when model behavior differs substantially across forms. The evaluation consequence is direct.A single- template score reflects both satisfaction of the operational predicate and compatibility with the selected linguistic realization. We consolidate the matched design as WISE (Wording Invariance and Sensitivity Evaluation), an eval- uation suite and reporting protocol for wording robustness in verifiable constraints. WISE complements formwise compliance with mean and worst-form performance, the best–worst wording gap, failure direction, model-level variation, and ranking stability. Our contributions are: 1. We introduce WISE, a matched evaluation design that isolates within-constraint wording effects while holding the task, target, checker, model, decoding configuration, and output scope fixed. 2.We show that construction-specific wording choices, including but not limited to avoidance and exclusion, can change compliance across verifier-equivalent constraints. Matched affirmative controls and a strict JSON-structure probe narrow the interpretation be- yond a simple polarity account. 3.We show that wording changes failure profiles and comparative model rankings, and summarize these effects through mean, worst-form, wording-gap, ranking-stability, multiple-comparison, and uncer- tainty measures. 2 Related Work Negation in language models.Negation remains chal- lenging for language models across factual probing, natural-language inference, and generation. Kassner and Schütze [2020] show that pretrained models often fail to distinguish affirmative and negated statements in cloze probes. Controlled inference benchmarks similarly reveal persistent weaknesses under sentential and subclausal negation [Hossain et al., 2020, Truong et al., 2022]. These difficulties remain visible in larger autoregressive models [Truong et al., 2023], while affirmative paraphrases can improve performance on semantically equivalent negated inputs [Rezaei and Blanco, 2024]. Prior work mainly studies whether models represent or infer the semantic effect of negation. We examine constrained generation, where the model must maintain an operational requirement throughout the complete vis- ible response. By separating direct negation, complex negation, avoidance, and exclusion, we expose wording forms that produce distinct compliance patterns under a fixed target and checker. Verifiable instruction-following evaluation.Automat- ically verifiable benchmarks evaluate requirements such as output length, keyword inclusion, formatting, and structural constraints through deterministic checkers [Chen et al., 2024, Ren et al., 2026, Gu et al., 2024, Ye et al., 2026]. IFEval establishes this paradigm [Zhou et al., 2023], while FollowBench and InFoBench extend it to fine-grained and compositional requirements [Jiang et al., 2024, Qin et al., 2024]. IFBench further studies generalization to unseen verifiable constraints [Pyatkin et al., 2025]. These benchmarks show that compliance varies with constraint type, composition, and novelty. Our study isolates another axis: the linguistic form used to express one fixed operational predicate. The base task, target, checker, model, decoding configuration, and output scope remain unchanged, allowing the effect of wording to be examined without changing the constraint inventory or composition. Promptsensitivityandevaluationreliability. Language-model evaluations are sensitive to prompt formulation, including task descriptions, templates, demonstrations, paraphrases, and instance-level varia- tions [Webson and Pavlick, 2022, Mizrahi et al., 2024, Zhuo et al., 2024, Yang et al., 2025, Chatterjee et al., 2024, Sun et al., 2024, Sclar et al., 2024, Cao et al., 2024, 2 Chataigner et al., 2026]. Evaluation procedures can also amplify apparent prompt sensitivity when scoring fails to recognize semantically valid responses [Hua et al., 2025]. Within instruction-following evaluation, IFEval++ examines robustness under related prompt variants [Dong et al., 2026], while MOSAIC studies effects associated with constraint position, quantity, and interaction [Purpura et al., 2026]. Compared with prior instruction-following robustness and multi-prompt evaluations, WISE isolates a narrower matched contrast: the base task, operational predicate, numerical target, deterministic checker, decoding config- uration, repetitions, and output scope are held fixed, while only the linguistic realization of the constraint changes. This enables paired form-level comparisons of compli- ance, failure direction, and model-ranking stability under verifier-equivalent requirements [Mizrahi et al., 2024, Dong et al., 2026, Chataigner et al., 2026, Purpura et al., 2026]. 3 Problem Setup Letiindex a base task,ma model,fa constraint wording form, andja repeated generation. Each task specifies an operational targett i , such as an exact word count, a keyword that must occur exactly once, or an admissible word-count range. The prompt combines a fixed content requestx i with a wording-specific realizationw f (t i )of that target. Model M m produces ˆy ijmf = M m (x i ,w f (t i )).(1) A deterministic verifierV i evaluates the complete visi- ble output: C ijmf = V i (ˆy ijmf )∈0, 1.(2) For exact word count, V i (y) = 1WordCount(y) = n i ,(3) whereWordCount(·)counts whitespace-delimited to- kens. Headings, Markdown, framing text, and wrapper content remain part of the evaluated output. The keyword experiment instead requires exactly one case-insensitive whole-word occurrence of a specified keyword, while the length-range experiment requires a word count in the inclusive interval [8, 12]. For wording formf, pooled compliance over a speci- fied model setM is ̄ C f = 1 |M|IJ X m∈M I X i=1 J X j=1 C ijmf .(4) Its contrast against the positive wording is ∆ f = ̄ C f − ̄ C POS ,(5) where negative values indicate lower compliance than under the positive form. For a tested set of wording formsF, we summarize wording robustness through C mean = 1 |F| X f∈F ̄ C f ,(6) C worst = min f∈F ̄ C f ,(7) G wording = max f∈F ̄ C f − min f∈F ̄ C f .(8) Here,C mean records average compliance across the tested realizations,C worst captures performance under the weak- est form, andG wording measures instability across forms. Model-specific versions of these quantities reveal whether pooled robustness reflects a shared response or an average over divergent systems. We interpret them together with form-specific failure profiles. For comparative analyses, we rank models separately under each wording form. Relative to POS, we report Kendall’sτ b [Kendall, 1938], the proportion of model pairs whose strict ordering reverses, the Jaccard overlap between top-three model sets, and the maximum absolute change in average rank. Pairs tied under either form are excluded from the reversal rate. Formal definitions appear in Appendix F. The experiments use matched comparisons. Within each task and model, the base request, operational target, verifier, decoding configuration, output scope, and num- ber of generations remain fixed; only the wording form changes. Differences acrossftherefore estimate wording sensitivity under a common task and checker. Aggregate summaries are reported alongside model-specific results because pooled effects can conceal large drops, reversals, ceiling behavior, or floor behavior. 4 Experimental Setup Tasks and target counts.WISE contains 100 base tasks, balanced across target counts of 5, 8, 10, and 12 words. The base tasks were developed from task patterns in existing instruction-following benchmarks [Zhou et al., 2023, Jiang et al., 2024, Qin et al., 2024] and rewritten as short factual or explanatory requests. We screened them for factual stability, answerability without specialized knowledge, and compatibility with all four target counts, 3 FormTemplate POSAnswer using exactly N words. NEGDo not use more or fewer than N words. NEVER Never use more or fewer than N words. AVOID1 Avoid using any number of words other than N . AVOID2Avoid giving an answer that is not exactlyNwords. CNEGDo not fail to answer using exactly N words. Table 1: Wording forms in the original WISE exact-count panel. All forms require the complete visible output to contain exactly N words. keeping the count constraint as the primary source of difficulty. Each item retains the same request and target across wording forms, with three independent generations collected for every model–item–form combination. Wording forms.The original WISE exact-count panel compares six natural realizations of the same word-count requirement, shown in Table 1. The forms were written for this study to span positive, prohibitive, avoidance, exclusion, and indirect-negative constructions while pre- serving the same operational predicate. They are eval- uated as complete instruction realizations, retaining the combinations of lexical choice, scope, and syntax that occur in natural prompts. The avoidance-family panel broadens this set with four additional avoidance and ex- clusion forms. Theexact-countdesignincludesamatched constructional-control panel with four verifier-equivalent forms: the original POS instruction, a longer affirmative form (Your answer should consist of a total of exactlyN words), an affirmative form that places the target later (The total number of words in your answer should be exactlyN), and AVOID1. This panel separates broad wording sensitivity from a simple negative-polarity ac- count while preserving the same 100 tasks, target counts, three repetitions, 120-token cap, and complete-output verifier. The primary analysis uses nine models for which the endpoint/configuration could be matched to the original exact-count collection. Models.We evaluate thirteen instruction-tuned models from seven providers: Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 [Anthropic, 2025, 2026b,a]; GPT-4o mini, GPT-4.1 mini, and GPT-5.5 [OpenAI, 2024, 2025, 2026]; Gemini 2.5 Flash and Gemini 3.1 Flash-Lite [Google, 2025, 2026]; DeepSeek Chat and DeepSeek V4 Pro [DeepSeek, 2026a,b]; Mistral Small [Mistral AI, 2026]; Qwen 3.6 35B A3B [Qwen Team, 2026]; and Kimi K2.6 [Moonshot AI, 2026]. The exact-count experiments use all thirteen models, whereas the keyword and range extensions use the seven models available when those panels were run. We supply no system prompt and use direct REST calls. Exact API identifiers, access dates, and provider-specific details appear in Appendix A. Decoding.For the original exact-count and avoidance- family experiments, outputs are capped at 120 tokens and three generations are collected for every model–item– form combination. Nine models use temperature 0.2; Kimi K2.6, DeepSeek V4 Pro, Claude Opus 4.8, and GPT-5.5 use the request configurations accepted by their respective endpoints, with GPT-5.5 reasoning effort set to none. The constructional-control panel uses each retained model’s endpoint configuration, held fixed across all four forms. Main metric. The primary outcome is strict exact- count compliance over the complete visible output. Word count is computed using Python whitespace splitting, without removing headings, Markdown, framing text, or wrappers. We also report mean absolute count error and the proportions of under-target and over-target outputs. Full counting conventions appear in Appendix R. Uncertainty. The exact-count study uses a provider– model–item hierarchical bootstrap with 2000 replicates. Each replicate resamples providers, models within sam- pled providers, and tasks within sampled models while retaining all forms and repetitions associated with each sampled item. Primary exact-count contrasts use the eleven models with nonfloor saved-output regimes; all- model results appear in Appendix O, and Gemini diag- nostics appear in Appendix I. The keyword panel uses an item-cluster bootstrap, and the range panel uses a model–item bootstrap, following the sampling hierarchy available in each experiment. These intervals summa- rize uncertainty within the studied panels. Alternative exact-count bootstrap schemes appear in Appendix P. Cross-constraint extensions. The keyword and range panels reuse the same 100 base tasks, a 120-token limit, and three generations per model–item–form combination. The keyword constraint requires a specified keyword ex- actly once, whereas the range constraint requires 8–12 whitespace-delimited words in the complete visible out- put. Primary contrasts use the six non-Gemini models 4 Figure 2: Strict compliance across three constraints for the six shared non-Gemini models. AVOID is lowest throughout. NEG has larger negative effects for exact count and keyword inclusion than for the 8–12 range, whose interval includes zero. Exact-count AVOID de- notes AVOID1. shared by both extensions, with all-model results reported in Appendix J. Structural-control extension. We include a strict JSON-structure probe as a structural extension of the matched design. The base request and target field spec- ification are held fixed, while the constraint is realized as POS, NEG, or AVOID. The complete-output verifier requires parseable JSON with exactly the requested key set and no extra prose. This probe tests whether wording sensitivity extends beyond count and quantity predicates. 5 Results Figure 2 provides the cross-constraint view for the three count and quantity settings. AVOID has the lowest pooled compliance in all three, whereas direct negation varies by constraint. The results follow the structure of the design. We begin with the original exact-count panel, use con- structional controls to separate polarity from length and target placement, expand the avoidance and exclusion family, examine consequences for model rankings, and then test transfer to keyword occurrence, a permissive length range, and a strict JSON-structure constraint. We close the section with human interpretation and multiplic- ity analyses. 5.1 Exact-Count Wording Sensitivity The original six-form experiment asks whether logically related instructions yield stable exact-count compliance. Table 2 shows a selective wording effect rather than a uniform response to negative language. AVOID1 produces the largest decrease, falling 20.5 percentage points below POS, while AVOID2 and CNEG remain close to the positive baseline. NEG and NEVER occupy an intermediate regime. The contrast between AVOID1 and AVOID2 motivates a broader test: does the observed loss belong to a recurrent wording family, or to one unusually difficult template? 5.2 Constructional Controls The original panel varies polarity, lexical choice, syntac- tic scope, and target position jointly. We next isolate two affirmative constructional changes: instruction length and target placement. Table 3 reports the nine-model analysis. The ordering remains POS>POS_LONG> POS_END>AVOID1. The longer affirmative form low- ers compliance by 6.7 points, and moving the target later lowers compliance by 18.2 points. AVOID1 remains the lowest pooled form, but only 2.9 points below POS_END. The result therefore rules out a simple account in which the effect is solely caused by negative polarity or the lexical item avoid. Instead, the operationally equivalent instruction changes difficulty through a bundle of con- structional properties, including length, target position, scope, and polarity. Including all eleven models preserves the pooled ordering and qualitative conclusion. 5.3Avoidance and Exclusion Family Robust- ness The avoidance-family panel retains POS and AVOID1 and introduces four related forms: Avoid exceeding or falling short ofNwords (AVOID3), Avoid an answer with a word count different fromN(AVOID4), Use no word count exceptN(EXCLUDE1), and Your answer should have no length other thanNwords (EXCLUDE2). All tasks, targets, models, decoding configurations, repe- titions, and checker rules remain unchanged. Table 4 establishes recurrence beyond AVOID1. Ev- ery tested avoidance or exclusion form falls below the corresponding POS baseline, while the close agreement between the two POS runs, 49.5% and 49.2%, indicates that the contrast is not driven by an unstable baseline. Ef- fect size remains strongly form-dependent, ranging from 12.2 to 30.5 percentage points. Model-specific vulnerability is heterogeneous. Large losses occur among both strong and weak POS perform- ers, and no single adverse form captures every model’s weakest case. The forms also change how generation departs from the target: AVOID1 and AVOID4 produce 5 FormComplianceDiff. vs. POS95% CIMean abs. error POS49.5−0.71 NEG39.0-10.4[-17.1, -4.4]1.00 NEVER40.8-8.6[-14.6, -2.6]0.91 AVOID128.9-20.5[-29.0, -12.9]2.96 AVOID246.9-2.6[-6.3, 3.2]0.83 CNEG50.1+0.7[-2.2, 4.5]0.67 Table 2: Original exact-count results for the eleven non-Gemini models. Values are percentages; intervals are 95% provider–model–item bootstrap intervals against POS. FormCompliance Diff. vs. POS95% CI POS54.9− POS_LONG48.2-6.7[-9.2, -4.3] POS_END36.7-18.2 [-21.3, -15.1] AVOID133.8-21.1 [-24.4, -17.9] Table 3: Constructional-control analysis for nine retained models. POS_LONG is a longer affirmative realization; POS_END is an affirmative realization with the target later in the sentence. substantial errors in both directions, AVOID3 and EX- CLUDE2 more often undershoot, and EXCLUDE1 has the highest rate of extremely short outputs. Appendix D provides the full distributional analysis and figures. 5.4Wording Alters Comparative Model Rank- ings Wording changes not only absolute compliance but also comparative conclusions across models. We rank the eleven primary non-Gemini models separately under each form in the avoidance-family panel and compare each ordering with the POS reference. Table 5 shows that ranking disruption is not limited to one form. Kendall’sτ b ranges from 0.514 to 0.844, while 7.4% to 24.1% of model pairs that are strictly ordered under both forms reverse direction. Top-three overlap ranges from 0.20 to 1.00, and the largest individual dis- placement ranges from two to five rank positions. EXCLUDE1 produces the largest pairwise disruption, with 24.1% of strictly ordered model pairs reversing rela- tive to POS. The top-ranked model changes from Claude Opus 4.8 under POS to Claude Sonnet 4.6 under EX- CLUDE1, and the two top-three sets have a Jaccard over- lap of only 0.20. Claude Opus 4.8 moves from rank 1 to rank 5, while DeepSeek V4 Pro moves from rank 7 to rank 3. The largest displacement across all forms occurs under AVOID1, where Claude Haiku 4.5 moves from rank 6 to rank 11. Figure 3 shows that rank changes occur throughout the evaluated panel rather than only among models near the bottom of the POS ordering. A fixed wording can therefore affect both measured compliance and the model comparisons derived from it. These results describe the tested model panel and wording set rather than a universal ordering of language models. 5.5 Transfer Beyond Exact Count Keyword occurrence. We next test whether wording sensitivity persists when the verifier counts a lexical item rather than output length. Each task is paired with a key- word absent from the task text, and compliance requires exactly one case-insensitive whole-word occurrence in the complete visible output. Across the six shared non-Gemini models, compliance falls from 77.4% under POS to 55.3% under NEG and 44.3% under AVOID. The corresponding decreases are 22.1 points for NEG, with a 95% interval of [-25.8, - 18.2], and 33.2 points for AVOID, with an interval of [-37.3, -29.1]. In the seven-model all-model analysis, the zero-occurrence rate rises from 22.8% under POS to 58.9% under AVOID, while repeated keyword use re- mains uncommon. Model-level effects vary substantially: Claude Haiku changes little and Claude Sonnet shows a small reversal, whereas GPT-4o mini, GPT-4.1 mini, DeepSeek Chat, and Mistral decline sharply. Complete results appear in Appendix J and Table 20. A permissive length range.We next relax exactness by accepting any output between 8 and 12 words. The tested forms are a positive requirement, a direct prohibition against falling outside the interval, and an avoidance instruction. Across the six shared non-Gemini models, compliance is 89.4% under POS, 85.8% under NEG, and 80.6% un- 6 FormComplianceDiff. vs. POS95% CIUnderOver POS49.2−31.519.3 AVOID127.9-21.3[-29.1, -14.0]33.738.4 AVOID334.0-15.2[-23.2, -9.6]52.113.9 AVOID425.2-23.9[-32.3, -16.6]40.134.6 EXCLUDE118.7-30.5[-38.9, -22.6]44.037.3 EXCLUDE236.9-12.2[-18.1, -6.1]49.213.8 Table 4: Avoidance-family panel for the eleven non-Gemini models. Values are percentages; intervals are 95% provider–model–item bootstrap intervals against POS. Formτ b 95% CI Reversal95% CI AVOID10.624 [0.367, 0.722]18.5 [13.2, 31.5] AVOID30.844 [0.587, 0.891]7.4[5.5, 20.4] AVOID40.673 [0.440, 0.782]16.4 [10.9, 27.8] EXCLUDE1 0.514 [0.273, 0.648]24.1 [17.0, 36.4] EXCLUDE2 0.745 [0.514, 0.855]12.7[7.3, 24.1] Table 5: Ranking stability relative to POS in the avoidance-family panel. Reversal is the percentage of model pairs whose strict ordering changes. Intervals are task-bootstrap 95% intervals. The corresponding top- three Jaccard overlaps are 1.00, 0.50, 0.50, 0.20, and 0.50; maximum rank shifts are 5, 2, 4, 4, and 3. der AVOID. The NEG contrast is -3.6 points, with a 95% interval of [-10.6, 1.2], while AVOID is -8.8 points below POS, with an interval of [-16.3, -3.2]. AVOID is lowest for all six models, although model-level best–worst gaps range from 1.3 points for Claude Sonnet to 24.3 points for Claude Haiku. The wider admissible interval there- fore attenuates wording sensitivity without eliminating it. Complete results appear in Appendix J and Appendix M. 5.6 Transfer to a Structural Constraint The preceding experiments use count and quantity predi- cates. We next test whether the same matched-design phe- nomenon appears under a structural constraint. Compli- ance requires parseable JSON with exactly the requested key set and no additional visible prose under a determinis- tic complete-output verifier. The pooled six-model results are shown in Table 6. The JSON probe provides a structural example of wording-dependent pooled compliance beyond count- ing, but its direction differs from the count experiments: AVOID has the highest pooled compliance rather than the lowest. The dominant failure mode is unparseable output, while wrong key sets are essentially absent. This FormCompliance Diff. vs. POS Parse fail POS13.1−86.9 NEG16.1+3.183.9 AVOID20.6+7.679.4 Table 6: Strict JSON-structure probe across six models. Compliance requires the complete visible output to be valid JSON with exactly the required key set. Values are percentages. result supports the narrower claim that verifier scores are sensitive to linguistic realization, not the stronger claim that avoidance wording always hurts compliance. 5.7 Human Interpretation of Wording Forms Five annotators independently evaluated the six exact- count wording forms across targets of 5, 8, 10, and 12 words. For each instruction, they judged whether it re- quired exactlyNwords and rated its clarity, naturalness, ambiguity, and complexity on five-point scales. Every form receives at least 90% exact-Njudgments. POS and EXCLUDE2 provide the cleanest behavioral comparison: all five annotators judge both forms as requiring exactlyNwords, yet EXCLUDE2 reduces pooled model compliance by 12.2 percentage points rel- ative to POS. The wording effects therefore cannot be explained solely by whether humans recognize the in- tended exact-count requirement. Full ratings, annotator- level judgment rates, and agreement statistics appear in Appendix Q. 5.8 Statistical Robustness The principal wording effects remain after multiplicity correction within each experimental family. In the origi- nal exact-count panel, NEG, NEVER, and AVOID1 re- main significant after Holm correction, whereas AVOID2 7 Figure 3: Model rankings across wording forms in the avoidance-family panel. Rows follow the POS ordering. Each cell reports average rank, with strict compliance shown below. Ties receive average statistical ranks. Wording changes both absolute compliance and comparative model ordering. and CNEG do not. All five avoidance-family contrasts and both keyword contrasts remain significant. For the permissive length range, AVOID remains significant while NEG remains inconclusive. Full results appear in Appendix G. 6 Discussion and Conclusion Wording sensitivity is construction-specific. The ex- periments do not support a single avoidance-only or negation-only mechanism. AVOID1 is disruptive, but AVOID2 and CNEG remain close to POS; affirmative controls show large drops when the instruction is longer or places the target later; and the JSON probe reverses the direction of the avoidance effect. The more stable con- clusion is evaluative: a verifier-equivalent requirement can yield different scores under different linguistic con- structions. Human judgments reinforce this result: every tested exact-count form receives at least 90% exact-N judgments, while POS and EXCLUDE2 receive unani- mous judgments yet differ by 12.2 percentage points in model compliance. Implications for evaluation. A deterministic verifier gives each output a reproducible score, but a single- template evaluation may still yield unstable compliance estimates and model rankings under alternative realiza- tions of the same predicate. WISE addresses this by holding the task, target, checker, model, decoding configuration, and output scope fixed while varying the constraint realization. It retains the conventional POS score and adds mean and worst- form compliance, the best–worst wording gap, ranking agreement, pairwise reversals, and failure direction. WISE is intended as a reporting protocol rather than a universal worst-form leaderboard, since worst-form compliance depends on the chosen variant set. It tests whether scores, failure directions, and rankings remain 8 stable across a declared matched variant set. What transfers across constraints.Across the studied settings, the transferable phenomenon is not that one surface form is always harmful. Instead, matched verifier- equivalent instructions can change measured compliance, and the direction and size of that change depend on the predicate being evaluated. Exact count and keyword inclusion show large losses under avoidance wording; the permissive range weakens but does not remove the effect; and the JSON setting shows that a structural verifier can respond differently. This pattern is why WISE reports a family of paired contrasts rather than a single robustness number. Limitations WISE is instantiated mainly on three English count and quantity predicates: exact word count, keyword inclusion exactly once, and an inclusive word-count range. The JSON probe adds one structural-control setting, but it is intentionally narrow. Extending the same matched de- sign to ordering, capitalization, citation format, tool-use schemas, and multi-constraint composition would estab- lish how wording effects vary across a broader constraint inventory. The wording inventory consists of natural instruction realizations that vary jointly in lexical choice, scope, syntax, and pragmatic form. The resulting contrasts characterize sensitivity to complete instructions. The constructional-control panel partially addresses this is- sue by varying affirmative length and target position, but it is not a full factorial design. A larger factorial exten- sion could separately estimate the contribution of polarity, length, target position, scope, and pragmatic naturalness. The exact-count and range verifiers use whitespace- delimited words over the complete visible output. This rule is transparent and reproducible, although alternative tokenization, language-specific segmentation, Markdown normalization, or answer-body extraction could change absolute compliance rates. The reported conclusions con- cern relative wording sensitivity under the stated verifier. The exact-count study covers thirteen models, whereas the keyword and range panels cover seven; cross- constraint comparisons therefore use the six non-Gemini models shared across experiments. Endpoint-specific de- coding configurations are held fixed within each model, but absolute rates are not directly comparable across providers. Two Gemini adapters exhibit saved-output floor behavior and are excluded from primary wording contrasts; Appendix I reports the associated diagnostics. The blinded wording validation contains five annota- tors. Although all forms receive at least 90% exact-N judgments and the agreement analyses show moderate to strong reliability, the study remains a small sample of English-speaking annotators. The ratings contextualize perceived clarity, naturalness, ambiguity, and complexity but do not establish how these properties would be judged across broader populations or languages. Ethical Considerations This work evaluates wording robustness using short, non-sensitive factual and explanatory prompts. The re- leased benchmark contains no personally identifying or intentionally offensive content. The experiments use hosted language-model APIs and therefore incur inference-related computational and environmental costs [Luccioni et al., 2024]. WISE is intended to comple- ment, rather than replace, broader evaluations of model capability. Reporting its results without the stated con- straint, language, model, and verifier scope could lead to overgeneralized model comparisons. References Anthropic.Introducing Claude Haiku 4.5, October 2025.URLhttps://w.anthropic.com/news/ claude-haiku-4-5. Anthropic.Introducing Claude Opus 4.8, May 2026a. URLhttps://w.anthropic.com/news/ claude-opus-4-8. Anthropic. Introducing Claude Sonnet 4.6, February 2026b. URLhttps://w.anthropic.com/news/ claude-sonnet-4-6. Bowen Cao, Deng Cai, Zhisong Zhang, Yuex- ian Zou, and Wai Lam.On the worst prompt performance of large language models.In Ad- vances in Neural Information Processing Sys- tems, volume 37, pages 69022–69042, 2024. doi: 10.52202/079017-2205. URLhttps://proceedings. neurips.c/paper_files/paper/2024/hash/ 7fa5a377b7ffabcce43cd00231b3f9c-Abstract-Conference. html. Clea Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Taik, Elliot Creager, and Golnoosh Far- 9 nadi.Say it another way: Auditing LLMs with a user-grounded automated paraphrasing framework. In Proceedings of the 19th Conference of the Euro- pean Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1441– 1467, Rabat, Morocco, March 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026. eacl-long.67.URLhttps://aclanthology.org/ 2026.eacl-long.67/. Anwoy Chatterjee, H S V N S Kowndinya Renduchin- tala, Sumit Bhatia, and Tanmoy Chakraborty. POSIX: A prompt sensitivity index for large language mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14550–14565, Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-emnlp.852. URLhttps://aclanthology. org/2024.findings-emnlp.852/. Yihan Chen, Benfeng Xu, Quan Wang, Yi Liu, and Zhen- dong Mao. Benchmarking large language models on controllable generation under diversified instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17808–17816, 2024. doi: 10.1609/aaai.v38i16.29734. URLhttps://ojs. aaai.org/index.php/AAAI/article/view/29734. DeepSeek. DeepSeek API Change Log, 2026a. URL https://api-docs.deepseek.com/updates/. DeepSeek.DeepSeek V4 Preview Release, April 2026b.URLhttps://api-docs.deepseek.com/ news/news260424/. Jianshuo Dong, Yutong Zhang, Liu Yan, Zhenyu Zhong, Tao Wei, Chao Zhang, and Han Qiu. Revisiting the reliability of language models in instruction-following. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7784–7812, San Diego, Cal- ifornia, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026. acl-long.354.URLhttps://aclanthology.org/ 2026.acl-long.354/. Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): 378–382, 1971. doi: 10.1037/h0031619. Google.Gemini 2.5 Flash,2025.URL https://ai.google.dev/gemini-api/docs/ models/gemini-2.5-flash. Google.Gemini 3.1 Flash-Lite, 2026.URL https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-flash-lite. Zihui Gu, Xingwu Sun, Fengzong Lian, Zhanhui Kang, Chengzhong Xu, and Ju Fan. DINGO: Towards diverse and fine-grained instruction-following evaluation. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pages 18108–18116, 2024. doi: 10.1609/aaai.v38i16.29768. URLhttps://ojs.aaai. org/index.php/AAAI/article/view/29768. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. An analysis of natural language inference bench- marks through the lens of negation.In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106–9118, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.732.URLhttps://aclanthology. org/2020.emnlp-main.732/. Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. Flaw or artifact? rethink- ing prompt sensitivity in evaluating LLMs. In Pro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 19889– 19899, Suzhou, China, November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025. emnlp-main.1006.URLhttps://aclanthology. org/2025.emnlp-main.1006/. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4667– 4688, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.257.URLhttps://aclanthology. org/2024.acl-long.257/. Nora Kassner and Hinrich Schütze. Negated and mis- primed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th An- nual Meeting of the Association for Computational Lin- guistics, pages 7811–7818, Online, July 2020. Asso- ciation for Computational Linguistics. doi: 10.18653/ 10 v1/2020.acl-main.698. URLhttps://aclanthology. org/2020.acl-main.698/. M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1–2):81–93, 1938.doi: 10.1093/ biomet/30.1-2.81.URLhttps://academic.oup. com/biomet/article/30/1-2/81/176907. Alexandra Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of AI deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, New York, NY, USA, 2024. Association for Computing Machinery. doi: 10. 1145/3630106.3658542. URLhttps://dl.acm.org/ doi/10.1145/3630106.3658542. Mistral AI.Mistral Small 4 model card, 2026. URLhttps://docs.mistral.ai/models/ model-cards/mistral-small-4-0-26-03. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt LLM evaluation. Transac- tions of the Association for Computational Linguistics, 12:933–949, 2024. doi: 10.1162/tacl_a_00681. URL https://aclanthology.org/2024.tacl-1.52/. Moonshot AI.Kimi K2.6 documentation, 2026. URLhttps://platform.kimi.ai/docs/guide/ kimi-k2-6-quickstart. OpenAI.GPT-4o mini model, 2024.URL https://developers.openai.com/api/docs/ models/gpt-4o-mini. OpenAI.GPT-4.1 mini model, 2025.URL https://developers.openai.com/api/docs/ models/gpt-4.1-mini. OpenAI.GPT-5.5model,2026.URL https://developers.openai.com/api/docs/ models/gpt-5.5. Alberto Purpura, Li Wang, Sahil Badyal, Gene Beaufrand, and Adam Faulkner.Deconstructing instruction- following: A new benchmark for granular evaluation of large language model instruction compliance abil- ities. In Proceedings of the 19th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 1334–1349, Rabat, Morocco, March 2026. Associa- tion for Computational Linguistics. doi: 10.18653/ v1/2026.eacl-long.62. URLhttps://aclanthology. org/2026.eacl-long.62/. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. Generaliz- ing verifiable instruction following.In Advances in Neural Information Processing Systems, vol- ume 38, 2025.URLhttps://proceedings. neurips.c/paper_files/paper/2025/hash/ 46499a0622ecf568b72d17b61e45dbd5-Abstract-Datasets_ and_Benchmarks_Track.html. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. InFoBench: Evalu- ating instruction following ability in large language models. In Findings of the Association for Computa- tional Linguistics: ACL 2024, pages 13025–13048, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-acl.772. URLhttps://aclanthology.org/ 2024.findings-acl.772/. Qwen Team.Qwen3.6-35B-A3B model card, 2026. URLhttps://huggingface.co/Qwen/Qwen3. 6-35B-A3B. Huimin Ren, Yan Liang, Baiqiao Su, Chaobo Sun, Hengtong Lu, Kaike Zhang, and Chen Wei. LexIn- structEval: Lexical instruction following evaluation for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 25117–25123, 2026. doi: 10.1609/aaai.v40i30. 39701. URLhttps://ojs.aaai.org/index.php/ AAAI/article/view/39701. MohammadHossein Rezaei and Eduardo Blanco. Para- phrasing in affirmative terms improves negation un- derstanding.In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 602–615, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. acl-short.55.URLhttps://aclanthology.org/ 2024.acl-short.55/. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Repre- sentations, 2024.URLhttps://proceedings. 11 iclr.c/paper_files/paper/2024/hash/ 6c0e99d736da621403018ca7b32b1a4d-Abstract-Conference. html. Patrick E. Shrout and Joseph L. Fleiss. Intraclass cor- relations: Uses in assessing rater reliability. Psycho- logical Bulletin, 86(2):420–428, 1979. doi: 10.1037/ 0033-2909.86.2.420. Jiuding Sun, Chantal Shaib, and Byron C. Wal- lace.Evaluating the zero-shot robustness of instruction-tuned language models. In The Twelfth International Conference on LearningRepre- sentations, 2024.URLhttps://proceedings. iclr.c/paper_files/paper/2024/hash/ d3221cdb27e49d9c1cd35ad254feccfe-Abstract-Conference. html. Hung Thinh Truong, Yulia Otmakhova, Timothy Bald- win, Trevor Cohn, Jey Han Lau, and Karin Verspoor. Not another negation benchmark: The NaN-NLI test suite for sub-clausal negation. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 883–894, Online only, November 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. aacl-main.65. URLhttps://aclanthology.org/ 2022.aacl-main.65/. Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. Language models are not naysayers: An analysis of language models on negation bench- marks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 101–114, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10.18653/ v1/2023.starsem-1.10. URLhttps://aclanthology. org/2023.starsem-1.10/. Albert Webson and Ellie Pavlick. Do prompt-based mod- els really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States, July 2022. Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 2022.naacl-main.167. URLhttps://aclanthology. org/2022.naacl-main.167/. Jiuding Yang, Hui Liu, Weidong Guo, Yu Xu, and Di Niu. Enhancing and assessing instruction-following with fine-grained instruction variants. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 3773–3782, New York, NY, USA, 2025. Association for Computing Ma- chinery. doi: 10.1145/3746252.3761322. URLhttps: //dl.acm.org/doi/10.1145/3746252.3761322. Junjie Ye, Caishuang Huang, Zhuohan Chen, Wenjie Fu, Chenyuan Yang, Leyi Yang, Yilong Wu, Peng Wang, Meng Zhou, Xiaolong Yang, Tao Gui, Qi Zhang, Zhongchao Shi, Jianping Fan, and Xuanjing Huang. MulDimIF: A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. In Findings of the Associa- tion for Computational Linguistics: ACL 2026, pages 2078–2104, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.99. URLhttps: //aclanthology.org/2026.findings-acl.99/. Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason E Weston, and Jing Xu. Following length constraints in instructions. In Pro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 24232– 24243, Suzhou, China, November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025. emnlp-main.1233.URLhttps://aclanthology. org/2025.emnlp-main.1233/. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URLhttps://arxiv.org/ abs/2311.07911. Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. ProSA: Assessing and understanding the prompt sensitivity of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976, Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-emnlp.108. URLhttps://aclanthology. org/2024.findings-emnlp.108/. A Model and API Details Table 7 lists the evaluated models, exact API identifiers, and access dates. 12 ProviderPaper nameAPI model IDAccessed AnthropicClaude Haiku 4.5 claude-haiku-4-5June 12, 2026 AnthropicClaude Sonnet 4.6 claude-sonnet-4-6June 12, 2026 AnthropicClaude Opus 4.8 claude-opus-4-8June 20, 2026 OpenAIGPT-4o mini gpt-4o-miniJune 12, 2026 OpenAIGPT-4.1 mini gpt-4.1-miniJune 12, 2026 OpenAIGPT-5.5 gpt-5.5-2026-04-23June 20, 2026 GoogleGemini 2.5 Flash gemini-2.5-flashJune 13, 2026 GoogleGemini 3.1 Flash Lite gemini-3.1-flash-liteJune 17, 2026 DeepSeekDeepSeek Chat deepseek-chatJune 13, 2026 DeepSeekDeepSeek V4 Pro deepseek-v4-proJune 17, 2026 MistralMistral Small mistral-small-latestJune 13, 2026 QwenQwen 3.6 35B A3B Qwen/Qwen3.6-35B-A3BJune 17, 2026 KimiKimi K2.6 kimi-k2.6June 17, 2026 Table 7: Evaluated models and API identifiers. Dynamic aliases are reported exactly as called. The exact-count study uses all thirteen models; the keyword and range extensions use the seven models available when those runs were executed. B Paired Failure Examples Table 8 gives matched examples in which POS satisfies the strict count while another wording fails for the same model, item, target, and repetition. CAvoidance-Family Failure Examples Table 9 complements Figure 6 with representative outputs. AVOID1 and AVOID4 retain substantial over-target mass, while AVOID3, EXCLUDE1, and EXCLUDE2 more often undershoot. D Surface-Behavior Audit We examine visible output patterns in the avoidance- family panel without using hidden states, token prob- abilities, or other model-internal information. Across the eleven primary non-Gemini models, wording forms produce distinct count-error distributions, as shown in Figure 4. AVOID3 and EXCLUDE2 are concentrated below the target, while AVOID1 and AVOID4 exhibit errors in both directions. EXCLUDE1 has the broadest distribution and the highest rate of outputs containing at most two words, 9.3%, compared with 0.5% under POS. Meta-discourse openings such as Here is, Here’s, and The answer is are rare, occurring in fewer than 0.4% of outputs under every form. They therefore provide only limited descriptive context and do not explain the main form-level compliance differences. POSAVOID1AVOID3AVOID4EXCLUDE1EXCLUDE2 Form −7.5 −5.0 −2.5 0.0 2.5 5.0 7.5 Observed words - target Figure 4: Distribution of observed word count minus the requested target in the avoidance-family panel for the eleven primary non-Gemini models. AVOID3 and EXCLUDE2 are concentrated below the target, while AVOID1, AVOID4, and especially EXCLUDE1 exhibit broader bidirectional errors. Boxes show the interquartile range, whiskers extend to 1.5 times the interquartile range, and outliers are not displayed. E Additional Figures Figures 5, 6, 7, and 8 visualize model-specific gaps, fail- ure directions, and pooled contrasts. F Ranking Stability Metrics Let ̄ C mf = 1 IJ I X i=1 J X j=1 C ijmf (9) 13 ModelFormTarget POS outputCount Contrast output Claude Haiku 4.5AVOID15Jupiter is the largest planet.5Jupiter is the largest planet here. (6, over) DeepSeek ChatAVOID45Jupiter is the largest planet.5Jupiter (1, under) GPT-5.5EXCLUDE15Gold’s chemical symbol is Au.5Au (1, under) Table 8: Matched avoidance-family examples in which POS is compliant but an alternative wording fails for the same model, item, target, and repetition. The examples include both over-target and under-target failures. ModelFormTargetCountOutput GPT-4o miniAVOID156The capital of France is Paris. GPT-4o miniAVOID456The capital of France is Paris. Claude Sonnet 4.6 EXCLUDE154William Shakespeare wrote it. Claude Sonnet 4.6 EXCLUDE187Vaccines train immunity, stopping transmission be- tween people. GPT-4.1 miniEXCLUDE254William Shakespeare wrote Hamlet. Table 9: Representative failures from the avoidance-family panel under strict entire-output whitespace counting. 020406080100 Strict exact count compliance (%) Mistral Small Qwen 3.6 35B A3B DeepSeek Chat Kimi K2.6 DeepSeek V4 Pro Claude Haiku 4.5 GPT-4o mini Claude Sonnet 4.6 GPT-4.1 mini GPT-5.5 Claude Opus 4.8 EXCLUDE1 -19.7 p EXCLUDE1 -22.3 p AVOID4 -32.7 p EXCLUDE1 -35.3 p AVOID3/AVOID4 -30.7 p AVOID1/AVOID4 -49.7 p AVOID1 -33.3 p AVOID1 -16.3 p EXCLUDE1 -27.0 p EXCLUDE1 -31.3 p EXCLUDE1 -56.7 p POS Worst avoidance/exclusion Figure 5: POS-to-worst-form compliance gaps for the eleven primary non-Gemini models in the avoidance- family panel. Circles show POS compliance, diamonds show each model’s weakest avoidance or exclusion form, and annotations report the corresponding decrease. denote modelm’s compliance under formf. Relative to POS, the strict pairwise reversal rate is R f = X a<b 1 ̄ C a,POS − ̄ C b,POS ̄ C af − ̄ C bf < 0 X a<b 1 ̄ C a,POS ̸= ̄ C b,POS , ̄ C af ̸= ̄ C bf . (10) Pairs tied under either form are excluded. We report Kendall’sτ b , top-three Jaccard overlap, and maximum absolute rank displacement. Figure 6: Under-target, exact, and over-target rates in the avoidance-family panel for the eleven non-Gemini models. G Statistical Robustness We assess multiplicity within each primary experimental family using Holm correction. Raw two-sided bootstrap p-values are computed from the same bootstrap draws used for the corresponding manuscript confidence inter- vals. Table 10 reports the complete results. HWISE Audit Summary Table 11 reports the complete audit summaries for the original exact-count panel, avoidance-family panel, con- structional controls, keyword constraint, inclusive length 14 FamilyContrastDiff.95% CI Raw p Holm p Original exact-count NEG–POS-10.4[-17.1, -4.4]<.001<.001 NEVER–POS-8.6[-14.6, -2.6].006.018 AVOID1–POS-20.5 [-29.0, -12.9]<.001<.001 AVOID2–POS-2.6[-6.3, 3.2].357.714 CNEG–POS+0.7[-2.2, 4.5].689.714 Avoidance familyAVOID1–POS-21.3 [-29.1, -14.0]<.001<.001 AVOID3–POS-15.2[-23.2, -9.6]<.001<.001 AVOID4–POS-23.9 [-32.3, -16.6]<.001<.001 EXCLUDE1–POS -30.5 [-38.9, -22.6]<.001<.001 EXCLUDE2–POS -12.2[-18.1, -6.1]<.001<.001 Keyword primaryNEG–POS-22.1 [-25.8, -18.2]<.001<.001 AVOID–POS-33.2 [-37.3, -29.1]<.001<.001 Range primaryNEG–POS-3.6[-10.6, 1.2].237.237 AVOID–POS-8.8[-16.3, -3.2]<.001<.001 Table 10: Multiplicity analysis for the primary wording contrasts. Differences are percentage points relative to POS. Raw two-sided bootstrapp-values and 95% confidence intervals are computed from the same bootstrap draws used in the primary analysis. Holm correction is applied separately within each experimental family. Figure 7: Original-form compliance contrasts against POS for the eleven non-Gemini models. Error bars are provider–model–item bootstrap 95% confidence inter- vals. range, and JSON probe. I Gemini Saved-Output Diagnostics The two Gemini models require separate interpretation. Their saved outputs are very short across forms, and exact compliance is at or near zero for targets 8, 10, and 12. We define a saved-output floor regime as exact compliance of at most 1% at every target above the minimum and a change of less than one mean observed word as the Figure 8: Avoidance-family compliance contrasts against POS for the eleven non-Gemini models. requested target increases from 5 to 12. Both Gemini models satisfy this criterion in both experiments. Ta- ble 12 gives the Gemini 2.5 target-level results; Gemini 3.1 shows the same qualitative pattern, with exact com- pliance of 21.6%, 0.9%, 0.7%, and 0.0% at targets 5, 8, 10, and 12 in the original experiment and mean output length between 2.36 and 2.88 words. A post hoc audit finds no API-error rows, and saved output-token counts are far below the 120-token limit, at most 9 tokens for Gemini 2.5 and 15 for Gemini 3.1, which provides no evidence of length-limit truncation. Gemini 2.5 has no empty saved outputs; Gemini 3.1 has empty-text rates of 9.7% and 10.7% in the original and 15 PanelPOS C mean C worst G wording Original exact count49.542.528.921.2 Avoidance family49.232.018.730.5 Constructional controls 54.943.433.821.1 Keyword exactly once77.459.044.333.2 Inclusive 8–12 range89.485.380.68.8 JSON structure13.116.613.17.6 Table 11: WISE audit summaries across experimental panels. Values are percentages over each panel’s primary model set. Gaps are computed from unrounded rates. For JSON, POS is the original positive form even though it is not the best-performing form. Experiment Target Exact Mean words Mean error Original530.43.66-1.34 Original80.02.85-5.15 Original100.02.89-7.11 Original120.02.93-9.07 Family530.93.56-1.44 Family80.02.76-5.24 Family100.02.91-7.09 Family120.02.90-9.10 Table 12: Gemini 2.5 Flash saved-output floor analysis. Exact is strict compliance in percent; mean error is ob- served word count minus requested target. avoidance-family runs. The adapter excludes usage meta- data from evaluated text. The runs did not retain raw re- sponse bodies,finishReason, or per-part thought flags, so provider-specific response-part extraction cannot be fully excluded as a contributor to short or empty saved outputs. We therefore exclude both Gemini models from the main wording-specific contrasts. J Cross-Constraint Pooled Results Tables 13 and 14 report the primary pooled analyses for the six non-Gemini models shared by the keyword and range extensions. Tables 15 and 16 report the correspond- ing seven-model analyses, including Gemini 2.5 Flash. K Constructional-Control Model-Level Results Table 17 reports the four-form exact-count constructional- control panel by model. The primary analysis uses the nine retained models shown here and excludes Qwen and FormComplianceDiff.95% CI POS77.4 − NEG55.3 -22.1 [-25.8, -18.2] AVOID44.3 -33.2 [-37.3, -29.1] Table 13: Primary keyword-exactly-once results for the six shared non-Gemini models. Values are percentages; differences and item-cluster bootstrap 95% confidence intervals are relative to POS. FormCompliance Diff.95% CI POS89.4 − NEG85.8-3.6[-10.6, 1.2] AVOID80.6-8.8 [-16.3, -3.2] Table 14: Primary inclusive 8–12 word range results for the six shared non-Gemini models. Values are percent- ages; differences and model–item bootstrap 95% confi- dence intervals are relative to POS. Kimi because their endpoint/configuration could not be matched to the original collection. Including all eleven models preserves the pooled ordering. L JSON-Structure Model-Level Re- sults Table 18 reports the strict JSON-structure probe by model and form. The main failure mode is unparseable complete visible output rather than wrong key sets. M Length-Range Model-Level Results Table 19 reports the complete model-by-form results. Claude Sonnet is near ceiling and Gemini 2.5 Flash is at floor, while the remaining models show varying gaps. The experiment contains 6,300 unique generations, zero API errors, zero missing rows, and zero duplicate keys. N Keyword Model-Level Results Table 20 reports the complete model-by-form keyword results. 16 FormComp. Diff.95% CI Zero Multi. POS69.7 − 22.87.5 NEG49.2 -20.5 [-24.0, -16.7] 47.53.2 AVOID38.7 -31.1 [-34.8, -27.3] 58.92.5 Table 15: Keyword-exactly-once robustness across all seven models, including Gemini 2.5 Flash. Values are percentages. Diff. and confidence intervals are relative to POS; Zero and Multi. denote zero and multiple keyword occurrences. FormComp. Diff.95% CI Below Above POS76.6 −17.26.2 NEG73.6-3.0[-9.4, 1.0]17.19.3 AVOID69.1-7.5 [-14.3, -2.2]22.08.8 Table 16: Inclusive 8–12 word range robustness across all seven models, including Gemini 2.5 Flash. Values are percentages; intervals use the model–item bootstrap. O Exact-Count Model-Level Results Tables 21 and 22 report complete model-by-form com- pliance values. These tables include Gemini; the main text separates the Gemini models because they satisfy the floor-effect criterion in Section I. P Bootstrap Sensitivity Analyses The main text uses provider–model–item hierarchical bootstrap intervals. Because there are only six providers in the non-Gemini analysis and the hierarchy is unbal- anced, we also compute two sensitivity analyses on the non-Gemini set: a model–item bootstrap, which treats model as the highest resampling level, and an equal- model item bootstrap, which bootstraps items within each model and averages model-level contrasts with equal weight. Table 23 shows that the main conclusions hold for the key contrasts. Q Human Wording Validation We conducted a blinded wording-validation study to ex- amine whether the six exact-count forms are perceived as expressing the same operational requirement. The val- idation set contains 24 randomized items, crossing six forms with target counts of 5, 8, 10, and 12 words. An- notators saw neutral item identifiers, target counts, and ModelPOSPOS_LONGPOS_ENDAVOID1 Claude Haiku 4.549.334.721.75.0 Claude Opus 4.880.764.335.783.0 Claude Sonnet 4.659.363.060.044.7 DeepSeek Chat49.334.316.026.3 DeepSeek V4 Pro59.358.746.028.7 GPT-4.1 mini53.753.047.739.3 GPT-4o mini58.048.331.324.0 GPT-5.558.355.056.742.0 Mistral Small26.322.715.311.3 Table 17: Per-model compliance in the constructional- control panel. Values are percentages. ModelPOSNEGAVOID Claude Haiku 4.50.00.00.0 Claude Sonnet 4.60.00.00.0 DeepSeek Chat70.762.062.3 GPT-4.1 mini0.76.05.3 GPT-4o mini7.028.755.0 Mistral Small0.00.01.0 Table 18: Strict JSON-structure compliance by model and wording form. Values are percentages. instructions without form labels. For each item, annotators judged whether the instruc- tion required exactlyNwords and provided ratings of clarity, naturalness, ambiguity, and complexity on five- point scales. Five annotators completed all 24 items, yielding 120 judgments for each measure. Q.1 Annotator-Level Exact-Count Judgments Table 24 shows that the exact-count interpretation is broadly shared. POS and EXCLUDE2 receive exact- Njudgments for all twenty annotator and target com- binations. The four remaining forms receive exact-N judgments in 90% to 95% of combinations. Variation is therefore limited to a small number of target-specific judgments rather than a general rejection of the exact- count interpretation. Q.2 Inter-Annotator Agreement We summarize agreement using Fleiss’κfor the binary exact-Njudgment [Fleiss, 1971] and ICC(2,k) for the continuous ratings [Shrout and Fleiss, 1979]. Table 25 reports moderate agreement for the binary exact-Njudgment, with Fleiss’κ = 0.54and a bootstrap 95% interval of [0.34, 0.72]. Agreement is higher for clarity and naturalness, with ICC values of 0.79 and 0.75, 17 ModelPOS NEG AVOID Gap Claude Haiku 4.573.754.349.3 24.3 Claude Sonnet 4.6 100.0 100.098.71.3 GPT-4o mini97.095.388.38.7 GPT-4.1 mini97.097.393.34.0 Gemini 2.5 Flash0.00.00.70.7 DeepSeek Chat92.090.389.03.0 Mistral Small76.777.764.7 13.0 Table 19: Strict 8–12 word range compliance by model and wording form. Gap is the best–worst difference in percentage points. ModelPOSNEGAVOID Claude Haiku 4.578.777.777.7 Claude Sonnet 4.688.088.391.0 GPT-4o mini70.312.39.3 GPT-4.1 mini91.779.738.3 Gemini 2.5 Flash23.312.75.0 DeepSeek Chat73.744.341.0 Mistral Small62.329.78.3 Table 20: Keyword-exactly-once strict compliance by model and wording form. Values are percentages. These are the seven models included when the keyword experi- ment was run. and somewhat lower for ambiguity and complexity, with ICC values of 0.68 and 0.64. These statistics support using the ratings as descriptive evidence while preserving uncertainty about fine-grained differences among forms. The central behavioral comparison does not depend on fine-grained rating agreement. POS and EXCLUDE2 receive unanimous exact-Njudgments, yet EXCLUDE2 lowers pooled model compliance by 12.2 percentage points relative to POS. AVOID1, AVOID4, and EX- CLUDE1 also receive 90% exact-Njudgments while pro- ducing considerably larger compliance reductions. Hu- man recognition of the target and model compliance are therefore related but distinct properties. R Reproducibility Details All experiments use the same 100-task manifest, which records the item ID, base task, and target count. Separate form manifests record the wording templates, and the released raw generation files include the instantiated in- struction and complete prompt for each generation. The original exact-count study and its avoidance-family panel ModelPOSNEGNEVERAVOID1AVOID2CNEG Claude Haiku 4.554.738.338.74.754.052.7 Claude Sonnet 4.658.360.067.041.746.760.7 Claude Opus 4.882.378.379.379.081.380.3 GPT-4o mini53.738.338.321.048.054.7 GPT-4.1 mini55.339.039.344.054.360.0 GPT-5.562.767.064.743.052.756.7 Gemini 2.5 Flash9.75.78.79.36.75.7 Gemini 3.1 Flash Lite6.76.34.05.36.36.0 DeepSeek Chat36.328.731.713.339.339.3 DeepSeek V4 Pro55.029.735.338.048.751.7 Mistral Small24.77.79.310.719.724.7 Qwen 3.6 35B A3B26.725.024.316.726.727.0 Kimi K2.634.317.321.06.044.343.7 Table 21: Original six-form strict compliance by model and form. Values are percentages. ModelPOSAVOID1AVOID3AVOID4EXCLUDE1EXCLUDE2 Claude Haiku 4.553.74.036.34.07.736.0 Claude Sonnet 4.656.340.057.750.341.754.3 Claude Opus 4.880.778.367.759.724.067.0 GPT-4o mini55.021.740.027.323.327.7 GPT-4.1 mini57.342.744.341.030.337.3 GPT-5.563.742.757.754.332.362.3 Gemini 2.5 Flash8.78.76.78.35.09.0 Gemini 3.1 Flash Lite7.34.36.07.35.37.3 DeepSeek Chat34.012.023.31.34.033.3 DeepSeek V4 Pro52.739.022.022.031.732.7 Mistral Small23.79.36.36.04.014.3 Qwen 3.6 35B A3B27.012.310.06.34.721.0 Kimi K2.637.04.78.35.31.720.3 Table 22: Avoidance-family strict compliance by model and form. Values are percentages. pair each task with six wording forms and collect three generations per model and form. The constructional- control panel uses four forms and three generations for each retained model. The keyword and range extensions use three forms and three generations for each of seven models, and the JSON probe uses three forms with a strict full-output parser. Theexact-countandrangecheckerscount whitespace-delimitedtokensusinglen((output or "").strip().split()).Any Unicode whites- pace recognized by Python separates words, while multiple spaces and newlines collapse undersplit(). Punctuation-attached strings, contractions, hyphenated words, Markdown markers, headings, and wrappers count as visible tokens when separated by whitespace; empty or whitespace-only output has count zero. The released WISE package includes raw genera- tions, strict verifier results, word or keyword counts, compliance indicators, directional failure flags, boot- strap summaries, model-level profiles, paired examples, constructional-control and JSON-control summaries, and audit summaries. The WISE audit script combines the constraint panels while preserving their constraint- specific metrics. The human-validation release contains 18 ExperimentContrastMethod95% CIDiff. OriginalAVOID1− POSProvider–model–item[-29.0, -12.9]-20.5 OriginalAVOID1− POSModel–item[-28.7, -13.6]-20.5 OriginalAVOID1− POSEqual-model item[-23.1, -17.8]-20.5 OriginalNEG− POSProvider–model–item[-17.1, -4.4]-10.4 OriginalNEG− POSModel–item[-16.0, -4.6]-10.4 OriginalNEG− POSEqual-model item[-12.9, -7.9]-10.4 FamilyAVOID4− POSProvider–model–item[-32.3, -16.6]-23.9 FamilyAVOID4− POSModel–item[-31.5, -17.1]-23.9 FamilyAVOID4− POSEqual-model item[-26.7, -21.3]-23.9 FamilyEXCLUDE1− POSProvider–model–item[-38.9, -22.6]-30.5 FamilyEXCLUDE1− POSModel–item[-38.4, -23.5]-30.5 FamilyEXCLUDE1− POSEqual-model item[-33.2, -27.7]-30.5 Table 23: Bootstrap sensitivity analyses for key non- Gemini contrasts. Differences and confidence intervals are percentage points. FormA1A2A3A4A5 Exact-N POS1.00 1.00 1.00 1.00 1.00100 AVOID11.00 0.75 1.00 0.75 1.0090 AVOID31.00 1.00 0.75 1.00 1.0095 AVOID41.00 1.00 0.75 0.75 1.0090 EXCLUDE1 1.00 0.75 1.00 0.75 1.0090 EXCLUDE2 1.00 1.00 1.00 1.00 1.00100 Table 24: Annotator-level exact-count judgment rates by wording form. Each annotator entry is the proportion of exact-Njudgments across target counts of 5, 8, 10, and 12 words. The final column aggregates all twenty judgments for each form. verified form-level summaries, annotator-level exact-N rates, and agreement statistics, together with release scripts that reproduce the reported tables. MeasureStatisticEstimate95% CI Exact-N judgment Fleiss’ κ0.54 [0.34, 0.72] ClarityICC(2,k)0.79 [0.62, 0.90] NaturalnessICC(2,k)0.75 [0.56, 0.88] AmbiguityICC(2,k)0.68 [0.45, 0.84] ComplexityICC(2,k)0.64 [0.40, 0.81] Table 25: Inter-annotator agreement across all 24 wording and target combinations. Fleiss’κis used for the binary exact-Njudgment. Continuous ratings use ICC(2,k), a two-way random-effects, absolute-agreement, average- measures specification. Confidence intervals are obtained from 2000 bootstrap replicates that resample the 24 items while retaining all five annotators for each sampled item. 19 Qualitative Examples of Wording-Sensitive Failures Matched examples from the released generation outputs. Longer Affirmative Form Changes the Count Model: GPT-4o miniTarget: 5 words Task: Name the process plants use to make food. POS outputcount = 5 Photosynthesis converts sunlight into food. POS_LONG outputcount = 6, over Photosynthesis is how plants make food. Same polarity and same verifier; the longer affirmative realization fails while POS succeeds. Late Target Placement in an Affirmative Form Model: GPT-4.1 miniTarget: 5 words Task: What is H2O commonly called? POS outputcount = 5 H2O is commonly called water. POS_END outputcount = 4, under H2O is called water. The target appears near the end of the instruction; the matched POS generation is compliant. AVOID1 Can Overshoot Model: Claude Haiku 4.5Target: 5 words Task: Name the largest planet in the solar system. POS outputcount = 5 Jupiter is the largest planet. AVOID1 outputcount = 6, over Jupiter is the largest planet here. Avoidance wording can fail by overshooting rather than by omitting content. AVOID1 Can Also Undershoot Model: Mistral SmallTarget: 5 words Task: What is the square root of 64? POS outputcount = 5 Eight times eight equals sixty-four. AVOID1 outputcount = 1, under 8 The same avoidance form can undershoot, showing that the failure direction is not uniform. EXCLUDE1 Can Produce an Extreme-Short Output Model: DeepSeek ChatTarget: 5 words Task: Name the largest planet in the solar system. POS outputcount = 5 Jupiter is the largest planet. EXCLUDE1 outputcount = 1, under Jupiter The exclusion construction can trigger answer suppression rather than execution of the count constraint. JSON Wrapper Failure Under Strict Full-Output Parsing Model: Claude Haiku 4.5Form: AVOID Task: Name the capital of France. Visible outputparse failed ```json "answer": "Paris", "reason": "Paris is the capital city of France, located in the north-central part of the country on the Seine River." ``` The object is valid-looking, but Markdown fencing makes the complete visible output fail the strict JSON parser. JSON Wording Can Improve Strict Compliance Model: GPT-4o miniTask: Hamlet author Task: Who wrote the play Hamlet? POS outputparse failed ```json "answer": "William Shakespeare", "reason": "William Shakespeare is widely regarded as the author of Hamlet, which is considered one of his most famous and significant works in English literature." ``` AVOID outputcompliant "answer": "William Shakespeare", "reason": "Hamlet is one of the most famous tragedies written by William Shakespeare, attributed to his work in the early 17th century." This structural example shows why the claim is directional only within a constraint setting: avoidance wording can reduce wrapper failures for JSON.

Instant research discovery