Paper deep dive

Language models can notice an impossible engineering problem yet still report it as solved

Shaoliang Yang, Jun Wang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 149

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language models draft engineering calculations, but answer accuracy does not show whether they reject an impossible problem. We tested 14 models on 30 pairs of mechanics problems, each with a valid version and one made impossible by changing a given value or assumption. Two independent solvers verified every answer key and showed that each flawed problem was physically impossible. We scored solving of valid problems separately from rejection of their flawed counterparts. Each reply required a "solved" or "cannot solve" status; rejection meant "cannot solve" or withholding an answer. The initial prompts did not warn that problems could be flawed. Across three recent models, 12 of 90 replies failed to reject a flawed problem. In 11 of these replies, the model stated the flaw, answered a corrected problem and still reported the original as "solved", according to artificial intelligence raters and numerical checks. We later retested four models from one provider, offering "flawed" instead of "cannot solve" and asking them to name and explain the defect. Three models showed statistically significant increases in rejection, but valid-problem solving fell in three. Evaluations therefore need to score both versions and distinguish flaw recognition from the reported status.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

145,964 characters extracted from source content.

Expand or collapse full text

LANGUAGE MODELS CAN NOTICE AN IMPOSSIBLE ENGINEERING PROBLEM YET STILL REPORT IT AS SOLVED A PREPRINT Shaoliang Yang Santa Clara University Santa Clara, CA 95053, USA syang11@scu.edu Jun Wang * Santa Clara University Santa Clara, CA 95053, USA jwang22@scu.edu ABSTRACT Language models draft engineering calculations, but answer accuracy does not show whether they reject an impossible problem. We tested 14 models on 30 pairs of mechanics problems, each with a valid version and one made impossible by changing a given value or assumption. Two independent solvers verified every answer key and showed that each flawed problem was physically impossible. We scored solving of valid problems separately from rejection of their flawed counterparts. Each reply required a “solved” or “cannot solve” status; rejection meant “cannot solve” or withholding an answer. The initial prompts did not warn that problems could be flawed. Across three recent models, 12 of 90 replies failed to reject a flawed problem. In 11 of these replies, the model stated the flaw, answered a corrected problem and still reported the original as “solved”, according to artificial intelligence raters and numerical checks. We later retested four models from one provider, offering “flawed” instead of “cannot solve” and asking them to name and explain the defect. Three models showed statistically significant increases in rejection, but valid-problem solving fell in three. Evaluations therefore need to score both versions and distinguish flaw recognition from the reported status. Keywords large language models· benchmark· engineering mechanics· flawed premises· evaluation Introduction A calculation can be arithmetically careful and still be the wrong thing to hand to an engineer. In practice the statement itself can be the weak part: a load past collapse, a gauge reading that contradicts the kinematics, a spin test above the burst speed, a shrink fit described as tight after the test has opened it. An unqualified numerical answer that accepts such a statement is not a small accuracy loss. It invites the engineer to proceed with a state that does not exist. We ask whether a model that can solve a mechanics problem also rejects it when one datum or premise makes it impossible. To test this, each problem comes in two versions, a consistent one and a flawed twin, and every reply must end with a status, “solved” or “cannot solve”. Benchmarks for engineering and physics score a final number against a single key. On them, current models range from high accuracy on foundational questions to below 60% for the best model on harder strength-of-materials and physics sets [1–3], and an agent framework now outscores the top human competitor in olympiad physics [4]. Benchmarks that grade a model against a simulator, a finite-element code or a design checker still score the answer, not the premise [5–11]. The keys used to judge those answers can themselves be wrong. An audit of Humanity’s Last Exam [12] revised almost half of its items [13], and a review of FrontierMath found wrong author answers [14]. This study therefore verifies its own keys. A second line of work tests premise checking and error detection, and often finds that models answer past a flaw. Premise-critique tests find that models seldom object to a flawed premise unprompted but object more often when told ∗ Corresponding author. arXiv:2610.06668v1 [cs.AI] 5 Oct 2026 Noticing an impossible problem yet reporting it solvedA PREPRINT to check [15]. Similar failures occur with altered and unanswerable medical questions [16] and with mathematical problems made unanswerable or misleading [17–19]. Benchmarks also test detection of flawed steps in mathematical and general reasoning [20,21]. Models struggle to repair their own reasoning errors [22,23], although feedback locating the error can help [24]. Two accounts have been offered for these failures. Accuracy-only scoring rewards an answer over an abstention [25]. Models also comply with illogical requests they know to be wrong, although explicit permission to refuse raised rejection in some models [26]. This paper carries the paired design into engineering mechanics. There, an independent solver can certify that the flaw makes the stated state impossible, and the consistent twin, graded against its own independently verified key, shows whether the model could solve the problem at all. Each problem comes as a pair: a consistent statement and a flawed twin that changes one datum or premise so that the stated state cannot exist, for example a load factor above the collapse load of a two-span beam. The main set of 30 pairs comes from five families of multi-regime problems, which couple two regimes that textbook shortcuts treat separately, such as elastic and plastic behaviour. Both twins are posed with the ordinary solve prompt, and the reply ends with a status, “solved” or “cannot solve”. The score is rejection: a “cannot solve” status or a withheld answer, which is correct for a flawed twin and a false alarm for a consistent one. Restricting the comparison to pairs whose consistent twin the model solved tests the premise check only where the model has shown it can solve. To check the numerical keys, two independent methods solve each problem and must agree to one part in a million on every graded answer. A third-party solver also checks three of the five multi-regime families, including every planted flaw in those families. A deterministic parser grades the answers of 14 models from three providers against the verified keys, with no model judge. For the separate recognition assessment, artificial intelligence (AI) raters read replies with model identities hidden and label whether they state the flaw. Numerical checks show whether the returned answers fit a corrected problem or the impossible state. The paired test shows how a model can recognise a flaw yet report the original problem as solved. Three recent models each rejected 23 to 28 of the 30 flawed twins. Across these models there were 12 missed rejections; in 11 of these replies, the model stated the flaw, answered a corrected problem and still reported “solved”. Under the ordinary solve prompt, newer versions rejected more flawed twins in two comparisons with older versions of the same model. Both gaps remained on pairs whose consistent twin both versions solved, but only one was reproduced on retesting. The offered status also changed rejection. We retested four models from one provider, replacing “cannot solve” with “flawed” and adding a defect type and a reason. This significantly increased rejection in three models but reduced solving of consistent twins in three. A score read from the status alone therefore mixes whether a model notices a flaw with how it reports that judgment, and this study measures the two separately. Results The results first show what a reply to a flawed twin does, then the benchmark, solving, rejection, its stability and the effect of the answer format. Some recent models state the flaw while returning a solved status Figure 1a shows one pair from the multi-regime set (Tier 2): a beam on a bed that pushes but never pulls (family T04), whose flawed twin states a dial-gauge reading that contradicts the computed lift-off. Every model received it with the ordinary solve prompt (the unwarned arm). Figure 1b shows four replies. GPT-6-Astra reports “cannot solve” and writes that the model predicts about 3.49 m of uplift, contradicting the gauge. Fable 5.1 writes that the gauge “is inconsistent with the stated no-tension Winkler model, which predicts the left end lifting off”, and returns the lift-off solution with a solved status. Sonnet 5 writes that the gauge “directly confirms the left end remains in contact”; the revealing observation is in the reply and the conclusion is backwards. Haiku 4.5 takes the gauge as the end displacement and builds its answer on it. These outcomes distinguish recognition from rejection. A miss is any outcome other than a rejection on a flawed twin. Every unwarned miss was coded into one of three classes, miss∈noted, misjudged, unexamined.(1) 2 Noticing an impossible problem yet reporting it solvedA PREPRINT a One flaw twin (T04, beam on a no-tension bed) consistent twin L = 7.8 m; P 1 = 130 kN at 3.7 m; P 2 = 510 kN at 6.2 m E = 210 GPa; I = 500 × 10 6 m 4 b = 0.46 m; k s = 66 MN m −3 left-end dial gauge g (downward +): g = -3.49 m the left end lifts 3.49 m: agrees with the mechanics flawed twin L = 7.8 m; P 1 = 130 kN at 3.7 m; P 2 = 510 kN at 6.2 m E = 210 GPa; I = 500 × 10 6 m 4 b = 0.46 m; k s = 66 MN m −3 left-end dial gauge g (downward +): g = 2.84 m the left end moves down: contradicts the lift-off Only the gauge datum differs; both twins ask the same five questions (contact, contact length, peak reaction, sagging moment, end uplift). The other data fix the answer: the left end lifts off (reproduced by OpenSees). b Four replies to the flawed twin (no warning given) rejected GPT-6-Astra (status: cannot solve) “The stated beam and foundation model predicts approximately 3.49 m of left-end uplift, contradicting the specified gauge measurement of 2.84 m downward.” JSON answer: no numbers returned noted, then answeredFable 5.1 (status: solved) “Note: the dial-gauge reading (2.84 m downward at the left end) is inconsistent with the stated no-tension Winkler model, which predicts the left end lifting off ...” JSON answer: numbers = the lift-off solution with the gauge set aside (repair) checked, wrong conclusion Sonnet 5 (status: solved) “... settlement (g = 2.84 m, downward), which directly confirms the left end remains in contact (a lifted end would necessarily show zero/upward displacement ...” JSON answer: numbers for full contact unexamined: datum usedHaiku 4.5 (status: solved) “The left end clearly remains in contact: w(0) = 2.84 m > 0 (measured positive downward).” JSON answer: numbers built on the gauge datum compression-only bed lift-off 2.15 m P 1 P 2 consistent: g = -3.49 m (end lifts) flawed: g = 2.84 m (end moves down) 020406080100 share of flawed twins (%) Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash Gemini 3.1 Pro gpt-oss-120b Tier 2 flaws (30 flawed twins, unwarned) 020406080100 share of flawed twins (%) not run Tier 1 premise flaws (20 flawed twins, unwarned) c rejected noted: repaired input, answered noted: answered for the flawed state noted (Tier 1, not split) checked, wrong conclusion unexamined no usable answer Figure 1: What a reply to a flawed twin does. a, One evaluation pair from the no-tension-bed family (T04). The statements differ only in the dial-gauge datum. The sketch draws the pair to scale along the beam: the left 2.15 m of the beam lifts off the compression-only bed, so a gauge reading of−3.49m (upward) agrees with the mechanics and 2.84 m downward contradicts them. The other data fix a lift-off of the left end, which OpenSees (Open System for Earthquake Engineering Simulation) reproduces. In the consistent twin the gauge value equals the graded uplift of that end, so that twin states one graded answer (Methods). b, Four replies to the flawed twin, unwarned, one per outcome class, with what the JavaScript Object Notation (JSON) answer contains. Quotes are verbatim, with typography normalised. GPT-6-Astra rejects the statement. Fable 5.1 names the inconsistency, returns the lift-off solution with the gauge set aside, and marks the problem solved. Sonnet 5 treats the gauge as confirmation that the end stayed down. Haiku 4.5 builds its answer on the gauge. c, Outcome of every unwarned flawed twin, Tier 2 (30, multi-regime) and Tier 1 (20, standard single-topic): rejected; noted, split on Tier 2 into a repaired input and answers for the flawed state; checked with a wrong conclusion; unexamined; or no usable answer. Classes and the split are from blinded coding (κ = 0.89 and 0.92; Methods). “Noted” means the reply states the flaw and still returns a solved status. “Misjudged” means it runs a revealing check and draws the wrong conclusion. “Unexamined” means it neither checks nor mentions the flaw. Two blinded AI coders labelled 232 misses, with Cohen’s [27]κ = 0.89, and an AI adjudicator resolved disagreements (Methods; Supplementary Fig. 5). The three-class scheme was chosen after disagreements were visible and has no human validation; it is exploratory and does not affect rejection scoring. To check repairs separately, we compared returned numbers with the keys. In the first collection (the panel’s original runs), Fable 5.1, Sonnet 5.5 and Opus 5 together missed 12 flawed twins. All 12 replies stated the flaw; 11 answered a corrected problem, according to blinded AI coding and separate numerical checks (Supplementary Table 12). Eight returned the T04 lift-off solution with the contradictory gauge set aside. The other repairs and the one reply whose numbers fit no state of the beam are detailed in Supplementary Note 2. Six repairs were disclosed in the structured reason field, and five only in prose: a system reading just the status would see “solved” in every case. Stating a flaw without rejecting the problem is not confined to the T04 family: on the 20 single-topic premise pairs of Tier 1, 11 of the same three models’ 23 misses also stated the flaw, on six items from five families (Fig. 1c). 3 Noticing an impossible problem yet reporting it solvedA PREPRINT a Construct (each evaluation set frozen before its model calls) b Evaluate c Analyse Parametric problem families plastic beam, settlement no-tension bed snap-through strip spin-tested disk shrink-fitted disk Tier 2: 5 multi-regime families (above) Tier 1: 13 single-topic families seeded sampling in realistic ranges Two independent references method A closed form or modes method B elements or shooting agree to 10 −6 (relative) third-party solver (OpenSees) T02, T04: keys and flaws T08: mode, force and flaws 62 checks, no failures physical certificates inside both references Acceptance gates R1 references agree R2 decision margin ≥ 5 % R3 sampled class = target R4 worked solution = ref. A R5 every shortcut > 10 % off R6 alternative readings agree R7 conditioning κ ≤ 50; tolerance max(1%, 0.1%κ) R5-R7: Tier 2 Frozen item sets solve flaw twin consistent statement one datum or premise changed Tier 2 25 solve items, 30 flaw-twin pairs (false premise, contradictory or impossible data) Tier 1 52 solve items, 21 critique pairs, 20 premise pairs (Hard-10) Isolated command-line harnesses Claude CLI Codex CLI Antigravity CLI tools off* no web no user config fixed prompt wall-clock timer *Antigravity cannot disable tools: a call that logs a tool step is graded as a failure 14 models, 3 providers Anthropic OpenAI Google open weight 2-4 generations per provider; repeated runs for consistency Deterministic grader (no model judge) reply full text last JSON block parser per-answer tolerance numbers, categories rejection rule cannot solve / flawed, or a withheld answer audits: 50 of 50 sampled replies parsed as intended; no model pair reverses order at 1-5 x tolerance Solving 1)accuracy with Wilson 95 % intervals 2) trap signatures: which textbook shortcut a wrong answer matches 3) consistent twin solved: the control for the flawed twin of each pair Flaw handling (every unwarned flawed twin) rejects the statement? no yes rejected states the flaw? no yes noted runs the check? no yes misjudged unexamined misses coded blind by two AI coders (232 cases, κ = 0.89); noted Tier-2 replies split blind into repair and continuation (26 cases, κ = 0.92) Comparison and reliability 1) paired exact McNemar tests, Holm-adjusted 2)six contrasts, confirmatory on the 21 added pairs 3) same contrasts on pairs both models solve 4) item random-intercept logistic model 5) pass³ over three runs 6) wall time and output tokens per item Figure 2: Benchmark architecture. a, Parametric generators produce five multi-regime and 13 single-topic families. Two independent reference methods and the acceptance gates check each key; OpenSees provides 62 third-party checks for the two-span beam (T02), no-tension bed (T04) and clamped cosine strip (T08). Each evaluation set was frozen before its model calls. A twin pair changes one datum or premise between a consistent and a flawed statement. b, The 14-model panel receives solve, critique and premise-check prompts. A deterministic grader reads the final structured answer, applies the numerical tolerance and scores rejection. The first scored reply counts; harness settings and prompt variants are specified in Methods. c, Separate analyses measure solve accuracy, rejection, recognition and repair. Blinded AI coding describes the misses; paired tests compare models on the same items and on pairs whose consistent twins both models solved. Repeats measure consistency across three runs. CLI, command line interface; JSON, JavaScript Object Notation. Older and smaller models have other failure patterns (Fig. 1c; Supplementary Note 1). Sonnet 5 runs the revealing check and concludes wrongly on 14 of 30 twins. Opus 4.8 states the flaw in 8 misses, but unlike the newer models, 5 of those 8 return numbers for the state that does not exist, such as a residual moment for a load past collapse. Each flawed problem has a verified, solvable twin The paired problems sit in a two-tier benchmark (Fig. 2). Tier 1 tests single-topic mechanics: 52 solve items and 20 consistent/flawed pairs across 13 families, selected as the hardest tenth by a score independent of model answers. Tier 2 couples mechanical regimes that textbook shortcuts treat separately, such as elastic and plastic response. It contains 25 solve items and 30 consistent/flawed pairs across the five families in Fig. 3 and Table 1, frozen before model calls. Methods gives the selection rules and the separate critique task. Two independent methods must agree to one part in a million on every graded answer (Table 1). Acceptance gates also exclude ambiguous statements and cases where a named shortcut lands near the key; the grading tolerance accounts for numerical sensitivity (Methods). OpenSees provides 62 further checks for T02, T04 and T08, including every planted flaw in those families (Supplementary Fig. 2). The turbine-disk (T11) and shrink-fit (T12) families rely on the two internal references. 4 Noticing an impossible problem yet reporting it solvedA PREPRINT Δ λP 1 λP 2 A B C M p1 M p2 T02 Settled two-span beam 0510 x (m) -200 0 200 400 600 800 M (kN m) shortcuts: ignore the settlement; shakedown = collapse; first yield settlement only first hinge, λ = 1.22 collapse, λ = 1.43 residual flaw twin (6 pairs): the load factor exceeds the collapse factor; the requested state cannot be reached P 1 P 2 lifts k s (compression only) T04 No-tension bed 0246 x (m) -5 0 5 10 w , down (m) shortcuts: a bed that also pulls; a rigid beam bed cannot pull (key) bed pulls (shortcut) flaw twin (6): a gauge reports the end moving down while that end lifts (contradictory data) P h inverted state (dashed) T08 Bistable strip 012 stroke (m) 0 1 2 3 4 5 6 7 P (N) one-mode shortcut: M cl -68 % shortcuts: a one-mode model; the symmetric path only symmetric path key: asymmetric one-mode shortcut flaw twin (6): rise too small (h/t 0.75-1.0); the 'switch' never snaps (false premise) ω blade load c plastic zone T11 Disk spin test 100200 r (m) -200 0 200 400 600 800 1000 stress (MPa) yield shortcuts: Hooke's law in the plastic zone; no rim load; elastic only hoop, at overspeed hoop, residual flaw twin (6): the overspeed exceeds the burst speed (physically impossible) ω p, δ shaft in disk T12 Shrink-fitted disk 0510 n (1,000 rpm) 0 20 40 60 80 p (MPa) n op shortcuts: no loosening; the plane-strain yield face as fitted after test (key) wrong yield face no loosening flaw twin (6): said to stay tight, but the spin test grows the bore beyond the interference a b c Figure 3: The five Tier-2 families. a, Sketches. T02, two-span steel beam with a settled middle support and plastic hinges. T04, free–free beam on a bed that pushes and never pulls; in the instance drawn, the left end lifts. T08, clamped cosine strip under a midpoint force; the dashed curve is the inverted stable state. T11, annular turbine disk in a spin test, plastic zone shaded, blade load at the rim. T12, disk shrink-fitted on a shaft. b, Reference-A solutions for one development instance per family, with the answers of the textbook shortcuts. Each shortcut misses at least one graded answer by more than 10%; in T08 the one-mode shortcut lands within 0.3% of the critical force and misses the clamp moment by 68%. Shaded bands mark the plastic-moment range (T02) and the contact zone (T04). c, One flaw twin per family. Only the stated datum or premise differs from the consistent twin. The five instances drawn here are development items, not evaluation items. Six evaluation pairs are used per family (30 pairs). Each flaw twin changes one thing (Fig. 3c; Table 1): a load factor above collapse (T02), a gauge reading that contradicts the lift-off (T04), a rise too small to snap (T08), an overspeed above burst (T11) or a fit called tight that the spin test has loosened (T12). The unwarned arm uses the same prompt as an ordinary solve item, including one closed-book sentence on Tier 2; the reply can report its status as “solved” or “cannot solve”. Other prompt versions are described where they are used. A reply rejects the statement when its status says that the problem cannot be solved or is flawed, or when it withholds a required answer; the exact rule is in the Methods. On a flawed twin a rejection is the scored outcome. On a consistent twin it is a false alarm. The endpoint is the returned status or a withheld answer, which is what a pipeline that reads the JavaScript Object Notation (JSON) output would act on. The endpoint does not require the reply to name the flaw, and a reply that names the flaw and still returns numbers is not a rejection. Recognition is therefore measured separately, from the blinded coding of the reason of each rejection and of each missed reply (Methods). Fourteen models from three providers completed the Tier-2 solve set (Table 2). The comparison is a snapshot of these available models, rather than a random provider sample. Harness settings and incomplete runs are detailed in Methods and Supplementary Tables 2 and 9. Standard problems cluster the models; multi-regime problems separate them On Tier 1, which every model except Gemini 3.1 Pro ran, 11 of the 13 models solved 45 to 52 of the 52 items (Fig. 4a). Four of them, Fable 5.1, Opus 5.5, Sonnet 5.5 and GPT-6-Astra, solved all 52. Haiku 4.5 solved 13 and the open-weight model 11. The best models exceeded the protocol’s 80% bound on this set, so Tier 1 was frozen as the standard-problem anchor and the multi-regime set was built. Tier-1 premise rejection still separates the panel (Supplementary Fig. 4). On Tier 2 the 14 models run from 0/25 to 25/25 (Cochran’sQ = 173, 13 degrees of freedom,P = 4.6× 10 −30 ) [28]. Solve accuracy spans a wide range, but this set cannot resolve the ranking of the top models: Wilson intervals at 23/25 run from 75% to 98%, so 25 items cannot separate the seven models that solve 22 to 25 (Fig. 4a; Supplementary 5 Noticing an impossible problem yet reporting it solvedA PREPRINT Table 1: Benchmark composition and verification. Two independent methods verify each key. Acceptance gates test decision margins, shortcut errors, alternative readings and numerical sensitivity (Methods). Supplementary Table 8 lists the graded quantities. T11 and T12 have no OpenSees check. Family ProblemWhat the two references doSolve items Flaw twin T02Two-span beam, support settlementForce method with kinematic theorems; stiffness method with linear programming of the static shakedown theorem 5Load above collapse T04Beam on a compression-only bedExact exponential basis with nested bracketing; transfer matrices and shooting 5Gauge contradicts lift-off T08Clamped cosine stripBuckling-mode expansion, 2000 modes; Hermite elements, displacement control 5Rise too small to snap T11Turbine-disk spin testClosed form with a root for the plastic boundary; Navier integration through the plastic zone 5Overspeed above burst T12Shrink-fitted disk on a shaftLamé fit plus the T11 disk; shooting for disk and shaft 5Fit claimed tight is loose Tier 113 single-topic familiesPer family, two methods (Supplementary Table 1) 5220 premise pairs Table 2: Model panel. Time limit per call: 1800 s on Tier 1 and 2700 s on Tier 2; the first scored reply per item counts (Methods). Tools, web search and user configuration are off where the harness allows it (Supplementary Table 2). Model identifiers are in Supplementary Table 9, output caps in Supplementary Table 2, and the handling of calls that ended without a reply in the Methods. Repeats are three independent runs. CLI, command line interface. ModelProviderHarnessTier 1Tier 2 solve Flaws (30) Repeats Fable 5.1AnthropicClaude CLI completecompletecompletesolve, flaws Opus 5.5AnthropicClaude CLI completecompletecompletesolve Sonnet 5.5AnthropicClaude CLI completecompletecompletesolve, flaws Opus 5AnthropicClaude CLI completecompletecompletesolve, flaws Sonnet 5AnthropicClaude CLI completecompletecomplete— Opus 4.8AnthropicClaude CLI completecompletecomplete— Haiku 4.5AnthropicClaude CLI173/174completecomplete— GPT-6-AstraOpenAICodex CLIcompletecompletecompletesolve GPT-6-SolOpenAICodex CLIcompletecompletecompletesolve GPT-5.6-SolOpenAICodex CLIcompletecompletecomplete— GPT-5.5OpenAICodex CLIcompletecompletecomplete— Gemini 3.8 Flash GoogleAntigravity completecompletecompletesolve; flaws on 9 Gemini 3.1 ProGoogleAntigravitynot runcompletecomplete— gpt-oss-120bOpen weight Antigravity completecompletecomplete— Table 3). Below them sit a middle group, from Opus 4.8 at 19/25 to Sonnet 5 at 13/25, and a floor of two models at 0/25. A random-intercept model of item difficulty against model level is in the Supplementary Information. Wrong answers in the middle of the panel are mostly recognisable textbook shortcuts, while Haiku 4.5 and gpt-oss-120b mostly fail in no named way (Fig. 4b; Supplementary Note 1). Rejection separates models that solving ranks together Rejection also spreads the panel widely (Fig. 5a; Supplementary Table 3). In the first collection, it runs over the 30 unwarned flawed twins from 30 (Opus 5.5, GPT-6-Astra, GPT-5.6-Sol) through 23 (Opus 5) to 7 (Opus 4.8), 3 (Sonnet 5) and 2 (Haiku 4.5) (Cochran’sQ = 220, 13 degrees of freedom,P = 1.2× 10 −39 ). Solving accuracy alone does not set this order: GPT-5.5 solved 16 of the 25 solve items yet rejected 28 of 30 flawed twins, whereas Opus 4.8 solved 19 and rejected 7 (Fig. 4a). Across the 14 models the two orders still agree broadly (Spearmanρ = 0.70, permutation P = 0.007; exploratory), but among the five models that solved 13 to 19 of the 25 solve items, rejection runs from 3 to 28 of 30. False alarms, rejections of a consistent twin, are rare (Fig. 5b), except for gpt-oss-120b, which rejects 14 consistent twins and 10 flawed ones and each time cites only the difficulty of computing by hand; its rejections are refusals, not checks. Of the 279 rejections by the other 13 models, 275 name the planted flaw (Methods). 6 Noticing an impossible problem yet reporting it solvedA PREPRINT 020406080100 solve items correct (%) Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash Gemini 3.1 Pro gpt-oss-120b Gemini 3.1 Pro: Tier 1 not run Tier 1 (52 items)Tier 2 (25 items) 0510152025 Tier-2 solve items (of 25) one-mode 4 yield face 3 one-mode 1 settlement 1 yield face 1 elastic only 1 one-mode 1 one-mode 2 elastic only 1 correct near-miss (within 5 %) named textbook shortcut other error no usable answer ab Figure 4: Standard problems bunch the panel; multi-regime problems spread it. a, Solve accuracy on Tier 1 (open circles, 52 items) and Tier 2 (filled circles, 25 items). Bars are Wilson 95% intervals. Model names are coloured by provider. Eleven of the 13 models run on Tier 1 solve 45 to 52 of 52; Haiku 4.5 (13) and gpt-oss-120b (11) do not. Gemini 3.1 Pro was not run on Tier 1. On Tier 2 the same panel runs from 0 to 25 correct (Cochran’sQ = 173, 13 degrees of freedom,P = 4.6× 10 −30 , 14 models). b, What each Tier-2 answer is: correct; near-miss (every quantity within 5%); a named textbook shortcut; another error; or no usable answer (time limit, output budget or harness timeout). The version tests were planned after the original nine pairs had been scored and before the other 21 pairs were added, so the confirmatory tests use only these 21 added pairs. In the first collection, rejection separated both Anthropic version pairs on them. Each split counts pairs only the first model rejected against pairs only the second rejected: Opus 5 rejected 15 flawed twins that Opus 4.8 did not, while the reverse happened once, and Sonnet 5.5 led Sonnet 5 by 18 to 0 (Holm-adjustedP = 0.0026and4.6× 10 −5 ; Fig. 5d; Supplementary Table 4). The planned OpenAI contrasts sit at the ceiling, where rejection cannot separate the models, and pooling all 30 pairs gives the same direction for the Anthropic pairs (Supplementary Note 2). In the first collection, the version gaps in rejection differed from the gaps in solving. On the 25 solve items, accuracy separates only the Sonnet pair, in an exploratory test that favours that gap because the families were chosen on Sonnet 5’s failures (11–0; Holm-adjustedP = 0.0059) [29,30]; Opus 5 against Opus 4.8 is 6–0 (P = 0.16; Methods). The 30 consistent twins give the same picture: 13–1 for the Sonnet pair and 5–1 for the Opus pair (exactP = 0.0018and 0.22; exploratory). To compare rejection where both models can solve the valid problem, we restrict the added pairs to those whose consistent twin both models solved. This leaves 17 Opus pairs and 10 Sonnet pairs. The rejection gaps remain, 13–1 and 9–0 (Holm-adjustedP = 0.011and 0.020; Supplementary Table 10). That restriction is exploratory; its caveats are in Supplementary Note 2 and three further checks in Supplementary Note 1. Only one of the two version gaps held on retesting Repeated runs test whether a single score hides inconsistent answers [31,32]. GPT-6-Astra solved all 25 Tier-2 solve items in every run, whereas GPT-6-Sol solved only 16 in all three, despite overlapping intervals for their first-run scores (Fig. 6a). Rejection varied too: Opus 5 rejected 23, 26 and 29 of 30 flawed twins in three runs, and Sonnet 5.5 28, 29 and 28. The next comparison asks whether Opus 5’s advantage over Opus 4.8 persists in a fresh collection. The same figure shows that median wall time does not order solve accuracy, so we claim no cost ranking (Fig. 6b,c; Supplementary Note 2). Both version gaps rest on one run of each older model, so a second collection, fixed before its first call and started about a day after the first collection, repeated them (Methods; Table 3). Opus 5, Opus 4.8, Sonnet 5.5 and Sonnet 5 were each scheduled for two runs on all 30 pairs: a fresh unwarned run with the ordinary solve prompt, and a schema-only run 7 Noticing an impossible problem yet reporting it solvedA PREPRINT 020406080100 flawed twins rejected (%) Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash Gemini 3.1 Pro gpt-oss-120b 27/30 30/30 28/30 23/30 3/30 7/30 2/30 30/30 29/30 30/30 28/30 20/30 22/30 10/30 Tier 2, unwarned (30)Tier 1, unwarned (20) 051015 consistent twins rejected (of 30) 0 0 0 0 0 0 0 0 0 0 1 4 3 14 false alarms 0369 same nine twins rejected; orange: warned false alarms 1 3 2 2 3 5 unwarnedwarned 3051015 discordant items: only second | only first Opus 5.5 vs Opus 5 Opus 5 vs Opus 4.8 Sonnet 5.5 vs Sonnet 5 GPT-6-Sol vs GPT-5.6-Sol GPT-6-Astra vs GPT-6-Sol Opus 5.5 vs GPT-6-Astra n.s. n.s. n.s. n.s. 0.0026 0.011 0.0059 < 0.001 0.02 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. Holm P solve, 25 items (exploratory) rejection, 21 added pairs same 21, both solved abcd Figure 5: Unwarned rejection in the first collection. a, Share of flawed twins rejected (the reply marks the statement unsolvable or withholds an answer). Filled circles: Tier 2, unwarned, 30 twins, with Wilson 95% intervals; counts at the right. Ticks: Tier 1 premise twins, unwarned, 20 pairs. b, False alarms: consistent Tier-2 twins rejected with no warning, out of 30. c, The warned arm on the original nine pairs: rejections of the same nine flawed twins without (open circle) and with (diamond) the warning; orange numbers are the consistent twins rejected with the warning. The warning changes both the instruction and the status field; Supplementary Table 15 separates them for five models. d, Six contrasts fixed before the 21 added pairs existed. Bars count discordant items (only the first model right, right of zero; only the second, left). Grey: solve accuracy, 25 items (exploratory). Blue: rejection on the 21 added pairs (confirmatory). Hatched: the same 21 pairs restricted to those whose consistent twin both models solved (exploratory). Pvalues are exact McNemar tests (two-sided binomial tests on the discordant items), Holm-adjusted for multiple testing within each set of six; values below 0.001 are in Supplementary Table 4. 406080100 items (%) Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 GPT-6-Astra GPT-6-Sol Gemini 3.8 Flash Fable 5.1 Sonnet 5.5 Opus 5 Gemini 3.8 Flash 20/25 22/25 20/25 24/25 25/25 16/25 17/25 25/30 28/30 23/30 4/9 solve items, solved flawed twins, rejected single run every run (pass 3 ) 10 2 10 3 median wall time per item (s) 0 20 40 60 80 100 Tier-2 solve items correct (%) Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash Gemini 3.1 Pro gpt-oss-120b 10 4 10 5 median output tokens per item 0 20 40 60 80 100 tokens as reported by each harness; definitions differ (Supplementary Table 2) abc AnthropicOpenAIGoogleopen weight Figure 6: Reliability and time. a, Items solved (25 solve items) or flawed twins rejected (30; nine for Gemini 3.8 Flash) in every one of three runs (pass 3 , squares) against the individual runs (circles). Solved at least once in three runs (pass@3): Fable 5.1, Opus 5.5, Opus 5 and GPT-6-Astra 25 of 25, Sonnet 5.5 and GPT-6-Sol 24, Gemini 3.8 Flash 19. Item-level repeats are in Supplementary Fig. 6. b, Tier-2 solve accuracy against median wall time per item. c, The same solve accuracy against median output tokens as each harness defines them. Token definitions differ by harness (Supplementary Table 2), so wall time is the common unit. Across this panel, at one effort setting per model, accuracy is not ordered by time. 8 Noticing an impossible problem yet reporting it solvedA PREPRINT Table 3: Second collection of the two confirmatory contrasts. Unwarned prompt: the reply can report “solved” or “cannot solve”. Schema-only prompt: “solved” or “flawed” with a defect type, and no check instruction. Earlier runs: 29–30 September, flawed twins rejected of 30. Fresh runs: times are in Coordinated Universal Time (UTC); calls started 30 September 21:31 UTC to 1 October 07:43 UTC, both arms of each model started together, compared on the same pairs (30, or for Sonnet 5 the 16 pairs with a final unwarned call). Rejected: flawed twins rejected. Consistent: consistent twins solved. Joint: consistent twin solved and flawed twin rejected. FA: false alarms on consistent twins. Over all 29 of its schema-only pairs Sonnet 5 rejected 25, solved 12 consistent twins and raised 3 false alarms. Prespecified tests are in Supplementary Table 16. Earlier unwarnedFresh unwarnedFresh schema only Modelfirstrepeatsrejected consistent joint FA rejected consistent joint FA Opus 52326, 2912/302812030/3030300 Opus 4.87—9/30248030/3019192 Sonnet 5.5 2829, 2827/302926130/3025250 Sonnet 53—0/16110015/16772 whose status field offers “flawed” (next section). A quota limit stopped Sonnet 5’s fresh unwarned run at 16 flawed twins and its schema-only run at 29 pairs (Supplementary Note 2). The Sonnet gap held on the prespecified test, but on fewer pairs. Because of the quota stop, only 10 of the 21 added pairs have a final Sonnet 5 call, and three of those calls ended without a reply, which the scoring counts as no rejection. On these 10 pairs, Sonnet 5.5 rejected 9 flawed twins that Sonnet 5 did not, and the reverse never happened (Holm-adjusted P = 0.016). Counting only the 7 pairs on which both models replied still gives a 6–0 split, but on so few pairs it is not significant (Holm-adjusted P = 0.125; Supplementary Table 16). The Opus gap did not hold. Opus 5 rejected 12 of the 30 flawed twins, against 23, 26 and 29 in its three earlier runs, and Opus 4.8 rejected 9; on the 21 added pairs the split was 2–0 (Holm-adjustedP = 1). Their solving gap barely moved: Opus 5 solved 28 consistent twins and Opus 4.8 24, against 29 and 25 in the first collection (paired split 6–2, exactP = 0.29; Table 3; Supplementary Table 10). Prompts, reported model identifier and command-line binary were unchanged, and the new replies were not given the blinded coding. The drop exceeds the spread of the three earlier runs and was not shared by Opus 4.8 (7 to 9) or Sonnet 5.5 (28 to 27), but these data cannot tell whether the served Opus 5 changed or its run-to-run variation is larger than three runs showed. The answer format changes how often models reject The second collection tests whether the reporting field changes rejection without adding a check instruction. Each model receives the unwarned and schema-only prompts on the same available pairs (Table 3). The schema-only arm raised rejection to near ceiling: Opus 5 from 12 to 30 of 30 flawed twins, Opus 4.8 from 9 to 30 and Sonnet 5 from 0 to 15 of 16, while Sonnet 5.5 already rejected 27 (Table 3; Holm-adjustedP ≤ 1.2× 10 −4 within each of the three models). Under this arm neither version gap was significant. The field also had costs. Consistent twins solved fell for Opus 4.8 (24 to 19), Sonnet 5.5 (29 to 25) and Sonnet 5 (11 to 7 of 16), and false alarms rose from none to two for Opus 4.8 and Sonnet 5. A joint success requires both a correct answer to the consistent twin and rejection of the flawed twin. Under this measure the schema-only prompt helped Opus 5, Opus 4.8 and Sonnet 5, but cost Sonnet 5.5 one pair (Table 3). These are effects of the whole reporting field, which requests a defect type and reason as well as changing the status label. In the first collection, an earlier warned prompt on the original nine pairs both instructed models to check the statement and replaced “cannot solve” with “flawed”, requesting a defect type and reason. Every model that had missed a flaw rejected more under this prompt, but five models gained false alarms (Fig. 5c; Supplementary Table 13). Later, the instruction-only and schema-only prompts each gave 43 rejections of 45 flawed twins across five Anthropic models. Because those arms ran at different times, they do not isolate the components’ causal effects (Supplementary Table 15; Methods). Discussion A verified key plus a flawed twin change the question from “can the model solve it?” to “does the model reject the flawed version of a problem it can solve?”. On this panel the two questions agree broadly (Spearmanρ = 0.70), but among models with similar solving the second separates sharply: GPT-5.5 solved 16 of the 25 solve items and Opus 4.8 19, yet they rejected 28 and 7 of the 30 flawed twins. In the first collection, rejection also separated two newer Anthropic 9 Noticing an impossible problem yet reporting it solvedA PREPRINT models from their older versions, Opus 5 from Opus 4.8 and Sonnet 5.5 from Sonnet 5, and in an exploratory restriction both gaps remained on pairs whose consistent twin both models solved. The retest repeated only the Sonnet gap, on 10 of the 21 added pairs and not significantly on the 7 with a reply from both models, and solve accuracy also separates that pair. The Opus gap, which solve accuracy did not separate significantly (6–0, Holm-adjustedP = 0.16), shrank: Opus 5 rejected 12 flawed twins and Opus 4.8 9, against 23 and 7 before, while the consistent twins they solved barely changed (28 and 24 of 30, against 29 and 25). The evidence that rejection measures something beyond solving therefore rests on such differences across the panel; no version gap that solving lacks has yet been replicated. Accuracy still matters: Haiku 4.5 and gpt-oss-120b solve none of the multi-regime problems, and no rejection score makes up for that. The misses differ by model. Haiku 4.5 and gpt-oss-120b mostly show no premise check in the returned reply, and gpt-oss-120b’s rejections are refusals to compute. Sonnet 5 examines the premise and concludes the wrong way. Opus 4.8 states the contradiction in 8 of its 23 misses and in 5 of those 8 returns numbers for the impossible state. The three recent models, Fable 5.1, Sonnet 5.5 and Opus 5, state it and, in all but one case, answer for a repaired reading, which an engineer reading the reply can see. A pipeline that reads only the status field sees a solved problem with numbers attached, and in five of the eleven repairs the repair appears only in the prose, not in the structured answer. Which of the two readings matters depends on who consumes the answer; a status-level veto is what an automated workflow can act on. In the second collection, a status field that can say “flawed”, with a defect type to name, raised rejection in three of four models against an unwarned run collected alongside it. Neither version gap was significant under it, but it lowered consistent-twin solving in three models. Its net gain in joint successes, pairs with the consistent twin solved and the flawed twin rejected, was therefore large for three models and absent for Sonnet 5.5. Under this field, Opus 5 and Sonnet 5.5, whose first-collection misses had all stated the flaw, rejected every flawed twin; so did Opus 4.8, most of whose misses had not stated it, so the field may prompt checking as well as reporting. Two published accounts are consistent with this: accuracy-only scoring rewards an answer over an abstention [25], and models comply with requests they know to be wrong, while a prompt that explicitly allows a refusal raises rejection in some models [26]. The arms here were not designed to test either account. Several limits bound these claims. The multi-regime sets are small (25 solve items; 30 pairs from five families), too small to rank neighbours inside the top cluster. The families were screened on Sonnet 5, so every comparison holds for these five families. Eight of the 12 missed rejections by the three recent models come from one family, the beam on a no-tension bed (T04), where setting aside a gauge reading that contradicts the mechanics is a natural repair. The confirmatory tests are the six rejection contrasts on the 21 pairs added after the analysis plan was written. The analyses on pairs that both models solved, the rejection-reason classes and the repair split are exploratory and were defined after the replies were read. The result of the warned prompt, which asked for a check and offered “flawed”, rests on nine twins and one wording per arm. The collection and response format limit the comparisons. The status-field comparison rests on four Anthropic models; its two arms ran over the same two periods but were not interleaved call by call, and Sonnet 5’s fresh unwarned run is incomplete. The second collection did not reproduce the Opus gap, and the planned extra runs of the older models were not made. Three harnesses wrap the providers, with different residual system text and incomparable token accounts. Access was by subscription, at one effort setting, on the versions served from 28 September to 1 October 2026; the Claude CLI version of each call was not logged, and a change in the served models between collections cannot be excluded. The miss taxonomy and the reason and repair classes come from AI coders and an AI adjudicator, not from expert raters. Problems were text, closed-book, with no tools and no drawing; transfer to an agent that can call a solver is untested. The panel is a snapshot. Relative to answer-only mechanics benchmarks, the instrument adds a key that two methods must share to one part in a million and a twin that changes one premise within a shared problem. Relative to premise-critique benchmarks outside engineering, the flaws are certified by the reference solvers and the consistent twin is scored as a control. Those peers validate their labels or automatic judges against human annotators [15,20,21]; here the semantic labels are validated only between AI coders, and an expert check of the noted replies is the most direct next validation. The price of the control is sample size, because each pair costs two calls of up to 45 minutes per model and run. Larger draws from the same generators are the direct extension; a private hold-out and a canary string in the release [33] help detect contamination of the set (Data availability). Where a model solves the consistent problem and still returns numbers for the flawed twin, give the answer format a status that can say the problem is flawed, ask for the premise check, or put a verifier in the loop that can refuse the answer, and re-measure on both twins whenever the model, its service or the format changes, because a format that raises rejection can also lower solving. The twin is how to test whether that check happens. 10 Noticing an impossible problem yet reporting it solvedA PREPRINT Methods The evaluation has five steps: 1. Generate problems and verify each key with two independent numerical methods. 2.Build consistent/flawed twins, changing one datum or premise. Tier 1 supplies standard-problem comparisons; Tier 2 supplies the multi-regime paired tests. 3. Collect model replies under fixed solving prompts and prespecified prompt variants. 4.Grade numerical answers and rejection deterministically; measure recognition and repair separately by blinded AI coding and numerical checks. 5.Compare models on the same pairs. Confirmatory rejection tests use only the 21 added pairs; other scopes are specified under Statistics. Problem construction and independent keys Tier 1 has 13 families (Supplementary Table 1), from thermal buckling of a stepped column to an indeterminate truss with a heated member. Each family draws numerical data from seeded ranges, builds a statement, and computes a key from the standard theory of its problem [34–45]. For the cantilever, Blevins supplies the component vibration theory [46]; the benchmark derives the combined tip-mass and intermediate-spring configuration from the beam equation and its boundary and point conditions. The beam on an elastic foundation uses a Winkler bed [47]. Tier 2 has five families. Generators draw from the same kind of seeded ranges, then accept an instance only when gates R1–R7 pass. The governing models below are what the reference solvers implement. Reference A and reference B are independent numerical implementations of the same governing model. Table 1 maps each family to those methods and its planted flaw. Supplementary Note 4 gives the complete assumptions, governing equations and flaw ranges. T02A settled two-span beam combines elastic response, plastic collapse and cyclic shakedown. Force/kinematic and stiffness/static methods provide independent keys [48–51]. T04A beam rests on a bed that pushes but never pulls. Independent contact solvers locate its lifted ends; the flaw contradicts an end’s uplift with a downward gauge reading [47, 52, 53]. T08 A clamped cosine strip can lose stability through a symmetric force maximum or an asymmetric bifurcation. Mode expansion and finite elements check both; the flawed strip has insufficient rise to snap [54–57]. T11An annular disk combines Tresca yielding with elastic unloading. The free-bore yield face must obey associated flow; the solid-disk shortcut is inappropriate [58–60]. The flaw requests an overspeed beyond burst. T12A shrink fit can open during overspeed and remain loose after unloading. Independent disk/shaft calculations check residual pressure and service lift-off [61]; the flaw claims that a loosened fit stays tight. Tier-1 premise defects are one of five types, fixed in the generator: missing data, contradictory data, physically impossible data, a mechanism, or a false premise. Critique pairs implant one wrong step in an otherwise correct worked solution; the first step at which the corrupted solution leaves the clean solution is part of the key. Tier-2 flaws carry the certificates named above. The unwarned prompt does not mention flaws. The warned prompt (described under Model calls and answer formats) was used on the original nine Tier-2 pairs and on the Tier-1 premise pairs. Hard-10 selection uses shortcut failure, conditioning and the number of crossed regimes to select the hardest tenth of each family without a model score. The frozen set has 52 solve items, 21 critique pairs and 20 premise pairs. Tier-2 families were screened on Sonnet 5 development items, so all comparisons remain conditional on that model-informed choice. Evaluation items use fresh, non-overlapping seed streams; the solve items and consistent twins are separate draws. The extension from 9 to 30 pairs was frozen before any model saw the 21 added pairs. The warned arm was not repeated on those added pairs. Selection chronology and thresholds are retained in Supplementary Note 3. Acceptance gates An instance is kept only when the applicable gates pass. R1–R4 apply to both tiers; R5–R7 apply only to Tier 2. R1 the two references agree to 10 −6 relative on every numeric answer, and categorical answers agree exactly. R2 the decision margin, the relative distance of the governing ratio from its switching value, is at least 5%, so the instance is not perched on a class boundary. 11 Noticing an impossible problem yet reporting it solvedA PREPRINT R3 the sampled class is the class requested, which keeps the categorical labels balanced. R4 the worked solution reproduces reference A at 10 −6 . R5every core shortcut misses at least one numeric answer by more than ten times the base grading tolerance (more than 10%). R6every alternative reading of the statement agrees within one third of the grading tolerance, so an ambiguity in wording cannot move the key. R7 κ q ≤ 50 for every answer, with κ q from equation (3). Tier 1 uses a flat 1% relative tolerance with the same absolute floors, and gates R1–R4. Third-party verification OpenSeesPy 3.8.0.0, the Python interface to OpenSees, provides third-party checks for T02, T04 and T08 [62,63]. It tests beam yielding, collapse and cyclic shakedown; contact geometry and response; and snap-through mode and critical force. T08 uses corotational beams [64] with a small antisymmetric imperfection. Agreement tolerances are 0.1–0.2% for T02/T04 and 3% for T08, whose key and finite-element model use different kinematics. Every flaw is checked positively: premature collapse, end uplift or a monotonic force path. T11 and T12 are outside this suite. Supplementary Note 3 specifies discretisation, quantities, cyclic brackets and limits. Model calls and answer formats Table 2 identifies the models, harnesses, time limits and repeat coverage. The first scored reply counts. Timeouts, exhausted output budgets and replies using a tool are failures. Provider isolation differs by harness, so no cross-provider result isolates the model alone. Isolation probes checked Claude and Codex; the probe records will be released with the code after this preprint is posted. Supplementary Note 3 and Tables 2, 7 and 9 retain the settings, probes and deviations. Every call uses a fixed mechanical-engineer instruction and requests one final structured answer. Tier 2 also instructs the model to work without tools, code execution or internet access. The exact prompt components are reproduced at the end of the Supplementary Information. The unwarned answer schema offers the status valuessolvedandcannot_solve. The warned arm inserts, before the schema, an instruction to check for missing, contradictory or impossible data, for a mechanism, and for a question about a state that does not exist, and not to solve a flawed statement; its schema offerssolvedandflawedwith a defect type. Instruction-only adds the check instruction while keeping the unwarned status options. Schema-only changes the status field without adding that instruction. A schema-only comparison therefore tests the whole reporting field, including the requested defect type and reason. Run groups and tests The first collection’s 30 pairs comprise nine original and 21 added pairs. The map separates the comparison scopes. Both second-collection rows use the same fresh calls. Collection / testAnswer formatPair setScope and adjustment First: modelsUnwarned21 addedConfirmatory: six model contrasts; Holm over six. First: other comparisons Unwarned; warned30 pooled; warned on 9 original Solve and pooled comparisons, twin controls and warning tests are exploratory. Component checksInstruction-only; schema-only 9 original; five Anthropic models Four paired contrasts; exploratory. Second: versionsFresh unwarned; fresh schema-only 21 added planned; 10 available for Sonnet 5 Confirmatory: four version contrasts; Holm over four. Second: reportingSame fresh arms 30 planned; 16 matched for Sonnet 5 Confirmatory: four within-model arm contrasts; Holm over four. Selected models repeat the first collection’s unwarned arm. Component arms were fixed before their calls but collected later than the first collection, so they cannot isolate the components’ causal effects. In the second collection, each model’s arms start together and follow the same item order over two time blocks. Reporting tests use the same available pairs within each model; missing calls are excluded and new replies are not coded. 12 Noticing an impossible problem yet reporting it solvedA PREPRINT The first plan was written after the solve items and original nine pairs were scored, before the added pairs existed. The second collection was fixed before its calls. Supplementary Note 3 records the plans, call interruptions, unavailable repeats and time-block limitation; Supplementary Tables 4, 15 and 16 give the contrasts. Answer parsing and grading The parser takes the last fenced JSON block and also accepts bare JSON (26 replies, all from gpt-oss-120b); the parse census of every run is in Supplementary Table 5. A numeric reply ˆy q is correct for key y q when |ˆy q − y q |≤ max r q |y q |, a q , r q = max 0.01, 10 −3 κ q ,(2) where a q is a family-specific absolute floor, used only for quantities that can sit near zero, and κ q = max j ∂ lny q ∂ lnx j (3) is estimated by±0.1% finite differences through reference A. Categorical answers must match exactly. In a seeded random sample of 50 replies, stratified by harness, every graded value matched the reply’s own final answer, and no reply flagged by an automatic screen was a parser error (Supplementary Table 5). Under a lenient parser, gpt-oss-120b rejections could rise by at most 3; no conclusion changes. The tolerance sweep replaces(r q ,a q ) bys(r q ,a q )fors ∈ 0.25, 0.5, 1, 2, 5, 10; no model pair reverses order betweens = 1ands = 5(Supplementary Fig. 1). A wrong solve answer is labelled a trap when it matches a named shortcut on at least 70% of the quantities where that shortcut differs from the key. It is a near-miss when every quantity lies within 5% of the key. Otherwise it is “other”, or a named non-answer (time limit, budget, harness timeout). Trap labels describe the answer. They are not used in the rejection score. Rejection scoring A reply counts as a rejection when its status iscannot_solve,flawed,ill_posedorunsolvable(the last two are accepted but never offered), or when it withholds a required answer: the answer object or a required value is missing or null, a numeric value is not a number, or a categorical value is not one of the allowed choices. A call that ends without a reply (a time limit, an exhausted budget or a harness timeout) is a failure, not a rejection. Scoring the status alone, without the withheld-answer rule, gives the same result on all 840 final unwarned Tier-2 twin calls of the panel and on all 456 final calls of the second collection. One wording was used per arm. The prompt components are in the Supplementary Information. Recognition, repairs and twin controls Twin controls, rejection reasons and repairs were defined after reading the replies and are exploratory. The analysis code and every assignment will be released with the benchmark after this preprint is posted. Co-solved pairs are those whose consistent twin both models solve; version contrasts are repeated on that subset. Word lists distinguish a reason naming the planted flaw from one citing only computational difficulty. Recognition counts a rejection naming the planted flaw or a miss coded “noted”; claiming a different defect does not qualify. Supplementary Note 3 records the word-list procedure. The T04 consistent twin states a dial-gauge reading that equals the graded uplift in five of its six pairs and whose sign shows which end lifts, so the statement itself gives one graded answer and part of another. We therefore repeat the co-solved contrasts without T04; Supplementary Tables 10 and 14 show that sensitivity and the answer-level comparison. A noted reply is a repair when its numbers match a physically attainable, corrected problem within equation(2). Supplementary Note 3 defines the attainable states, continuations and undecided replies and gives the family-specific numerical checks. Two blinded AI coders label misses, rejection reasons and repairs, with an AI adjudicator resolving disagreements. Model, run and item identifiers are hidden. The miss taxonomy was revised after disagreements were visible; the semantic labels have no human expert validation and remain exploratory. Supplementary Note 3 retains the codebooks, blinding, agreement statistics and comparison with the separate numerical rules; Supplementary Tables 6 and 12 report the labels and repaired answers. 13 Noticing an impossible problem yet reporting it solvedA PREPRINT Statistics Intervals are Wilson 95% intervals [65]. Exact McNemar tests compare two models on the same items: withbitems only the first gets right andconly the second,Pis the two-sided binomial probability of a split at least as uneven asb–c among b + c discordant items. Cochran’s Q is the omnibus test across models on the same items. A random-intercept logistic model [66], Pr(y im = 1) = σ(β m + u i ), u i ∼N (0,σ 2 ), β m ∼N (0, 2.5 2 ),(4) withσ(·)the logistic function, was fitted by maximum a posteriori estimation, the marginal likelihood taken by non- adaptive 30-point Gauss–Hermite quadrature. It is a description of item difficulty against model level, not a substitute for the paired tests. Consistency ispass 3 , the items solved or the flaws rejected in all three runs, and pass@k, the items solved at least once [32,67]. Coder agreement is Cohen’sκ = (p o − p e )/(1 − p e ). Analysis code uses NumPy, SciPy and Matplotlib [68–70]. All-pairs comparisons and twin controls are exploratory. Two further exploratory checks were defined after the replies were read: Spearman and Kendall rank correlations between Tier-2 solve accuracy and unwarned rejection across the 14 models, with permutationPvalues from 200,000 seeded permutations, and exact McNemar tests of consistent-twin solving for the two version contrasts. The design-level check, whether the newer model rejects more flawed twins in each of the five flaw designs (one per family), uses an exact two-sided sign test (Supplementary Table 11). The collection-specific confirmatory scopes and adjustment sets are given in the run map above; the full analysis plan remains in Supplementary Note 3. Use of AI tools Language models appear in this study in three roles. (1) The models in Table 2 are the objects of measurement. (2) GPT-6-Sol and Claude Sonnet 5.5 coded missed flaws, rejection reasons and noted replies, and GPT-6-Astra adjudicated their disagreements; these are measurement procedures, described above and reported with their agreement statistics. (3) The authors conceived the study, made every protocol and design decision, approved each model run, wrote the manuscript and are responsible for all conclusions. Working under their direction, an AI coding assistant (Claude Code, Anthropic, running Claude Opus 5.5) assisted in implementing the item generators, the evaluation harness and the analysis and plotting scripts, which the authors reviewed and edited before adoption; and edited manuscript text, which the authors reviewed and edited. Its unblinded labelling of 42 misses is not used. Every key is checked by two independent reference solvers, and for three families by OpenSees; every figure and table is regenerated from the stored model replies by the analysis code. No AI system is an author. The authors take full responsibility for the content. Reproducibility Evaluation sets, keys, prompts, parser and grader are frozen, and each is identified by a checksum. The Tier-2 evaluation seeds do not overlap the screening seeds. The stored record of each call holds the prompt hash, the model identifier, the harness version where the CLI reports it, and the wall time. The analysis environment is Python 3.12.13, NumPy 2.4.3, SciPy 1.17.1 and OpenSeesPy 3.8.0.0. Data availability Benchmark items, answer keys, flaw-twin pairs, build manifests and every model reply, with grades and both coders’ labels, will be released athttps://github.com/nbbllxx0/Paired-flawed-engineering-problemsunder a Creative Commons Attribution 4.0 International (C BY 4.0) licence after this preprint is posted. An archived copy with a digital object identifier will be issued on journal publication. Each released file will carry a canary string. A private hold-out regenerated from new seeds is retained to detect contamination and is not in the public item set. Source data for the graphs in Figs. 1, 4, 5 and 6 accompany this preprint as ancillary files and will also be released in the same repository. Code availability Generators, reference solvers, acceptance gates, harness, grader and the scripts that regenerate the figures and tables will be released in the same repository under the permissive MIT software licence. 14 Noticing an impossible problem yet reporting it solvedA PREPRINT Acknowledgements Subscription access to the three command-line harnesses provided the model calls. This work is supported by Santa Clara University. Author contributions S.Y. designed the study, developed the methods, ran the experiments, directed the implementation and analysis, and wrote and edited the manuscript with AI assistance (Methods, Use of AI tools). J.W. conceptualized the ideas, provided funding, and reviewed and edited the manuscript. Competing interests The authors declare no competing interests. References [1] Zhou, X. et al. EngiBench: A benchmark for evaluating large language models on engineering problem solving. In Findings of the Association for Computational Linguistics: ACL 2026, 36308–36334 (Association for Computational Linguistics, San Diego, California, United States, 2026). [2] Wan, Q. et al. SoM-1K: A thousand-problem benchmark dataset for strength of materials. Preprint at https://arxiv.org/abs/2509.21079 (2025). [3] Zhang, Y. et al. ABench-Physics: Benchmarking physical reasoning in LLMs via high-difficulty and dynamic physics problems. Preprint at https://arxiv.org/abs/2507.04766 (2025). [4] Jian, D.-S. et al. Mastering olympiad-level physics with artificial intelligence. Preprint at https://arxiv.org/abs/2511.10515 (2025). [5] Gerstmayr, J., Weyrer, S., Möltner, T., Manzl, P. & Pieber, M. Large language models and their awareness of mechanics and spatial geometry. Preprint at https://arxiv.org/abs/2608.14615 (2026). [6] Mudur, N. et al. FEABench: Evaluating language models on multiphysics reasoning ability. Preprint at https://arxiv.org/abs/2504.06260 (2025). [7] Mohammadzadeh, S., Hamdi, E., Shor, J. & Lejeune, E. FEM-Bench: A structured scientific reasoning benchmark for evaluating code-generating LLMs. Preprint at https://arxiv.org/abs/2512.20732 (2025). [8] Ezemba, J., Pohl, J., Tucker, C. & McComb, C. A large-scale benchmark to assess vision-language model question answering capabilities in engineering simulations. Communications Engineering (2026). https://doi.org/10.1038/s44172-026-00786-2. [9] Singh, H., Dubey, A. & Herrera, J. A. A. CADEngBench: It looks like CAD, but does it work? Evaluating parametric design, assembly reasoning, and physics simulation. Preprint at https://arxiv.org/abs/2608.09296 (2026). [10] Guo, X. et al. Toward engineering AGI: Benchmarking the engineering design capabilities of LLMs. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track, 78582–78642 (2025). [11]Hang, Z. et al. PDEAgent-Bench: A multi-metric, multi-library benchmark for PDE solver generation. Preprint at https://arxiv.org/abs/2605.09636 (2026). [12]Center for AI Safety, Scale AI & HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature 649, 1139–1146 (2026). [13] Zhai, W. et al. HLE-Verified: A systematic verification and structured revision of Humanity’s Last Exam. Preprint at https://arxiv.org/abs/2602.13964 (2026). [14] Glazer, E. et al. FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI. Preprint at https://arxiv.org/abs/2411.04872 (2024). [15] Li, J., Li, G., Chang, Y. & Wu, Y. Don’t take the premise for granted: Evaluating the premise critique ability of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, 836–869 (Association for Computational Linguistics, Suzhou, China, 2025). 15 Noticing an impossible problem yet reporting it solvedA PREPRINT [16]Griot, M., Hemptinne, C., Vanderdonckt, J. & Yuksel, D. Large language models lack essential metacognition for reliable medical reasoning. Nature Communications 16, 642 (2025). [17]Sun, Y. et al. Benchmarking hallucination in large language models based on unanswerable math word problem. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2178–2188 (ELRA and ICCL, 2024). [18] Zhao, J. et al. Exploring the compositional deficiency of large language models in mathematical reasoning through trap problems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 16361–16376 (Association for Computational Linguistics, Miami, Florida, USA, 2024). [19] Petrov, I., Dekoninck, J. & Vechev, M. BrokenMath: A benchmark for sycophancy in theorem proving with LLMs. In International Conference on Machine Learning (ICML 2026). Preprint at https://arxiv.org/abs/2510.04721 (2026). [20] Zheng, C. et al. ProcessBench: Identifying process errors in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1009–1024 (Association for Computational Linguistics, Vienna, Austria, 2025). [21] Recchia, G., Mangat, C. S., Li, I. & Krishnakumar, G. FindTheFlaws: Annotated errors for detecting flawed reasoning and scalable oversight research. Proceedings of the AAAI Conference on Artificial Intelligence 40, 37867–37876 (2026). [22]Huang, J. et al. Large language models cannot self-correct reasoning yet. In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024) (2024). [23] Tsui, K. Self-correction bench: Uncovering and addressing the self-correction blind spot in large language models. In Third Conference on Language Modeling (COLM 2026) (2026). URL https://openreview.net/forum?id=rLscoLa2TM. [24]Tyen, G., Mansoor, H., Carbune, V., Chen, P. & Mak, T. LLMs cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics: ACL 2024, 13894–13908 (Association for Computational Linguistics, Bangkok, Thailand, 2024). [25]Kalai, A. T., Nachum, O., Vempala, S. S. & Zhang, E. Evaluating large language models for accuracy incentivizes hallucinations. Nature 653, 1047–1051 (2026). [26] Chen, S. et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digital Medicine 8, 605 (2025). [27]Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46 (1960). [28] Cochran, W. G. The comparison of percentages in matched samples. Biometrika 37, 256–266 (1950). [29] McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 153–157 (1947). [30] Holm, S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65–70 (1979). [31]Zhu, M. et al. Probing the critical point (CritPt) of AI reasoning: a frontier physics research benchmark. Preprint at https://arxiv.org/abs/2509.26574 (2025). [32] Yao, S., Shinn, N., Razavi, P. & Narasimhan, K. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations (ICLR 2025) (2025). URL https://openreview.net/forum?id=roNSXZpUDN. [33] Srivastava, A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (2023). URL https://openreview.net/forum?id=uyTL5Bvosj. [34] Timoshenko, S. P. & Gere, J. M. Theory of Elastic Stability (McGraw-Hill, New York, 1961), 2nd edn. [35] Timoshenko, S. P. & Goodier, J. N. Theory of Elasticity (McGraw-Hill, New York, 1970), 3rd edn. [36] Hibbeler, R. C. Structural Analysis (Pearson, Boston, MA, 2018), 10th edn. [37] Den Hartog, J. P. Mechanical Vibrations (McGraw-Hill, New York, 1956), 4th edn. [38] Budynas, R. G. & Nisbett, J. K. Shigley’s Mechanical Engineering Design (McGraw-Hill, New York, 2020), 11th edn. 16 Noticing an impossible problem yet reporting it solvedA PREPRINT [39] Tada, H., Paris, P. C. & Irwin, G. R. The Stress Analysis of Cracks Handbook (ASME Press, New York, 2000), 3rd edn. [40] Bisshopp, K. E. & Drucker, D. C. Large deflection of cantilever beams. Quarterly of Applied Mathematics 3, 272–275 (1945). [41] Frisch-Fay, R. Flexible Bars (Butterworths, London, 1962). [42] Johnson, K. L. Contact Mechanics (Cambridge University Press, Cambridge, 1985). [43]M’Ewen, E. Stresses in elastic cylinders in contact along a generatrix (including the effect of tangential friction). The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 40, 454–459 (1949). [44] Jones, R. M. Mechanics of Composite Materials (Taylor & Francis, Philadelphia, PA, 1999), 2nd edn. [45] Daniel, I. M. & Ishai, O. Engineering Mechanics of Composite Materials (Oxford University Press, New York, 2006), 2nd edn. [46] Blevins, R. D. Formulas for Natural Frequency and Mode Shape (Van Nostrand Reinhold, New York, 1979). [47] Hetényi, M. Beams on Elastic Foundation: Theory with Applications in the Fields of Civil and Mechanical Engineering (University of Michigan Press, Ann Arbor, MI, 1946). [48] Melan, E. Der Spannungszustand eines Mises-Henckyschen Kontinuums bei veränderlicher Belastung. Sitzungsberichte der Akademie der Wissenschaften Wien, Abteilung IIa 147, 73–87 (1938). [49] Neal, B. G. The Plastic Methods of Structural Analysis (Chapman and Hall, London, 1977), 3rd (SI) edn. [50] Horne, M. R. Plastic Theory of Structures (Pergamon Press, Oxford, 1979), 2nd edn. [51] Koiter, W. T. General theorems for elastic-plastic solids. In Sneddon, I. N. & Hill, R. (eds.) Progress in Solid Mechanics, vol. 1, 165–221 (North-Holland, Amsterdam, 1960). [52] Weitsman, Y. On foundations that react in compression only. Journal of Applied Mechanics 37, 1019–1030 (1970). [53] Tsai, N.-C. & Westmann, R. A. Beam on tensionless foundation. Journal of the Engineering Mechanics Division, ASCE 93, 1–12 (1967). [54] Fung, Y. C. & Kaplan, A. Buckling of low arches or curved beams of small curvature. Tech. Rep. NACA TN 2840, National Advisory Committee for Aeronautics (1952). [55] Qiu, J., Lang, J. H. & Slocum, A. H. A curved-beam bistable mechanism. Journal of Microelectromechanical Systems 13, 137–146 (2004). [56] Pandey, A., Moulton, D. E., Vella, D. & Holmes, D. P. Dynamics of snapping beams and jumping poppers. EPL (Europhysics Letters) 105, 24001 (2014). [57]Vangbo, M. An analytical analysis of a compressed bistable buckled beam. Sensors and Actuators A: Physical 69, 212–216 (1998). [58] Hill, R. The Mathematical Theory of Plasticity (Oxford University Press (Clarendon Press), Oxford, 1950). [59] Chakrabarty, J. Theory of Plasticity (Elsevier Butterworth-Heinemann, Oxford, 2006), 3rd edn. [60] Gamer, U. Tresca’s yield condition and the rotating disk. Journal of Applied Mechanics 50, 676–678 (1983). [61] Timoshenko, S. P. Strength of Materials, Part I: Advanced Theory and Problems (D. Van Nostrand, Princeton, NJ, 1956), 3rd edn. [62] McKenna, F. OpenSees: A framework for earthquake engineering simulation. Computing in Science & Engineering 13, 58–66 (2011). [63] Zhu, M., McKenna, F. & Scott, M. H. OpenSeesPy: Python library for the OpenSees finite element framework. SoftwareX 7, 6–11 (2018). [64] Crisfield, M. A. Non-linear Finite Element Analysis of Solids and Structures, Volume 1: Essentials (Wiley, Chichester, 1991). [65] Wilson, E. B. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22, 209–212 (1927). [66] Breslow, N. E. & Clayton, D. G. Approximate inference in generalized linear mixed models. Journal of the American Statistical Association 88, 9–25 (1993). [67] Chen, M. et al. Evaluating large language models trained on code. Preprint at https://arxiv.org/abs/2107.03374 (2021). 17 Noticing an impossible problem yet reporting it solvedA PREPRINT [68] Harris, C. R. et al. Array programming with NumPy. Nature 585, 357–362 (2020). [69] Virtanen, P. et al. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods 17, 261–272 (2020). [70] Hunter, J. D. Matplotlib: A 2D graphics environment. Computing in Science & Engineering 9, 90–95 (2007). 18 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Information This appendix accompanies the main text. Family codes identify the two-span beam (T02), no-tension bed (T04), clamped cosine strip (T08), turbine disk (T11) and shrink fit (T12). Supplementary Notes 1 and 2 give further results, controls, sensitivities and repair details; Note 3 records the selection, collection and coding protocols, and Note 4 gives the governing models and flaw certificates. Supplementary Figs. 1–6 and Tables 1–16 provide the family and harness definitions, rates and contrasts, parser and coding summaries, protocol deviations, graded quantities, model identifiers, twin controls, warning factorial and second-collection tests. The final section gives the prompt components. Supplementary Note 1: Further results This note holds result details that the main text summarises briefly. Wrong solve answers. Wrong answers in the middle of the panel are mostly recognisable (main-text Fig. 4b). Sonnet 5’s seven shortcut matches are four single-mode snap-through calculations, one Tresca face that swaps the hoop and radial stresses and two plane-strain readings of the shrink fit; Opus 4.8, Gemini 3.8 Flash and the two Sol models contribute one each. Haiku 4.5 and gpt-oss-120b mostly fail in no named way: 24 and 22 of their 25 answers fall in the residual class. Family-level counts are in Supplementary Fig. 3. Three further checks on the confirmatory contrasts.Three further checks bound the result. First, the 30 pairs are six draws each of five flaw designs, one per family. For both confirmatory contrasts the newer model rejects more twins in every one of the five designs (Supplementary Table 11). With five designs, no exact two-sided design-level test can fall belowP = 0.0625, so the claim is about these designs. Second, Sonnet 5 returned no usable answer on six flawed twins; excluding the pairs either model left unanswered, Sonnet 5.5 leads 19–0 over 24 pairs. Third, the older model in each version pair ran once in the first collection; the second collection (main text) repeated both contrasts. Misses of older and smaller models. Older and smaller models fail earlier in the same tree. Sonnet 5 runs the revealing check and concludes wrongly on 14 of 30 twins. Haiku 4.5 is unexamined on 21 of 30 and gpt-oss-120b on 16. Opus 4.8 splits across the tree: 7 rejections, 8 noted, 6 misjudged and 9 unexamined. Unlike the newer models, 5 of its 8 noted replies return numbers for the state that does not exist, such as a residual moment for a load past collapse, and 3 repair the input. Among the seven models that solve at least 22 of 25 Tier-2 items, none of the 13 Tier-2 misses is unexamined, while 25 of their 43 Tier-1 premise misses are. The flaw types differ between the tiers, so this is a description, not a test. Supplementary Note 2: Controls, sensitivities and repair details The main text gives each result with the numbers that carry it. This note gives the full account of each, with every count, test and caveat, in the order of the main text. First-run rejection and planned contrasts.Supplementary Tables 3 and 4 give the full rejection counts, false alarms and all planned contrasts. The main Results state the two confirmed version comparisons; the controls and additional estimates below delimit them. Combining the original nine pairs with the 21 added pairs gives the same direction: Opus 5 leads Opus 4.8 by 17–1 and Sonnet 5.5 leads Sonnet 5 by 25–0 (P = 7.2× 10 −4 and3.6× 10 −7 ; Supplementary Table 4). These pooled comparisons provide context; the confirmatory tests use only the added pairs. Pairs whose consistent twin both models solved. The twin allows a stricter test: rejection on the pairs whose consistent version the model solved. A model that cannot solve the consistent problem may fail the flawed one for the same reason, and the families were screened on Sonnet 5. Opus 5 solves 29 of the 30 consistent twins and rejects 22 of the matching flawed twins. Opus 4.8 solves 25 and rejects 5 of those; Sonnet 5 solves 16 and rejects 1; Sonnet 5.5 solves 28 and rejects 26 (Supplementary Table 10). Restricted to pairs that both models solved, the splits on the 21 added pairs are 13–1 for Opus 5 over Opus 4.8 (17 pairs; Holm-adjustedP = 0.011) and 9–0 for Sonnet 5.5 over Sonnet 5 (10 pairs;P = 0.020). The rejection gap is therefore not the solve gap measured a second time. This restriction is exploratory and selects pairs on an outcome. It is also weaker for the no-tension bed (T04), whose consistent twin states one graded answer (Methods; Supplementary Table 10); without T04 the splits are 12–0 over 15 pairs and 9–0 over 9 pairs (Supplementary Table 14). 19 Noticing an impossible problem yet reporting it solvedA PREPRINT Mechanical checks of the twelve noted misses.All 12 Tier-2 misses of Fable 5.1 (3), Sonnet 5.5 (2) and Opus 5 (7) are noted. They concentrate where a stated number contradicts the model’s own calculation: 8 are the T04 gauge and 3 the T02 load past collapse. On the two flaw designs of the added pairs, a rise too small to snap (T08) and a fit that the spin test loosens (T12), these three models reject 35 of 36 twins. Two blinded coders classed every noted Tier-2 reply as a repair or a continuation (25 of 26 agreed,κ = 0.92; Supplementary Table 12). In 11 of the 12, by that coding and by comparing the returned numbers with the keys, the answer belongs to an attainable reading of the statement. The eight T04 replies return the lift-off solution of the stated loads and bed, which is the key of the consistent twin, with the contradictory gauge set aside. Two of Opus 5’s T02 replies give the residual moment after unloading from the collapse state, the largest load the beam can carry, and say so in the JavaScript Object Notation (JSON) reason. Fable 5.1’s T12 reply returns zero residual pressure for a fit it calls loose. The remaining reply, Opus 5 on one T02 twin, unloads the collapse moments by the unattainable load factor; that number belongs to no state of the beam. Six of the 11 repairs are stated in the JSON reason field; five appear only in the prose. Wall time. Median wall time per Tier-2 solve item runs from 40 s (gpt-oss-120b) to 616 s (Sonnet 5) and does not order accuracy: Sonnet 5 solves 13 of 25, Haiku 4.5 at 433 s none and GPT-6-Astra at 223 s all 25 (main-text Fig. 6b). Wall time on subscription services includes queueing, and token counts differ in definition between harnesses (main-text Fig. 6c), so no cost ranking is claimed. The second collection. The confirmatory contrasts rest on one run of each older model, so a second collection, fixed before its first call and started about a day after the first collection, repeated them (Methods; main-text Table 3; Supplementary Table 16). Opus 5, Opus 4.8, Sonnet 5.5 and Sonnet 5 each received a fresh unwarned run and a schema-only run (used in the next section) on all 30 pairs, with the two arms of each model started together. A quota limit stopped 23 of Sonnet 5’s unwarned calls, so its fresh unwarned run covers 16 flawed twins, six of which ended without a reply, and the planned second unwarned runs of Opus 4.8 and Sonnet 5 were not made. The Sonnet gap held. On the 10 added pairs with a final Sonnet 5 call, three of them non-answers, Sonnet 5.5 rejected 9 flawed twins that Sonnet 5 did not, and the reverse never happened (Holm-adjustedP = 0.016); counting only calls with a reply weakens this to 6–0 over 7 added pairs (Supplementary Table 16). The Opus gap did not hold. Opus 5 rejected 12 of the 30 flawed twins, against 23, 26 and 29 in its three earlier runs, and Opus 4.8 rejected 9; on the 21 added pairs the split was 2–0 (Holm-adjustedP = 1). Prompts, reported model identifier and command-line binary were unchanged, and Opus 5 still solved 28 of 30 consistent twins. The new replies were not coded, so whether its new misses state the flaw is not known; a word-list screen cannot decide it, because for T02 and T04 the same words occur in its replies to the consistent twins. These data cannot tell a change in the service from run-to-run variation. Opus 5’s drop from its first run, 11 flawed twins, is about two thirds of its first-run gap over Opus 4.8, 16 twins. The warning and its two parts.How a reply may report its status changed rejection sharply. In the first collection, a warned arm on the original nine pairs changed two things at once: it told the model to check the statement and not to solve a flawed one, and it offered a “flawed” status with a defect type in place of “cannot solve” (Supplementary Information, Prompts). On the same nine twins (main-text Fig. 5c; Supplementary Table 13), the warning raised rejection in every model that had missed any of them: Opus 5 from 3 to 9, Opus 4.8 from 1 to 9 and Sonnet 5 from 0 to 8; five models already rejected all nine. It also added false alarms in five models, up to three of nine. Counting a pair as a success only when the consistent twin is solved and the flawed twin rejected, the warning helps Opus 5 (2 to 9) and Opus 4.8 (1 to 8) and costs GPT-5.5 (7 to 4) and Gemini 3.1 Pro (7 to 6). PCBench and the trap study report the same dependence on an explicit instruction for flawed mathematical premises. The later 21 twins were run unwarned only, so the warning result rests on nine twins of three designs. Two further arms, instruction-only and schema-only, gave each part alone to five Anthropic models, about a day and a half after the unwarned and warned arms (Supplementary Table 15; Methods). Either part alone gave 43 rejections of 45 flawed twins, against 44 with both and 18 with neither in the earlier runs, with no false alarm. Because the arms ran at different times and unwarned runs vary (Opus 5 rejected 3, 6 and 9 of the same nine twins in three runs), these counts alone do not separate the causal contributions of the two parts. The status field. The schema-only arm of the second collection, run alongside the fresh unwarned arm, isolates the status field. Within each model, on the same pairs, it rejected more flawed twins than the unwarned arm: 18–0 for Opus 5, 21–0 for Opus 4.8 and 15–0 for Sonnet 5 (Holm-adjustedP ≤ 1.2× 10 −4 ), and 3–0 for Sonnet 5.5, which already rejected 27 unwarned; the three gains hold within each time window (Supplementary Table 16). On the 21 added pairs both Opus models rejected all 21 flawed twins, Sonnet 5.5 21 and Sonnet 5 18, so neither version gap passed the adjusted test (0–0 and 3–0; Holm-adjustedP = 1and 0.75), although Sonnet 5 still rejected three fewer. The format also had costs. Consistent twins solved fell for Opus 4.8 (24 to 19), Sonnet 5.5 (29 to 25) and Sonnet 5 (11 to 7 of 16), and false alarms rose from none to two for Opus 4.8 and Sonnet 5. Counting a pair as a success only when the consistent twin is solved and the flawed twin rejected, the schema-only arm gained for Opus 5 (12 to 30), Opus 4.8 20 Noticing an impossible problem yet reporting it solvedA PREPRINT (8 to 19) and Sonnet 5 (0 to 7) and lost one pair for Sonnet 5.5 (26 to 25). The schema lists defect types and asks for a reason, so these are effects of the whole status field, not of a single word. Supplementary Note 3: Selection, collection and coding protocols Selection historyHard-10 selection uses a model-free score: shortcut failure, conditioning, and the number of regimes the item actually crosses. The top of each family, about 10% of the 1280-item pool, was frozen before any model call. That Hard-10 set has 52 solve items, 21 critique pairs and 20 premise pairs (134 items, each pair counted as two). Tier-2 families were screened on development items with Sonnet 5. A family was kept when at least two of three development items failed at the 1% tolerance and at least one failed at 5%. T02, T04, T08 and T11 were kept. T03 was dropped because Sonnet 5 solved it. T12 was developed in later calibration runs. Evaluation items are a fresh seed stream with no parameter overlap against the screening draws. New draws do not undo the model-informed choice of families: every result in this paper holds for these five families. The extension from 9 flaw pairs to 30 was designed after the original nine pairs had shown that nine could not support a pairwise claim, in order to add flaw types (T08 and T12) and to balance the original three at six pairs each. It was frozen, on its own seed stream, before any model saw those items. The Tier-2 evaluation draw was frozen with 256-bit Secure Hash Algorithm (SHA-256) manifests before any model saw it; its solve items and consistent twins are separate draws. The warned arm was not repeated on the 21 added pairs. Run failures and excluded models Reasoning effort was high where the command line interface (CLI) exposes the setting, and no sampling temperature or seed was set. Time limits were 1800 s on Tier 1 and 2700 s on Tier 2, where a call stopped at the limit is a failure. One Haiku 4.5 Tier-1 call exceeded the output cap twice (173 of 174 calls final). A stalled Codex connection is retried once and then graded as a harness timeout (seven GPT-5.5 calls: three solve items and four twin calls; one flawed-twin reply stalled after a rejecting status and counts as a non-rejection); an exhausted output budget, including an Antigravity call with no answer after at least 60,000 thinking tokens, is final. These non-answer rules were set or corrected during the runs (Supplementary Table 7). A call that failed in transport was retried; the first scored reply per item counts. Opus 4.6 produced no answer within 2700 s on each of 3 calls and Sonnet 4.6 recorded 23 of 61 calls, 16 at the limit; both were stopped and enter no contrast. Single-part warning armsTwo single-part arms separate the two parts of the warning; both were fixed before their first call (Supplementary Table 7). The instruction-only arm adds the check instruction to the unwarned prompt and keeps the unwarned status options. The schema-only arm uses the warned status field and defect type without the instruction. Both ran once on the nine original pairs for Fable 5.1, Sonnet 5.5, Opus 5, Sonnet 5 and Opus 4.8, with the Tier-2 settings, on 30 September, about a day and a half after the first collection; the unwarned and warned cells are these models’ first runs. The unwarned repeats of Fable 5.1, Sonnet 5.5 and Opus 5 on the same nine twins give the run-to-run reference. Four exploratory contrasts on the flawed twins test the instruction with and without the warned schema and the schema with and without the instruction (exact McNemar, Holm-adjusted over the four). Second collection protocol and interruptions A second collection tested the two confirmatory contrasts again; it was fixed before its first call (Supplementary Table 7). Opus 5, Opus 4.8, Sonnet 5.5 and Sonnet 5 each received a fresh unwarned run and a schema-only run on all 30 pairs, flawed and consistent twin, with the Tier-2 settings. For each model the two arms started together and ran concurrently in the same item order. A subscription usage limit paused all calls for about seven hours, so collection fell into two blocks (Supplementary Table 16); every arm has calls in both blocks except Opus 4.8’s schema-only arm, which finished in the first. At the restart, 137 calls failed with a command-line sign-in error before any model reply; they were rerun. A quota guard, set to keep a share of the weekly plan free, stopped the collection after 456 of the 480 planned calls: 23 of Sonnet 5’s unwarned calls and one of its schema-only calls are missing. The planned second fresh unwarned runs of Opus 4.8 and Sonnet 5 (120 calls) were not made. The prespecified tests are the two contrasts on the 21 added pairs in each arm (exact McNemar, Holm over the four), the same contrasts on all 30 pairs as combined estimates, the schema-only arm against the fresh unwarned arm within each model (Holm over the four models) and the rejection counts of every unwarned run of each model; pairs with a missing call are left out of a contrast. The new replies are scored at the status level and are not coded. Miss-coding protocol and superseded analysesThe export lists every unwarned miss from each model’s primary run, Tier-2 and Tier-1, with a random case identifier, the problem, the planted flaw and the reply. Timeouts, budget failures and harness failures are classed automatically and are not sent to the coders. The codebook’s decision order is: an explicit statement of the contradiction is “noted”; an explicit revealing check with the wrong conclusion is “misjudged”; a reply that uses the flawed datum as given is “adopted”; a reply that never touches it is “silent”. The three-class scheme of the main text merges adopted and silent, which the coders did not apply uniformly, into “unexamined”. It was chosen after the 28 four-class disagreements were visible, so it is a post-hoc primary analysis; on the 232 cases the 21 Noticing an impossible problem yet reporting it solvedA PREPRINT coders agreed withκ = 0.89on three classes andκ = 0.81on four (Supplementary Fig. 5). The coders are artificial intelligence (AI) models. Coder 1 was GPT-6-Sol through the Codex CLI at low effort. Coder 2 was Claude Sonnet 5.5 through the Claude CLI at medium effort. Both used the isolated backends. Agreed codes were kept. Each four-class disagreement was adjudicated by GPT-6-Astra through the isolated Codex CLI at high effort; it is a panel model, neither a coder nor the coding assistant, and was not involved in building or analysing the study (Use of AI tools). The adjudicator saw the codebook, four written rules, the problem, the planted flaw, the full reply and both coders’ labels, and was blind to model, run, item and the study’s aims; this arrangement was fixed before its output existed. An earlier adjudication by Claude Opus 5.5, the model behind the coding assistant, gave the same class on 26 of 28 cases and is superseded. An unblinded labelling of 42 cases by the coding assistant, made before the codebook was fixed, is not used (Supplementary Table 6). Reason and repair coding protocolThe word lists and the repair rules above were written after the replies had been read, so both were checked by a second blinded double coding, fixed before any coder output existed (Supplementary Table 7). The export held every unwarned Tier-2 rejection of a flawed twin (289) and every Tier-2 miss coded noted (26), each with a random case identifier, the problem, the planted flaw and the full reply, and no model, run or item. A rejection reason was coded planted (the reply names the planted flaw in any wording), other defect (it claims a different defect), inability (it claims no defect and cites only the difficulty of computing) or unclear. A noted reply was coded repair (it states a corrected or attainable version and its answers fit that version), continuation (its answers are for the state the flaw makes impossible or use the flawed datum) or unclear. Coders, adjudicator, backends and effort settings were those of the miss coding. The coders agreed on all 289 reason classes (κ = 1.00) and on 25 of the 26 noted replies (κ = 0.92); GPT-6-Astra adjudicated the one disagreement. The coded reason class equals the word-list class for all 289 rejections, so every contrast is unchanged. For noted replies the coded class is used: it equals the rule class for 21 of the 22 replies the rules decide and assigns the four they leave as other. Third-party solver checks OpenSees (Open System for Earthquake Engineering Simulation), through its Python interface OpenSeesPy 3.8.0.0, repeats T02, T04 and T08 (main-text references 62, 63). T02 is a frame with zero- length elastic–perfectly plastic hinges; it checks the first-hinge factor and location, the collapse factor and the residual moment, and brackets the shakedown factor by cyclic simulation at 0.97λ sd and 1.03λ sd , up to 150 cycles. T04 uses compression-only springs at 5 m spacing and checks contact length, peak reaction, largest sagging moment and end uplift. T08 uses corotational beams (main-text references 64) under displacement control, with a10 −3 hantisymmetric imperfection, and checks the instability mode and critical force but not the midpoint displacement, axial force or clamp moment. Agreement tolerances are 0.1–0.2% for T02 and T04, and 3% for T08, where the keys use shallow-arch kinematics and the finite-element model does not. Flaw certificates are positive checks: for a T02 flaw, OpenSees must collapse before the stated load factor; for a T04 flaw, the gauged end must lift; for a T08 flaw, the force must rise monotonically to a stroke of twice the rise. T11 and T12 are not in this suite. Provider harnesses and isolation main-text Table 2 lists the panel, Supplementary Table 9 the model identifiers passed to each CLI and Supplementary Table 2 the harness settings. Anthropic models used the Claude Code CLI with an empty tool list, no project memory and no hooks; the CLI binary on the machine was dated 25 September 2026, and the version string of each call was not recorded. OpenAI GPT-6 and GPT-5 models used the Codex CLI (0.158.0) with tools, web search and user configuration disabled, in a read-only sandbox and an ephemeral session. Gemini and gpt-oss-120b used the Antigravity CLI (1.2.13), which cannot disable tools but logs every tool step. Nine Gemini 3.8 Flash calls logged a tool step; none returned a usable answer, and each is graded as a failure. No reply that used a tool is counted as correct or as a rejection. Isolation probes confirmed the Claude and Codex settings; the probe records will be released with the code after this preprint is posted. Twin-control and reason definitions These analyses use the existing replies and were defined after the replies had been read. They are exploratory; the analysis code and every assignment will be released with the benchmark after this preprint is posted. A pair is co-solved when both models of a contrast answer its consistent twin correctly; the contrasts are repeated on co-solved pairs. The reason of each rejection is matched against two word lists: phrases that name the planted flaw of the family (for example “gauge”, “inconsistent” and “contradict” for T04, or “burst”, “collapse” and “no equilibrium” for T11) and phrases that cite only the difficulty of computing (“without computational tools”, “by hand”, “beyond the scope”). A rejection whose reason matches only the second list is an inability rejection. Recognition of the planted flaw counts a rejection whose reason names the planted flaw, or a miss coded “noted”; a rejection that claims a different defect is not recognition. Deterministic repair rules Each Tier-2 miss coded “noted” is compared with the attainable readings of its flawed statement. For T04, the consistent twin shares every datum except the gauge, so its key is the lift-off solution with the gauge set aside; a reply whose answers all pass main-text equation 2 against that key is a repair. For T02, only 22 Noticing an impossible problem yet reporting it solvedA PREPRINT the residual moment depends on the load factor; a residual within tolerance of unloading from the collapse state, M (λ c )− λ c M E , is a repair, and one within tolerance of the collapse moments unloaded by the unattainable factor, M (λ c )−λ 1 M E , is a continuation. For T12, a reply that returns zero residual fit pressure and zero service lift-off speed treats the fit as loose and is a repair. T08 and T11 have no attainable reading of the requested state, so numbers returned for them are continuations. Anything else is “other”. Analysis plan and additional endpointsThe analysis plan was written on 29 September 2026, after the solve items and the original nine flaw pairs had been scored and before the 21 added pairs existed. Six contrasts were named then: Opus 5.5 versus Opus 5, Opus 5 versus Opus 4.8, Sonnet 5.5 versus Sonnet 5, GPT-6-Sol versus GPT-5.6-Sol, GPT-6-Astra versus GPT-6-Sol, and Opus 5.5 versus GPT-6-Astra. Holm adjustment is applied inside each family of six. The six solve-accuracy contrasts are exploratory, because those items already existed. The six rejection contrasts are confirmatory only on the 21 added pairs; the pooled 30-pair splits are the combined estimate. All-pairs tests, 91 on solve accuracy and 91 on rejection across the 14 models, are exploratory, as are the twin controls above. The design-level statement uses an exact two-sided sign test over the five flaw designs. Four claims of the study protocol are not tested here: accuracy against cost across effort levels, the correction step of the critique arm, self-correction follow-ups and defects invisible to final-answer grading (Supplementary Table 7). Additional scoring and collection details The following protocol details accompany the concise Methods in the main text. Call settings and collection details. Main-text Table 2 lists the panel. Anthropic models used Claude Code with tools, memory and hooks disabled; OpenAI models used Codex with tools, web search and configuration disabled, in ephemeral sessions. Gemini and gpt-oss-120b used Antigravity, whose tools cannot be disabled. No reply using a tool counted as correct or as a rejection. Isolation probes checked Claude and Codex; the probe records will be released with the code after this preprint is posted. Binary identifiers, isolation settings and observed tool calls are specified in Supplementary Note 3 and Supplementary Tables 2 and 9. Reasoning effort was high where configurable; no sampling temperature or seed was set. Time limits were 1800 s on Tier 1 and 2700 s on Tier 2. Timeouts and exhausted output budgets are failures. Transport retries, non-answer rules, incomplete models and deviations are recorded in Supplementary Note 3 and Table 7. The first scored reply counts. The system prompt is fixed: an instruction to act as a mechanical engineer and to finish with one JSON object in a fenced block. Tier 2 appends one sentence: no tools, no code execution, no internet, calculations in the reply itself. The sentence was added after a probe returned a fabricated tool call and no answer. Two single-part arms test the warning components: instruction-only adds the check instruction while keeping the unwarned status options; schema-only changes the status field without the check instruction. Each ran once on the original nine pairs for five Anthropic models, using Tier-2 settings. They were fixed before their first call but collected about a day and a half after the original unwarned and warned runs, so their counts alone cannot separate the components’ causal effects. The four exploratory paired contrasts and collection details are specified in Supplementary Note 3 and Supplementary Table 15. A second collection retested the two version contrasts in fresh unwarned and schema-only arms, fixed before calls (Supplementary Table 7). Each of four models received 30 consistent/flawed pairs; its arms started together and followed the same item order over two time blocks. Quotas left 456 of 480 calls; Sonnet 5 lacks 23 unwarned calls and one schema-only call. Further older-model repeats were not made. Prespecified paired tests compare versions on the 21 added pairs, and arms within models, each Holm-adjusted over four contrasts. Missing calls are excluded; new replies are not coded. Supplementary Note 3 and Table 16 give interruptions, timings and combined estimates. Twin-control and repair scoring details. In T04 the consistent twin states the dial-gauge reading that the flawed twin contradicts. In five of the six T04 pairs the gauged end is the end that rises most, so the stated reading equals the graded uplift to within 0.2% and the sign of the reading shows which end lifts. The T04 consistent twin therefore states one graded answer and part of another. The co-solved contrasts are repeated without T04 (Supplementary Table 14), and Supplementary Table 10 compares the uplift answers on T04 solve items and consistent twins: older models score that answer more often on the twins than on the solve items. A noted reply is a repair when its numbers match a physically attainable, corrected problem within main-text equation 2. Rules test the gauge-free lift-off solution (T04), elastic unloading from collapse (T02), or a loose fit (T12). Numbers for the impossible state are continuations; undecided replies are “other”. T08 and T11 have no attainable version of the requested state. Supplementary Note 3 gives the exact numerical rules. 23 Noticing an impossible problem yet reporting it solvedA PREPRINT Blinded miss-coding details. Two blinded AI coders labelled every usable unwarned miss from the primary runs, with model, run and item identifiers hidden. Non-answers were classed automatically. The three-class scheme merges adopted and silent replies after four-class disagreements were visible, so it is exploratory. On 232 cases, agreement wasκ = 0.89for three classes and 0.81 for four (Supplementary Fig. 5). GPT-6-Sol and Claude Sonnet 5.5 coded; GPT-6-Astra adjudicated blindly under a fixed protocol. No human expert validated the labels. Supplementary Note 3 retains the codebook, effort settings, adjudicator inputs and superseded analyses. Blinded reason and repair checks. A separately fixed blinded double coding checked the post-hoc rejection word lists and repair rules (Supplementary Table 7). It covered all 289 unwarned Tier-2 rejections and 26 noted misses, using the same coders and adjudicator. Agreement was complete for rejection reasons (κ = 1.00) and 25 of 26 for repairs (κ = 0.92). Word-list and coded rejection classes match throughout, leaving contrasts unchanged. Repairs use the coded class. Supplementary Note 3 defines classes, export fields and comparison with the numerical rules. Prespecified analysis scope. The plan was written after the solve items and original nine pairs were scored, but before the 21 added pairs existed. It named six contrasts (Supplementary Table 4), with Holm adjustment within each set. Solve contrasts are exploratory; rejection contrasts are confirmatory only on the added pairs, with 30-pair results as combined estimates. All-pairs comparisons and twin controls are exploratory. The five-design analysis uses an exact two-sided sign test. Supplementary Note 3 retains contrast definitions and planned endpoints outside this paper. Supplementary Note 4: Governing models and flaw certificates T02, settled two-span beam. Hinges can form only at the five sections A, underP 1 , B, underP 2 , and C. Sagging moments are positive. The plastic moment at B is the smaller of the two span values. Elastic response at load factorλ, with settlement ∆ at B, is the force-method compatibility condition FX + λδ P +δ ∆ = 0,(S1) andM (x) = M 0 (x;λ) + P j X j m j (x) . The first-hinge factorλ y is the smallestλat which|M|reachesM p at a hinge section. Collapse is the minimum, over mechanisms, of internal plastic work over external work. Shakedown uses Melan’s theorem:λ sd is the largest factor for which a residual moment field keepsλM E + ρinside the plastic moments for every load path in the domain (main-text references 48, 49, 50). Reference A uses Koiter’s kinematic dual (main-text references 51); reference B uses linear programming of the static form. The residual moment after loading to λ 1 and elastic unloading is M (λ 1 )− λ 1 M E . The flawed twin sets λ 1 > λ c . T04, compression-only bed. EI w ′ + k wH(w) = X i P i δ(x− x i ), k = k s b,(S2) withHthe Heaviside function. In the Winkler reactionkw,k = k s bis the foundation modulus of a bed of widthb (main-text references 47). The compression-only switch, the bed pushing only where the deflection is downward, is the unilateral contact of Weitsman and of Tsai and Westmann (main-text references 52, 53). The contact interval is unknown. Lifted parts carry zero moment and shear and stay straight. The sign conditions are the optimality conditions of the convex potentialΠ(w) = 1 2 R EI w ′2 + 1 2 R k⟨w⟩ 2 − P i P i w(x i ), with⟨w⟩ = max(w, 0). Two minimisers can differ only by a rigid-body motiona + bx, and such a motion changes the bed energy wherever the beam is in contact. Equilibrium under the downward loads requires a contact zone of positive length, so the contact solution is unique. The flawed twin places a downward gauge reading at an end the other data lift off the bed. T08, clamped cosine strip. Both ends are clamped. As in the shallow-arch theory of Fung and Kaplan (main-text references 54), axial compression N is the integral of the change in squared slope, and the transverse equilibrium is EI (y− y 0 ) ′ + N y ′ =−P δ(x− L/2),(S3) with the clamped cosine shapey 0 = (h/2)(1− cos 2πx/L)of Qiu, Lang and Slocum (main-text references 55). The strip loses stability at the earlier of a force maximum on the symmetric path and the asymmetric bifurcation N = N 2 ; low arches show this competition between symmetric and asymmetric modes (main-text references 54), and a clamped buckled beam switches to the asymmetric mode when the axial load reachesN 2 (main-text references 56). Reference A expands the deflection in the symmetric buckling modes (2000 modes kept) (main-text references 57). Reference B uses Hermite finite elements under midpoint displacement control. The flawed twin setsh/tbetween 0.75 and 1.0. On that range the symmetric path has no force maximum andN < 0.9N 2 throughout, so the strip the prompt calls a snap-through switch never loses stability. 24 Noticing an impossible problem yet reporting it solvedA PREPRINT T11, spinning disk. Plane stress, free bore, rim tension from the blades proportional toω 2 . The elastic field is the standard rotating-disk solution (main-text references 35). In the plastic zonea≤ r ≤ cthe Tresca face isσ θ − σ z = Y with σ θ ≥ σ r ≥ 0, and radial equilibrium integrates to rσ r = Y (r− a)− ρω 2 3 r 3 − a 3 .(S4) The associated flow rule gives zero plastic radial strain, so the radial displacement in the plastic zone follows fromσ r andYalone (main-text references 58, 59). For a solid disk, Gamer showed that the textbook Tresca solution violates the associated flow rule (main-text references 60). For the annular disk with a free bore used here we keep the face σ θ − σ z = Ywithσ θ ≥ σ r ≥ 0, and every accepted instance certifies that the plastic hoop strain is non-negative and grows with speed. The faceσ θ − σ r = Yis a shortcut that gate R5 catches. Unloading is elastic. Residual stress and permanent bore growth are the elastic–plastic state minus the elastic solution at the same overspeed. The flawed twin sets the overspeed above the burst speed. T12, shrink-fitted disk.The fit pressure falls linearly withω 2 until lift-off. Above lift-off the disk is a free-bore T11 disk. After the overspeed the residual interference is the installed interference minus the permanent bore growth, and the service lift-off speed scales with the square root of the residual pressure (main-text references 61). Certificates required at build time: the fit opens before any yielding, and re-contact is elastic. The flawed twin states that the fit stays tight, with the interference set to 0.70–0.85 of the permanent bore growth, so the disk is loose. Supplementary figures 0.25x0.5x1x2x5x10x Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash Gemini 3.1 Pro gpt-oss-120b 212323232323 222223232525 202224252525 252525252525 101213141717 131719212121 000000 252525252525 232323232323 222222222222 151616161717 171717181921 141415151719 000000 Tier 2 (25 solve items): items correct 0.25x0.5x1x2x5x10x Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash gpt-oss-120b 515152525252 525252525252 525252525252 494949494949 484949495151 505051515151 91313131414 525252525252 515151515151 515151515151 454545454545 505151515151 101011111113 Tier 1 Hard-10 (52 solve items): items correct ab Tolerance scale multiplies every key tolerance (1x = frozen rule, bold). Colour: change from 1x (red fewer, blue more). Only models with complete solve runs are shown. Supplementary Figure 1: Tolerance sensitivity. Solve items counted correct when every key tolerance is multiplied by a factor from 0.25 to 10. The frozen rule is the factor 1. Colour is the change from that rule. No model pair reverses order between factors 1 and 5. The largest move in that range is four additional items, for Sonnet 5. 25 Noticing an impossible problem yet reporting it solvedA PREPRINT 10 −9 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 |OpenSees - key| / |key| T02 residual moment at B T02 collapse factor T02 first-hinge factor T04 contact length T04 largest sagging moment T04 peak reaction T04 end uplift T08 critical force 1 % grading tolerance Qualitative checks passed: T02 first-hinge location matches: 11 T02 flaw reproduced: 6 T02 shakedown bracket: 11 T04 flaw reproduced: 6 T08 flaw reproduced: 6 T08 mode matches: 11 T02, T04: elastic-plastic frame and compression-only springs (check tolerance 0.1-0.2 %). T08: corotational beams (exact kinematics), so its differences include the keys' shallow-arch model. The two T08 points above 1 % (2.2-2.4 %) are asymmetric-bifurcation strips: the imperfect-path peak sits below the bifurcation load; check tolerance 3 %. Supplementary Figure 2: Third-party verification. Relative difference between OpenSees and the answer key for every checked quantity, with counts of the qualitative checks (shakedown bracket, flaw reproduced, mode match). T08 compares exact beam kinematics with the keys’ shallow-arch model. The two points above 1% are imperfection-sensitive bifurcation loads, accepted at a 3% tolerance. Sixty-two checks, no failures. T11 and T12 are not in this figure. T02T04T08T11T12 solve items correct (of 5 per family) Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash Gemini 3.1 Pro gpt-oss-120b 55544 55544 54555 55555 51142 42355 00000 55555 45554 54544 33154 43055 50055 00000 Supplementary Figure 3: Tier-2 solve accuracy by family. Items correct out of five in each family. On the primary run, Gemini 3.8 Flash solved none of the five T08 items. 26 Noticing an impossible problem yet reporting it solvedA PREPRINT critique: detect critique: locate step critique: false alarm premise, unwarned: reject premise, unwarned: false alarm premise, warned: reject premise, warned: false alarm Fable 5.1 Opus 5.5 Sonnet 5.5 Opus 5 Sonnet 5 Opus 4.8 Haiku 4.5 GPT-6-Astra GPT-6-Sol GPT-5.6-Sol GPT-5.5 Gemini 3.8 Flash gpt-oss-120b 20/2120/210/2114/200/2018/200/20 21/2121/210/2118/200/2020/200/20 20/2120/210/2115/200/2020/200/20 19/2119/210/218/200/2017/200/20 21/2120/210/215/200/2016/200/20 21/2121/210/216/200/2017/200/20 14/2114/210/213/200/19*17/202/20 21/2121/210/2116/200/2020/201/20 21/2121/210/2113/200/2020/201/20 21/2121/210/2113/200/2020/200/20 21/2121/212/2113/200/2019/200/20 21/2121/210/2112/200/2020/200/20 20/2119/210/217/200/2015/208/20 * one call without an answer; critique pairs: 21, premise pairs: 20 Supplementary Figure 4: Tier-1 flaw handling. Critique pairs (detect the implanted error, locate the first wrong step, false alarm on the clean solution) and premise pairs, unwarned and warned (reject: the flawed statement is rejected; false alarm: the consistent statement is rejected). Premise rejection shows the same warned-versus-unwarned gap as Tier 2, at 20 pairs. notedadopted misjudged silent coder GPT-6-Sol noted adopted misjudged silent coder Sonnet 5.5 57300 0104911 01240 04019 232 missed flaws; kappa 0.81 (4 classes), 0.89 (3 classes: adopted + silent merged) Supplementary Figure 5: Coder agreement. Confusion matrix of the two blinded coders over 232 missed flaws. Three-class κ = 0.89. Four-classκ = 0.81. The off-diagonal mass in the four-class matrix sits on the adopted/silent boundary, which the primary scheme merges. 27 Noticing an impossible problem yet reporting it solvedA PREPRINT T02-000T02-001T02-002T02-003T02-004T04-000T04-001T04-002T04-003T04-004T08-000T08-001T08-002T08-003T08-004T11-000T11-001T11-002T11-003T11-004T12-000T12-001T12-002T12-003T12-004 Fable 5.1 run 1 Fable 5.1 run 2 Fable 5.1 run 3 Opus 5.5 run 1 Opus 5.5 run 2 Opus 5.5 run 3 Sonnet 5.5 run 1 Sonnet 5.5 run 2 Sonnet 5.5 run 3 Opus 5 run 1 Opus 5 run 2 Opus 5 run 3 GPT-6-Astra run 1 GPT-6-Astra run 2 GPT-6-Astra run 3 GPT-6-Sol run 1 GPT-6-Sol run 2 GPT-6-Sol run 3 Gemini 3.8 Flash run 1 Gemini 3.8 Flash run 2 Gemini 3.8 Flash run 3 correctwrong Supplementary Figure 6: Item-level consistency. Each Tier-2 solve item in each of three runs, for the seven models that were repeated. A cell is one item in one run. GPT-6-Astra has no flips. GPT-6-Sol flips on eight items, which is whypass 3 falls from a primary score of 23/25 to 16/25. Supplementary tables Supplementary Table 1: Tier-1 families. Hard-10 is the hardest tenth of a 1280-item pool, selected by a model-free score and frozen before any model call. The evaluation slice used here is 52 solve items, 21 critique pairs and 20 premise pairs. FamilyProblem F01Thermal buckling versus first yield of a clamped stepped column F02Two-span beam with a settling compression-only bearing and a thermal gradient F03Torsional resonance of a three-disk rotor on a stepped shaft F04Finite free–free beam on an elastic foundation, off-centre load F05Fatigue versus first-cycle yield of a notched shaft, with a minimum-diameter redesign F06Edge crack in a restrained strip under thermal contraction: fracture versus net-section yield F07Two-material shrink fit under internal pressure and heating F08Large deflection of a cantilever strip: elastica versus linear theory F09Hertz line contact: subsurface shear and the load at first yield F10Cross-ply laminate: cure residual stress and first-ply failure F11Rotating disk with a radial temperature gradient F12Cantilever with a tip mass and an intermediate spring F13Indeterminate truss with a heated member and a fabrication misfit 28 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 2: Harness settings that affect comparability. Wall time is the common unit across providers. Token counts are not. “Output tokens” include thinking in all three harnesses, on different tokenizers. CLI, command line interface. Claude Code CLICodex CLI 0.158.0Antigravity CLI 1.2.13 ModelsAnthropic panelGPT-6 and GPT-5 seriesGemini, gpt-oss-120b System text Replaces the system promptPrepended; the CLI keeps its own agent instructions Prepended; the CLI keeps its own agent prompt ToolsNoneDisabled, read-only sandboxCannot be disabled; a call that logs a tool step is graded as a failure (nine Gemini 3.8 Flash calls) EffortHigh (Haiku 4.5: no flag)HighEncoded in the model identifier Output cap Tier 2: 128,000 (Haiku 4.5: 64,000). Tier 1: CLI default Service defaultNot settable SamplingCLI default; no temperature or seed set CLI default; no temperature or seed set CLI default; no temperature or seed set SessionNo persistenceEphemeralOne turn Supplementary Table 3: Tier-2 rates with Wilson 95% intervals. Solve items are the 25 Tier-2 solve problems. Unwarned rejection (flawed twins) and false alarms (consistent twins) are out of 30 pairs for all 14 models. Gemini 3.1 Pro was not run on Tier 1 and is included here. ModelSolve itemsUnwarned rejectionFalse alarms Fable 5.123/25 (75–98%)27/30 (74–97%)0/30 (0–11%) Opus 5.523/25 (75–98%)30/30 (89–100%)0/30 (0–11%) Sonnet 5.524/25 (80–99%)28/30 (79–98%)0/30 (0–11%) Opus 525/25 (87–100%)23/30 (59–88%)0/30 (0–11%) Sonnet 513/25 (33–70%)3/30 (3–26%)0/30 (0–11%) Opus 4.819/25 (57–89%)7/30 (12–41%)0/30 (0–11%) Haiku 4.50/25 (0–13%)2/30 (2–21%)0/30 (0–11%) GPT-6-Astra25/25 (87–100%)30/30 (89–100%)0/30 (0–11%) GPT-6-Sol23/25 (75–98%)29/30 (83–99%)0/30 (0–11%) GPT-5.6-Sol22/25 (70–96%)30/30 (89–100%)0/30 (0–11%) GPT-5.516/25 (45–80%)28/30 (79–98%)1/30 (1–17%) Gemini 3.8 Flash17/25 (48–83%)20/30 (49–81%)4/30 (5–30%) Gemini 3.1 Pro15/25 (41–77%)22/30 (56–86%)3/30 (3–26%) gpt-oss-120b0/25 (0–13%)10/30 (19–51%)14/30 (30–64%) Supplementary Table 4: Contrasts. Exact McNemar tests, Holm-adjusted within the six contrasts for that outcome. Counts are discordant items (only the first model right, then only the second). The six contrasts were fixed on 29 Septem- ber 2026, after the solve items and the original nine flaw pairs had been scored and before the 21 added pairs existed. Solve accuracy is exploratory. Rejection on the 21 added pairs is the confirmatory test. Rejection on all 30 pairs is the combined estimate. The same contrasts under the twin controls are in Supplementary Table 14. OutcomeContrastOnly first Only secondHolm P Solve items (exploratory) Opus 5.5 vs Opus 5021 Solve items (exploratory) Opus 5 vs Opus 4.8600.16 Solve items (exploratory) Sonnet 5.5 vs Sonnet 51100.0059 Solve items (exploratory) GPT-6-Sol vs GPT-5.6-Sol321 Solve items (exploratory) GPT-6-Astra vs GPT-6-Sol201 Solve items (exploratory) Opus 5.5 vs GPT-6-Astra021 Rejection, 30 pairsOpus 5.5 vs Opus 5700.063 Rejection, 30 pairsOpus 5 vs Opus 4.81717.2× 10 −4 Rejection, 30 pairsSonnet 5.5 vs Sonnet 52503.6× 10 −7 Rejection, 30 pairsGPT-6-Sol vs GPT-5.6-Sol011 Rejection, 30 pairsGPT-6-Astra vs GPT-6-Sol101 Rejection, 30 pairsOpus 5.5 vs GPT-6-Astra001 Rejection, 21 added pairs Opus 5.5 vs Opus 5101 Rejection, 21 added pairs Opus 5 vs Opus 4.81510.0026 29 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 4, continued OutcomeContrastOnly first Only secondHolm P Rejection, 21 added pairs Sonnet 5.5 vs Sonnet 51804.6× 10 −5 Rejection, 21 added pairs GPT-6-Sol vs GPT-5.6-Sol011 Rejection, 21 added pairs GPT-6-Astra vs GPT-6-Sol101 Rejection, 21 added pairs Opus 5.5 vs GPT-6-Astra001 An item random-intercept logistic model on the solve items was fitted by maximum a posteriori estimation, the marginal likelihood taken by non-adaptive 30-point Gauss–Hermite quadrature, with a normal prior of mean 0 and standard deviation 2.5 on the model effects. The item-difficulty standard deviation is 0.79 on the logit scale. Model effects (logit, standard error) run from+3.89(1.16) for Opus 5 and GPT-6-Astra, through+0.03(0.45) for Sonnet 5, to−3.98(1.16) for Haiku 4.5 and gpt-oss-120b. The fit converged. It is a description of the spread, not a ranking test. Supplementary Table 5: Parser census, scored replies. The grader uses the last fenced JavaScript Object Notation (JSON) block and also accepts bare JSON. A seeded random sample of 50 replies, stratified by harness (seed 20260930), matched the graded value in every case. The flagged set (0 unit-scale slips, 4 sign flips, 35 answers whose correct value appears elsewhere in the reply) was reviewed separately; none was a parser error. The census covers the primary panel runs only. The single-part arms (180 calls) gave 178 fenced replies and two non-answers; the second collection (456 final calls) gave 447 fenced replies, one unparsed reply and eight non-answers. Replies Scored replies, primary panel runs3679 Fenced JSON3643 of which two JSON blocks1 Bare JSON (all gpt-oss-120b)26 Unparsed10 Seeded sample, graded as stated50 of 50 Supplementary Table 6: Coding of missed flaws. Both coders labelled 232 cases. The three-class scheme merges adopted and silent into “unexamined” and was adopted after the four-class disagreements were seen. The 28 four-class disagreements were adjudicated by GPT-6-Astra, neither a coder nor the coding assistant, blind to model, run, item and the study’s aims. A first adjudication by Claude Opus 5.5, the model behind the coding assistant, gave the same class on 26 of 28 cases and on 28 of 28 under the three-class scheme. An earlier unblinded labelling of 42 cases by the coding assistant, made before the codebook was fixed, is not used; it agrees with the two coders at three-classκ = 0.76and0.69(four-class 0.39 and 0.37), and all 13 cases it called noted are noted for both coders. SchemeCasesRaw agreementCohen’s κ Four classes (noted, misjudged, adopted, silent)2320.880.81 Three classes (noted, misjudged, unexamined)2320.940.89 Supplementary Table 7: Protocol deviations. Items, keys and prompts of a frozen set were not edited after its manifest. Every dated deviation is listed; three grading rules for non-answers were set or corrected during the runs. CLI, command line interface; JSON, JavaScript Object Notation. DateDecisionWhat it bounds 28 Sep Dataset v0.1 to v0.2: mechanism defects added in F01, F02 and F13Before any model call. 28 SepHard-10 subset chosen for the first model run; selection score fixed in advanceBefore any model call. 28 Sep Hard-10 is the evaluation set for every modelTier 1 was not re-cut after scores. 28 Sep Google and open-weight models run through Antigravity Tools cannot be disabled. A call that logs a tool step is graded as a failure; nine calls did, none with a usable answer. 28 Sep The 80% difficulty gate failed; Tier 2 was proposedTier 1 is the standard-problem anchor. 28 Sep Self-correction follow-ups moved to Tier 2 and were not run; a closed-book sentence is added to Tier-2 prompts No self-correction result is reported. Tier-1 prompts unchanged. 29 Sep Tier-2 rules fixed before any evaluation item existedPrompts and the grader travel with the frozen set. 29 Sep Tier-2 output budget is each model’s documented maximumClaude CLI 128,000 tokens; Haiku 4.5 64,000. 29 Sep Evaluation set frozen: 25 solve items and 9 flaw pairs. T02, T04, T08 and T11 kept because Sonnet 5 failed development items; T03 dropped because Sonnet 5 solved it; T12 from later calibration runs Solve-accuracy contrasts that include Sonnet 5 are biased. Evaluation items are fresh draws; every result holds for these five families. 29 Sep Missed flaws split into noted and silent, after the first scoresThe graded outcome stays the JSON status. 29 Sep Grading rule: an exhausted output budget is a final failureRegrading left the finished scores unchanged. 29 Sep Opus 4.6 stopped after 3 calls with no answer textReported, not ranked. The 65 tokens/s figure is a separate diagnostic. 29 Sep Budget rule extended to Antigravity; the first scored reply countsGemini 3.8 Flash scores are unchanged either way. 30 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 7, continued DateDecisionWhat it bounds 29 Sep OpenSees convergence horizon lengthened; Tier-2 flaws checkedTest code only. Keys unchanged. 30 of 30 frozen checks pass. 29 Sep Sonnet 4.6 stopped after 23 of 61 callsReported, not ranked. 29 Sep Six contrasts fixed after the original nine flaw pairs were scored Rejection contrasts are confirmatory on the 21 added pairs only. Solve-accuracy contrasts are exploratory. All-pairs tests are exploratory. 29 Sep Twenty-one new flaw pairs frozen before any model saw themThe warning result rests on nine twins. T08 and T12 flaws are unwarned only. 29 Sep OpenSees unloading step uses a force testTest code only. No item or key changed. 29 Sep Codex idle timeout is a separate harness-failure class, retried onceA transport limit, not the 2700 s timer. 29 Sep That harness-timeout class implemented in the graderSeven GPT-5.5 calls (three solve items, four twin calls) are final failures of that class. 30 Sep Blinded double coding fixed before coder output was readTimeouts, budget and harness failures are excluded from κ. 30 Sep Three-class scheme chosen after the disagreements were read; first adjudication by Claude Opus 5.5, the model behind the coding assistant Primary κ is post hoc. Four-class κ is reported beside it. 30 Sep Gemini 3.1 Pro not run on Tier 1 (quota). Flaw extension completedReported on Tier 2 only, including all 30 twins. 30 Sep One Haiku 4.5 Tier-1 call exceeded the output cap on both attempts173 of 174 calls final. Tier-1 counts use models with every solve item final. 30 Sep Independent re-adjudication by GPT-6-Astra, fixed before its output existed; its decisions are primary Same class as the first adjudication on 26 of 28 (four classes) and 28 of 28 (three classes). No reported number changes. 30 Sep Protocol claims not tested: accuracy against cost across effort levels, the correction step of the critique arm, self-correction, defects invisible to final-answer grading No result is reported for them. One effort setting was run. 30 SepTwin controls added after all replies had been read: contrasts on co-solved pairs, rejection-reason word lists, recognition endpoint, repair rules for noted misses Exploratory. No run, item, key, prompt or grading rule changed; the confirmatory tests are unchanged. 30 SepBlinded double coding of the 289 rejection reasons and 26 noted Tier-2 replies, fixed before any coder output existed Coded classes replace the word lists and repair rules where they differ: reasons in no case, noted replies in five. 30 Sep Instruction-only and schema-only arms on the nine original pairs for five Anthropic models, fixed before the first call Exploratory. Run about a day and a half after the unwarned and warned cells they are compared with. 30 SepSecond collection: fresh unwarned and schema-only runs of Opus 5, Opus 4.8, Sonnet 5.5 and Sonnet 5 on all 30 pairs, plus a second unwarned run of the older two; fixed before the first call Tests in Supplementary Table 16; status-level scoring. A usage pause split collection into two blocks; a quota guard stopped it at 456 of 480 calls (Sonnet 5); the second runs were not made. 30 Sep Recognition counts only rejections that name the planted flaw; co-solved contrasts repeated without T04 Recognition 3 to 2 (Haiku 4.5), 27 to 26 (GPT-5.5); no planned contrast changes; without T04 the co-solved contrasts hold. Supplementary Table 8: Graded quantities. A solve item is correct when every quantity in its row passes the tolerance rule in the main text. The field name in the JavaScript Object Notation (JSON) answer is given in parentheses. FamilyGraded quantities T02Load factor at the first plastic hinge (lam_y); location of the first hinge (first_hinge); collapse load factor (lam_c); shakedown load factor (lam_sd); residual moment at B (M_B_res) T04Which ends lift off (lift); contact length (L_c); largest ground reaction per unit length (p_max); largest sagging moment (M_max); upward displacement of the end that rises most (v_up) T08Mode of the first instability (mode); force at that point (P_cr); midpoint displacement (v_cr); axial force (N_cr); clamp moment (M_cl) T11Speed at first yield (n_y); radius of the elastic–plastic boundary at the overspeed (c_os); residual hoop stress at the bore (s_res); permanent bore growth (du_perm); operating hoop stress at the bore (s_op) T12Fit pressure at rest (p0); lift-off speed during the spin test (n_lift); speed at first yield (n_y); radius of the elastic–plastic boundary at the overspeed (c_os); fit pressure after the test (p_res); lift-off speed in later service (n_lift_s) Supplementary Table 9: Model identifiers. The identifier passed to each command-line harness, as stored with each call, and the reasoning-effort flag. Haiku 4.5 has no effort flag; for Gemini and gpt-oss-120b the setting is part of the identifier. No sampling temperature or seed was set. CLI, command line interface. ModelIdentifierHarnessEffort Fable 5.1 claude-fable-5-1Claude Code CLIhigh Opus 5.5 claude-opus-5-5Claude Code CLIhigh Sonnet 5.5 claude-sonnet-5-5Claude Code CLIhigh Opus 5 claude-opus-5Claude Code CLIhigh Sonnet 5 claude-sonnet-5Claude Code CLIhigh Opus 4.8 claude-opus-4-8Claude Code CLIhigh Haiku 4.5 claude-haiku-4-5Claude Code CLI— GPT-6-Astra gpt-6-astraCodex CLI 0.158.0high GPT-6-Sol gpt-6-solCodex CLI 0.158.0high GPT-5.6-Sol gpt-5.6-solCodex CLI 0.158.0high GPT-5.5 gpt-5.5Codex CLI 0.158.0high Gemini 3.8 Flash gemini-3.8-flash-highAntigravity CLI 1.2.13— Gemini 3.1 Pro gemini-3.1-pro-highAntigravity CLI 1.2.13— gpt-oss-120b gpt-oss-120b-mediumAntigravity CLI 1.2.13— 31 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 10: Twin controls per model, 30 Tier-2 pairs, unwarned. Consistent: consistent twins solved. Rejected: flawed twins rejected. Given consistent solved: flawed twins rejected among the pairs whose consistent twin the model solved. Inability: rejections whose reason claims no defect and cites only the difficulty of computing. Other reason: rejections that claim a defect other than the planted flaw. Both from the blinded coding, which equals the word lists for every rejection. Noted: misses coded noted, split by the blinded coding into repair / continuation. Recognised: rejections whose coded reason names the planted flaw, plus noted misses. Scoring the status field alone, without withheld answers, gives the same rejection and false-alarm counts for every model. Lower block: in family T04 the consistent twin states a gauge reading that equals the graded upliftv up in five of six pairs; the uplift answers and the whole-item results on the five T04 solve items and the six T04 consistent twins. Exploratory; rules in the Methods. ModelConsistent Rejected Given consistent solved Inability Other reason Noted Recognised Fable 5.1302727/30003 (3/0)30 Opus 5.5303030/3000030 Sonnet 5.5282826/28002 (2/0)30 Opus 5292322/29007 (6/1)30 Sonnet 51631/16002 (2/0)5 Opus 4.82575/25008 (3/5)15 Haiku 4.502—112 (0/2)2 GPT-6-Astra303030/3000030 GPT-6-Sol262925/2600029 GPT-5.6-Sol303030/3000030 GPT-5.5182816/1811026 Gemini 3.8 Flash192010/1900020 Gemini 3.1 Pro202217/20002 (1/1)24 gpt-oss-120b010—10000 T04 solve items (5)T04 consistent twins (6) Modelv up correct / item correct v up correct / item correct Fable 5.15 / 56 / 6 Opus 5.55 / 56 / 6 Sonnet 5.54 / 46 / 6 Opus 55 / 56 / 6 Sonnet 52 / 14 / 3 Opus 4.82 / 25 / 4 Haiku 4.50 / 05 / 0 GPT-6-Astra5 / 56 / 6 GPT-6-Sol5 / 56 / 6 GPT-5.6-Sol4 / 46 / 6 GPT-5.53 / 35 / 4 Gemini 3.8 Flash3 / 33 / 1 Gemini 3.1 Pro1 / 05 / 3 gpt-oss-120b0 / 00 / 0 32 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 11: Rejection per flaw design. Top: flawed twins rejected, unwarned, out of six per family (one flaw design per family). Bottom: the six planned contrasts, rejections of the first model minus the second per design; designs where the first rejects more, fewer or the same; exact two-sided sign test over the five designs. ModelT02 T04 T08 T11 T12 Fable 5.164665 Opus 5.566666 Sonnet 5.564666 Opus 532666 Sonnet 502100 Opus 4.801420 Haiku 4.501010 GPT-6-Astra66666 GPT-6-Sol66665 GPT-5.6-Sol66666 GPT-5.566655 Gemini 3.8 Flash36560 Gemini 3.1 Pro64453 gpt-oss-120b05122 ContrastT02 T04 T08 T11 T12 More/fewer/same Sign test P Opus 5.5 vs Opus 5+3+4+0+0+02/0/30.50 Opus 5 vs Opus 4.8+3+1+2+4+65/0/00.0625 Sonnet 5.5 vs Sonnet 5+6+2+5+6+65/0/00.0625 GPT-6-Sol vs GPT-5.6-Sol+0+0+0+0-10/1/41 GPT-6-Astra vs GPT-6-Sol+0+0+0+0+11/0/41 Opus 5.5 vs GPT-6-Astra+0+0+0+0+00/0/51 Supplementary Table 12: Tier-2 misses coded noted, by the numbers returned. Every unwarned Tier-2 miss coded “noted” (the reply states the flaw and marks the problem solved). Rule: the comparison of the returned numbers with the attainable readings of the flawed statement (Methods). Blinded: the class from the blinded double coding, which is used in the paper. Pair: family, set (original: the nine pairs frozen first; added: the 21 pairs frozen later) and pair number within the family in that set. JavaScript Object Notation (JSON) reason: whether the reply’s JSON reason field is non-empty; otherwise the statement of the flaw is only in the prose. ModelPairRuleBlindedWhat the returned numbers are, by the rule comparisonJSON reason Fable 5.1T04, original pair 1 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideno Fable 5.1T04, original pair 2 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideyes Fable 5.1T12, added pair 6repairrepairresidual fit pressure and service lift-off speed returned as zero (fit loose)yes Sonnet 5.5T04, original pair 1 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideno Sonnet 5.5T04, original pair 2 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideno Opus 5T04, original pair 1 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideno Opus 5T02, original pair 1 continuation continuation residual moment 126.0 = collapse moments unloaded by the unattainable factorλ 1 = 2.61 (126.0 kN m) no Opus 5T02, original pair 2 repairrepairresidual moment -23.8 = unloading from the attainable collapse state (-23.8 kN m)yes Opus 5T04, original pair 2 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideno Opus 5T02, original pair 3 repairrepairresidual moment 62.3 = unloading from the attainable collapse state (62.3 kN m)yes Opus 5T04, original pair 3 repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideyes Opus 5T04, added pair 2repairrepairanswers equal the solution of the stated mechanics with the gauge datum set asideyes Sonnet 5T02, added pair 2otherrepairresidual moment -42.1 matches neither readingno Sonnet 5T04, added pair 2otherrepairanswers differ from the solution of the stated mechanicsno Opus 4.8T02, original pair 3 continuation continuation residual moment 146.8 = collapse moments unloaded by the unattainable factorλ 1 = 2.39 (146.8 kN m) no Opus 4.8T02, added pair 1repairrepairresidual moment -5.1 = unloading from the attainable collapse state (-5.1 kN m)yes Opus 4.8T11, added pair 1continuation continuation numbers returned for a state that does not exist (no attainable reading)no Opus 4.8T02, added pair 2continuation continuationresidual moment 113.6 = collapse moments unloaded by the unattainable factorλ 1 = 1.99 (113.6 kN m) no Opus 4.8T11, added pair 2continuation repairnumbers returned for a state that does not exist (no attainable reading)no Opus 4.8T02, added pair 3continuation continuation residual moment 45.9 = collapse moments unloaded by the unattainable factor λ 1 = 2.15 (45.9 kN m) no Opus 4.8T12, added pair 4repairrepairresidual fit pressure and service lift-off speed returned as zero (fit loose)yes Opus 4.8T12, added pair 6continuation continuation returns a residual fit pressure of 3.7 for a fit the data leave looseno Haiku 4.5T02, original pair 1 othercontinuation residual moment -98 matches neither readingyes Haiku 4.5T02, added pair 1othercontinuation residual moment -75 matches neither readingyes Gemini 3.1 Pro T11, original pair 1 continuation continuation numbers returned for a state that does not exist (no attainable reading)no Gemini 3.1 Pro T12, added pair 6repairrepairresidual fit pressure and service lift-off speed returned as zero (fit loose)no 33 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 13: Warned and unwarned arms on the same nine pairs. Rejected: flawed twins rejected (of 9). False alarms: consistent twins rejected (of 9). Joint: pairs with the consistent twin solved and the flawed twin rejected (of 9). The warned arm changes the instruction and the status field together; Supplementary Table 15 separates them for five models. RejectedFalse alarmsJoint Modelunwarned warned unwarned warned unwarned warned Fable 5.1790079 Opus 5.5990099 Sonnet 5.5790069 Opus 5390029 Sonnet 5080004 Opus 4.8190118 Haiku 4.5140300 GPT-6-Astra990099 GPT-6-Sol990089 GPT-5.6-Sol990099 GPT-5.5990274 Gemini 3.8 Flash690247 Gemini 3.1 Pro890376 gpt-oss-120b475500 Supplementary Table 14: Planned contrasts under the twin controls. Discordant items (only the first model, only the second) with the number of pairs in brackets, and Holm-adjusted exact McNemarPwithin the six contrasts. Co-solved: rejection on pairs whose consistent twin both models solved. No T04: the same without family T04, whose consistent twin states a graded answer (Methods). Recognition: a rejection whose coded reason names the planted flaw, or a noted miss. Answered: rejection on pairs where neither model’s reply to the flawed twin was a non-answer. All exploratory. PairsContrastCo-solved PNo T04 PRecognitionPAnsweredP 21 added Opus 5.5 vs Opus 51–0 (21)10–0 (18)10–0 (21)11–0 (21)1 21 added Opus 5 vs Opus 4.813–1 (17)0.01112–0 (15) 0.00298–0 (21)0.03915–1 (21)0.0026 21 added Sonnet 5.5 vs Sonnet 59–0 (10)0.0209–0 (9)0.02016–0 (21)1.8× 10 −4 15–0 (18) 3.7× 10 −4 21 added GPT-6-Sol vs GPT-5.6-Sol0–1 (18)10–1 (15)10–1 (21)10–1 (21)1 21 added GPT-6-Astra vs GPT-6-Sol1–0 (18)11–0 (15)11–0 (21)11–0 (21)1 21 added Opus 5.5 vs GPT-6-Astra0–0 (21)10–0 (18)10–0 (21)10–0 (21)1 all 30Opus 5.5 vs Opus 57–0 (29)0.06253–0 (23)10–0 (30)17–0 (30)0.0625 all 30Opus 5 vs Opus 4.814–1 (24) 0.0049 13–0 (20) 0.001515–0 (30)3.1× 10 −4 17–1 (30) 7.2× 10 −4 all 30Sonnet 5.5 vs Sonnet 512–0 (15) 0.0029 12–0 (12) 0.002425–0 (30)3.6× 10 −7 19–0 (24) 2.3× 10 −5 all 30GPT-6-Sol vs GPT-5.6-Sol0–1 (26)10–1 (20)10–1 (30)10–1 (30)1 all 30GPT-6-Astra vs GPT-6-Sol1–0 (26)11–0 (20)11–0 (30)11–0 (30)1 all 30Opus 5.5 vs GPT-6-Astra0–0 (30)10–0 (24)10–0 (30)10–0 (30)1 Supplementary Table 15: Instruction and schema factorial, nine original pairs. Flawed twins rejected, with consistent twins rejected (false alarms) in brackets, each of 9 per model. Unwarned and warned: each model’s first run. Instruction only: the check instruction with the unwarned status field. Schema only: the warned status field and defect type without the instruction. Both ran on 30 September. Repeats: flawed twins rejected in the two unwarned repeat runs on the same nine twins (—: no repeat run). Lower block: exact McNemar tests on the 45 paired flawed-twin replies, discordant pairs (only the first cell named, only the second) and Holm-adjustedPover the four. The instruction-only and schema-only cells ran about a day and a half after the unwarned and warned cells, so these tests compare different collection periods; they do not separate the causal contributions of the two parts, and the near-equal counts do not show equal effect. Supplementary Table 16 compares the schema-only arm with a fresh unwarned run collected over the same two periods. Joint successes (consistent twin solved and flawed twin rejected), pooled over the five models: 39 of 45 warned, 40 instruction only, 38 schema only. Exploratory. ModelUnwarnedInstruction onlySchema onlyWarnedRepeats Fable 5.17 (0)9 (0)9 (0)9 (0)9, 6 Sonnet 5.57 (0)9 (0)9 (0)9 (0)8, 7 Opus 53 (0)9 (0)9 (0)9 (0)6, 9 Sonnet 50 (0)7 (0)7 (0)8 (0)— Opus 4.81 (0)9 (0)9 (0)9 (1)— Pooled (of 45)18 (0)43 (0)43 (0)44 (1)— 34 Noticing an impossible problem yet reporting it solvedA PREPRINT Supplementary Table 15, continued ContrastFirst onlySecond only P (Holm) Instruction only vs unwarned2502.4× 10 −7 Schema only vs unwarned2502.4× 10 −7 Warned vs instruction only101 Warned vs schema only101 Supplementary Table 16: Second collection: prespecified tests. Fresh unwarned and schema-only runs of Opus 5, Opus 4.8, Sonnet 5.5 and Sonnet 5, started together, with call times in Coordinated Universal Time (UTC): 30 September 21:31–23:05 and 1 October 06:28–07:43 UTC. Top: the two confirmatory contrasts in each arm, discordant flawed twins (only the first model rejects, only the second), pairs in brackets, exact McNemarPHolm-adjusted over the four tests on the 21 added pairs; all 30 pairs as combined estimates (Holm over the same four on 30). Bottom: schema only against fresh unwarned within each model, 30 flawed twins, Holm over the four models, with false alarms on consistent twins and joint successes (consistent twin solved and flawed twin rejected), each given as schema only / unwarned. Pairs with a missing call are left out: 23 unwarned calls and one schema-only call of Sonnet 5 were not made (quota), and six of its 16 unwarned flawed-twin calls ended without a reply. Status-level scoring; the new replies were not coded. Exploratory sensitivities: counting only calls with a reply, Sonnet 5.5 versus Sonnet 5 is 6–0 over 7 added pairs (Holm-adjustedP = 0.125, same family of four) and 8–0 over all 10 pairs with a reply (P = 0.031); restricting the within-model comparison to pairs whose two calls started in the same window gives 17–0 (26 pairs), 14–0 (19), 3–0 (30) and 10–0 (11) for Opus 5, Opus 4.8, Sonnet 5.5 and Sonnet 5 (Holm-adjusted P = 6.1× 10 −5 , 3.7× 10 −4 , 0.25 and 0.0039). ArmContrast21 added Holm PAll 30Holm P Fresh unwarned Opus 5 vs Opus 4.82–0 (21)13–0 (30)0.50 Fresh unwarned Sonnet 5.5 vs Sonnet 5 9–0 (10)0.01614–0 (16) 4.9× 10 −4 Schema onlyOpus 5 vs Opus 4.80–0 (21)10–0 (30)1 Schema onlySonnet 5.5 vs Sonnet 5 3–0 (21)0.754–0 (29)0.38 ModelDiscordant (schema / unwarned)Holm PFalse alarmsJointPairs Opus 518 / 02.3× 10 −5 0 / 030 / 1230 Opus 4.821 / 03.8× 10 −6 2 / 019 / 830 Sonnet 5.53 / 00.250 / 125 / 2630 Sonnet 515 / 01.2× 10 −4 2 / 07 / 016 Prompts The system prompt on every call was: You are an expert mechanical engineer. Solve engineering mechanics problems carefully and give numerical answers in the units requested. Finish every reply with one JSON object in ajsonfenced block, exactly as the user specifies. Tier 2 appends one closed-book sentence to the user prompt: No tools, code execution or internet access are available in this conversation. Do all calculations yourself in this reply; do not write tool calls or simulated tool output. Always end the reply with the JSON block. The unwarned arm uses that prompt and this status schema. Nothing in it mentions flaws. "status": "solved" or "cannot_solve" "reason": "one sentence; required when status is cannot_solve" The warned arm inserts the following instruction before the answer schema, and replaces the status field: Before solving, check the problem statement. It may be flawed: data may be missing, contradictory or physically impossible, the structure may be unable to carry the load (a mechanism), or a question may ask for something that does not exist. If it is flawed, do not solve it. 35 Noticing an impossible problem yet reporting it solvedA PREPRINT The warned status schema is: "status": "solved" or "flawed" "defect_type": null, or one of missing_data, contradictory_data, physically_impossible, mechanism, false_premise "reason": "one sentence naming the flaw; required when status is flawed" When the status is flawed or cannot_solve, the answer object is null. The two factorial arms recombine these parts without new wording. The instruction-only arm inserts the instruction above into the unwarned prompt and keeps the unwarned status schema. The schema-only arm uses the warned status schema without the instruction. Item statements, numerical data and keys will be released with the benchmark after this preprint is posted. The evaluation pair drawn in main-text Fig. 1 is a free–free beam on a compression-only bed, identical in the two versions except the dial-gauge datum at the left end. 36

Instant research discovery