Paper deep dive

PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning

Kecheng Liang, Haoyang Liu, Zexin Chen, Zirong Liu, Weixing Chen, Qiufeng Wang, Yang Liu, Liang Lin

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 137

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A key challenge in physics diagram understanding is correctly associating visual information with the physical entities, relations, and conditions it describes. Even when a value, symbol, or other local element is accurately recognized, assigning it to the wrong entity or scope can distort the underlying physical premise and lead to incorrect reasoning. To systematically study this challenge, we introduce \textbf{PhysAlign}, a benchmark designed to assess whether multimodal models correctly associate information recognized from physics diagrams with its intended physical role. By disentangling visual recognition from physical-role assignment through localized probes and controlled variants, PhysAlign isolates correspondence errors from recognition failures. It contains 3,341 human-validated probes spanning 986 physics problems, enabling systematic evaluation of visual recognition and physical-role correspondence at scale. We further introduce five complementary evaluation metrics, including CAcc, GAcc, and JAcc, which provide a comprehensive assessment of models' ability to recognize diagram content, establish correct physical correspondences, and solve the underlying physics problem. Across our evaluated multimodal models, PhysAlign reveals a consistent gap between local visual recognition and physical-role grounding. Even when the queried content is correctly recognized, the conditional correspondence error rate remains 13.8\% for GPT-6-Astra and rises to about 50.6\% for InternVL3.5-8B. These findings indicate that strong perception alone does not ensure reliable physical interpretation, exposing a distinct grounding bottleneck that is largely hidden by answer-level accuracy and highlighting the need for future models to better align recognized visual evidence with its physical meaning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

133,669 characters extracted from source content.

Expand or collapse full text

PHYSALIGN: A BENCHMARK FOR EVIDENCE-GROUNDED ROLE ALIGNMENT IN MULTIMODAL PHYSICS REASONING A PREPRINT Kecheng Liang 1,∗ Haoyang Liu 2,∗ Zexin Chen 1 Zirong Liu 1 Weixing Chen 1 Qiufeng Wang 3 Yang Liu 1,† Liang Lin 1 1 Sun Yat-sen University 2 Dalian University of Technology 3 Xi’an Jiaotong-Liverpool University õ huggingface.co/datasets/Jetson888/PhysAlign physalign-lab.github.io/ § github.com/HCPLab-SYSU/PhysAlign ABSTRACT A key challenge in physics diagram understanding is correctly associating visual information with the physical entities, relations, and conditions it describes. Even when a value, symbol, or other local element is accurately recognized, assigning it to the wrong entity or scope can distort the underlying physical premise and lead to incorrect reasoning. To systematically study this challenge, we intro- duce PhysAlign, a benchmark designed to assess whether multimodal models correctly associate information recognized from physics diagrams with its intended physical role. By disentangling visual recognition from physical-role assignment through localized probes and controlled variants, PhysAlign isolates correspondence errors from recognition failures. It contains 3,341 human-validated probes spanning 986 physics problems, enabling systematic evaluation of visual recognition and physical-role correspondence at scale. We further introduce five complementary evaluation metrics, including CAcc, GAcc, and JAcc, which provide a comprehensive assessment of models’ ability to recognize diagram content, establish correct physical correspondences, and solve the underlying physics problem. Across our evaluated multimodal models, PhysAlign reveals a consistent gap between local visual recognition and physical-role grounding. Even when the queried content is correctly recognized, the conditional correspondence error rate remains 13.8% for GPT-6-Astra and rises to about 50.6% for InternVL3.5-8B. These findings indicate that strong perception alone does not ensure reliable physical interpretation, exposing a distinct grounding bottleneck that is largely hidden by answer-level accuracy and highlighting the need for future models to better align recognized visual evidence with its physical meaning. 1 Introduction Physics reasoning tests whether multimodal large language models (MLLMs) can interpret a physical situation from text and diagrams and apply the appropriate physical laws. A diagram specifies relations among objects and associates quantities with particular entities, while the accompanying text provides further conditions and constraints. These correspondences determine the physical premises on which a solution rests. Evaluating physics reasoning therefore requires examining whether a model recovers the stated problem setup as well as whether it derives the correct answer. Recent benchmarks examine both the use of visual evidence and intermediate steps in physics reasoning. SeePhys [Xiang et al., 2025] distinguishes problems that require visual information from those that can be solved from text alone. ∗ Equal contribution. † Corresponding author: liuy856@mail.sysu.edu.cn. arXiv:2609.33319v1 [cs.AI] 27 Sep 2026 A uniform rod of length L and mass M is hinged at O and supported by a cable attached to the wall. A small block of mass m rests on the rod at a distance from O. The system is in static equilibrium. m T M x O L g E4: support cable E2: block E3: rod R1: symbol T R1: local evidence anchor E2:block (mass m) E3:rod (mass M) E4:support cable (b)Two interpretations from same visual evidence. Reference physical premise T R1 The symbol T denotes the tensionin the supporting cable. read: T owner: Model-reported premise T R1 M The symbol T is assigned tothe rod. read: T owner: E3 (rod) (a)Problem evidence E4 (support cable) (c) Deterministic local scoring Scoring only—not provided to the model. FieldReferencePredictionScore ContentTTCⱼ= 1 GroundingE4E3Gⱼ= 0 Jointboth correctwrongJⱼ= 0 Figure 1: Correct recognition does not guarantee physical-role grounding. (a) The problem evidence specifies a rod supported by a cable, withR1marking the symbolT. (b) The reference associatesTwith the supporting cable (E4), whereas the illustrated response assigns it to the rod (E3). (c) Recognition is correct (C j = 1), but grounding and joint correctness fail (G j = 0,J j = 0). Reference targets and scores are used only for evaluation and are not provided to the model. PhysicsArena [Dai et al., 2025] evaluates variable identification, physical process formulation, and solution derivation, while SeePhys Pro [Xiang et al., 2026] studies visual variable grounding through semantically aligned input variants that transfer information from text to images. However, diagnosing errors in the recovered problem setup requires a more localized distinction between reading content and assigning it to the correct physical role. Final-answer accuracy cannot determine whether an error originates in recognition, role assignment, or subsequent reasoning. Reconstructing a complete physical graph can also entangle an incorrect correspondence with unrelated entity omissions, naming differences, and structured-output errors. We focus on measuring recognition and role assignment separately at the same evidence location. Correct recognition of a symbol does not necessarily imply correct grounding to the corresponding physical entity. As illustrated in Figure 1, a rod carrying a block is held in static equilibrium by a hinge atOand a supporting cable. The symbolTat evidence anchorR1denotes the tension in the supporting cable (E4). The illustrated response recognizesT correctly but assigns it to the rod (E3). Here, the queried correspondence refers to the cable associated with the tension symbol, rather than to the body on which the tension force acts. We call the assignment of local content to the physical entity or role supported by the problem evidence physical-role grounding. This distinction motivates testing whether role-assignment errors persist when the evidence location is given and even when the correct local reading is supplied. We introduce PhysAlign, a benchmark for diagnosing physical-role grounding through localized probes with explicit evidence anchors. PhysAlign contains 3,341 probes constructed from 986 physics problems across six source datasets. We convert human-reviewed observational annotations into tests of entity reference and physical-quantity ownership. Each probe retains the original problem statement, diagram, and physical conditions, and uses an image region or text span to identify the queried evidence. For each probe, the model identifies the physical role associated with the queried content and, when an independent reading target is available, also reports the corresponding local content. This design enables recognition and role assignment to be evaluated against their respective evidence without requiring full diagram-graph reconstruction. PhysAlign combines separate recognition and grounding scores with a paired reading control. For probes with independently defined reading and role targets, we use deterministic rules to score recognition and grounding separately; joint correctness requires both targets to be correct. These scores distinguish correctly recognized content from a correctly reported physical premise. A subset of 553 probes also supports paired input variants: the original problem, evidence anchor, question, candidates, and scoring rules remain fixed, while the augmented condition supplies predefined correct local readings. We select these readings using model-independent rules that prohibit disclosure of the queried role correspondence. The comparison measures whether supplying correct content improves grounding. This decomposition concerns observable responses and does not assume separate recognition and grounding modules within a model. Results from six MLLMs of different scales and model families show that grounding errors persist even when local content is recognized correctly. On the joint evaluation set, the lowest grounding error rate conditional on correct recognition is approximately 13.8%. Paired grounding point estimates increase by 1.3–8.1 percentage points on the 2 observed model-specific subsets, but item-level analysis of four open-weight models reveals both repairs and harms. The highest observed +GT accuracy is 84.9%. These findings support evaluating reported physical premises against their source evidence alongside original-problem solving, without treating supplied readings as perfect perception. Our contributions are as follows: (1) We introduce PhysAlign, a benchmark for local physical-role grounding that links tests of entity reference and physical-quantity ownership to explicit source evidence and human-reviewed role targets. (2) We develop deterministic recognition, grounding, and joint scores, together with a paired reading control that tests the effect of supplying correct local content without disclosing the target role correspondence. (3) Across model families and scales, we identify correctly recognized content with incorrect physical-role assignments. This discrepancy persists on probes where a frozen proximity rule fails, while paired readings reveal both repairs and harms. These findings expose response-level failures that final-answer accuracy alone cannot localize. 2 Related Work Multimodal scientific and physics reasoning. Scientific reasoning benchmarks span science question answering in ScienceQA [Lu et al., 2022], multidisciplinary reasoning in MMMU [Yue et al., 2024], visual mathematics in MathVista [Lu et al., 2024], and undergraduate physics in PhysUniBench [Wang et al., 2025a]. SeePhys [Xiang et al., 2025] distinguishes Vision-Essential from Vision-Optional problems, while SeePhys Pro [Xiang et al., 2026] studies robustness as task-critical information moves from text to diagrams. PhysicsArena [Dai et al., 2025] further evaluates intermediate stages such as variable identification and process construction. These benchmarks primarily assess final solutions or broad reasoning stages, where perception, entity correspondence, and derivation remain coupled. PhysAlign instead examines the pre-reasoning recovery of the problem setup: whether content read from a local visual region is assigned to the correct physical entity or role. Controlled visual diagnosis and role grounding.Controlled vision–language evaluations show that recognizing indi- vidual elements does not ensure correct relational interpretation. Winoground [Thrush et al., 2022], ARO [Yuksekgonul et al., 2023], VL-CheckList [Zhao et al., 2022], and SugarCrepe [Hsieh et al., 2023] isolate failures in compositional structure, attributes, relations, and order. Related diagnostics for mathematical and scientific reasoning vary the availability or modality of visual evidence: MathVerse [Zhang et al., 2024] changes the visual–textual composition of matched problems, MMStar [Chen et al., 2024] emphasizes vision-essential evaluation and textual-shortcut control, and SeePhys Pro [Xiang et al., 2026] probes modality transfer and visual variable grounding. PhysAlign instead fixes the original problem and evidence anchors, separately measures Content Recognition and Role Grounding, and uses paired controls that supply correct local readings without revealing the grounding target. A protocol-level comparison with related benchmarks is provided in Appendix E (Table 18). 3 PhysAlign: An Evidence-Grounded Role Alignment Benchmark 3.1 Task Definition and Scope Each PhysAlign probe evaluates a local physical correspondence using the original problem text and diagram, an evidence anchor, a local question, and candidates identified by neutral aliases. The anchor specifies an image region or an exact text span. The model selects the corresponding physical role target and, when a separate reading target is defined, reports the content at the anchor. In Figure 1, anchorR1marks the tension symbolT, and the queried correspondence is the supporting cable (E4). A response with read: T and owner: E3 correctly recognizes the symbol but incorrectly assigns it to the rod. For this query, the target is the cable associated with the tension, rather than the body on which the force acts. The benchmark uses three probe types:T02-singleresolves an anchored mention to an entity;T03-image transcribes an anchored quantity label and identifies its owner; andT03-textidentifies the owner of an anchored textual quantity. Each query preserves the relation specified by its source annotation, and a quantity’s owner may be an object, force, or field. Joint evaluation requires separately defined targets for content recognition and role grounding; probes without an independent reading target receive grounding scores only. This decomposition concerns observable responses and makes no assumption about separate recognition and grounding modules within a model. 3.2 Evidence-Linked Annotations and Probe Construction We construct probes from reviewed annotations linked to the original problem evidence. Figure 2 summarizes the pipeline, and Appendix A provides provenance records and admission criteria. 3 1.1 Source Curation & Standardization Source datasets Example: physics problem Unified format LiveK12Bench SeePhys OlympiadBench PhysElite A block of mass m₁ on a rough incline (angle θ, friction coefficient μ) is connected by a light rope over a pulley to a hanging mass m₂. What is the acceleration of the system? θ μ m₁ m₂ text image source id version 1.2 Evidence-linked Annotation (Answer-blind) Passes 1–4 are answer-blind Pass 1 Visual primitives Pass 2 Entity grounding Pass 3 Text & quantity Pass 4 퐺 표푏푠 construction Pass 5 Solution–evidence alignment θ μ m₁ m₂ θ μ m₁ E1: block E3: pulley E4: incline E2: mass θ μ m₁ m₂ T • bounding boxes • OCR / text regions • arrows / geometry • link entities to regions • categorize objects • recognize symbols • extract quantities on hangs E1 block E3 pulley E4 incline E2 mass m μ θ m₂ Human review & correction Solution derivation ... m₂ 2.1 Deterministic Probe Compilation Region–Entity–View (R / E / V) ... θ μ m₁ m₂ R Candidate entities E E1: block (m₁) E2: mass (m₂) E3: pulley ... Localized view V m₁ Probe types T02-singleT03-imageT03-text Resolve R → E Read R + Groundto E ... Read anchored label and select its owner. Ground the anchored mention to an entity. Given T→ Groundto E Select the owner of the anchored textual quantity. 2.2 Paired Input Construction for Evaluation Base(model reads content) +GT content(provided) m₁ “m₁” Input: question + image Model reads local text from the image. Input: question +image +transcription Readingsis provided, entity owner is hidden. Structural checks (per probe) uniquenessvalid localization candidate distinction dependency completeness θ μ m₂ 3 Quality Control & Packaging 1 Public-input answer first, then check with private answer and source evidence. Model answer Private answer + source evidence Accept CorrectRegenerate 2 Public Private questions ground truths (content and roles) Probes (anchors and candidates) mappings (evidence–role) images/ (figures) membership (splits & metadata) ... ... 3 • Parent-problem split • Same-source probes stay together. • Scores aggregated by parent problem. PhysAlign Benchmark Base+GT PublicPrivate Blind-first QC Public / Private Split & Aggregation Check evidence, entity-text grounding, quantity accuracy, and relations Approved 퐺 표푏푠 (observational annotations) T T T Gaokao-M -Physics PhyX-OE Figure 2: Data construction and quality assurance in PhysAlign. Standardized source problems undergo answer-blind annotation (Passes 1–4), human review, and probe compilation. Pass 5 aligns reference solutions with evidence for analysis only and does not supply probe targets. Paired base/+GT inputs share the original problem, evidence anchors, questions, and candidates; +GT supplies correct local readings without disclosing the queried role target. Structural checks and instance audits precede packaging, which separates public inputs from private targets and scoring mappings. We use GPT-5.6 Sol [OpenAI, 2026a] with high reasoning effort to draft structured annotations, which are then reviewed and corrected against the original text and diagrams. Passes 1–4 record visible primitives, entity references, explicit quantities and conditions, and their supporting evidence without access to reference answers or solutions. Only these reviewed observational records supply probe targets. Pass 5 separately aligns reference solutions with evidence for analysis. Fixed templates convert supported annotation fields into local questions. Candidates use neutral aliases linked to image regions or text spans. Alias assignments and candidate order are randomized once and held fixed across models. Single-target probes require at least two valid, type-compatible candidates; set-valued templates retain all correct targets. Candidate descriptions exclude answer-bearing role associations. Reference readings, role targets, alias mappings, and source-annotation links are stored separately from public inputs. Eligible probes also receive the paired +GT input defined in Section 4.1. Quality assurance comprises source-annotation review, model-assisted probe screening, and human verification of exported probes. Checks assess evidence localization, question semantics, candidate completeness and distinguishability, target support, and information leakage. During human verification, two coauthors independently answer the public probe before consulting private targets and source annotations; a third reviewer adjudicates disagreements. Ambigu- ous or unsupported records are withheld pending resolution. Appendix A.3 details construction-stage verification; Appendix A.5 reports the post-construction audit. 3.3 Dataset Scale and Coverage PhysAlign contains 986 parent problems from six sources and 3,341 unique localized probes; 553 probes from 385 parents also support paired +GT inputs, without adding unique probes. Table 1 reports source coverage and the parent-disjoint release partition. Evaluation pools are drawn from the full release (development and test combined); metric-specific eligibility and observed coverage are defined in Section 4.2. The source problems span mechanics, electromagnetism, optics, thermal physics, waves, modern physics, and astro- physics (Figure 3). Mechanics and electromagnetism account for most parent problems under the estimated domain assignments. The illustrated problems provide context for the local correspondences evaluated by PhysAlign. Ap- 4 A Diverse Benchmark of Physics Problems SourceProblems LocalizedProbes Paired+GTInputs 0% 10% 20% 30% 40% 50% 986 SourceProblems 3,341 LocalizedProbes 553 Paired+GTInputs Mechanics 478/1,672/296 Electromagnetism 267/893/168 Optics 50/153/22 Thermodynamics 48/183/16 Waves & Acoustics 35/97/11 Astronomy & Astrophysics 11/25/1 Modern, Atomic & Nuclear Physics 19/72/6 Representative Problem Examples Mechanics A block moves at constant speed under a horizontal force F, and also when F is applied at 60∘above the horizontal. What is the kinetic friction coefficient μ? Optics For the three-lens telescope shown (f1=10cm, f2 =f3= 2cm; D1=4cm, D2=D3 =1.2 cm ), what are the position and diameter of the exit pupil? Electromagnetism Two identical charged part - icles, a and b, enter a unif- orm magnetic field directed into the page at O and exit at M and N , respctively, with OM=MN.What is v b / v a ? Thermodynamics Moist air rises adiabatically over a mountain, forming clo -uds at M 1 where P=84.5kPa. Given an initial pressure of 100kPa, what is the temperat -ure T 1 at M 1 ? Waves & Acoustics A string of length L = 1.0 mis fixed at both ends. If thewave speed is v = 100 m/s,what is the fundamentalfrequency? Astronomy & Astrophysics A blackbody moon of mass M s orbits a reflecti- ve, polarizing planet of mass M p and radius R p at orbital radius R s . A laser of power P 0 ,offset by R p /2, reflects onto the moon. If the polarizat -ion and reflection angles are equal, how much energy does the moon absorb before aligning with the observer? Modern, Atomic & Nuclear Physics A nucleus consists of closely packed nucleons of radius rN=0.85fm, with packing factor f=π/6. Given A nucle - ons (Z≈A/2) of mass mN= 1.67×10 −27 kg each, what are the nuclear radius R, mass density ρm, and charge density ρ c ? MiddleSchool 7 knowledge domains in total University-level Physics Cross-disciplinary 78/246/33 Figure 3: Domain coverage and representative physics problems in PhysAlign. Left: Distributions of source problems, localized probes, and paired +GT inputs across seven physics domains and a cross-disciplinary group. Each radar series shows percentages within its corresponding collection; counts beside each group follow the same order. Right: Representative question–diagram examples from the seven domains, illustrating an educational span from middle school to university-level physics. Table 1: Source coverage and release partition of PhysAlign. (a) Source coverage Source datasetParents Parents with +GT SeePhys [Xiang et al., 2025]268119 LiveK12Bench [Wang et al., 2026]25088 PhysElite [Xu et al., 2026]22079 OlympiadBench a [He et al., 2024]20968 Gaokao-M-Physics [Zong and Qiu, 2024]2317 PhyX-OE [Shen et al., 2025]1614 Full release986385 (b) Development/test partition SplitParents Local probes Paired +GT Development7920890 Test9073,133463 Full release9863,341553 Evaluation. Both splits are used jointly. Paired +GT inputs are variants, not additional unique probes. a English physics competition subset. pendix A.6 reports estimated domain composition, probe density, and paired coverage, with parent-level and probe-level statistics reported separately. 4 Evaluation Protocol and Metrics 4.1 Paired Evaluation with Ground-Truth Local Content For each eligible probe, the Base input contains the original problem, local question, evidence anchors, and candidates. The matched +GT input additionally supplies verified ground-truth (GT) readings of the specified local content while withholding the queried physical role. Both conditions use the same question, candidates and aliases, output schema, inference settings, and scoring rules, and are run in independent sessions without shared history, predictions, or feedback. Supplied readings are inputs and are therefore excluded from recognition scoring. Model-independent rules determine which readings may be supplied. In Figure 1, +GT may state that R1 readsT, but not thatTdenotes the tension in the supporting cable (E4), since the latter reveals the grounding target. Candidate- side information, when present, is selected symmetrically across candidates. The paired comparison measures how grounding changes when verified local content is supplied. It does not provide a complete physical description or isolate the causal effect of correcting OCR errors. 5 Table 2: Recognition, grounding, paired local-reading controls, and independent original-problem solving. Joint (L)All (G)Reading control (P m )Solving ModelCAcc ↑GAcc base L ↑ JAcc ↑GAcc ↑GAcc base P m ↑GAcc +GT P m ↑ ∆G P m SolveAcc ↑ Qwen3.5-4B53.9746.5132.0940.2146.5154.61+8.109.94 Qwen3.5-9B65.8154.5537.5540.5254.5556.53+1.9812.12 Qwen3.5-27B84.1480.2968.1284.1180.2982.00 +1.7118.39 InternVL3.5-8B2.1233.361.0523.5233.3634.81+1.457.47 GPT-6 Astra89.4483.3177.1381.9483.5984.90+1.3271.72 Gemini 3.8 Flash85.2079.3370.5990.7178.6780.86+2.2021.31 Scores are percentages;SolveAccis mean normalized credit, not binary accuracy (Appendix B.4). Parent/probe counts:L, 311/412; G, 986/3,341;P m , 311/412 (open weights), 304/397 (APIs); solving, 986 parents.GAcc = GAcc base G ;∆G P m is the paired change in percentage points, rounded independently. Coverage details: Appendix B.5. Blue bold/underlined red: column maxima/minima; green: positive changes (largest in bold), not significance. 4.2 Deterministic Scoring and Balanced Aggregation LetGcontain all 3,341 grounding probes from 986 parents. The joint setL⊆Gcontains 412 probes from 311 parents with independent reading targets; the paired-eligible release setP ⊆Gcontains 553 probes from 385 parents with valid +GT inputs. Recognition and joint metrics useL, and overall grounding usesG. Base/+GT comparisons use modelm’s T03-image probes with both response records available,P m ⊆L∩P (Appendix B.5). For predicted contentˆc j and candidate identifierˆu j , letc j andr j denote the reference content and role target. Given the fixed content normalizer N j and candidate mapping φ j , C j = 1[N j (ˆc j ) = N j (c j )], G j = 1[φ j (ˆu j ) = r j ], J j = C j G j .(1) C j andJ j are defined onL, whereasG j applies toG. Content must match the specified anchor rather than an unordered inventory. Grounding is scored independently, whileJ j requires correct recognition and grounding for the same local premise. Base and +GT use the same grounding rule. Normalization, multi-field and set matching, and format-error handling are fixed before evaluation; these local scores use no open-ended language-model judge. For an evaluation setS , Agg S first averages probes within each parent–task-type group, then parents within each task type, and finally task types using prespecified weights (see Appendix B.3 for the formal definition). We report CAcc = Agg L (C), JAcc = Agg L (J ), GAcc v S = Agg S (G v ), v ∈base, +GT.(2) This aggregation prevents annotation-rich parents from dominating a task type. Overall grounding isGAcc base G . Paired scores use identical Base/+GT support and weights within eachP m , with∆G P m = GAcc +GT P m − GAcc base P m ; +GT performance is not assumed to be an upper bound. OnL,C + G − denotes the eventC j = 1,G j = 0, i.e., correct content recognition with incorrect grounding; the other sign combinations are defined analogously. Under the same aggregation, its weighted share isCAcc− JAcc, notCAcc− GAcc, andJAccis not the product of the marginal accuracies. We also reportGAcc base L for same-set comparisons. 5 Experiments and Diagnostic Analysis 5.1 Experimental Setup and Main Results We evaluate Qwen3.5-4B, -9B, and -27B [Qwen Team, 2026], InternVL3.5-8B [Wang et al., 2025b], GPT-6 Astra [Ope- nAI, 2026b], and Gemini-3.8-flash [Google DeepMind, 2026] using pools drawn from both release partitions, not test alone. Table 2 reports local metrics onL,G, andP m , and independent solving on all 986 parents.SolveAcc averages normalized problem credit: scripted binary credit for choice and fill-in items, and expert-assigned partial credit for short-answer and calculation items (Appendix B.4). Local metrics use balanced weights; solving weights parents equally. Repeated Base scores reflect shared probes and weights; API deviations reflect response coverage (Appendix B.5). Inference settings are in Appendix C.1. Two patterns emerge from the aggregate results. Within the Qwen3.5 family, scaling from 4B to 27B coincides with substantial gains in both content recognition and joint performance:CAccincreases from 53.97% to 84.14%, while JAccrises from 32.09% to 68.12%. Yet stronger content recognition does not eliminate the gap between reading local content and grounding its physical role. On the same joint setL, GPT-6 Astra [OpenAI, 2026b] obtains 89.44%CAcc, 6 0255075100 Weighted joint share on L (%) 32.121.914.431.6 37.628.317.017.2 68.116.012.2 32.365.6 77.112.3 70.614.6 050100 46.51 54.61 54.55 56.53 80.29 82.00 33.36 34.81 83.59 84.90 78.67 80.86 −5051015 Change (p) +8.10 +1.98 +1.71 +1.45 +1.32 +2.20 (a) Reading-grounding decomposition(b) Paired local-reading control Δ (95% CI) C+ G− Qwen3.5-4B 21.88 Qwen3.5-9B 28.26 Qwen3.5-27B 16.01 InternVL3.5-8B 1.07 GPT-6 Astra 12.32 Gemini-3.8-flash 14.61 C+ G+C+ G−C− G+C− G−+GT All models: N = 311, n = 412 Base Open weights: 311/412; APIs: 304/397 (N/n) GAcc on P m (%) Figure 4: Recognition–grounding decomposition and paired controls. (a) Weighted outcome shares onL(N = 311,n = 412). (b) Base/+GT grounding and paired changes with 95% confidence intervals onP m ; model-specific coverage is shown above the panel (Appendix B.5). N /n count parents/probes. +GT supplies local readings, not perfect perception. 83.31%GAcc base L , and 77.13%JAcc; the corresponding results for Gemini-3.8-flash [Google DeepMind, 2026] are 85.20%, 79.33%, and 70.59%. Thus, gains in scale and aggregate accuracy do not by themselves ensure consistent recovery of local content and its physical role. Marginal accuracies also cannot reveal whether a grounding error follows a recognition error or occurs after the content has already been read correctly. We therefore decompose both outcomes on the same local probes. 5.2 Recognition–Grounding Decomposition and Local-Reading Controls Figure 4(a) decomposes the joint setLinto four outcomes according to whether content recognition and grounding are correct. The central result is that correctly reading the content at a specified location does not guarantee correct grounding of its physical role. Excluding InternVL3.5-8B [Wang et al., 2025b], whose local recognition is nearly absent, the weighted share of content-correct but grounding-wrong events (C + G − ) ranges from 12.32% to 28.26% across the other five models. The shares are 21.88%, 28.26%, 16.01%, 12.32%, and 14.61% for Qwen3.5-4B [Qwen Team, 2026], Qwen3.5-9B [Qwen Team, 2026], Qwen3.5-27B [Qwen Team, 2026], GPT-6 Astra [OpenAI, 2026b], and Gemini-3.8-flash [Google DeepMind, 2026], respectively. Its recurrence across model families and scales shows that this is not an isolated failure: models can read the specified text or value correctly and still assign it to the wrong physical entity or role. TheC + G − quantity is a weighted event share over the full joint set, not a conditional grounding error rate given correct recognition. Its magnitude therefore also depends on how often a model reads the content correctly. InternVL3.5- 8B [Wang et al., 2025b], for example, has only 1.07%C + G − but also only 2.12%CAcc; the small event share does not imply more reliable role grounding. The four-way decomposition diagnoses failure modes hidden by marginal accuracy rather than providing another model ranking. Figure 4(b) reports positive point estimates for Base-to-+GT changes of 1.32–8.10 percentage points on model-specific observed setsP m . The transition analysis for the four open-weight models (Appendix D.1, Table 12) reveals both repairs and harms. For Qwen3.5-4B [Qwen Team, 2026], +GT repairs 13.35% and harms 5.25% of the weighted probe mass, yielding a net gain of 8.10 points (95% CI [3.88, 12.50]). The other three net-change intervals include zero. Net gains therefore do not imply uniform item-level benefits. Even under +GT, Qwen3.5-27B [Qwen Team, 2026], GPT-6 Astra [OpenAI, 2026b], and Gemini-3.8-flash [Google DeepMind, 2026] retain weighted grounding error rates of 18.00%, 15.10%, and 19.14%, respectively, on their observed P m . Supplied readings therefore neither guarantee role assignment nor define a grounding upper bound. Since +GT withholds the correct owner or role and leaves other visual and spatial uncertainties unresolved, the paired changes do not isolate a pure perception effect, and residual errors cannot be attributed exclusively to physical reasoning. 7 Table 3: Grounding validity and robustness: rule-based baselines, image removal, and candidate reordering. Rule-based baselines (GAcc ↑) Random candidates: 18.69%Nearest region: 76.82% Information controls and interface robustness ReferenceNo imageReordered ModelGAcc ↑GAcc ↑∆G (p)GAcc ↑ Qwen3.5-4B [Qwen Team, 2026]40.2138.78−1.4341.93 Qwen3.5-9B [Qwen Team, 2026]40.5243.53 +3.0143.02 Qwen3.5-27B [Qwen Team, 2026]84.1163.72−20.3984.48 InternVL3.5-8B [Wang et al., 2025b]23.5249.87+26.357.24 GPT-6 Astra [OpenAI, 2026b]81.9475.57−6.3784.27 Gemini 3.8 Flash [Google DeepMind, 2026]90.7176.90−13.8190.82 Accuracies are percentages (↑). Reference is each model’s GAcc (all) in Table 2; Reordered permutes candidates. ∆G = GAcc no image − GAcc base is in percentage points, rounded independently. Random is the evaluation-weighted expectation of1/K j on single-target probes; rule scores here use their original pools. Same-support comparisons appear in Appendix D.2. Blue bold/underlined red: model-accuracy maxima/minima; green/red shading: positive/negative changes (extremes in bold), not significance or better/worse rankings. 5.3 Validity: Visual Evidence, Geometry, and Interface Robustness Table 3 evaluates three alternative explanations for the grounding results: dependence on visual input, simple geometric regularities, and the candidate interface. Each control addresses a distinct question and is interpreted within its own scope. Removing the image substantially reduces grounding accuracy for the strongest models. Qwen3.5-27B [Qwen Team, 2026], GPT-6 Astra [OpenAI, 2026b], and Gemini-3.8-flash [Google DeepMind, 2026] lose 20.39, 6.37, and 13.81 percentage points, respectively. The consistent declines show that their high reference scores cannot be explained entirely by the text-only interface or candidate priors. Weaker models do not follow the same pattern:Qwen3.5-9B[Qwen Team, 2026] and InternVL3.5-8B [Wang et al., 2025b] gain 3.01 and 26.35 points without the image. Visual input therefore interacts strongly with the model and the evaluation interface rather than conferring a uniform benefit. Removing the image can also alter the evidence anchors and the identifiability of candidate targets. This control diagnoses visual-input sensitivity only when the remaining input still defines a valid task; it does not isolate the exact contribution of visual reasoning. The spatial baseline requires a same-support comparison rather than subtraction from overallGAcc. On the fixed applicable subset (665 parents, 1,639 probes), Qwen3.5-27B [Qwen Team, 2026] achieves 86.35% against 76.82% for nearest-region, a 9.53-point margin (95% CI [6.10, 13.06]). We further define H-hard as the probes on which the frozen nearest-region rule is incorrect, independently of model predictions (Appendix D.2). On this subset (271 parents, 394 probes), Qwen3.5-27B [Qwen Team, 2026] achieves 73.22% grounding accuracy against 18.76% for same-support random selection. On its joint-reading intersection (69 parents, 74 probes), the model’s C + G − share remains 29.71% [20.14, 39.84], with a 34.75% [23.77, 46.55] grounding error rate conditional on correct reading. The tested proximity rule thus does not fully explain this model’s grounding performance, and reading–grounding disagreement persists where the rule fails. This does not exclude other shortcuts. Candidate reordering changes aggregateGAccby at most 2.50 percentage points for five models. InternVL3.5-8B [Wang et al., 2025b] is the exception, falling from 23.52% to 7.24%. Thus, five models exhibit limited aggregate sensitivity to the tested reordering, not established permutation invariance. Similar accuracies can conceal offsetting item-level changes; alias preferences and sensitivity to other orderings remain untested. Together, these controls distinguish sensitivity to image evidence, performance beyond the tested proximity rule, and robustness to the tested candidate reordering. They motivate reporting geometry-defined hard-subset performance alongside aggregate grounding, without identifying internal strategies or establishing a shortcut-free benchmark. We next examine whether successful original-problem solving certifies accurate local grounding. 5.4 Association with Original-Problem Solving We next compare parent-level grounding by binary final-answer correctness, a diagnostic distinct from the partial-credit SolveAcc. Figure 5 uses 311 parents and 412 local probes from the full release. Solving and probing use independent contexts; Appendix D.3 details the grouping and aggregation. 8 0255075100 Mean grounding accuracy (%) 39.19 41.31 40.67 44.61 85.51 87.89 23.37 25.88 83.05 85.53 92.08 92.36 024 Change (p) +2.12 +3.94 +2.39 +2.51 +2.48 +0.28 (a) Grounding by solving outcome(b) Difference All models: N=311, n=412 Correct − wrong Qwen3.5-4B Qwen3.5-9B Qwen3.5-27B InternVL3.5-8B GPT-6 Astra Gemini-3.8-flash Solve-wrongSolve-correct Figure 5: Local grounding by binary final-answer correctness, not by partial-creditSolveAcc. (a) Parent-level means for solve-wrong and solve-correct groups (N = 311parents,n = 412probes). (b) Correct-minus-wrong difference in percentage points. Estimates use independent solving and probing runs. All six models have higher grounding point estimates on correctly solved problems, but the differences span only 0.28–3.94 percentage points (Figure 5(b)). This common direction suggests a modest descriptive association, not equivalence between the two tasks. The absolute scores provide the sharper contrast (Figure 5(a)): on correctly solved problems, Qwen3.5-27B [Qwen Team, 2026] and GPT-6 Astra [OpenAI, 2026b] still incur local grounding error rates of 12.11% and 14.47%, respectively. Conversely, Gemini-3.8-flash [Google DeepMind, 2026] retains 92.08% grounding accuracy on incorrectly solved problems. Correct answers thus do not certify accurate local reports, and high local grounding accuracy does not certify successful problem solving. PhysAlign therefore complements solution accuracy by testing reported evidence–role correspondences that final answers do not resolve. This conclusion concerns observable behavior, not the premises used internally: the model- specific grouping and independent contexts neither identify which local evidence entered a solution nor establish that repairing a grounding error would change its answer. Appendix D.3 provides the aggregation details, full estimates, and interpretive boundaries. 6 Conclusion PhysAlign complements solution accuracy with anchored probes that separate content recognition from physical-role grounding. Our results show that correct recognition does not guarantee grounding, reference readings resolve only some grounding errors, and correct final answers can coexist with incorrect reported premises. These findings motivate models that verify evidence–role correspondences through query-conditioned local checks and are trained with hard negatives that preserve content while swapping entities or roles. Our findings concern explicit reports under fixed anchors and candidates in static diagrams, not complete scene understanding, internal model states, or premises used in a particular solution. Domain coverage is uneven, and whether such targeted training improves grounding and problem solving or transfers to dynamic settings remains to be tested. References Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, et al. SeePhys: Does seeing help thinking? – benchmarking vision-based physics reasoning. Advances in Neural Information Processing Systems, 38, 2025. Song Dai, Yibo Yan, Jiamin Su, Dongfang Zihao, Yubo Gao, Yonghua Hei, Jungang Li, Junyan Zhang, Sicheng Tao, Zhuoran Gao, et al. PhysicsArena: The first multimodal physics reasoning benchmark exploring variable, process, and solution dimensions. In EMNLP (Findings), pages 17290–17316, 2025. Kun Xiang, Terry Jingchen Zhang, Zirong Liu, Bokai Zhou, Yueling Tang, Junjie Yu, Jiacong Lu, Shangrui Huang, Heng Li, Likui Zhang, et al. SeePhys Pro: Diagnosing modality transfer and blind-training effects in multimodal rlvr for physics reasoning. arXiv preprint arXiv:2605.09266, 2026. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. 9 Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. Lintao Wang, Encheng Su, Jiaqi Liu, Pengze Li, Peng Xia, Jiabei Xiao, Wenlong Zhang, Xinnan Dai, Xi Chen, Yuan Meng, et al. PhysUniBench: An undergraduate-level physics reasoning benchmark for multimodal models. arXiv preprint arXiv:2506.17667, 2025a. URL https://arxiv.org/abs/2506.17667v1. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5238–5248, June 2022. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2023. URLhttps://arxiv.org/abs/2210. 01936. Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. VL- Checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221, 2022. Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. SugarCrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36:31096– 31116, 2023. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024. OpenAI. GPT-5.6 Sol model. OpenAI API Documentation, 2026a. URLhttps://developers.openai.com/ api/docs/models/gpt-5.6-sol. Accessed: 2026-09-15. Xiaohan Wang, Mingze Yin, Yilin Zhao, Gang Liu, and Dian Li. LiveK12Bench: Have large multimodal models truly conquered high school-level examinations? arXiv preprint arXiv:2605.26781, 2026. Ruoran Xu, Wending Gao, Liyunfeng Chen, Aixin Shi, Haoyu Cheng, Zixiang Fang, Yiqiang Zou, and Qiufeng Wang. PhysElite: How far are LLMs from solving olympiad-level physics problems? arXiv preprint arXiv:2608.25097, 2026. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850, 2024. Yi Zong and Xipeng Qiu. Gaokao-m: A chinese human-level benchmark for multimodal models evaluation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8817–8825, 2024. Hui Shen, Taiqiang Wu, Qi Han, Yunta Hsieh, Jizhou Wang, Yuyue Zhang, Yuxin Cheng, Zijian Hao, Yuansheng Ni, Xin Wang, et al. Phyx: Does your model have the" wits" for physical reasoning? arXiv preprint arXiv:2505.15929, 2025. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen.ai/blog?id= qwen3.5. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025b. OpenAI. GPT-6 Astra model. OpenAI API Documentation, 2026b. URLhttps://developers.openai.com/ api/docs/models/gpt-6-astra. Accessed: 2026-09-14. Google DeepMind.Gemini 3.8 Flash: Model card.Google DeepMind Model Card, 2026.URLhttps: //deepmind.google/models/model-cards/gemini-3-8-flash/ . Published September 2, 2026. Accessed September 16, 2026. 10 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain- of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. PhysReason: A comprehensive benchmark towards physics-based reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16593–16615, 2025. Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. AI2D-RST: a multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55(3):661–688, 2021. Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693–6702. IEEE, 2019. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. URL https://arxiv.org/abs/1904.08920. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14375–14385. IEEE, 2024. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé, I, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. 11 A Dataset Construction, Annotation, and Quality Assurance This appendix supplements the construction protocol in Section 3.2 and the dataset overview in Section 3.3. Source coverage and split statistics are reported in Table 1 in the main text; here we provide provenance records, annotation details, review procedures, release accounting, and a post-construction quality audit. Unless an individual split or candidate pool is specified, counts refer to the full retained release, combining development and test. Evaluation pools are drawn from this release; observed paired coverage is distinguished from release eligibility in Appendix B.5. Split-specific counts describe the release inventory, not separate performance evaluations. Section A.5 reports an audit of the retained inventory and the disposition of flagged probes in a subsequent release candidate. A.1 Source Problems and Provenance Source records. Table 1 lists the six source datasets and their retained parent counts, rather than the sizes of the upstream datasets. For OlympiadBench [He et al., 2024], we use its English physics competition subset. Each source record preserves the original image, statement, question, answer options when present, reference answer or solution, and provenance identifiers. PhysAlign augments these records with evidence-linked observational annotations, local probes, candidate mappings, recognition packets, and scoring records. The original physics problems are reused rather than newly authored. Deduplication and split propagation. Multiple probes can originate from the same parent problem and therefore do not represent independent source problems. The merge procedure deduplicates exactly identical public inputs and propagates split assignments and isolation decisions through original-image hashes and existing recorded groups. The current merge introduced no additional exclusions. All retained probes from a parent remain in the same split, with membership recorded inprivate/membership.json. These checks address exact-input duplication and recorded grouping; they do not establish the absence of semantic near-duplicates or prior model exposure to the original problems. A.2 Observational Annotation and Probe Compilation Pass-level outputs. The observational stage in Figure 2 comprises four answer-blind passes. Pass 1 records visible primitives, text, and locations; Pass 2 links them to physical entities; Pass 3 records textual mentions, explicit quantities, constraints, and query targets; and Pass 4 consolidates directly supported observations and evidence links intoG obs . Correct answers and reference solutions are excluded from these four passes. Pass 5 aligns atomic steps of an existing reference solution with the observations they use. It is reserved for analysis: it neither re-solves the problem nor supplies probe targets or solution-derived corrections to G obs . Exported fields and mappings. The interfaces in Section 3.1 exportreferentforT02-single,readand ownerforT03-image, andownerforT03-text. Each instance associates an evidence anchor (R), neutral candidate entities (E), and an original or localized view (V). Targets are compiled from reviewed observational fields, while alias-to-entity mappings remain on the scoring side. Templates preserve the physical meaning of their source fields. In particular, a quantity’sowner_ididentifies the entity to which that quantity belongs; the owner may be an object, force, field, or another physical entity. Identifying the force entity whose magnitude is shown, for example, differs from identifying the object on which that force acts. The compiler does not substitute one relation for the other. Appendix B specifies the input/output interfaces and +GT information rules. A.3 Quality Control and Admission Rules Verification and correction. Quality control checks both the source annotations and the exported probes. Source verification compares transcription, localization, entity correspondence, quantity ownership, and evidence support against the original problem. Exported-probe review examines the complete public question, evidence anchor, candidate set, and scoring target. Structural checks validate references, dependencies, mappings, and localization; semantic correctness, answer uniqueness, and candidate distinguishability require review against the original text and images. Human verification follows the blind-first procedure in Section 3.2. Two coauthors independently answer the public probe before consulting the private target and source annotations. Direct approval requires agreement between their independent answers and successful quality checks; disagreements proceed to a third reviewer for adjudication. Review also checks candidate completeness, transcription, localization, and information leakage. Corrections require 12 regeneration and rechecking, while shared-source or rule-level issues require examination of related instances. The construction-stage procedures above are distinct from the post-construction audit of retained probes in Section A.5. Revision of further-review candidates. We used GPT-6 Astra [OpenAI, 2026b] to revise the 1,089 candidates marked as requiring further review at initial screening. The first stage checked probe content against the original text and images and corrected transcription and reading-target errors where applicable. The second stage compared evidence regions, candidate entities, and target relations with the original images to determine the required localization and semantic corrections. The revised probes then underwent item-by-item human verification using the same blind-first procedure. These decisions apply to the revised versions; the original screening states remain part of the initial candidate accounting. Section A.4 reports the final review outcomes and distinguishes the supplementary candidates from the retained benchmark. Admission and audit interpretation. The recorded construction-stage test-parent admission condition is n rej i ≤ 2, wheren rej i counts generated candidate probes for parentirecorded as rejected. This condition governs parent eligibility; it neither admits rejected probes nor overrides subsequent risk isolation. Candidates with unresolved screening states or incomplete calls were excluded from the reported benchmark. Subsequent revisions and their review outcomes are accounted for separately. Table 4(a) records model-screening states, whereas panel (c) reports final human-review outcomes for revised probes. Model approval does not establish agreement between human reviewers, and a further-review state does not itself indicate human disagreement. Neither the screening counts nor final post-revision approvals are used to infer pre-adjudication agreement or a human-performance ceiling. Input–scoring separation. Model-visible files contain base and +GT inputs together with the images available for inference. Reference targets, source and candidate mappings, parent membership, source snapshots, and review records are stored separately for scoring and audit. A.4 Candidate Screening and Release Retention Table 4(a) accounts for 5,594 generated probes from 1,084 candidate parents. Initial model screening yielded 3,370 approvals, 1,128 rejections, 1,089 candidates requiring further review, and seven incomplete or failed screening calls. These four states are mutually exclusive. Risk isolation subsequently excluded 29 model-approved candidates, leaving 3,341 retained probes from 986 parents, as shown in panel (b). The retained probes constitute 59.72% of generated candidates. The other 2,253 candidates were not admitted to the reported benchmark at that stage: 1,128 model rejections, 1,089 further-review cases, seven incomplete calls, and 29 risk-isolated approvals. These initial states do not imply that every excluded candidate was semantically incorrect; subsequent revisions are reported separately. Among the 986 retained parents, 184 retain their complete generated probe set, whereas 802 retain only an approved subset. Here, complete means that all generated candidates for the parent are retained, not that every physical fact in the problem is annotated. Partial retention means that only probes satisfying admission requirements enter the benchmark; candidates rejected or unresolved at screening were not admitted in those versions. The remaining1,084− 986 = 98 candidate parents contribute no retained probe. The available accounting does not attribute this parent-level difference to a single exclusion cause. Human re-review and supplementary candidates. Panel (c) reports the outcomes of the revision procedure in Section A.3. All 1,089 revised probes underwent item-by-item human review and received final approval; none remained rejected or unresolved. These are final outcomes after revision, not first-pass acceptance rates or measures of inter-reviewer agreement. The approved revisions form a separate supplementary candidate pool and were not used in the reported model evaluation. Dataset counts, evaluation denominators, and experimental results therefore refer to the retained 3,341 probes from 986 parents. They characterize these retained local tasks rather than the full unfiltered candidate pool. A.5 Post-construction Quality Audit Sampling protocol.We audited 2,400 probes from the retained 3,341-probe inventory (71.83%), separately from the revision of the 1,089 supplementary candidates in Section A.4. Table 5(a) reports the allocation using the source-group labels recorded in the audit report. All 201 probes in the three smallest groups were inspected; the remaining 2,199 audit 13 Table 4: Candidate-probe screening, release admission, and human re-review. Panel (a) records the initial model-screening states of 5,594 generated probes; panel (b) reports admission within the model-approved subset. Panel (c) reports final human-review outcomes for revised versions of the 1,089 candidates initially requiring further review. Recorded status or outcomeProbes (a) Initial model-screening states Approved by model screening3,370 Rejected by model screening1,128 Further review required at initial screening1,089 Incomplete / call error7 Total generated candidates5,594 (b) Admission within model-approved candidates Withheld through risk isolation29 Retained probes3,341 (c) Human re-review of revised candidates Revised probes reviewed by humans1,089 Approved after review; retained in supplementary pool1,089 Rejected or unresolved after review0 Added to the reported benchmark0 The states in (a) sum to 5,594. The 29 isolated candidates are included in the 3,370 approvals, so (b) sums to29 + 3,341 = 3,370; panels must not be added together. Panel (c) follows revised versions of the 1,089 further-review candidates in (a). Its coverage and outcome rows describe the same probes and must not be summed. These revised probes form a separate supplementary candidate pool and were not used in the reported evaluation. Human outcomes are final post-revision decisions, not first-pass acceptance or inter-reviewer agreement. Incomplete calls concern construction-stage screening. slots were allocated approximately in proportion to the sizes of the three largest groups. Within partially audited groups, probes were sampled without replacement using a fixed-seed hash ordering oflogical_probe_id. Selection did not depend on model scores, problem difficulty, current gold answers, or previous review outcomes. First-pass findings.Of the audited probes, 2,332 passed without revision (97.17%); 68 (2.83%) required correction. The flagged cases involved incomplete local transcriptions (including symbols, subscripts, superscripts, and units), inaccurate evidence localization or entity ownership, and inconsistencies among candidate entities, gold answers, and private mappings. These semantic errors require comparison with source evidence and cannot be detected by file-existence, checksum, or schema checks alone. Correction and reinspection. The 68 flagged probes underwent two-stage correction with GPT-6 Astra [OpenAI, 2026b] at high reasoning effort. The first stage corrected local readings and clearly displaced reading locators; the second jointly checked evidence regions, candidate entities, entity ownership, and gold answers against the source images. Reinspection applied the original admission criteria: accurate reading, valid localization, unambiguous ownership, and consistent answer mapping. Of the 68 flagged probes, 66 passed reinspection (97.06%); two remained unresolved and were excluded from the subsequent release candidate. The audited subset therefore contained 2,398 accepted probes after correction and reinspection (99.92%; Table 5(b)). For each flagged probe, the audit trail records its identifier, source image, original annotation, issue description, correction history, and reinspection decision. Interpretation and release scope. The 97.17% first-pass rate describes annotations before audit-driven correction, whereas 99.92% is the post-correction acceptance rate within the audited subset. Neither measures inter-reviewer agreement, human task performance, or model accuracy. Because sampling fractions differ across groups, these pooled rates are descriptive audit statistics rather than sampling-weighted estimates for the full inventory; the audit does not assess the remaining 941 probes. The 3,341-probe sampling frame and the subsequent release candidate are distinct accounting stages. The audit alone does not establish the effect of annotation changes on the model results in Section 5. A.6 Composition, Problem Content, and Evaluation Coverage Counting units and paired coverage. The following statistics describe the retained benchmark; the supplementary candidate pool is excluded. We use the evaluation sets defined in Section 4.2. The retained dataset contains|G| = 3,341 unique grounding probes, each with a base input. A subsetP ⊆Gwith|P| = 553also has a paired +GT input variant. 14 Table 5: Post-construction quality audit of retained probes. (a) Sampling allocation. (b) Review outcomes within the audited set. (a) Sampling allocation Audit source group a Total probesAuditedCoverage (%) Local1,6791,17670.04 PhysElite79255469.95 OlympiadBench66946970.10 Expansion-v3123123100.00 Phyx-OE4040100.00 Gaokao3838100.00 Total3,3412,40071.83 (b) Audit disposition Review outcome b ProbesShare (%) Passed first review without revision2,33297.17 Passed after correction and reinspection662.75 Unresolved after reinspection; excluded20.08 Final accepted within audited set c 2,39899.92 a Labels follow the audit report; counts refer to probes, not the parent-level source counts in Table 1. b The first three rows partition the 2,400 audited probes. All shares in (b) use 2,400 as the denominator. c Final acceptance combines the first two outcome rows; it is not an additional category or an estimate of full-dataset accuracy. The 553 +GT instances are paired input variants rather than additional independent probes. Similarly, thereadand owneroutputs of aT03-imageinstance are two fields of one probe rather than two separate probes. The remaining 3,341− 553 = 2,788localized probes are valid retained tasks without a paired +GT variant; they should not be confused with the 2,253 original candidates excluded from the reported release. There are 385 parent problems with at least one retained paired +GT variant and 601 without one. Parent-level coverage is385/986 ≈ 39.05%, whereas probe-level paired coverage is553/3,341 ≈ 16.55%. These ratios answer different questions and are neither model-accuracy scores nor annotation-agreement measures. The joint reading–grounding set Lcontains 412 probes from 311 parents. Its reading-target eligibility differs from the paired-input eligibility ofP(553 probes from 385 parents); these two evaluation populations must not be conflated. Estimated domain composition. Table 6 reports the estimated parent-level domain composition of the retained benchmark. It distinguishes seven named physics-domain groups from the residual classifications “Cross-domain / uncertain” and “Unclassified.” The assignments are descriptive estimates rather than a verified fine-grained subject taxonomy. Figure 3 combines the two residual groups as “Cross-disciplinary” (78 parents), whereas Table 6 reports them separately. Mechanics and electromagnetism contribute 478 and 267 parents, respectively, accounting for approximately 75.6% of the release. Their 208 and 112 parents with paired +GT variants account for approximately 83.1% of parents with paired variants. Neither the full parent distribution nor its paired-input coverage is therefore domain-balanced. These are parent-level statistics and do not determine the domain distribution of the 553 paired probes, since a parent may contribute multiple +GT variants. Problem-content vocabulary and concept coverage. The coarse domain labels summarize where the source problems belong, but do not identify the physical quantities, objects, relations, and geometric concepts within them. Figure 6 complements the domain inventory with a lexical view of the retained source problems. The left panel organizes recurring problem vocabulary into concept families; the right panel shows prominent physics-related terms in the problem text. This analysis is descriptive: lexical frequency is not used to assign domain labels, construct candidates, determine probe eligibility, or compute evaluation scores. Probe density and split-specific coverage. Table 7 reports probe density and paired coverage for the development and test splits. For splits, localized-probe density is|G s |/N s and paired coverage is100|P s |/|G s |, whereN s is the number of retained parent problems. These are inventory summaries rather than performance metrics. 15 Table 6: Estimated domain composition and parent-level paired-input coverage of PhysAlign. Counts refer to parent problems; domain assignments are estimated, and paired-input coverage is measured within each group. All parentsPaired-input coverage Domain groupCountShare (%)PairedUnpairedRate (%) Mechanics a 47848.4820827043.51 Electromagnetism a 26727.0811215541.95 Optics505.07153530.00 Thermal physics / thermodynamics484.87103820.83 Waves and acoustics353.5592625.71 Modern, atomic and nuclear physics191.9341521.05 Astronomy and astrophysics111.121109.09 Residual classification groups b Cross-domain / uncertain606.09213935.00 Unclassified181.8351327.78 Full release986100.0038560139.05 Definitions. For groupd, letN d denote its parent count andH d the number of parents with at least one retained paired +GT variant. Share= 100N d /986; Rate= 100H d /N d ; Unpaired= N d − H d . Percentages are rounded to two decimals and may not sum to 100. a Gray-tinted rows highlight sample concentration: mechanics and electromagnetism together account for approximately 75.6% of all parents and 83.1% of parents with paired +GT variants. b These are residual classification labels rather than additional physics domains. All domain-level counts and derived percentages use estimated assignments. Figure 6: Problem-content profile of the retained PhysAlign source problems. The left panel hierarchically groups recurring physics-related vocabulary into broad concept families, while the right panel visualizes prominent terms appearing in the problem text. This lexical view complements the parent-level domain statistics in Table 6; it is not used for domain assignment, probe construction, eligibility filtering, or benchmark scoring. Across the retained release, the mean localized-probe count per parent is3,341/986≈ 3.39, with a median of 3 and a range of 1–6. Among the 385 parents with paired +GT variants, the mean paired-variant count is553/385≈ 1.44; this denominator excludes parents without paired variants. Using the split counts in Table 1, development and test have localized-probe densities of 2.63 and 3.45 probes per parent and paired-probe coverage rates of 43.27% and 14.78%, respectively. The release contains 553 paired-eligible probes from 385 parents inP. Reported Base/+GT comparisons use the observedT03-imagesubsetsP m ⊆ P, with identical probes and weights across conditions within each model 16 Table 7: Probe density, paired-input coverage, and supplementary statistics of PhysAlign. Panel (a) reports split-level density and paired-probe coverage; panel (b) provides full-release summaries with explicit counting units. Paired +GT variants are not additional unique probes. (a) Split-level density and paired-probe coverage SplitLocalized probes / parentPaired-probe coverage (%) a Development2.6343.27 Test 3.4514.78 Full release3.3916.55 (b) Full-release summaries and counting units StatisticValueDenominator / counting unit Parents with paired +GT inputs39.05%385/986 parents Paired +GT variants per paired parent b 1.44553/385 Localized probes per parent: median; range3; 1–6Parent-level summary Localized probes without a paired +GT variant2,7883,341− 553 probes Packaged image files c 18,640File-level count Definitions. For splits,N s is the parent count,G s is the localized-probe pool, andP s ⊆G s is the paired subset. Localized-probe density is|G s |/N s . a Paired-probe coverage is100|P s |/|G s |:90/208for development,463/3,133for test, and553/3,341overall. Gray-tinted cells highlight the split difference, not performance or statistical significance. b The denominator is the 385 parents with paired +GT variants, not all 986 parents. Reported Base/+GT scores instead use observed subsetsP m (Appendix B.5). c The 18,640 image files are not 18,640 unique source diagrams or independent physics problems. (Appendix B.5), not separate split-level evaluations. Comparing +GT onP m with Base on all ofGwould confound the input intervention with a change in evaluation population. Balanced performance aggregation and matched base/+GT comparisons follow Section 4.2. B Probe Specifications and Scoring This appendix explains how local tests are presented, how model outputs are converted into scores, and how results on different evaluation sets are aggregated. Data sources, annotation review, and admission statistics are provided in Appendix A; prompt templates, model configurations, and execution records are provided in Appendix C. Local-probe scoring covers theT02-single,T03-image, andT03-textinterfaces. Independent original-problem scoring is specified in Appendix B.4. B.1 Public Inputs, Evidence Anchors, and Candidates Each public probe comprises the original problem text and diagram, a local question, an evidence anchor, candidates, and output requirements. Image anchors identify explicit regions in a specified original image; text anchors identify exact spans in the original text. Localized views display these locations without modifying the original problem content. Candidates are presented through probe-specific neutral aliases linked to their image regions or text spans. Mappings from aliases to reference entities are retained on the scoring side; models need not generate internal entity identifiers. In the static-equilibrium example in Figure 1, anchor R1 marks the symbolT, while E2, E3, and E4 identify the block, rod, and supporting cable, respectively. The local question asks “What is the symbol in R1, and to which entity does it belong?” The reference associatesTwith the supporting cable (E4), whose tension it denotes. The illustrated response instead reportsread:Tandowner: E3 (rod). The resulting scores areC j = 1,G j = 0, andJ j = 0: content recognition is correct, but the physical ownership of the recognized symbol is not preserved. Entity identifiers are used here for exposition; actual responses use the corresponding public candidate aliases. The reference targets and scores shown in the figure are evaluation annotations, not inputs to the illustrated probe. When an eligible probe has a paired +GT variant, that variant may additionally supply “the symbol in R1 isT”, but must not reveal its ownership, such as “T denotes the tension in the supporting cable.” Candidate construction and display order are fixed before evaluation. Single-target queries contain at least two valid candidates, and the correct target must have a valid, distinguishable public representation. Existing labels in the 17 original image are retained, but answer-revealing descriptions derived from private annotations, such as “E4 is the cable associated withT”, are not added. Unresolved entity correspondences, ambiguous references, and indistinguishable candidates are not forced into single-target queries.T02-singlereturnsreferent;T03-imagereturnsread and owner; and T03-text returns owner. Reusable prompts are provided in Appendix C. B.2 Ground-Truth Local Content and Evaluation Pools The Base condition (also labeled Raw in the figures) and the +GT condition use the same original text and diagram, question, anchor, candidates, and output requirements, and are run in independent contexts; +GT only adds predefined ground-truth local recognition information. The supplied content is selected by rules independent of the tested model and its errors, and must neither directly give the tested role nor select only the correct candidate’s label or leak associations through field combinations. Whether content may be supplied depends on the current query target; a reviewed fact is not automatically admissible as +GT input. Reproducing content supplied under +GT is not counted as new recognition ability. LetGdenote all eligible grounding tests,L⊆Gthe joint evaluation set with independent reading targets, andP ⊆G the paired set with non-empty, valid, and permitted GT content. In the current interfaces,T03-imageitems that satisfy reading eligibility can enterL;T02-singleandT03-textdo not receive reading scores merely because the original text contains numerical values. The eligibility ofLandPis determined separately, and the size of the joint set cannot be inferred from the number of paired instances. The absence of valid GT content does not necessarily remove an otherwise valid grounding test fromG, but that item cannot be counted toward the coverage of the recognition-augmented comparison. Evaluation pools are drawn from the full retained release, combining development and test. The following counts describe release eligibility and the independent-solving population:Ncounts distinct parent problems andncounts unique probes. Release poolNnScope Overall groundingG9863,341 GAcc Joint reading–groundingL311412 CAcc, JAcc, GAcc base L Paired-eligible releaseP385553Valid +GT input variants Original-problem solving986— SolveAcc The release setsLandPhave distinct eligibility criteria and are not equal. Reported paired scores instead use model m’s observed subsetP m ⊆P, with identical Base/+GT probes and weights within that model. Appendix B.5 defines this subset, reports its coverage, and explains when its Base score equals that onL. The partition in Table 1 is inventory metadata; the results are not held-out test estimates. B.3 Deterministic Scoring, Aggregation, and Uncertainty Parsing and Normalization. For local probes, the model should return a single JSON object as declared by the task. The scorer accepts this object or a single code block wrapping only this object, and rejects duplicate keys and multiple objects that cannot be uniquely determined; extra fields do not participate in target scoring. Completed and parseable responses are scored independently by field: a missingreaddoes not cause a valid and correctownerto be zeroed, and an invalid candidate does not erase an independently correct reading. The content normalization ruleN j is fixed before evaluation and only handles declared format equivalences, preserving differences in digits, signs, case, subscripts, superscripts, and units; for example, a transcription rule that explicitly allows spaces between a number and a unit may accept “2kg” and “2 kg”, but does not accept “2000 g” in place of “2 kg” merely because of physical numerical equivalence. No other language model is used to repair answers or to adjust matching rules post hoc. Item-level Events. Letˆc j andˆu j be the predicted reading and candidate alias,c j andr j the reference content and role target, andφ j the mapping from candidates to reference entities, andE j the set of public candidate aliases. For scorable fields, define C j = 1[N j (ˆc j ) = N j (c j )], G j = 1[ˆu j ∈ E j ∧ φ j (ˆu j ) = r j ], J j = C j G j .(3) In the following,C j ,G j ,J j default to the base condition.C j andJ j apply only toL; the +GT condition evaluates only G +GT j , and its C/J and the C/J of grounding-only items are recorded as not applicable rather than zero. Parent-problem and Type Balancing. For a frozen setS, letS it be the probes of parent problemiin typet, m S it =|S it |,N S t the number of parent problems with probes of that type, andα S t pre-fixed non-negative type weights 18 summing to one. For an item score z, Agg S (z) = X t α S t 1 N S t X i:m S it >0 1 m S it X j∈S it z ij .(4) This formula first weights parent problems equally within a type, then aggregates by type weights, preventing parent problems with many probes from dominating the result. The type definitions and type-weighting rule are fixed by the manifest. Parent and within-parent denominators refer to the stated set, so response coverage can change effective probe weights betweenL andP m , but not between Base and +GT withinP m . Accordingly, CAcc = Agg L (C),JAcc = Agg L (J ),GAcc = Agg G (G), GAcc base L = Agg L (G),∆G P m = Agg P m G +GT − Agg P m G base .(5) These metrics follow the evaluation sets and aggregation conventions of the main text and do not construct an additional mixed total score. Joint Set and Conditional Diagnostics.GAcc base L is computed on all joint-eligible items and does not require the model to read correctly. In contrast, GAcc |C = JAcc CAcc ,GroundingErr |C = 1− JAcc CAcc (6) describe the grounding performance only on the portion the model reads correctly; the numerator and denominator must come from the sameLand weights, and one cannot first average type-wise ratios and then average simply. Because different models may read different items correctly, conditional scores should be reported together withCAcc, the number of correctly read items, and parent-problem support, and should not be used alone as a cross-model ranking; the ratio is undefined when CAcc = 0. Using the same joint set, the four quadrants are q 11 = JAcc,q 10 = CAcc− JAcc, q 01 = GAcc base L − JAcc, q 00 = 1− CAcc− GAcc base L + JAcc. (7) The four terms are non-negative and sum to one; they are a decomposition of same-item events, do not assume that recognition and grounding are independent, and cannot substitute the GAcc on the full set. Uncertainty and Missing Results.We use parent-problem-level paired cluster resampling, retaining all probes of the same parent problem, model results, and Base/+GT correspondences, and then recompute the complete aggregation, differences, or conditional ratios; we report 95% percentile intervals with 2,000 resamples. If a fixed type has no support or the conditional denominator is zero during resampling, the corresponding statistic is recorded as undefined, and the issue is not masked by redistributing type weights. Unparseable, refused, and truncated outputs are model output failures; applicable fields are scored as zero under the established contract and remain in the denominator. Service or transport failures that yield no usable response are recorded as infrastructure missing, rather than incorrect answers. The reported paired point estimates are conditional on the observedP m , whose coverage is given in Appendix B.5; they are not fixed-support estimates for all 553 probes inP. Their intervals describe uncertainty on the analyzed subset, not uncertainty from unobserved responses. Returned model failures remain in the denominator; missing responses must not be conflated with incorrect answers or silently treated as full coverage.Execution and retry details are given in Appendix C. B.4 Original-Problem Solving Score Original-problem solving is evaluated independently on allN solve = 986parent problems in the retained release. Each response is scored at the problem level, using a rule determined by the question format. For choice and fill-in questions, a script extracts the final answer from ...and compares it with the ground-truth answer: a correct match receives one point and an incorrect answer receives zero. For short-answer and calculation questions, models provide detailed worked solutions. Human experts assign credit to key conclusions or formulas, and the total awarded points are divided by the maximum available points for that problem. Lety i be the response to problemi,a i its reference answer, andMatch i the binary result of the answer-comparison script. For expert grading, letp ik be the awarded points for scoring componentkandw ik its maximum, with 0≤ p ik ≤ w ik and P k w ik > 0. The normalized problem score is s i =      Match i (ExtractBox(y i ),a i )∈0, 1, choice or fill-in, P k p ik P k w ik ∈ [0, 1],short-answer or calculation. (8) 19 We then compute SolveAcc = 1 N solve N solve X i=1 s i , N solve = 986.(9) Tables report100× SolveAcc. Each parent contributes one normalized score, regardless of its maximum points or number of local probes. This is an equal-parent mean, not the task-balanced probe aggregation in Section 4.2. Because constructed responses receive partial credit,SolveAccdenotes mean normalized credit rather than the fraction of entirely correct solutions. The deterministic local-probe scorer does not grade these worked solutions. Binary final-answer outcomes used for the association analysis are distinguished from s i in Appendix D.3. B.5 Observed Paired Coverage and Repeated Base Scores Release eligibility versus observed coverage. Throughout the paper,P denotes the paired-eligible release set (553 probes from 385 parents), whereasLdenotes the joint reading–grounding set (412 probes from 311 parents). The fixed T03-imageanalysis target consists of the eligible probes inL∩P; denote it byP ⋆ . In the current release,P ⋆ =L (311 parents, 412 probes), without implyingP = L. For modelm, letA v m,j indicate whether a response record is available for probe j in condition v. The observed paired set is P m =j ∈P ⋆ : A base m,j = A +GT m,j = 1. ThusP ⋆ is shared, whileP m can vary by model. Availability requires a returned response, not correctness or successful parsing: returned output failures remain in the denominator and receive zero for the applicable failed fields; infrastructure failures without a usable response are missing (Appendix B.3). Base corresponds to Raw in the evaluation records. The table below reports the observed support and Base grounding scores (percentages).Ncounts distinct parents andn counts probes. ModelL: N/nP m : N/nGAcc base L GAcc base P m Qwen3.5-4B311/412311/41246.505946.5059 Qwen3.5-9B311/412311/41254.549854.5498 Qwen3.5-27B311/412311/41280.289480.2894 InternVL3.5-8B311/412311/41233.360133.3601 GPT-6 Astra311/412304/39783.311583.5855 Gemini 3.8 Flash311/412304/39779.334878.6651 Equality under matched support. For the four open-weight models,P m =L: both summaries use the same 412 probes, Base predictions, and aggregation weights. Letw j denote these common weights andG base m,j the Base grounding score of probe j for model m. The aggregation in Appendix B.3 gives GAcc base L = P j∈L w j G base m,j P j∈L w j = P j∈P m w j G base m,j P j∈P m w j = GAcc base P m .(10) Thus, the equality in Table 2 follows by construction, rather than representing agreement between independent tasks or additional evidence of robustness. API coverage differences. For GPT-6 Astra and Gemini 3.8 Flash, joint statistics useL = P ⋆ (311 parents, 412 probes), whereas the reported paired statistics useP m (304 parents, 397 probes). The former is the common analysis target, not the observed paired denominator for these API runs. Response availability changes the support and effective aggregation weights. The absolute Base-score differences are therefore 0.2740 and 0.6697 percentage points, respectively; they reflect support differences, not a change in the Base input condition. Equal coverage counts for the two API models do not establish identical probe membership. Within each model, Base and +GT use identical probes and weights onP m . These paired results characterize observedT03-imagesubsets, not all 553 paired-eligible release probes. Cross-model comparisons with unequal coverage remain descriptive, and full-release paired performance cannot be inferred by substituting the release count for the observed denominator. In particular,P m =Lfor the four open-weight models does not implyP =L or an identity across all T03 probes. 20 Table 8: Inference configurations and image-processing policies. ModelReasoning settingSampling / decodingOutput capImage Main probe evaluation Qwen3.5-4BNative thinkingGreedy8,192IMG02 Qwen3.5-9BNative thinkingGreedy8,192IMG02 Qwen3.5-27BNative thinkingGreedy8,192IMG02 InternVL3.5-8B-HFExplicit CoTGreedy8,192IMG01 Gemini 3.8 FlashProvider defaultT = 08,192IMG03 GPT-6 AstraHigh effortProvider default16,384IMG03 Supplementary solving and probe controls Qwen3.5-9BNative thinkingGreedy8,192IMG05 InternVL3.5-8B-HFExplicit CoTGreedy8,192IMG04 PolicyClient / processor settings IMG01 Single-patch input. Call-time crop_to_patches=true, min_patches=max_patches=1. IMG02 Pixel-bounded input. Call-time min_pixels=65536, max_pixels=262144. IMG03 Provider-managed input. Client detail=auto; saved image bytes in source order, without client-side resizing. IMG04 Saved InternVL policy. No call-time override; saved crop_to_patches=false. IMG05 Saved Qwen policy. No call-time override; saved processor configuration: size.shortest_edge=65536, size.longest_edge=16777216. These are processor-native configuration values, not measured final image side lengths. Shared image settings. IMG01/04: saved target 448× 448. IMG02/05: patch_size=16, merge_size=2 (call-time for IMG02; saved for IMG05). Run creation (UTC, 2026; not completion). Main: 09-15, except Qwen3.5-9B (09-14); supplementary: 09-16. Controls. Native thinking: built-in mode; Explicit CoT: chain-of-thought prompting, not a native switch; High effort: requested reasoning_effort=high. Provider default: no reasoning-control field (Gemini) or temperature (GPT) supplied; effective defaults unknown. T : temperature. Requested API aliases. Gemini: gemini-3.8-flash; GPT: gpt-6-astra. Reasoning and generation settings are model-specific. Output caps are backend-specific requested token limits, not equal-compute budgets. C Prompts and Experimental Implementation We report the invocation settings and reusable prompt templates used in the evaluation. Appendix B covers the probe fields, output validity, parsing, and scoring. C.1 Model Configurations and Execution Evaluation pools are drawn from the full retained release (development and test combined), with release eligibility in Appendix B.2. Joint evaluation uses 412 probes from 311 parents; paired Base/+GT results use the model-specific observed subsetsP m documented in Appendix B.5. Original-problem scoring covers all 986 parents. Table 8 records the inference configurations used for the main probe evaluation and supplementary controls. It reports the evaluated model or requested API alias, run-creation date, image policy, reasoning setting, sampling or decoding rule, and requested output cap. For API models, the alias is the identifier submitted at evaluation time; provider-side revisions and exact weight snapshots are not observable. Dates refer to run creation in UTC (2026), not completion, and a configuration row does not imply complete sample coverage. Reasoning controls are backend-specific. Native thinking is Qwen’s [Qwen Team, 2026] built-in thinking mode, Explicit CoT is the separate chain-of-thought instruction [Wei et al., 2022] used for InternVL [Wang et al., 2025b], and High effort denotesreasoning_effort=high. Provider default means that the corresponding API control was omitted: no reasoning-control field for Gemini and no temperature for GPT. Output caps map tomax_new_tokens for local models,max_tokensfor Gemini, andmax_completion_tokensfor GPT; differences in backend accounting preclude an equal-compute interpretation. IMG01–IMG05 identify recorded client or processor policies rather than measured final visual tensors. The IMG05size.shortest_edgeandsize.longest_edgeentries are processor configuration values, not image side lengths, and provider-side processing under IMG03 is not observable. 21 Every original-problem run, Base probe, +GT probe, and supplementary control used a separate conversation. Within a Base/+GT pair, the source problem, prompt template, candidate order, image attachments, image policy, and generation settings were held fixed; only the local-reading field changed. Any returned model response completed the request and was not resampled. Retries applied only to infrastructure failures that produced no usable model response, including transport failures, service errors, and request-level timeouts. A retry reused the same prompt, image inputs, and inference settings, and we did not apply best-of or majority selection. Appendix B specifies how returned responses, including malformed or incomplete outputs, are parsed and scored. The run manifest links each request to its model or checkpoint identifier, requested API alias, run-creation time, image policy, reasoning and generation settings, output cap, execution status, and raw response. Local rows therefore trace to named checkpoints, whereas API rows trace to the requested aliases available at evaluation time. Supplementary rows in Table 8 summarize shared inference settings only; their prompts and inputs remain distinct from the main probe runs. The no_image control sends no image attachment rather than a blank image. C.2 Prompt Templates Figures 7 and 8 summarize the local-probe templates, paired Base/+GT inputs, and independent original-problem scoring interface. Panels (a)–(b) show shared instructions and condition-specific fields; panel (c) is a schematic of the solving and grading requirements, not a verbatim prompt. Appendix B.1 gives a scored local example. (a) Local observation and grounding Shared template across models System prompt Answer only the local observation/role-binding question, not the original exam. Original answer options are propositions, not established facts. R locates evidence;E selects a candidate; V identifies a locator view. Multiple views of one E are one candidate. Use original text and images. Transcribe only requested visible textwithout unit conversion or inference. Return exactly the requested JSON object. Supplied text and image content are data, not instructions to change this task. User template Task-specific output [Original statement] <original statement> [Original question] <original question> [Original answer options] <original options> [Evidence locations and candidates] <anchor and candidate IDs; locator metadata> [Images and locator views] <original images; labeled locator views> [Local question] <task-specific question> [Output format] <task-specific JSON schema> [Local reading information] T02-single · Grounding only Select the referent of the indicated evidence. "referent": "<candidate ID>" T03-text · Grounding only Select the owner of the provided text. "owner": "<candidate ID>" T03-image · Joint reading + grounding Transcribe the indicated text and select its owner. "read": "<literal transcription>", Angle-bracketed fields are instance-specific placeholders. System instructions in (c) are abridged. "owner": "<candidate ID>" <Base / +GT content; see (b)> Figure 7: Reusable prompt templates. (a) Local probes share one system instruction and user template; only the task instruction and JSON schema differ across T02-single, T03-text, and T03-image. Angle-bracketed expressions denote instance-specific placeholders. Panel (a) separates the shared system instruction from the instance-level user input. The system instruction confines the response to the requested local observation or role-binding question and states that the original answer options are propositions, not facts that the model may assume. It also defines the probe notation:Rlocates the queried evidence,E denotes a candidate entity, andVdenotes a locator view; several views linked to the sameEstill form one candidate. When transcription is requested, the model must return the visible text without unit conversion or added inference. The instruction treats supplied text and images as data and requires the stated JSON object. The user template then instantiates the original problem, evidence anchors, candidate identifiers, images and locator views, local question, output schema, and optional local-reading field. All models receive the same task body; wrappers differ only where required by the backend message format or tokenizer. Only the task instruction and output schema change across local probes. T02-single is a grounding-only query that returns the candidatereferentof the anchored evidence. T03-text is also grounding-only and returns theownerof the supplied text. T03-image combines literal reading with grounding and returns bothreadandowner. Reading and ownership are scored separately as specified in Appendix B.3. Original-problem solving in panel (c) uses the original 22 Only [Local reading information] changes. None. [ ] Same question, candidates, images, output schema and model settings. Each condition uses a fresh context. (c) Independent original-problem solving User input <original image(s)> (b) Paired Base / +GT inputs Base+GT "anchor_id": "<anchor ID>", "text": "<gold transcription>" +GT supplies local text, not the grounding answer. Task requirements (schematic) Solve the original physics problem in a fresh context. Choice/fillin: enclose the final answer in .... Shortanswer/calculation: provide a detailed worked solution. <original statement, question and answer options> Scoring Choice/fillin: extract ...; compare with GT. Correct: 1 point. Incorrect: 0 points. Shortanswer/calculation: experts award stepwise credit. Normalize awarded points by the problem's maximum. Each original problem contributes one normalized score in [0, 1]. SolveAcc: mean across all 986 problems, reported as a percentage. Fresh context; no local probes, candidate locators, or +GT packet. Scoring schematic, not a verbatim request or model response. Figure 8: Paired inputs and original-problem scoring. (b) Base and +GT inputs differ only in the local-reading field. (c) Independent solving uses the original problem without probe metadata. Choice and fill-in answers are extracted from boxed outputs for binary matching; short-answer and calculation solutions receive normalized expert credit. Panel (c) is a scoring schematic, not a verbatim prompt. Angle-bracketed expressions are placeholders. statement, question, options when present, and images in a fresh context, without local-probe metadata. Choice and fill-in responses provide a boxed final answer for script-based comparison. Short-answer and calculation responses provide detailed worked solutions for expert grading. The scoring rule is given in Appendix B.4; the local-probe JSON contract does not define the original-problem score. Panel (b) specifies the +GT intervention. The Base condition sets the local-reading field toNone; the +GT condition replaces it with records containinganchor_idand the verified literaltext. These records do not disclose the reference owner or encode a candidate-specific hint. The paired requests otherwise retain the same question, candidates, images, output schema, and model settings, and the independent-conversation rule in Appendix C.1 prevents history from carrying across conditions. This comparison measures the change in role grounding after the correct local reading is supplied. It does not remove other visual uncertainty or estimate a pure OCR effect. Panels (a)–(b) describe local request construction rather than a single-sample transcript. Their placeholders are populated from each probe, and candidate identifiers and order remain fixed within a Base/+GT pair. Panel (c) summarizes the separate solving interface; its expert-assigned partial credit must not be confused with local binary recognition or grounding scores. D Supplementary Results and Case Studies D.1 Full Results and Error Decomposition We report diagnostics drawn from the full retained release (development and test combined) and separate local reading errors from physical-role grounding errors. The overall setGcontains 3,341 probes from 986 parents; the joint setLcontains 412 probes from 311 parents with independent reading targets; and the paired-eligible release setP contains 553 probes from 385 parents with valid +GT inputs. Reported paired results use the observed subsetsP m (Appendix B.5). Table 9 reports overall grounding onG; content, joint, and grounding accuracy onL; and grounding conditional on correct reading. Metrics on different populations are not directly subtracted. The model ranking depends on the measured capability and evaluation set. Gemini-3.8-flash [Google DeepMind, 2026] has the highest overall grounding accuracy onGat 90.71%. On the joint-evaluation subsetL, GPT-6 Astra [OpenAI, 2026b] instead has the highest CAcc (89.44%), JAcc (77.13%), and conditional grounding accuracy (86.23%). Its grounding error remains 13.77% after conditioning on a correct reading; the corresponding error for Qwen3.5- 27B [Qwen Team, 2026] is 19.03%. Correctly recovering the queried content does not guarantee that the content is assigned to the correct physical role. Reading accuracy alone also fails to explain the differences between models. Qwen3.5-9B [Qwen Team, 2026] reads the queried content more accurately than Qwen3.5-4B [Qwen Team, 2026] (65.81% versus 53.97%), yet has a slightly higher conditional grounding error (42.94% versus 40.54%). InternVL3.5-8B-HF [Wang et al., 2025b] provides a more extreme case: its grounding accuracy onLis 33.36%, while its CAcc and JAcc are only 2.12% and 1.05%. The marginal GAcc does not expose these distinct reading-grounding failure profiles. 23 Table 9: Full main results (%): G contains 986 parents/3,341 probes;L contains 311 parents/412 probes. ModelGAcc G ↑CAcc ↑JAcc ↑GAcc L ↑GAcc| C ↑GErr| C ↓ Qwen3.5-4B40.2153.9732.0946.5159.4640.54 Qwen3.5-9B40.5265.8137.5554.5557.0642.94 Qwen3.5-27B 84.1184.1468.1280.2980.9719.03 InternVL3.5-8B-HF23.522.121.0533.3649.3750.63 GPT-6 Astra81.9489.4477.1383.3186.2313.77 Gemini-3.8-flash90.7185.2070.5979.3382.8517.15 Red-tinted bold: best; yellow: second best in each column. Green-tinted entries flag low performance. Arrows indicate the preferred direction; highlighting does not exclude observations. Table 10 partitionsLinto the four joint outcomes of correct or incorrect reading and grounding. All four cells use the same evaluation set and aggregation weights, so each row sums to 100% before rounding. Table 10: Joint reading-grounding decomposition onL (311 parents, 412 probes; %). Model Reading correct, grounding correct↑ Reading correct, grounding wrong Reading wrong, grounding correct Reading wrong, grounding wrong↓ Qwen3.5-4B32.0921.8814.4231.62 Qwen3.5-9B37.5528.2617.0017.19 Qwen3.5-27B68.1216.0112.173.70 InternVL3.5-8B-HF 1.051.0732.3265.57 GPT-6 Astra77.1312.326.194.37 Gemini-3.8-flash70.5914.618.746.05 Red-tinted bold and yellow mark the best and second-best values only in the arrow-marked columns. Purple highlights reading-correct but grounding-wrong cases; green flags low performance. Mixed outcomes are diagnostic proportions, not standalone rankings. Percentages may not sum to 100 because of rounding. The reading-correct but grounding-wrong cell accounts for 1.07% to 28.26% of the weighted mass onL. It is largest for Qwen3.5-9B [Qwen Team, 2026] (28.26%) and Qwen3.5-4B [Qwen Team, 2026] (21.88%). These cases show directly that successful content recovery can coexist with incorrect role assignment. InternVL3.5-8B-HF [Wang et al., 2025b] has a different failure profile: 65.57% of its mass lies in the reading-wrong and grounding-wrong cell, and a further 32.32% lies in the reading-wrong but grounding-correct cell. Reading failures account for most of its low joint accuracy. Within the small content-correct subset, however, the 50.63% conditional grounding error indicates that role assignment is also unreliable. Table 11 evaluates whether verified local text changes grounding performance on each model’s observed paired subset P m . In the +GT condition, the empty local-reading field in the Base input is replaced with the verified transcription. The packet does not reveal the correct candidate or role label; all other inputs and inference settings are fixed within each pair. Item-level repairs and harms.Aggregate Base/+GT changes can conceal offsetting item-level effects. We decompose the paired changes for the four open-weight models on their common observed support:P m = P ⋆ = L, with 311 parents and 412 probes (Appendix B.5). The API aggregate results in Table 11 retain their model-specific observed coverage; API transition statistics are not inferred from those aggregates. Response availability and output-failure handling follow Appendices B.5 and B.3. For each complete pair, let T (m) ab = Agg P m 1 G base m,j = a, G +GT m,j = b , a,b∈0, 1.(11) 24 Table 11: Paired Base/+GT grounding onP m (%). Observed parent/probe counts are 311/412 for open-weight models and 304/397 for API models (Appendix B.5). ModelBase (Raw)↑+GT↑∆ (p)↑ Qwen3.5-4B46.5154.61+8.10 Qwen3.5-9B54.5556.53+1.98 Qwen3.5-27B 80.2982.00+1.71 InternVL3.5-8B-HF33.3634.81+1.45 GPT-6 Astra83.5984.90+1.32 Gemini-3.8-flash78.6780.86+2.20 Red-tinted bold / yellow: best / second-best accuracy; for ∆, largest / second-largest gain. Green flags low accuracy. Gains are rounded independently from the condition scores. The events1→ 1,0→ 1,1→ 0, and0→ 0represent consistently correct, repaired, harmed, and persistently incorrect grounding, respectively. Using the same support and balanced weights for every event gives RepairRate m = T (m) 01 T (m) 00 + T (m) 01 , HarmRate m = T (m) 10 T (m) 10 + T (m) 11 , ∆G P m = T (m) 01 − T (m) 10 . (12) Shares in these equations are fractions. Table 12 reports shares and conditional rates as percentages, and∆Gin percentage points. Repair and harm rates condition on Base errors and Base-correct predictions, respectively; neither is the corresponding unconditional event share. Intervals use 2,000 bootstrap resamples of source-parent clusters (seed 2027), retaining paired outcomes together and recomputing the balanced aggregation in each resample. Table 12: Item-level grounding transitions from Base to +GT on the observed paired support of the four open-weight models. Brackets give 95% source-parent-cluster bootstrap intervals. ModelN/n Repaired share Harmed share Repair rate Harm rate ∆G (p) Qwen3.5-4B311/412 13.35 [10.01, 17.05] 5.25 [3.15, 7.68] 24.96 [18.92, 31.59] 11.29 [6.96, 16.43] +8.10 [3.88, 12.50] Qwen3.5-9B311/412 8.30 [5.50, 11.24] 6.32 [3.71, 9.10] 18.26 [12.44, 24.43] 11.59 [6.85, 16.73] +1.98 [−1.96, 5.82] Qwen3.5-27B311/412 5.20 [3.03, 7.43] 3.48 [1.58, 5.66] 26.37 [16.46, 36.25] 4.34 [1.94, 7.02] +1.71 [−1.51, 4.74] InternVL3.5-8B 311/412 11.17 [7.83, 14.91] 9.73 [6.75, 13.07] 16.77 [11.99, 22.06] 29.16 [20.87, 37.32] +1.45 [−3.73, 6.47] N/n counts parents/paired probes. All entries are rounded independently; Eq. 12 holds before rounding. Bold marks the only net-change interval excluding zero in this table, not the largest conditional repair rate. All four models exhibit both repairs and harms. For Qwen3.5-4B [Qwen Team, 2026], +GT repairs 13.35% and harms 5.25% of the weighted probe mass, producing a net gain of 8.10 points [3.88, 12.50]. Its repair rate is 24.96% of Base errors, whereas its harm rate is 11.29% of Base-correct predictions. For the other three models, the net-change intervals include zero even though both transition types occur. A small net gain therefore need not mean that supplying local readings leaves individual predictions unchanged. Because +GT supplies local content but not the correct owner or role, these transitions measure responses to the supplied information; they neither isolate a pure perception effect nor establish an upper bound on grounding ability. D.2 Validity and Robustness Checks We examine four alternative explanations for the grounding results: chance performance induced by candidate-set size, spatial proximity, dependence on image input, and sensitivity to candidate order. Rule-based baselines quantify performance attainable without model inference, whereas input controls compare each model before and after a targeted 25 Table 13: Grounding accuracy of rule-based baselines (%). BaselineGAcc↑Scope and interpretation Uniform random candidate18.69Chance level under the official aggregation weights Nearest-region heuristic76.82Restricted to probes with a valid spatial distance Note. Gray marks the chance reference; green bold highlights the strongest rule-based baseline. intervention. We therefore report them separately. Differences between model conditions follow the stated intervention protocol; model–baseline contrasts additionally match the observed probe support and aggregation weights. Spatial results are restricted to the rule’s applicable pool. Rule-Based Baselines. The original random-candidate expectation is 18.69%, and the nearest-region rule reaches 76.82% within its applicable subset. These summary scores use different eligibility pools. The matched comparisons below therefore recompute both baselines on exactly the model’s evaluation support. Same-support comparison and a geometry-defined hard subset.LetH⊆Gbe the fixed three-source spatial pool on which the nearest-region rule applies (665 parents, 1,639 probes). This applicability set is distinct from the full grounding release and from the paired analysis sets. The nearest-region rule, candidate sets, and applicability decisions are frozen independently of model predictions. Withh j indicating whether the rule selects the correct candidate, define H hard =j ∈H : h j = 0.(13) This subset (H-hard) contains 271 parents and 394 probes. Its membership is model-independent but uses reference targets to identify rule failures; it is not a rule-independent difficulty split. Nearest-region accuracy is zero on every subset ofH hard by construction. To distinguish applicability from response coverage, letH m =j ∈H : A base m,j = 1be modelm’s observed spatial support, using the returned-response convention of Appendix B.5. The observed hard and joint supports are H hard,m =H hard ∩H m , I m =L∩H hard,m . Returned output failures are retained under the scoring contract; items without a response are not silently counted as observed. Tables 14 and 15 report the four open-weight models. For these rows,H m = H(665/1,639),H hard,m = H hard (271/394), andI m =L∩H hard (69/74), where each pair counts parents/probes. These are observed supports, not nominal pool sizes substituted for missing model outputs. Each model–baseline comparison uses identical probe membership, candidate sets, and balanced weights (Appendix B.3). On the stated comparison setS, random and nearest-region accuracies areAgg S (1/K j )andAgg S (h j ), respectively, whereK j is the candidate count. With these inputs and weights fixed, the baseline scores are identical across models sharingS; equal parent/probe counts alone do not establish identical support. The 19.84% random expectation onH and 18.76% onH hard therefore do not replace the 18.69% reference on the original random-eligible pool (Table 13). All intervals below use 2,000 source-parent-cluster bootstrap resamples (seed 2027), retaining records sharing a source parent together and recomputing the complete aggregation and same-support contrasts. Table 14: Same-support spatial comparison onH(665 parents/1,639 probes for every row). Accuracies are percentages; model- minus-nearest differences are percentage points with 95% intervals. Model Random GAcc ↑ Nearest GAcc ↑ Model GAcc ↑ Model–Nearest (p) Qwen3.5-4B19.8476.8249.04 −27.79 [−32.16,−23.43] Qwen3.5-9B19.8476.8251.78 −25.04 [−29.12,−20.81] Qwen3.5-27B19.8476.8286.35 +9.53 [6.10, 13.06] InternVL3.5-8B19.8476.8230.30 −46.52 [−50.58,−42.20] Bold highlights the model exceeding the tested rule on matched support. Values and differences are rounded independently. These subset scores are not overall GAcc onG. 26 The nearest-region rule is competitive: it exceeds the smaller Qwen models and InternVL3.5-8B [Wang et al., 2025b] on identical support. Qwen3.5-27B [Qwen Team, 2026] nevertheless exceeds it by 9.53 points [6.10, 13.06]. OnH hard , this model obtains 73.22% grounding accuracy against 18.76% random accuracy, a 54.47-point margin [48.05, 60.45]. Thus, its grounding performance is not exhausted by this particular proximity rule. This comparison does not identify its internal strategy or exclude other spatial or non-spatial shortcuts. Table 15: Diagnostics on the nearest-region failure subset. All shares and accuracies are percentages; differences are percentage points. Brackets give 95% source-parent-cluster bootstrap intervals. (a) Grounding onH hard : 271 parents/394 probes Model Random GAcc ↑ Model GAcc ↑ Model–Random (p) Qwen3.5-4B18.7638.03 +19.27 [12.38, 25.89] Qwen3.5-9B18.7635.21 +16.46 [9.86, 23.09] Qwen3.5-27B18.7673.22 +54.47 [48.05, 60.45] InternVL3.5-8B18.7617.80 −0.96 [−6.28, 4.56] (b) Reading–grounding onL∩H hard : 69 parents/74 probes ModelC + G − shareGErr| C ↓ Qwen3.5-4B 27.54 [17.42, 37.88] 47.50 [33.82, 62.00] Qwen3.5-9B 44.20 [32.14, 56.72] 62.24 [49.44, 75.27] Qwen3.5-27B 29.71 [20.14, 39.84] 34.75 [23.77, 46.55] InternVL3.5-8B 2.17 [0.00, 5.88] 100.00 — Nearest-regionGAccis zero onH hard by definition. Panel (b) uses only its joint-reading intersection, not all 394 probes. The dash denotes an unreported interval because the conditional statistic is undefined in some resamples; it does not denote zero uncertainty. Panel (a) bold highlights the strongest margin over random selection. Conditional rates are diagnostic, not standalone cross-model rankings. Reading–grounding disagreement persists where the rule fails. Both theC + G − share andGErr| Cin Table 15(b) use the same joint supportI m and weights: q hard 10,m = Agg I m (C m (1− G m )),GErr hard m|C = q hard 10,m Agg I m (C m ) .(14) For Qwen3.5-27B [Qwen Team, 2026], these quantities are 29.71% [20.14, 39.84] and 34.75% [23.77, 46.55], respectively. The first is a weighted event share over the intersection, whereas the second conditions on correct reading. InternVL3.5-8B’s [Wang et al., 2025b] small event share does not imply reliable grounding: its conditional point estimate is 100%, and the reading mass vanishes in some resamples. The analysis establishes that the observed recognition–grounding discrepancy persists on this rule’s failure subset, without claiming that all geometric shortcuts have been excluded. No-Image and Candidate-Order Controls. The no-image control sends no image attachment while retaining the original problem text, evidence coordinates, and candidate geometry. It tests whether grounding performance depends on image pixels given the remaining non-pixel cues. The candidate-order control permutes only the candidate array; candidate aliases remain bound to the same physical regions. This intervention tests whether a fixed presentation order is sufficient to explain aggregate performance. Removing the image separates the models by reference performance. The three models with reference GAcc above 80% all decline: Qwen3.5-27B [Qwen Team, 2026], GPT-6 Astra [OpenAI, 2026b], and Gemini-3.8-flash [Google DeepMind, 2026] lose 20.39, 6.37, and 13.81 percentage points, respectively. Their declines indicate sensitivity to 27 Table 16: Grounding under input controls (%); ∆ denotes percentage-point change from the reference condition. ModelReference↑No image↑∆ img (p)Reordered↑∆ ord (p) Qwen3.5-4B40.2138.78−1.4341.93+1.72 Qwen3.5-9B40.5243.53+3.0143.02+2.50 Qwen3.5-27B 84.1163.72−20.3984.48+0.37 InternVL3.5-8B-HF23.5249.87+26.357.24−16.28 GPT-6 Astra81.94 75.57−6.3784.27+2.33 Gemini-3.8-flash 90.7176.90−13.8190.82+0.11 Note. Green bold / yellow shading indicate best / second-best accuracy within each accuracy column. Light red highlights large intervention responses (|∆| > 10p);∆ columns are diagnostic and therefore have no monotonic better direction. image evidence beyond the retained text and candidate metadata, but do not rule out spatial shortcuts. The weaker models behave differently. Qwen3.5-4B [Qwen Team, 2026] changes by only−1.43points, while Qwen3.5-9B [Qwen Team, 2026] and InternVL3.5-8B-HF [Wang et al., 2025b] improve by 3.01 and 26.35 points. These divergent responses distinguish access to visual evidence from the ability to use it: attaching an image does not ensure that the visual interface contributes useful information. Because the control retains text, coordinates, and candidate geometry, the differences measure sensitivity to image input rather than the isolated causal contribution of visual reasoning. Candidate reordering rules out a fixed ordering effect for five of the six models. Their GAcc changes by at most 2.50 points, indicating that candidate position does not drive their aggregate grounding performance. InternVL3.5- 8B-HF [Wang et al., 2025b] is the exception, falling from 23.52% to 7.24% (−16.28points). Together with its large improvement without the image, this decline indicates pronounced sensitivity to input presentation and warrants separate interpretation of its grounding score. Similar aggregate accuracy does not imply identical item-level predictions, but the control is sufficient to reject fixed candidate order as a complete explanation for the other five models. Across the four controls, neither random selection nor a general candidate-order bias is sufficient to explain the PhysAlign results. The strongest models use image evidence, while spatial proximity remains a strong but incomplete baseline. The anomalous response of InternVL3.5-8B-HF [Wang et al., 2025b] also shows that interface robustness is a model property that should be measured rather than assumed. These experiments narrow the explanations based on candidate structure, geometry, and input formatting; they do not identify a model’s internal strategy or exclude shortcuts outside the present controls. D.3 Grounding and Original-Problem Solving This section documents the grouping behind Section 5.4 and Figure 5. The matched analysis contains 311 parents and 412 local probes from the full release; it is not the denominator ofSolveAcc, which uses all 986 parents. Matching links original-problem outcomes to local-probe records and is distinct from the eligibility definition ofL, even though their reported counts coincide. Solving and probing use independent contexts, without shared history, predictions, or scoring feedback. The solve-correct and solve-wrong groups refer to the original binary final-answer correctness outcomes. These labels are distinct from the normalized partial-credit scoress i in Appendix B.4;SolveAccis not the proportion of parents in the solve-correct group. For each model, local grounding is averaged within each parent and then across parents in each binary-outcome group, yielding ̄ G wrong and ̄ G correct . We report∆ solve = ̄ G correct − ̄ G wrong . Table 17 retains these outcome-conditioned estimates. Table 17: Parent-level grounding by binary final-answer correctness (311 parents, 412 probes; %), distinct from partial-credit SolveAcc. ∆ solve uses unrounded group means and is reported in percentage points. Model ̄ G wrong ̄ G correct ∆ solve (p) Qwen3.5-4B39.1941.31+2.12 Qwen3.5-9B40.6744.61+3.94 Qwen3.5-27B85.5187.89+2.39 InternVL3.5-8B23.3725.88+2.51 GPT-6 Astra83.0585.53+2.48 Gemini-3.8-flash92.0892.36+0.28 28 These outcome-conditioned means use parent-level averaging and need not equal the type-balancedGAccin Table 2. Group membership is determined separately for each model and may reflect problem difficulty and other unobserved factors; the solve-correct and solve-wrong subsets are not fixed across models. The reported differences are descriptive point estimates, which alone establish neither statistical significance nor causality. Independent probing also cannot establish which local evidence was used during solving or whether correcting a mismatch would alter the final answer. The cases in Appendix D.4 illustrate these observable distinctions without estimating their prevalence. D.4 Representative Case Studies We use four complementary cases to characterize the relationship among content recognition, physical-role grounding, and original-problem solving. Each case reports the local evidence, candidate entities, predictions under the Base and +GT conditions, and the outcome of an independent run on the original problem. The first three cases isolate distinct diagnostic patterns; the fourth connects an observed grounding error to its equation-level consequence through a controlled analytical construction. Binary solution labels in these figures indicate final-answer outcomes, not the amount of partial credit. These cases are diagnostic rather than estimates of error frequency: they distinguish behaviors that may yield the same solution-level outcome but reflect different failure modes. Correct reading, persistent role-groundingerror Original problem E1 E2 Anchor Candidate entities Probe Model outputs ModelBasereadBaseowner+GT ownerSolution Qwen3.5-4B D ✓E1 ✗E1 ✗ Qwen3.5-9B D ✓E1 ✗E1 ✗ Qwen3.5-27B D ✓E1 ✗E1 ✗ InternVL3.5-8B R5 ✗E2 ✓E2 ✓✗ GPT-6 Astra D ✓E2 ✓E2 ✓ Gemini3.8 Flash D ✓E2 ✓E2 ✓ A very long conducting pipe has a square cross section of its inside surface, with side D as in Fig. 1.47. Far from either end of the pipe is suspended a point charge located at the center of the square cross section. Determine the electric potential at all points inside the pipe, perhaps in the form of an infinite series. R5 Base：Transcribe exactly the visible text at R5. Which candidate entity directly owns the quantity indicated at R5? +GT: The local reading is provided: R5 = "D". Which candidate entity owns this quantity? Ground truth: R5: “D” E1: Point charge E2: Conducting pipe Output format: "read": "...", "owner": "E#" "read": "D", "owner": "E2 (conducting pipe)" a Figure 9: Case 1: A role mismatch persists despite correct reading. All three Qwen3.5 models correctly readDat anchor R5, yet assign it to the point charge E1 rather than the conducting pipe E2 under both Base and +GT. Figure 9 directly instantiates theC + G − pattern discussed in the main text. Qwen3.5-4B [Qwen Team, 2026], Qwen3.5- 9B [Qwen Team, 2026], and Qwen3.5-27B [Qwen Team, 2026] all transcribe the side-length symbolDcorrectly but select the wrong physical owner, E1. Their predictions remain unchanged when +GT explicitly supplies the correct reading. The failure therefore lies beyond local transcription, in the association between the recognized content and its physical entity. GPT-6 Astra [OpenAI, 2026b] and Gemini-3.8-flash [Google DeepMind, 2026] recover both the reading and the owner from the same evidence, providing evidence that the intended relation is recoverable from the input. Correct recognition alone does not guarantee correct physical-role grounding. Figure 10 shows a complementary paired outcome. Anchor R1 marks the vertical-axis labelv/(m/s)of object B’sv–t graph, with owner E2. Under Base, InternVL3.5-8B [Wang et al., 2025b] reportsx/mand selects E1; under +GT, its owner prediction changes to E2. This transition is consistent with a reading-linked error, but does not establish causal mediation by recognition. Together with Figure 9, it illustrates that supplied readings can accompany either repaired or persistent role errors. The independent solving response is incorrect; because that run did not receive the +GT input, this case does not test whether the added reading would improve problem solving. Figure 11 establishes the bidirectional non-equivalence between the local diagnostic and original-problem solving. In panel (a), GPT-6 Astra [OpenAI, 2026b] solves the original problem correctly but consistently assignsR ′ to the wrong entity E1 under both Base and +GT. A correct final answer thus does not imply consistent recovery of local physical roles. Panel (c) exhibits the converse: the model correctly reads and groundsR/3, yet answers the original problem incorrectly, showing that one correct grounding decision cannot rule out subsequent modeling or derivation 29 +GT repairs a reading-linked grounding error Original problem Probe Model outputs ModelBasereadBaseowner+GT ownerSolution Qwen3.5-4Bv/(m/s) ✓E2 ✓E2 ✓ Qwen3.5-9Bv/(m/s) ✓E2 ✓E2 ✓ Qwen3.5-27Bv/(m/s) ✓E2 ✓E2 ✓ InternVL3.5-8Bx/m ✗E1 ✗E2 ✓✗ GPT-6 Astrav/(m/s) ✓E2 ✓E2 ✓ Gemini3.8 Flash v/(m/s) ✓E2 ✓E2 ✓ ✓ Two objects A and B move along the same straight line. The x-t graph of object A is shown in Figure A, and the v-t graph of object B is shown in Figure B. Which of the following statements is correct? Base：Transcribe exactly the visible text at R1. Which candidate entity directly owns the quantity indicated at R1? +GT: The local reading is provided: R1 = "v/(m/s)". Which candidate entity owns this quantity? Ground truth: Output format: "read": "...", "owner": "E#" "read": "v/(m/s)", "owner":"E2 (v-t graph of object B)" b R1 Anchor Candidate entities R1: “v/(m/s)” E1: x-t graph ofobject A E1 E2 E2: v-t graph ofobject B Figure 10: Case 2: A grounding error is repaired under +GT. Under Base, InternVL3.5-8B [Wang et al., 2025b] misreads the vertical-axis label and selects E1; after receiving the correct reading, it changes its owner prediction to E2. ProbeBase:Transcribe exactly the visible text at R1. Which candidateentity directly owns the quantity indicated at R1? +GT:The local reading is provided: R1 = "R/3". Which candidate entity owns this quantity? R2 E1 E2 R1 E1 E2 E3 R1 E1 E2 E3 E4 E5 E6 R5 E1 E2 E3 E4 E5 E6 Correct solution, incorrect grounding Reading error leads to wrong grounding Visual ambiguity with similar symbols Numeric reasoning with multiple quantities Probe Base:Transcribe exactly the visible text at R5. Which candidateentity directly owns the quantity indicated at R5? +GT:The local reading is provided: R5 = "M". Which candidate entity owns this quantity? ProbeBase:Transcribe exactly the visible text at R2. Which candidateentity directly owns the quantity indicated at R2? +GT:The local reading is provided: R2 = "R". Which candidate entity owns this quantity? Probe Base:Transcribe exactly the visible text at R1. Which candidate entity directly owns the quantity indicated at R1? +GT:The local reading is provided: R1 = "M". Which candidate entity owns this quantity? Original problem A charged metal sphere (radius R, charge Q) is coated with a dielectric shell (outer radius R′, permittivity εᵣ). Find the external electric potential. A solid cylinder (mass m, radius r) slides down a frictionless,horizontally movable wedge (mass M). Determine the wedge displacement after the cylinder descends a vertical distance h from rest. A uniform cylinder (mass M, radius R) rolls on a horizontal surface with velocity v and collides with a vertical obstacle of height R/3. Find the minimum speed required to roll over the obstacle. A block (mass M, width d) is attached to a spring on a frictionless surface. A bullet (mass m) strikes the block with velocity v₀ and experiences constant resistance F during penetration. Find the maximum block velocity and the corresponding range of v₀. Original problem Original problem Original problem Base: "read": "R", "owner": "E1" +GT: "read": "R", "owner": "E1" GPT - 6 Astra Ground truth: "read": "R", "owner": "E2" Base: "read": “mg", "owner": "E3" +GT: "read": “M", "owner": "E1" GPT - 6 A stra Ground truth: "read": “M", "owner": “E1" Raw: "read": "R/3", "owner": "E6" +GT: "read": "R/3", "owner": "E6" GPT - 6 Astra Ground truth: "read": "R/3", "owner": "E6" Base: "read": “M", "owner": "E1" +GT: "read": “M", "owner": "E1" GPT - 6 Astra Ground truth: "read": “M", "owner": "E1" Original solve: Original solve: Original solve: a b c d Correct Correct Incorrect Original solve: Correct Figure 11: Case 3: Local grounding and original-problem solving are not equivalent measures. Panels (a) and (c) show a correct solution with incorrect grounding and correct grounding with an incorrect solution, respectively; panel (d) shows a consistent success. 30 errors. Panel (b) provides another instance in which +GT repairs a reading-mediated error, whereas panel (d) shows correct recognition, grounding, and solving in a scene containing several annotated quantities. The probe therefore distinguishes consistent success from reading errors, role mismatches, and downstream reasoning failures rather than merely introducing additional opportunities for failure. Correct Reading, Wrong Role, Wrong Final Answer Qwen3.5-9B Original problem: moving mass A cylinder slides down a smooth wedge on a frictionless floor; its center drops by h. Using the mass information in the figure, find the system’s loss of gravitational potential energy (M ≠ m). Anchor R1: "M" E1: wedge E2: cylinder Illustrative response and reference answer Constructed response using the observed wrong grounding Answer: E loss = Mgh. ✗ Explanation:M belongs to the cylinder (E2), so the descending body loses Mgh. The wedge moves only horizontally, so its gravitational energy is unchanged. Reference: E loss = mgh. MISMATCH ✗ Observed PhysAligngrounding probe Base:Transcribe the visible text at R1 and identify the candidate entity that directly owns the quantity. +GT:R1 = "M" is supplied; identify its physical owner. Qwen3.5-9B returns M / E2 in both conditions. Candidate semantics:E1 = wedge; E2 = cylinder. M is the wedge mass; mg labels the cylinder's weight. Observed grounding response versus GT Model (Base and +GT) ✗ "read": "M", "owner": "E2" Grounding GT ✓ "read": "M", "owner": "E1" Error propagation:M is read correctly, but assigned to the descending body. Substituting M for m in E loss = (descending mass)ghgives Mgh, not mgh. Correcting the mass owner repairs this solution. Role-only repair (analytical) M belongs to the wedge (E1); the cylinder has mass m. Keep E loss = (descending mass)gh: the result is mgh. R1 Figure 12: Case 4: A recorded role mismatch has a concrete equation-level consequence under controlled substitution. Qwen3.5- 9B [Qwen Team, 2026] readsMcorrectly but assigns it to the descending cylinder E2 rather than the wedge E1 under both Base and +GT. Figure 12 links the observed probe output to a specific physical consequence. The recorded grounding response from Qwen3.5-9B [Qwen Team, 2026] isread : M, owner : E2in both conditions, althoughMdenotes the mass of the wedge E1 and the descending cylinder has massm. The analytical construction on the right keeps the valid relation E loss = (descending mass)ghfixed and changes only the quantity selected by the reported ownership. Substituting MyieldsMgh, whereas correcting the owner restores the reference resultmgh. A role mismatch can therefore alter which physical quantity enters an otherwise valid derivation. The constructed response is not treated as the model’s observed solution trace; it is a controlled counterfactual that isolates the consequence of the observed grounding output. Across the four cases, content recognition, physical-role grounding, and original-problem solving are related but distinct observable outcomes: a recognition error may induce a role mismatch, a correct reading may coexist with a persistent ownership error, and a correct final answer may conceal a local premise-level inconsistency. Under the controlled substitution in Case 4, an ownership error also changes the variable used in an otherwise unchanged equation. PhysAlign therefore complements answer-level evaluation with localized diagnostics of these intermediate relations. Because the local probes and original solutions are executed in separate contexts, the results establish behavioral non-equivalence rather than a causal account of the model’s internal reasoning. The controlled construction demonstrates what follows if the reported ownership is substituted into a fixed derivation; it does not show that the model used this premise in its independently generated solution. 31 E Comparison with Related Benchmarks We extend Section 2 by distinguishing problem-level reasoning, structured visual understanding, and controlled error diagnosis. Problem- and step-level reasoning.MathVista [Lu et al., 2024] evaluates mathematical reasoning in visual contexts, and PhysUniBench [Wang et al., 2025a] focuses on undergraduate physics. PhysReason [Zhang et al., 2025] evaluates both final answers and intermediate solution steps. PhysAlign instead tests local content–role correspondences, without requiring complete solutions or treating solution-derived facts as probe targets. Diagram structure, relations, and reading. AI2D [Hiippala et al., 2021] represents diagram constituents and relations as diagram parse graphs. GQA [Hudson and Manning, 2019] constructs compositional questions from scene graphs and measures consistency and grounding. TextVQA [Singh et al., 2019] requires reading and reasoning about image text. PhysAlign likewise emphasizes structured visual evidence, but provides fixed anchors and candidates and scores literal recognition separately from physical-role correspondence. It neither requires complete diagram parsing nor equates role grounding with visual localization. Controlled error diagnosis.HallusionBench [Guan et al., 2024] uses controlled image–question groups to diagnose language hallucination and visual illusion. PhysAlign tests whether supplying a verified local reading changes the reported role, keeping the original problem, anchor, candidates, and scoring fixed. This control neither removes all perceptual uncertainty nor reveals internal reasoning. Table 18 compares the protocols of selected closely related benchmarks. Its criteria concern diagnostic interfaces, not overall benchmark quality. Associations between PhysAlign scores and independently evaluated problem solving are descriptive, not causal. Table 18: Protocol-level comparison with closely related benchmarks. Benchmark General evaluationLocalized diagnosisUtility Input Ctrl. Interm. Eval. Fixed Anchor Rec./Grd. Split Paired Rec. Ctrl. Det. Local Score Solve Link SeePhys [Xiang et al., 2025]✓✗ SeePhys Pro [Xiang et al., 2026]✓△✗ PhysicsArena [Dai et al., 2025]✗✓✗✓ MathVerse [Zhang et al., 2024]✓✗ PhysAlign (Ours)✓ Legend:✓explicit;△indirect or related;✗no corresponding protocol under the definitions below. Symbols indicate protocol features, not model performance or overall benchmark quality. Input Ctrl.: controlled changes to evidence availability or modality. Interm. Eval.: evaluation of intermediate reasoning stages or outputs. Fixed Anchor: evaluation at a predefined local image region or text span in an unchanged source problem. Rec./Grd. Split: separate recognition and role-grounding scores on the same probe. Paired Rec. Ctrl.: matched probes with and without correct local readings, with the grounding target withheld. Det. Local Score: rule-based local scoring without an LLM judge. Solve Link: association between separately evaluated diagnostic scores and original-problem correctness. For SeePhys Pro,△denotes modality-transfer-based diagnosis rather than direct intermediate-output scoring. F Scope, Limitations, and Release Documentation Measurement Scope. PhysAlign measures whether a model can map locally visible content to its correct physical role when given evidence anchors, candidate entities, and a constrained output interface. The protocol scores content recognition and role grounding separately, but it does not assess unconstrained, holistic diagram understanding. The +GT condition supplies only a verified local transcription: it reveals neither the correct candidate nor the role label, and it leaves uncertainty in entity boundaries, spatial relations, and physical semantics unresolved. The Base/+GT comparison therefore measures the change in grounding after a correct transcription is supplied; it is not an upper bound under perfect perception. 32 Interpretive Boundaries. Our measurements concern explicit local correspondences reported under fixed inputs. They do not recover a model’s latent representation, complete physical world model, or internal reasoning process. Local probes and original-problem solving are evaluated in separate contexts. Agreement or disagreement between them supports a distinction at the level of observable behavior, but cannot determine which local premise the model used, ignored, or propagated incorrectly in a particular solution. Likewise, a controlled substitution based on an incorrect role characterizes a possible consequence within a fixed derivation; it does not causally identify the model’s actual reasoning trace. Coverage and Release Boundaries. The current evaluation focuses on static educational physics diagrams and is skewed toward mechanics and electromagnetism. Retained probes cover reviewed local facts rather than every expressible relation in each problem, and paired +GT inputs are available only for a subset. Because the parent problems come from public datasets, we cannot exclude prior exposure to the original problems or semantically similar variants during pretraining. Parent-level split propagation keeps probes from one parent together, and exact-input deduplication removes identical public inputs; neither certifies a contamination-free evaluation. The reported experiments combine development and test within the retained release and should therefore be interpreted as release-derived diagnostics, not held-out test estimates. Appendix A documents provenance, review, retention, and coverage. These documentation priorities are consistent with Datasheets for Datasets [Gebru et al., 2021], which advocates explicit reporting of dataset composition, collection, and recommended uses. We plan to release the probe inputs and metadata, prompt templates, evaluation code, and recorded run configurations for which redistribution is permitted, while keeping public inputs separate from private reference targets and scoring mappings. Source problems and images will be redistributed only where upstream licenses permit; otherwise, the release will provide provenance identifiers and reconstruction instructions without asserting additional rights over third-party content. 33

Instant research discovery