Paper deep dive

One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling

Geigh Zollicoffer, Minh Vu, Rajiv Ranasinghe, Manish Bhattarai

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 45

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:World models, systems that generate what happens next given current environmental conditions, are increasingly being implemented with multi-modal generation in mind. However, generating multiple modalities simultaneously, such as visual simulations alongside physical state predictions in the form of text, introduces the risk of cross-modal inconsistency. Tested separately, both outputs may look convincing while still disagreeing: a model can calculate that a ball should rebound in one modality, then generate no rebound in another modality, to say nothing of diverging from real-world dynamics entirely. In this work we focus on two failures explicitly: \emph{Internal misalignment}, the disagreement between the world model's generated video and the same world model's prediction in a different modalities, and \emph{external misalignment} the disagreement between the world model's generation and an analytic physical environment. We derive common contracts of event, magnitude, timing, and construct a physics grounded pipeline to make comparisons measurable in both external and internal settings. We then ask whether progressively supplying the model's own contract (the A ladder for the internal setting) or a corrected physical contract (the B ladder for the external setting) closes the respective gaps. Across four mechanisms and 20 settings, we find that while language answers all 22 text probes correctly with respect to the true environment, the neutral video is often in disagreement, suggesting that the current unified backbones may not be capable of correct reasoning, internal consistency, and external physical fidelity all at once.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

41,854 characters extracted from source content.

Expand or collapse full text

One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling Geigh Zollicoffer Affiliation: Theoretical Division, Los Alamos National Laboratory Minh Vu Affiliation: Computing and Artificial Intelligence Division, Los Alamos National Laboratory Rajiv Ranasinghe Affiliation: Earth and Environmental Sciences Division, Los Alamos National LaboratoryLos Alamos, NM 87545 Manish Bhattarai Affiliation: Theoretical Division, Los Alamos National Laboratory Abstract World models, systems that generate what happens next given current environmental conditions, are increasingly being implemented with multi-modal generation in mind. However, generating multiple modalities simultaneously, such as visual simulations alongside physical state predictions in the form of text, introduces the risk of cross-modal inconsistency. Tested separately, both outputs may look convincing while still disagreeing: a model can calculate that a ball should rebound in one modality, then generate no rebound in another modality, to say nothing of diverging from real-world dynamics entirely. In this work we focus on two failures explicitly: Internal misalignment, the disagreement between the world model’s generated video and the same world model’s prediction in a different modalities, and external misalignment the disagreement between the world model’s generation and an analytic physical environment. We derive common contracts of event, magnitude, timing, and construct a physics grounded pipeline to make comparisons measurable in both external and internal settings. We then ask whether progressively supplying the model’s own contract (the A ladder for the internal setting) or a corrected physical contract (the B ladder for the external setting) closes the respective gaps. Across four mechanisms and 20 settings, we find that while language answers all 22 text probes correctly with respect to the true environment, the neutral video is often in disagreement, suggesting that the current unified backbones may not be capable of correct reasoning, internal consistency, and external physical fidelity all at once. 1 Introduction Imagine asking a model how high a dropped ball should rebound, and it answers “2.7 diameters.” Give the same scene and parameters to a different modality of a system, and the ball may not rebound at all. An issue arises when interfaces belong to one shared backbone; both outputs may look individually convincing; together they tell incompatible physical stories. This inconsistency is consequential. A world model used for robot planning, counterfactual evaluation, or synthetic data generation must preserve the effect of an intervention, not merely produce a realistic-looking future. A planner can exploit a video model that ignores friction, delay, or contact magnitude even when a language interface can state the governing relation. Conversely, a correct verbal answer is not evidence that the sampled future obeys it. Figure 1: A schematic of one generated future, two alignment gaps. From the same robot scene, text predicts a secure grasp and the Isaac physical reference lifts the vial, while generated video fails to realize the hold/lift. The video-text disagreement is internal misalignment; the video-environment disagreement is external misalignment. Both the generated video and text contract are outputted from the same world model backbone. The motivating robot case is grounded in an originating paired hold/slip check. The vial tracker detected the object in every frame. Normalized vertical rise was 0.4480.448 frame-heights for the Isaac hold (roughly centered in the video), but 0.0040.004 for the Cosmos hold (towards the bottom of the screen); the Cosmos slip was similarly near-zero (0.0060.006). We distinguish two alignment questions. Internal alignment asks whether the world model generated video realizes the same predictions as what the same backbone states through its language or vision-language modalities. External alignment asks whether video agrees with the analytic environment. These references answer different questions even when they happen to predict the same outcome: one tests cross-modal consistency, while the other tests physical correctness with the ground truth environment. Our experimental flow follows that distinction. We first measure both gaps and then apply a tiered intervention, which we summarize in the form of two different ladders. The A ladder progressively supplies the model’s own text-derived magnitude and trajectory, testing whether more internal information closes the video–text gap. The B ladder supplies corrected physical magnitude and trajectory, testing whether more oracle information closes the video–environment gap (see Figure 2 for a full visualization). Concretely, our work aims to investigate four main questions about the world models generated video: • Q1 (External alignment): Does the generated video match TphysT_phys, the environment reference? • Q2 (Internal alignment): Does the generated video match TtextT_text, the same model’s explicit prediction? • Q3 (A ladder): Does progressively adding TtextT_text reduce internal error of the generated video? • Q4 (B ladder): Does progressively adding corrected TphysT_phys reduce external error of the generated video? To measure alignment against the world model’s generated video, we measure and define each modality’s explicit, verifiable prediction i.e its predictive contract. From the text, we extract a binary event (e.g., bounce/no bounce or slide/no slide), an event magnitude (e.g., bounce height), and, where appropriate, keyframes and timestamps specifying when and where an object should appear. From the video, we recover the object’s pixel trajectory and express it in the same object-relative units used by the text-derived predictions. Ground-truth measurements are provided by the simulation environment. We use Isaac/PhysX because it provides a controllable external environment in which we can systematically vary object properties, spatial configurations, and physical parameters, such as mass, distance, and static friction. Contributions. (1) We formalize internal video–text alignment and external video–environment alignment with predictive contracts (§3). (2) We construct a four-mechanism intervention testbed and validated RGB reader to measure both gaps. (3) We introduce an A/B tiered testing design that evaluates whether progressively supplying self-derived or oracle physical information can close the corresponding gap. (4) We substantiate the resulting claims through canonical scoring of 448 clips, sensor-validity checks, and multi-seed evaluation. Figure 2: Two references and two tiered interventions (schematic). The same measured video contract E⁡(V)E(V) is compared with the model’s text contract for internal alignment and with Isaac/analytic physics for external alignment. The A ladder adds self-derived event, magnitude, and keyframes; the B ladder adds corrected physical information. 2 Related Work Physical evaluation of video generators. Benchmarks score generated video for physical plausibility and commonsense (Bansal et al., 2025; Li et al., 2025), or against real recorded physical experiments with explicit auditing of prompt and ground-truth confounds (Motamed et al., 2026). These establish that generators violate physics; they compare a generator to an external reference. We add two things such benchmarks cannot express: a within-checkpoint comparison of the generator against the same system’s explicit reasoning, and one-parameter interventions that test directional response rather than pointwise plausibility. Our result that most seeds retain one outcome class across three parameter sweeps is about the sampled conditional distribution, not any single clip’s realism or universal model behavior. Understanding–generation gaps. GapEval directly quantifies bidirectional understanding–generation consistency in unified multimodal models (Wang et al., 2026). Our study is narrower in model breadth and broader in physical instrumentation: it holds the scene fixed, intervenes on a known parameter, and compares pixel trajectories with both an external reference and the model’s explicit prediction. The present A/B internal-target experiment is diagnostic rather than an equivalence result (Q3 and Appendix). Multi-modal world models. Cosmos 3 couples an autoregressive understanding stream with a diffusion generation stream in one jointly-trained checkpoint (NVIDIA, 2026); V-JEPA 2 and related work frame video models as predictors for understanding and planning (Assran et al., 2025; Mazzaglia et al., 2024). Our architectural audit (§4) uses the released modeling code to establish exactly which parameters the two pathways share, which is what licenses the “reasoning is available to the generator, yet not realized” framing. Steering and aligning generated physics. Force prompting, physics priors, inference-time alignment with latent world models (Hafner et al., 2024; Zollicoffer et al., 2025), and concept-vector steering all improve or control physical behavior in generated video (Gillman et al., 2025; Narayanan et al., 2026; Yuan et al., 2026; Alam, 2026; Joseph et al., 2026). Our crossed correct/incorrect directive design is complementary: before steering, it measures what the text channel actually controls, event occurrence, in both directions, and what it does not, magnitude on an unmodified checkpoint. 3 Physics Fidelity Framework Let x denote an initial scenario comprising of a conditioning frame and a scene description with grounded physical measurements, including, but not limited to, object mass or weight, object dimensions, distances between objects or surfaces. Let θ denote a physical parameter, such as the coefficient of restitution e, coefficient of friction μ, launch speed v, or pendulum length L; and let V⁡(x,θ,p)V(x,θ,p) denote a video sampled from the generation pathway using prompt p. A reference simulator or governing physical equations provide Tphys(x,θ)T_phys(x,θ), the physically expected trajectory of the tracked object under scenario x and physical parameters θ. Before evaluation, we verify that this trajectory can be determined by the specified initial conditions and governing dynamics defined by scenario x (see Appendix A.4 for an explicit inclined-plane example). In contrast, Ttext(x,θ)T_text(x,θ) denotes the trajectory predicted by the model’s text pathways. A calibrated pixel extractor E maps videos to observable trajectories with distance converted to object-relative units (the object’s own diameter), making measurements camera-calibration-free (for an illustration see Figure 3). Formally, we represent each trajectory using a common predictive contract C=(z,m,u,K,I)C=(z,m,u,K,I), where z denotes the event class (e.g., slide/stay or bounce/no-bounce), m the measurable continuous quantity of interest (e.g., displacement, amplitude, or period), u its units, K any observable keyframes or timestamps, and I scene invariants such as object size and camera pose. The physical reference Tphys(x,θ)T_phys(x,θ) and the model’s text prediction Ttext(x,θ)T_text(x,θ) are expressed under this same schema (Ctext,CphysC_text,C_phys), while the video extractor EvE_v maps the generated video V to the corresponding observed contract Cvideo=E⁡(V)C_video=E(V). Using a shared schema ensures that like quantities are compared across modalities; for example, displacement is compared with displacement rather than period, and that quantities that are missing, corrupted, or below the extractor’s measurement resolution remain unscored. Reasoning fidelity first checks whether TtextT_text agrees with TphysT_phys. Both alignment tests then use the same measured video contract, Cvideo=E⁡(V⁡(x,θ,p))C_video=E\! (V(x,θ,p) ), while changing only the reference: Dint(x,θ,p)=d⁡(Cvideo,Ttext(x,θ)),Dext(x,θ,p)=d⁡(Cvideo,Tphys(x,θ)).D_int(x,θ,p)=d\! (C_video,T_text(x,θ) ), 20.00003ptD_ext(x,θ,p)=d\! (C_video,T_phys(x,θ) ). Thus internal misalignment is disagreement with the same checkpoint’s text or image–text prediction, while external misalignment is disagreement with the environment, with mechanism-appropriate distances d, each on the observable the extractor can read reliably from pixels: bounce, absolute error in rebound apex (ball-diameters); incline, regime agreement (slide/stay) plus displacement up to the visible-ramp ceiling (in cube-widths; in practice a binary slide/stay result); push, pushed/not pushed; pendulum, absolute error in swing amplitude (bob-diameters). A clip is tolerance-correct when measurable and within max⁡(0.5 units,25%) (0.5 units,25\%) of the reference, or in the correct regime for incline; these rules were fixed before scoring. DintD_int asks whether the generator realizes what the model itself predicted; DextD_ext asks whether that realization is physically correct. The (Dint,Dext)(D_int,D_ext) plane separates faithful-and-correct, faithful-but-wrong, unfaithful-but-correct, and unfaithful-and-incorrect. When Ttext=TphysT_text=T_phys, the two distances coincide numerically even though their interpretations remain different; separating them empirically requires cases in which the two targets differ. We then construct a tiered intervention (see Figure 2 for an illustration) that incrementally adds context to the scenario x. Tier 0 supplies the scene and parameter; Tier 1 adds the event. At Tier 2 and Tier 3: the A ladder adds the model’s own magnitude and keyframes and is evaluated by DintD_int, whereas the B ladder adds corrected physical magnitude and keyframes and is evaluated by DextD_ext. This paired construction asks whether additional information closes the intended gap. Degeneration. Generated videos can fail physically and ontologically: objects vanish, spawn, morph, or teleport, and frame geometry can corrupt through letterboxing or other discontinuities, a class our tracker does not gate and which we report by inspection. Every clip is gated into dynamic, static, degenerate; only measurable clips receive a number, and we report P⁡(measurable)P(measurable), P⁡(correct∣measurable)P(correct ), and an unconditional score treating degeneration as failure. Figure 3: One measured video contract, two alignment tests (schematic). The frozen RGB reader maps a rollout to event, magnitude, units, keyframes, and invariants. Comparing that same contract with TtextT_text yields DintD_int; comparing it with TphysT_phys yields DextD_ext. Vanish, duplicate, morph, teleport, and non-physical-drift failures are rejected rather than assigned a numerical score. 4 Experiments World Model Cosmos3-Super is one jointly-trained Mixture-of-Transformers checkpoint with separate 31.2B understanding and generation streams, 1.6B shared parameters, and two-way joint attention. Text, image–text, and video modes load the same shards but execute largely distinct stream parameters. During generation, prompt tokens pass through the understanding stream and generation queries attend to its keys and values. Appendix A.1 gives the full architectural and inference audit. Tested Scenarios (see Appendix A.3 for visualizations) Bounce (restitution). A ball is dropped onto a fixed pedestal from height h and rebounds to e2he^2h for coefficient of restitution e. Observable: rebound-apex height above the landing level, in ball-diameters. Incline (Coulomb friction). A cube rests on a 25∘25 ramp and slides iff μ<tan⁡θμ< θ, accelerating as a=g⁡(sin⁡θ−μcos⁡θ)a=g( θ-μ θ). Observable: distance travelled down-slope from the start, in cube-widths. Push (kinetic friction). A cube on a flat table is given an initial horizontal speed v0v_0 and, opposed only by kinetic friction, decelerates at a=μkga= _kg and coasts to rest after d=v02/(2μkg)d=v_0^2/(2 _kg). Observable: horizontal distance travelled before stopping, in cube-widths. Pendulum (timing). A bob on a rigid rod of length L is released from θ0 _0 and swings with period T=2πL/gT=2π L/g. Observable: horizontal swing amplitude about the lowest point, in bob-diameters (with the period read from zero-crossings). Grasp (contact friction). A Franka gripper closes on a vial, which is lifted iff Coulomb friction overcomes gravity, μ⋅grip≥mgμ·grip≥ mg. Observable: vertical rise of the vial. An additional setting we also consider. Reference simulator Isaac Sim 6.0.1 (PhysX) supplies controllable bounce, incline, projectile, and pendulum scenes with one intervened parameter and one pixel-measurable observable each (Appendix Table 3). Evaluator and validation A conditioning-frame–locked tracker maps every clip to event, object-relative magnitude, object-relative displacement, and corruption status (Fig. 3). On six lossless Isaac feeds it recovers event class on 6/6, and in our experiments RGB and sensor traces agree framewise (R2=0.989/0.997R^2=0.989/0.997). Appendix A.3 reports sensor, camera, metric-amendment, and adjudication details. Protocol For each (mechanism, θ), the identical scenario is posed to the text and image+text pathways (temperature 0), and the video pathway generates from the same conditioning frame under a neutral prompt (sizes and parameter stated; outcome never stated). The tiered design shares T0 (scene/parameter) and T1 (event), then separates the model-derived A and corrected-physics B references at T2 (magnitude) and T3 (keyframes). The paired final-script A/B arm contains 32 clips at four seeds. Because most measurable TtextT_text and TphysT_phys targets coincide, it is diagnostic rather than a powered comparison of ladder slopes. Wrong directives (W1, W2) The wrong-directive arms replace the correct language with a false statement of the outcome while holding the physical parameter, conditioning frame, and duration fixed. W1 is qualitative-wrong, it asserts the event does not happen (“the ball does not bounce,” “the bob remains hanging and does not swing”), and W2 is magnitude-wrong, commanding a false extent (e.g. a 0.30.3-diameter rebound where physics dictates 2.72.7). Their purpose is to test the direction of language control: if the tier effect were the model recalling physics from the parameter, a false description should not matter. 5 Results 5.1 Prerequisite: Is the text reference well defined and correct? Across the 20-setting sweep plus two pendulum wording permutations, the text pathway is exact on 22/22 probes (e.g. 0.0333 diameters at e=0.1e=0.1, 31.9 diameters at v=5v=5), and image+text is 19/22 strict: all 19 parsed final answers are correct over binary event prediction (verified with the environment directly; three responses exhaust a 4096-token budget). 5.2 External and internal alignment under intervention Where text and physics declare the same observable and the text pathway is correct, a neutral video that misses the reference simultaneously exhibits external misalignment with TphysT_phys and internal misalignment with TtextT_text. Figure 4 shows the parameter-response sweeps: 20 settings (restitution e∈0.1,…,0.9e∈\0.1,…,0.9\; friction μ∈0.2,…,0.8μ∈\0.2,…,0.8\ bracketing the slide threshold tan⁡25∘=0.466 25 =0.466, at which PhysX itself transitions between μ=0.45μ=0.45 and 0.500.50; launch speed v∈1.5,…,5.0v∈\1.5,…,5.0\ m/s; length L∈0.25,…,1.0L∈\0.25,…,1.0\ m), eight seeds each, 160 neutrally-prompted 81-frame clips, scored in one frozen pass with the start box declared from each conditioning frame before scoring. Sensitivity to small prompt edits Changing the parameter value edits one token of a ∼ 120-token prompt at fixed noise, and one might expect any such edit to leave the sample nearly unchanged. However, edits of comparable size that name an event, i.e “the cube slides down the ramp” vs. “the cube stays in place,” “swings back and forth” vs. “remains hanging”, flip the outcome class of the same seeds (incline 0/8→6/80/8→ 6/8; pendulum 7/8→3/87/8→ 3/8), suggesting that insensitivity is not likely a cause of unresponsiveness. The conditioning channel is responsive to words about what happens; it is unresponsive to the number that determines what happens. Guidance affects swing occurrence but not amplification Regenerating the L=1L=1 neutral condition on Pendulum with a change of guidance (a hyperparameter which controls sensitivity to the prompt during video generation) to 3.0 and 9.0 (four seeds each) leaves seed 234 swinging with an over-large amplitude at all three scales. Seed 345 swings at guidance 6.0 but freezes at 3.0 and 9.0; seeds 123 and 456 remain frozen. Thus the gain persists conditional on a rendered swing. In the over-amplified clips the bob’s pixel diameter varies by only 16–26% across the swing (median 19–22 px), while the measured amplitude exceeds the physical maximum by ∼ 90%; a depth excursion large enough to inflate the amplitude that much would roughly double the apparent bob size. Table 1: Parameter-sensitivity summary under neutral prompting (Tier 0, 8 seeds per setting). Class-constant: seeds whose outcome class never changes across the sweep. Event-rate ρ: Spearman of P⁡(dynamic)P( dynamic) vs. parameter across settings (undefined when constant). Correct: unconditional tolerance-correct clips over settings above evaluator resolution. Mechanism settings class-constant seeds event-rate ρ correct (uncond.) Bounce 5 6/8 −0.58-0.58 (p=0.31p=0.31) 1/24 Incline 6 8/8 N/A slide 0/24; stay 24/24 Push 5 7/8 −0.06-0.06 (p=0.83p=0.83) 2/40 Pendulum 4 2/8 0.800.80 (p=0.20p=0.20) 0/32 Figure 4: Parameter-response sweeps under neutral prompting (Tier 0). Each panel: one mechanism, x = the main physical parameter stated in the prompt, y = the observable read from pixels in object-relative units; one line per diffusion seed; dashed = Isaac reference (for incline, the visible-ramp ceiling of the evaluator, 4.7 cube-widths, is drawn for the slide regime); × = corrupted, degenerate, or unscorable clip. Eight seeds per setting. Three mechanisms are flat and seed-striped; the pendulum swings only at L≥0.5L≥ 0.5, non-monotonically in L, with a 1.2–2× non-physical gain. 5.3 Can corrected information close misalignment? We cross the physical parameter (Q1–Q2) with language that states a correct outcome—neutral T0 (parameter only), T1 (qualitative), T2 (magnitude), T3 (keyframed)—or a deliberately incorrect one (W1 qualitative-wrong, W2 magnitude-wrong), holding parameter, conditioning frame, and duration (81 frames) fixed across eight seeds; all 192192 clips (44 axes × 66 conditions × 88 seeds) are scored in the canonical pass (Table 2). We call a clip valid when it is dynamic and its observable lies within tolerance of the commanded (or true) magnitude; corrupted or degenerate otherwise. On pendulum, W1 suppresses the swing in five of eight seeds (3/83/8 dynamic vs. 66–7/87/8 under every correct tier), and on incline W2’s small commanded slide collapses displacement to 1/81/8. The wrong directives thus establish that the tier unlocking is caused by the language content, the generator follows the stated outcome, in either direction, rather than by the physical parameter or the conditioning image, which by themselves leave the event unrendered. Language controls whether the binary event occurs, in both directions. On incline at μ=0.2μ=0.2 (truth: slide), neutral and T1 prompts leave the cube frozen (0/80/8 each, CI [0,0.32][0,0.32]), T2 unlocks motion in 3/83/8 seeds, and the keyframed T3 prompt unlocks it in 6/86/8 (CI [0.41,0.93][0.41,0.93]), saturating at the evaluator’s visible-ramp ceiling (4.7 cube-widths). On pendulum at L=1L=1, where six to seven of eight seeds swing under any correct-directive tier, the wrong directive “remains hanging, does not swing” suppresses the swing in five of eight seeds (W1: 3/83/8 dynamic, CI [0.14,0.69][0.14,0.69], vs. 66–7/87/8 under T0–T3). Neither effect is available to the physical parameter itself. Language does not control magnitude. A commanded rebound height of 2.72.7 diameters (T2) yields one valid bounce, four corrupted or degenerate clips, and three low-amplitude events with overlapping observables (T2: 0.620.62–2.362.36 diameters). Even an explicit “does not bounce” instruction (W1) produces two valid bounces and four corrupted trajectories. A commanded swing of 44 bob-diameters (T2) yields over-amplified swings (7/87/8 dynamic, median 7.07.0). The one suggestive magnitude effect is on incline, where a commanded 1515-cube-width slide (T2: 3/83/8 dynamic, up to 4.84.8 cube-widths) elicits more displacement than a commanded 22-cube-width slide (W2: 1/81/8, up to 1.11.1); at eight seeds this is a consistent direction with overlapping intervals, not a calibrated response. Rendered motion is not rendered physics. The push axis does move—33–44 of 88 seeds render a slide at every tier—but the motion never obeys the commanded magnitude: the median travel is a fixed ∼10 10 cube-widths (about 2.6×2.6× the true stopping distance) and is independent of the stated v0v_0, so only 00–1/81/8 clips fall within tolerance. Even the magnitude-wrong directive W2 still slides (4/84/8), and 11–55 of 88 clips per tier are degenerate or unscorable. Bounce T1 and T3 (the most explicit correct directives) are corrupted or degenerate in 6/86/8 and 7/87/8 seeds. Does a richer B ladder close the quantitative external gap? No tier is generally monotone in physical fidelity. Only incline shows a dose–response (0/8,0/8,3/8,6/80/8,0/8,3/8,6/8 slide), and this reaches the evaluator’s visible ceiling rather than the commanded magnitude. Bounce is non-monotone with only one tolerance-correct clip; pendulum remains over-amplified; push renders motion but stays parameter-blind, holding a fixed ∼10 10-cube-width slide regardless of tier or commanded v0v_0 (00–1/81/8 within tolerance). Richer specification therefore controls whether some renderable motion occurs more reliably than how much, while sometimes increasing degeneration. Table 2: Event occurrence vs. correctness across eight seeds, scored two ways: External (against the true physics value) and Internal (against the model’s own asserted value). Valid Event: seeds (of 88) that render any valid event; Valid Magnitude: seeds whose observable is additionally within tolerance of the target (±30%± 30\%; incline counts reaching the visible-ramp ceiling; push is stopping distance in cube-widths). Correct directives T0–T3 add increasingly explicit correct language; wrong directives W1 (qualitative-wrong) and W2 (magnitude-wrong) state a false outcome. External (vs. physics) Internal (vs. model’s claim) Axis T0 T1 T2 T3 W1 W2 T0 T1 T2 T3 W1 W2 Bounce Valid Event 2 1 4 1 4 4 2 1 4 2 4 4 Valid Magnitude 0 0 1 1 1 2 0 0 3 2 1 2 Incline Valid Event 0 0 3 6 0 1 0 0 3 6 0 1 Valid Magnitude 0 0 2 5 0 0 0 0 2 5 0 0 Push Valid Event 3 3 4 3 1 4 3 3 4 3 1 4 Valid Magnitude 0 0 1 1 0 1 0 0 1 1 0 1 Pendulum Valid Event 6 6 7 7 3 7 6 6 7 6 3 7 Valid Magnitude 3 2 1 1 1 1 0 0 0 0 0 0 6 Limitations Isaac/PhysX is simulation, not hardware truth. We test one model series, four mechanisms, and eight sweep seeds. Appendix A records the complete qualifications. 7 Conclusion We evaluate whether multimodal world models remain consistent across their own modalities and with external physical dynamics. Using shared contracts over event, magnitude, and timing, we compare generated video against both the model’s text-derived prediction, TtextT_text, and a physics-grounded reference, TphysT_phys, separating internal from external misalignment. Across four physical mechanisms and 20 settings, the text pathway answers all 22 physics probes correctly with respect to the reference environment, yet neutral video generation frequently fails to realize the corresponding dynamics, showing that correct reasoning in one modality does not guarantee either cross-modal consistency or physical fidelity in another. Intervention ladders that progressively supply the model’s own predicted contract or a corrected physical contract can alter behavior, but do not consistently close these gaps across mechanisms. These results suggest that current unified multimodal backbones may not be able to express a single coherent physical world model across modalities, and that evaluating world models requires testing not only whether each output is plausible in isolation, but whether their predictions agree with one another and with the environment they are intended to model. References Alam (2026) N. Alam Causal physics steering in video world models via concept activation vectors. arXiv preprint arXiv:2605.24322. External Links: Link Cited by: §2. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas V-JEPA 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. External Links: Link Cited by: §2. Bansal et al. (2025) H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y. Bitton, C. Jiang, Y. Sun, K. Chang, and A. Grover VideoPhy: evaluating physical commonsense for video generation. In International Conference on Learning Representations, Cited by: §2. Gillman et al. (2025) N. Gillman, C. Herrmann, M. Freeman, D. Aggarwal, E. Luo, D. Sun, and C. Sun Force prompting: video generation models can learn and generalize physics-based control signals. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §2. Hafner et al. (2024) D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap Mastering diverse domains through world models. External Links: 2301.04104, Link Cited by: §2. Joseph et al. (2026) S. Joseph, Q. Garrido, R. Balestriero, M. Kowal, T. Fel, S. Bakhtiari, B. Richards, and M. Rabbat Interpreting physics in video world models. arXiv preprint arXiv:2602.07050. External Links: Link Cited by: §2. Li et al. (2025) D. Li, Y. Fang, Y. Chen, S. Yang, S. Cao, J. Wong, M. Luo, X. Wang, H. Yin, J. E. Gonzalez, I. Stoica, S. Han, and Y. Lu WorldModelBench: judging video generation models as world models. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §2. Mazzaglia et al. (2024) P. Mazzaglia, T. Verbelen, B. Dhoedt, A. Courville, and S. Rajeswar GenRL: multimodal-foundation world models for generalization in embodied agents. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §2. Motamed et al. (2026) S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos Do generative video models understand physical principles?. In IEEE/CVF Winter Conference on Applications of Computer Vision, p. 948–958. External Links: Link Cited by: §2. Narayanan et al. (2026) S. Narayanan, Z. Jiang, S. Narasimhan, and M. Chandraker PhyCo: learning controllable physical priors for generative motion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 41892–41902. External Links: Link Cited by: §2. NVIDIA (2026) NVIDIA Cosmos 3: omnimodal world models for physical ai. arXiv preprint arXiv:2606.02800. External Links: Link Cited by: §2. Wang et al. (2026) C. Wang, Y. Chen, Z. Hu, D. Chen, W. Chen, S. Wiegreffe, and T. Zhou Quantifying the gap between understanding and generation within unified multimodal models. arXiv preprint arXiv:2602.02140. External Links: Link Cited by: §2. Yuan et al. (2026) J. Yuan, X. Zhang, F. Friedrich, N. Beltran-Velez, M. Hall, R. Askari-Hemmat, X. Han, N. Ballas, M. Drozdzal, and A. Romero-Soriano Inference-time physics alignment of video generative models with latent world models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16118–16129. External Links: Link Cited by: §2. Zollicoffer et al. (2025) G. Zollicoffer, K. Eaton, J. C. Balloch, J. Kim, W. Zhou, R. Wright, and M. Riedl Novelty detection in reinforcement learning with world models. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2. Appendix A Additional Analyses A.1 Architecture and inference audit Cosmos3-Super has 64 layers at hidden size 5120. Each layer contains an understanding stream (31.2B parameters; separate attention projections and MLPs) and a generation stream (31.2B; separate MMDiT-style projections and MLPs), with approximately 1.6B shared parameters, a 0.6B vision encoder, and a 0.7B video VAE. Text and image–text reasoning use temperature 0; video uses UniPC at 30 steps, guidance 6.0, flow-shift 10, 24 fps, 640×480640×480, and 49–81 frames. Released-code inspection verifies causal self-attention over prompt tokens in the understanding stream and generation attention over concatenated understanding/generation keys and values. This establishes a route, not where quantitative content is lost. The video-encoder patch is a numerically equivalent device-placement fix, verified on two input sizes. A.2 Scene and solver specifications The scenes are a 6 cm, 0.2 kg ball dropped 20 cm onto a 50×50×1050×50×10 cm pedestal (restitution combine mode max); a 6 cm, 0.2 kg cube on a 90 cm ramp at 25∘25 with static and dynamic friction equal (combine mode min); a 6 cm, 0.2 kg cube given an initial horizontal speed of 1.2 m/s on a flat table with kinetic friction μk=0.3 _k=0.3 (combine mode min); and a 10 cm pendulum bob released from rest at 25∘25 . Isaac Sim 6.0.1 uses a 60 Hz physics step, the default TGS solver, and no per-scene solver overrides. The originating Franka vial-grasp case motivates the study but is not counted among these four controlled mechanisms. Table 3: Controlled mechanisms, interventions, and pixel observables. Lengths are measured in the tracked object’s own diameter or width. Axis Scene θ values Observable Bounce 6 cm ball, 20 cm drop e∈0.1,0.3,0.5,0.7,0.9e∈\0.1,0.3,0.5,0.7,0.9\ rebound apex (e2⋅e^2·drop) Incline 6 cm cube, 25∘25 ramp μ∈0.2,0.35,0.45,0.5,0.6,0.8μ∈\0.2,0.35,0.45,0.5,0.6,0.8\ stay/slide; displacement Push 6 cm cube, flat table (μk=0.3 _k=0.3) v0∈0.6,0.9,1.2,1.5,1.8v_0∈\0.6,0.9,1.2,1.5,1.8\ m/s stopping distance (v02/2μkgv_0^2\!/2 _kg) Pendulum 10 cm bob, 25∘25 release L∈0.25,0.5,0.75,1.0L∈\0.25,0.5,0.75,1.0\ m amplitude (half peak-to-peak) A.3 Test Setting Visualizations We evaluate four parameterized mechanics settings, each realized as an Isaac Sim twin that supplies the exact ground-truth observable and a single conditioning frame (Figure 5). The four axes isolate distinct physical mechanisms—restitution, Coulomb friction, ballistic motion, and oscillatory timing—and each observable is expressed in the object’s own diameter so it can be read back from generated pixels and validated against exactly what the conditioning specified. Figure 5: The four controlled physics settings, shown as their Isaac-twin conditioning frames. Bounce (restitution, rebound =e2h=e^2h): a ball released above a pedestal. Incline (Coulomb friction, slides iff μ<tan⁡θμ< θ): a cube on a 25∘25 ramp. Push (kinetic friction, stop =v02/(2μkg)=v_0^2/(2 _kg)): a cube given an initial velocity along a flat table. Pendulum (timing, T=2πL/gT=2π L/g): a bob on a rigid rod released from 25∘25 . Each axis has one pixel-measurable observable in object-diameter units (rebound apex, down-slope displacement, stopping distance, swing amplitude). A.4 Explicit Validation Example Example verification of TphysT_phys: inclined plane. Scenario: Consider a cube of width w=0.06mw=0.06\,m placed at rest at the top of a ramp of length 0.90m0.90\,m inclined at α=25∘α=25 , with coefficient of friction μ=0.2μ=0.2. Solution: The cube slides when the downslope component of gravity exceeds the maximum frictional resistance, equivalently when μ<tan⁡α.μ< α. Here, 0.2<tan⁡25∘≈0.466,0.2< 25 ≈ 0.466, so the cube is expected to slide. Under the standard Coulomb-friction model, its downslope acceleration is a=g(sinα−μcosα)≈9.81(sin25∘−0.2cos25∘)≈2.37m/s2.a=g( α-μ α)≈ 9.81( 25 -0.2 25 )≈ 2.37\,m/s^2. Starting from rest, the expected displacement is therefore d⁡(t)=12at2.d(t)= 12at^2. The cube reaches the end of the 0.90m0.90\,m ramp after tedge=2(0.90)2.37≈0.87s.t_edge= 2(0.90)2.37≈ 0.87\,s. Expressed in the object-relative units used by our video extractor, the total expected displacement is 0.900.06=15 0.900.06=15 cube-widths. Thus, for this scenario, TphysT_phys predicts a monotonic downslope trajectory reaching approximately 1515 cube-widths of displacement at 0.87s0.87\,s. We additionally verify this analytical prediction against the reference simulator’s state trajectory, providing a direct check that the scenario specification and governing dynamics uniquely determine the physical reference used for evaluation. A.5 Original pilot and canonical replacement The originating four-point protocol crossed four mechanisms, six prompt cells (T0, T1, A-T2, A-T3, B-T2, B-T3), and two seeds (123 and 42), for 48 clips. Its validity gate found text–image+text agreement on all 7 comparable probes, while video matched physics on only 1/8; repairing the ambiguous short-pendulum wording yields the current 8/8 text and image+text prerequisite. In the two-seed ladder, B-T3 produced a dynamic incline in 1/2 clips and a dynamic pendulum in 2/2, whereas every post-T0 bounce clip was degenerate. Thus the pilot suggested that explicit language could switch a coarse event without establishing magnitude fidelity. The canonical study broadens the ladder to eight seeds; its horizontal-translation axis is push (slide-to-stop under kinetic friction), which renders a slide but at a fixed, parameter-blind magnitude—sharpening the same event-vs-magnitude dissociation on an axis that does move. A.6 Evaluator qualification details The canonical reader uses per-axis color segmentation and a box declared from the conditioning frame. Object loss yields degenerate; a rebound to release height or sustained upward drift yields corrupted. The incline camera exposes only 4.5 of 15 cube-widths, and the push camera’s Isaac feed is reliable only at its endpoints (replicator substeps corrupt the intermediate frames), so its pixel scale is fixed from the release-to-rest displacement (23.4 px per cube-width, recovering the 3.91-cube-width truth). We use pendulum half peak-to-peak amplitude because deviation from a temporal mean is phase-biased; applying it to 110 clips changes 72 magnitudes but no event class. Sensor checks find bounce apex error at most 13.8%, correct incline classification around tan⁡25∘=0.466 25 =0.466, push stopping-distance error at most 4.0% against the Isaac twin, and pendulum radius stability to 2.1×10−52.1×10^-5 m. Short-pendulum simulator amplitude is 58.5% from analytic truth (0.8% at L=1L=1), so analytic amplitude is the target. A.7 Stochastic realization With eight seeds per setting we can separate “the model cannot produce the correct dynamics” from “default sampling rarely selects them.” Using the tolerance rule of Q1–Q2 (degenerate and clamp-hit clips count as failures; sub-resolution settings excluded), per-seed success@1 vs. an oracle best-of-8 that accepts a setting if any seed is correct: bounce 0.04→0.330.04→ 0.33, pendulum 0.00→0.000.00→ 0.00, incline 0.50→0.500.50→ 0.50 (the frozen output is correct exactly in the stay regime, for every seed), push 0.05→0.200.05→ 0.20. The oracle number is a diagnostic capacity bound: it uses simulator truth to select and is not deployable. Thus a tolerance-correct bounce exists for one third of scored settings, while no tolerance-correct pendulum appears in the eight-seed support and push is tolerance-correct only at the highest speed (v0=1.8v_0=1.8 m/s, where its fixed ∼10 10-cube-width slide happens to fall within tolerance), and neutral prompting contains no incline slide. Invalid-trajectory probability is 0.300.30 (bounce), 0.280.28 (push), and 00 (incline, pendulum). A.8 Second checkpoint: Cosmos3-Nano We repeat the eight settings with the same frames, neutral prompts, 81-frame length, canonical scorer, and four seeds (32 clips). Nano changes how often motion appears, but not parameter sensitivity: bounce counts match across e=.3/.9e=.3/.9, incline slides at both μ=.2/.8μ=.2/.8, push slides in 3/43/4 seeds at both v0=.6/1.8v_0=.6/1.8 yet renders the same ∼1.5 1.5-cube-width displacement regardless of commanded speed, and the long pendulum swings in 4/44/4 seeds with the same 1.21.2–1.8×1.8× gain. This supports a shared failure family, not a scale comparison. Three neutral paraphrases per dynamic setting (24 clips) alter individual samples but not the seed-dominated conclusion.

Instant research discovery