Paper deep dive
CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations
Aman Mehta, Riya Baviskar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/14/2026, 4:53:46 AM
Summary
The paper introduces CALIPER, a benchmark designed to evaluate whether pretrained visual encoders genuinely infer physical properties (mass and friction) or rely on visual shortcuts. The authors demonstrate that in clean, fixed-camera scenes, all representations—including random ones—perform near-ceiling because displacement is directly visible in pixel coordinates, making it impossible to rank models. However, when camera, lighting, and clutter are varied (nuisance scene), the benchmark successfully separates representations, with V-JEPA 2 outperforming others. The study also critiques linear probing, showing that decodability does not imply usage, as erasing physical features affects performance differently depending on scene complexity.
Entities (7)
Relation Signals (6)
Clean Scene → failstorank → Visual Representations
confidence 96% · in the clean scene every representation lands within 0.02 R^2 of the ceiling... because a fixed camera exposes the object's displacement directly
Nuisance Scene → enablesranking → Visual Representations
confidence 95% · Resampling camera, lighting, and clutter... spreads the same representations across 0.50 R^2
CALIPER → evaluates → Physical Inference
confidence 95% · CALIPER is a direct test... to predict how far the object slides... checks that the evidence is actually used.
Swap Test → verifies → Physical Inference
confidence 93% · Swapping in another object's calibration clips checks that the evidence is actually used.
Linear Probing → isuninformative → Model Usage
confidence 92% · Linear probes track none of this... erasing the probed mass direction... costs nothing in one scene and 0.35 R^2 in the other.
V-JEPA 2 → outperforms → Random ViT
confidence 90% · V-JEPA 2 misses by 4 mm and the random ViT by 20 mm
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as the perception front end of world models for manipulation, and their physical competence is assessed with perturbation benchmarks and linear probes, almost always in a clean, fixed-camera scene. We show that these assessments cannot distinguish an encoder that infers physics from one that does not. CALIPER (calibrate, then predict) is a direct test: an object of unknown mass and friction is struck twice at known speeds, a third strike is shown only up to the moment of contact, and a linear readout on frozen features must predict how far the object slides. Swapping in another object's calibration clips checks that the evidence is actually used. Across 2,000 simulated episodes and eight representations, from V-JEPA 2 to a randomly initialised ViT and raw pixels, calibration adds +0.50 R^2 and the swap removes it. Yet in the clean scene every representation lands within 0.02 R^2 of the ceiling set by true simulator state, because a fixed camera exposes the object's displacement directly in pixel coordinates. Resampling camera, lighting, and clutter for every clip spreads the same representations across 0.50 R^2; when the readout chooses a push speed for a goal distance, V-JEPA 2 misses by 4 mm and the random ViT by 20 mm, no better than ignoring the object. Linear probes track none of this: a change in frame aggregation moves a probe more than pretraining does, and erasing the probed mass direction from the same representation costs nothing in one scene and 0.35 R^2 in the other. Whether a benchmark can rank models is an empirical property, and we give three checks that establish it.
Tags
Links
- Source: https://arxiv.org/abs/2609.08250v1
- Canonical: https://arxiv.org/abs/2609.08250v1
Trouble viewing inline? Open PDF directly →
Full Text
31,647 characters extracted from source content.
Expand or collapse full text
CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations Aman Mehta †thanks: heyamanmehta@gmail.com Affiliation: Independent Researcher Riya Baviskar †thanks: riyakbaviskar@gmail.com Affiliation: Independent Researcher Abstract How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are the usual perception front end of world models for manipulation, and their grasp of such latent physics is judged with perturbation benchmarks and linear probes, almost always in a clean, fixed-camera scene. We show that these evaluations cannot tell an encoder that infers physics from one that does not. CALIPER (calibrate, then predict) strikes an object of unknown mass and friction twice at known speeds, shows a third strike only up to the moment of contact, and asks a linear readout on frozen features to predict the slide; swapping in another object’s calibration clips checks that the evidence is used. Across 2,000 simulated episodes and eight representations, calibration adds +0.50+0.50 R2R^2 and the swap removes it, yet in the clean scene every representation, a randomly initialised ViT included, lands within 0.02 R2R^2 of the simulator-state ceiling, because a fixed camera exposes the displacement directly in pixel coordinates. Resampling camera, lighting, and clutter for every clip spreads the same representations across 0.50 R2R^2; asked to choose a push speed for a goal distance, V-JEPA 2 misses by 4.0 m and the random ViT by 19.6 m, no better than ignoring the object. Linear probes track none of this: a change in frame aggregation moves a probe more than pretraining does, and erasing the probed mass direction from the same representation costs nothing in one scene and 0.350.35 R2R^2 in the other. A benchmark’s ability to rank models has to be measured, and we give three checks that do so. 1 Introduction A light box and a heavy one of the same shape look identical until something moves them. A world model that is meant to support manipulation therefore has to infer mass and friction from the interactions it has watched, and then use that inference to predict what an action it has not yet seen will do. Pretrained visual encoders such as V-JEPA 2, VC-1, and R3M have become the standard front end for such models, and they are chosen on the strength of evaluations that are supposed to measure this ability. The two most common are single-parameter perturbation, where a scene is rendered twice with one property changed and a large change in the representation is read as physical sensitivity, and probing, where a linear map is fit from activations to the true property. Both are almost always run in a scene with one object, a fixed camera, and nothing else in view. We find that neither test, in that scene, can tell an encoder that infers physics from one that does not. They give nearly the same score to every encoder we tried, including one whose weights were never trained. The encoders are not at fault; the evaluations were simply never checked for whether they could rank anything. Our test, CALIPER (calibrate, then predict), is built so that the answer is not in the input (Figure 1). A target of unknown mass and friction is struck twice by a standard puck at known speeds, and the two resulting slides, the calibration clips, are shown in full. A third strike at a new speed, the query, is shown only up to the moment of contact, and the model has to predict how far the object will slide. Appearance is sampled independently of physics, so no frame gives the answer away. Because the calibration clips are separate inputs we can also swap them for another object’s; any accuracy that survives the swap did not come from physical inference. We run the task in two scenes, one with a fixed camera and an empty table and one in which camera pose, lighting, floor colour, and two distractor objects are resampled for every clip, and we compare eight representations under one linear readout: V-JEPA 2, VideoMAE, VC-1, DINOv2, R3M, a randomly initialised ViT, raw pixels, and frame differences. Every claim is checked against simulator state that no learned model receives. The test itself behaves as intended. Calibration raises R2R^2 by +0.50+0.50 for every representation and the swap removes all of it, so the readout is using the object’s own interaction history. What the clean scene then shows is that this is not enough to rank anything: the eight representations span only 0.020 R2R^2, raw pixels equal V-JEPA 2, and the random ViT is within 0.020 of both. The reason is mechanical. With a fixed camera the object’s displacement becomes a fixed pattern of pixel coordinates, and any injective feature map preserves that pattern, so a linear readout recovers the slide from almost anything. A perturbation benchmark we audit turns out to have the opposite problem: in 13 of 20 of its task-property cells the perturbation moves the simulated outcome by less than 10−410^-4, so there was nothing to measure. Once camera, lighting, and clutter vary from clip to clip, the same task does separate the representations, and by a wide margin: the spread grows to 0.498 R2R^2, V-JEPA 2 keeps 0.974, the per-frame encoders drop to about 0.70.7, and the random ViT loses its calibration gain entirely. Asked to choose a push speed that will reach a goal distance, V-JEPA 2 misses by 4.0 m and the random ViT by 19.6 m, which is what you get by ignoring the object. Linear probes track none of this. Concatenating frames instead of averaging them raises the random ViT’s friction probe from 0.361 to 0.672, above VC-1, without changing its prediction at all, and erasing the probed mass direction from raw pixels costs −0.001-0.001 R2R^2 in the clean scene but −0.350-0.350 under nuisance variation, from the same probe accuracy. Our conclusion is that a benchmark’s ability to separate representations has to be demonstrated rather than assumed. We end with three checks that demonstrate it, none of which the evaluations we started from would have passed. 2 Related Work How visual encoders for robotics are judged. VC-1 (Majumdar et al., 2023), R3M (Nair et al., 2022), V-JEPA 2 (Assran et al., 2025), VideoMAE (Tong et al., 2022), and DINOv2 (Oquab et al., 2024) are compared by downstream control success, which mixes perception, physical inference, and policy optimisation into one number. Freezing the encoder and fixing the readout isolates the middle term, at the cost of measuring only what is linearly available. Physics benchmarks. IntPhys (Riochet et al., 2018), Physion (Bear et al., 2021), and PHYRE (Bakhtin et al., 2019) score whether a model’s physical predictions are correct. None reports whether its design could have ranked the models it compares: what a trivial baseline scores, or whether the perturbations it applies change the outcome. Those are the questions we ask. Identifying physics from motion. Inferring object properties from how they move is a classical problem (Wu et al., 2015; Wu et al., 2016), and the same idea underlies learning by poking (Agrawal et al., 2016) and rapid adaptation from a short proprioceptive history (Kumar et al., 2021). We turn it into an evaluation. The calibration clips are the identification opportunity, and the swap tests whether it was taken. Decodability versus use. Control tasks and amnesic probing established, for language models, that a property being linearly decodable from a representation does not mean the model uses it (Hewitt and Liang, 2019; Belinkov, 2022; Elazar et al., 2021), and stated reasoning can diverge from actual reasoning (Turpin et al., 2023). We bring the erasure test to visual physics and add one observation: the erasure cost of the same property from the same representation changes with the scene, so decodability is uninformative about use even within a single model. 3 Calibration-Conditioned Prediction Figure 1: One CALIPER episode. A standard puck strikes a target of unknown mass and friction at two known speeds (calibration), and the resulting slides reveal the target’s physics. A third collision at a new speed is the query; the model’s input stops at the moment of contact, so it contains the action but not its consequence. The model predicts the displacement that follows. Appearance is sampled independently of mass and friction. Environment. Each episode contains one target: a box with mass m∈[0.15,0.75]m∈[0.15,0.75] kg (log-uniform), sliding friction μ∈[0.12,0.32]μ∈[0.12,0.32], and half-extents, colour, and aspect ratio drawn independently of (m,μ)(m,μ), simulated in MuJoCo (Todorov et al., 2012). A puck of fixed mass strikes the target head-on at a commanded speed. We use collisions rather than applied forces because a fixed force leaves objects below the friction threshold stationary and sends the rest out of frame, whereas collision recoil is bounded by the impact speed for any mass ratio (Appendix B). Two calibration collisions at 0.450.45 and 0.750.75 m/s are shown in full. The query collision, at a speed drawn from [0.40,0.85][0.40,0.85] m/s, is truncated at first contact. The regression target is log displacement during the query. Two scenes. In the clean scene the camera, lighting, and floor are fixed and the table holds only the puck and the target. In the nuisance scene camera pose, lighting, floor tint, and two distractor objects are resampled for every clip, so the three clips of an episode do not share a frame of reference; each draw is checked to keep the workspace in view. Physics, the distribution of (m,μ)(m,μ), and the readout are identical across the two scenes. Anything that changes between them is a property of the scene, not of the task. Readout. A frozen encoder maps each clip to one vector. Each clip’s vector is centred and reduced by PCA on the training rows, standardised, concatenated with the commanded query speed, and fit by ridge regression with the penalty chosen by inner cross-validation, under repeated 5-fold cross-validation. All conditions share one component budget, split equally across clips, so adding a clip never adds parameters. Four conditions nest: action only (the commanded speed); query (adds the truncated query clip); query++calibration (adds the object’s own calibration clips); and query++swapped calibration (another episode’s clips, assigned by a derangement). We report R2R^2 on held-out episodes and do not interpret differences of a few hundredths, since repeated K-fold understates variance. Simulator state sets the reference points: the true calibration displacements reach R2=0.992R^2=0.992, the true (m,μ)(m,μ) reach 0.988, and the commanded speed alone reaches 0.524. The gap between the last and the first is the headroom that physical inference can fill. On the same folds we fit linear probes for logm m and μ from the calibration features, and we remove a decoded property by iterative nullspace projection (Ravfogel et al., 2020; Elazar et al., 2021) to test whether prediction depends on it. 4 Results Table 1: Query-outcome R2R^2 on 2,000 episodes. Query: truncated query clip only; ++cal.: adds the object’s own calibration clips; ++swap: adds another object’s. The commanded speed alone gives 0.524 (clean) and 0.514 (nuisance); true simulator state gives 0.992 and 0.992. Goal err.: median miss when the readout selects the push speed for a goal distance and the simulator executes it (Table 2). The last row is the spread of the ++cal. column, which is what a benchmark needs before it can rank anything. Fixed camera, clean scene Varied camera, lighting, clutter Goal err. (m) Representation Query + cal. + swap Query + cal. + swap clean nuis. V-JEPA 2 0.495 0.995 0.503 0.472 0.974 0.470 2.8 4.0 VideoMAE 0.492 0.994 0.501 0.480 0.912 0.474 2.8 7.1 VC-1 0.486 0.985 0.495 0.476 0.712 0.470 3.2 13.2 DINOv2 0.487 0.980 0.484 0.466 0.673 0.476 4.0 13.5 R3M 0.489 0.990 0.492 0.479 0.703 0.472 2.8 12.7 Random ViT 0.483 0.977 0.491 0.484 0.476 0.476 4.1 19.6 Frame diff. 0.500 0.996 0.500 0.476 0.944 0.473 2.7 5.7 Raw pixels 0.496 0.996 0.496 0.480 0.861 0.477 2.7 8.5 True (m,μ)(m,μ) 2.9 2.8 Ignore object 18.0 19.0 Range of + cal. 0.020 0.498 The clean scene. Every representation gains about +0.50+0.50 R2R^2 from the object’s own calibration clips and loses all of it when those clips are swapped for another object’s (Table 1, left). The query clip alone adds nothing over the commanded speed, which confirms that appearance carries no usable physics. The swap is what lets us say the evidence is used: the readout is not keying on anything that survives replacing it. What the clean scene cannot do is separate the representations. The with-calibration column spans only 0.020, with raw pixels at 0.996, V-JEPA 2 at 0.995, and a ViT with random weights at 0.977. We attribute this to the scene rather than to the encoders. With a fixed camera, the target’s position after each calibration slide is a fixed function of pixel coordinates, so a linear readout can recover the slide distance from raw pixels directly, and equally from any injective transform of them, a random projection included; nothing has to be inferred. The perturbation benchmark audited in Appendix A fails in the opposite way. Simulated with no model at all, 13 of 20 of its cells move the outcome by less than 10−410^-4, so a sensitivity score computed there is fitting integrator noise. Both defects could have been seen before any model was run. Figure 2: Same task, two scenes. With a fixed camera and clean table (blue) every representation reaches the ceiling. With camera, lighting, floor, and distractors resampled per clip (orange) the same representations spread across 0.498 R2R^2, and the random ViT drops to the floor set by the commanded speed alone. The nuisance scene. Resampling camera, lighting, floor, and distractors for every clip changes neither the physics, the labels, nor the readout, yet the spread of the with-calibration column grows from 0.020 to 0.498 (Table 1, right; Figure 2). The swap still removes every gain, so the ordering that appears reflects physical inference rather than some new shortcut. V-JEPA 2 retains 0.974, frame differences 0.944, VideoMAE 0.912, and raw pixels 0.861; VC-1, R3M, and DINOv2 fall to 0.712, 0.703, and 0.673; the random ViT reaches 0.476 against a speed-only floor of 0.514, so its features no longer separate an object’s displacement from a camera move. The ordering holds at every training-set size (Figure 3), and it groups by pretraining objective: the two models trained to predict video lead, the pixel-space references, which encode motion but no invariance, come next, and the three per-frame encoders trail regardless of what they were trained on. R3M and VC-1 are trained on the same egocentric video with different objectives and land within 0.010.01 of each other here, which we take as weak evidence that data alone does not confer the needed invariance. One result surprised us: frame differences (0.944) beat VideoMAE (0.912) and every per-frame encoder. We suspect the per-clip camera change is small enough that a 32×3232× 32 difference image still localises the moving object, and we have not tested larger viewpoint changes. In other words, what the clean scene reported as “all representations are equivalent” was a statement about the scene, not about the representations. Figure 3: Query-outcome R2R^2 against training-set size. Left: in the clean scene every representation converges to the same plateau. Right: under nuisance variation the ordering of Table 1 holds at every training size. Data efficiency. In the clean scene only the approach to the plateau differs (Figure 3, left). At 75 training episodes VideoMAE reaches 0.931, V-JEPA 2 0.876, R3M 0.862, frame differences 0.833, VC-1 0.765, DINOv2 0.720, and the random ViT 0.741. The R3M and VC-1 pair, matched on data and differing in objective, separates by 0.100.10 here and by 0.400.40 in friction decodability. We do not build a claim on this: these are differences between 75-sample fits whose fold spreads understate variance, and under nuisance variation the same pair is within 0.010.01 at every size. The nuisance ordering at 75 episodes (V-JEPA 2 0.716, frame differences 0.595, VideoMAE 0.405, raw pixels 0.334, VC-1 0.176, DINOv2 0.167, R3M 0.143, random ViT 0.017) is the full-data ordering, which is why we read that ordering as a property of the representations rather than of the sample size. Table 2: Goal-conditioned speed selection. The fitted readout is given a goal distance, selects the push speed it predicts will reach it, and the simulator executes that push with the true physics. Err.: median ||actual −- goal||; ≤ 10 m: fraction of pushes within tolerance. True (m,μ)(m,μ): the same readout on simulator state; ignore object: speed alone. Goals are each episode’s own true query displacement, so every goal is reachable. Clean scene Nuisance variation Representation Err. (m) ≤10≤ 10 m Err. (m) ≤10≤ 10 m V-JEPA 2 2.8 94% 4.0 82% VideoMAE 2.8 93% 7.1 64% VC-1 3.2 89% 13.2 40% DINOv2 4.0 84% 13.5 39% R3M 2.8 93% 12.7 41% Random ViT 4.1 82% 19.6 27% Frame diff. 2.7 94% 5.7 71% Raw pixels 2.7 94% 8.5 56% True (m,μ)(m,μ) 2.9 89% 2.8 91% Ignore object 18.0 29% 19.0 28% The gap in task units. An R2R^2 does not say what a representation is worth to a controller, so we also ask the fitted readout a control-shaped question: given a goal distance, choose the push speed whose predicted displacement is closest, then let the simulator execute that push with the true physics (Table 1, right; Table 2). In the clean scene every representation, the random ViT included, lands within a few millimetres of the goal, at the linear readout’s own floor (2.9 m with true (m,μ)(m,μ)). Under nuisance variation V-JEPA 2 misses by 4.0 m and reaches 82% of goals within 10 m, the per-frame encoders miss by roughly 1313 m, and the random ViT misses by 19.6 m, matching the object-blind reference (19.0 m). This is open-loop, one step, and simulated, so it is not a policy; its purpose is to express the R2R^2 gap in millimetres. Table 3: Encoding versus use. Probe: linear decodability of the property from the calibration clips, with frames mean-pooled or, for the per-frame encoders, concatenated. ΔR2 R^2 after erasing: change in prediction once the property is made linearly undecodable, in the clean scene and (mass) under nuisance variation. Removing an equal number of random directions costs at most 0.004 (clean) and 0.006 (nuisance). Probe R2R^2 (mean-pooled) Concat. ΔR2 R^2 after erasing Nuis. Representation Mass Friction Friction Mass Friction Mass V-JEPA 2 0.965 0.904 – -0.219 -0.023 -0.333 VideoMAE 0.933 0.803 – -0.150 -0.018 -0.403 VC-1 0.791 0.390 0.633 -0.130 -0.011 -0.219 DINOv2 0.757 0.291 0.451 -0.190 -0.018 -0.208 R3M 0.930 0.792 0.792 -0.004 -0.000 -0.208 Random ViT 0.762 0.361 0.672 -0.025 -0.003 +0.000 Frame diff. 0.986 0.963 – -0.306 -0.030 -0.373 Raw pixels 0.983 0.940 – -0.001 -0.000 -0.350 Probes. Probing is often used to argue that a model knows a property, with any gap between probe accuracy and behaviour attributed to a failure to act on that knowledge. Two interventions show that probe R2R^2 cannot carry this argument in our setting (Table 3). The first is only a preprocessing change: re-encoding the per-frame models with frames concatenated rather than mean-pooled raises the random ViT’s friction probe from 0.361 to 0.672, above VC-1 (0.390) and DINOv2 (0.291), while its prediction changes by +0.000+0.000. The same frozen network reads as 0.360.36 or 0.670.67 depending on one line of code that papers rarely report, so a ranking of encoders by probe accuracy is, here, a ranking of aggregation conventions. The second intervention is erasure. Raw pixels and frame differences decode mass to 0.983 and 0.986 in the clean scene, yet removing that direction costs frame differences −0.306-0.306 and raw pixels −0.001-0.001. We attribute the difference to redundancy: raw pixels encode the target’s position at every sampled frame, so one erased direction leaves the property recoverable from the rest, while frame differences concentrate it. Under nuisance variation that redundancy is gone and the same erasure costs raw pixels −0.350-0.350; R3M moves from −0.004-0.004 to −0.208-0.208. So whether a decoded property is load-bearing depends on what else the scene leaves available, which a probe cannot see. Erasure never returns prediction to the query-only floor, and we read this as the readout depending on a sufficient statistic of the interaction, namely how far the object slid, rather than on an identified (m,μ)(m,μ); reading a factor out of an outcome is harder than reading the outcome itself. What a benchmark should report. Three inexpensive checks would have caught every failure described above. The first is an effect-size audit: simulate the perturbation with no model and publish |Δoutcome|| | for every cell. The second is the spread between a random-features baseline, raw pixels, and the best model; if it is a few hundredths, the scene rather than the encoder is setting the score. The third is a swap or an erasure behind any claim that a representation uses a property, since decodability is neither necessary nor sufficient for use. None of the evaluations we started from met these. 5 Limitations The environment is deliberately small: rigid boxes, one contact event per clip, a simulator. The numbers characterise this setting. A frozen encoder with a ridge readout measures what is linearly available, which is a lower bound on what fine-tuning could extract, and we evaluate each model’s encoder under a shared readout rather than its own rollout machinery, so a model whose native predictor uses physics differently could score differently. The nuisance scene varies viewpoint, lighting, and clutter but not object category or contact type, and its camera has to be wider than the nominal one to keep the workspace in view, so targets also appear smaller; we cannot attribute the drop to any single factor. The erasure intervention removes linearly decodable structure only; a property could survive non-linearly and still be used. Eight representations suffice to show that a scene can conceal a 0.498 spread, not to characterise which pretraining choices produce invariance in general. Finally, the audited perturbation benchmark is our own earlier design; we do not know how common its defects are in published benchmarks, only that they were invisible from its outputs. References Agrawal et al. (2016) P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine Learning to poke by poking: experiential learning of intuitive physics. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: §2. Bakhtin et al. (2019) A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick PHYRE: a new benchmark for physical reasoning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Bear et al. (2021) D. M. Bear, E. Wang, D. Mrowca, F. J. Binder, H. F. Tung, R. T. Pham, N. Haber, D. L. K. Yamins, et al. Physion: evaluating physical prediction from vision in humans and machines. In NeurIPS Datasets and Benchmarks Track, Cited by: §2. Belinkov (2022) Y. Belinkov Probing classifiers: promises, shortcomings, and advances. Computational Linguistics 48 (1), p. 207–219. Cited by: §2. Elazar et al. (2021) Y. Elazar, S. Ravfogel, A. Jacovi, and Y. Goldberg Amnesic probing: behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics 9, p. 160–175. Cited by: §2, §3. Hewitt and Liang (2019) J. Hewitt and P. Liang Designing and interpreting probes with control tasks. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2. Kumar et al. (2021) A. Kumar, Z. Fu, D. Pathak, and J. Malik RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems (RSS), Cited by: §2. Majumdar et al. (2023) A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V. Berges, P. Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier Where are we in the search for an artificial visual cortex for embodied intelligence?. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Nair et al. (2022) S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta R3M: a universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), Cited by: §2. Oquab et al. (2024) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research. Cited by: §2. Ravfogel et al. (2020) S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg Null it out: guarding protected attributes by iterative nullspace projection. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §3. Riochet et al. (2018) R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux IntPhys: a framework and benchmark for visual intuitive physics reasoning. arXiv preprint arXiv:1803.07616. Cited by: §2. Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa MuJoCo: a physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Cited by: §3. Tong et al. (2022) Z. Tong, Y. Song, J. Wang, and L. Wang VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Turpin et al. (2023) M. Turpin, J. Michael, E. Perez, and S. R. Bowman Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Wu et al. (2016) J. Wu, J. J. Lim, H. Zhang, J. B. Tenenbaum, and W. T. Freeman Physics 101: learning physical object properties from unlabeled videos. In British Machine Vision Conference (BMVC), Cited by: §2. Wu et al. (2015) J. Wu, I. Yildirim, J. J. Lim, W. T. Freeman, and J. B. Tenenbaum Galileo: perceiving physical object properties by integrating a physics engine with deep learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Appendix A Effect-size audit The single-perturbation recipe correlates representational distance with outcome change, which presupposes that the perturbation changes the outcome. We simulate matched pairs across objects and seeds for four tasks and five properties and record |Δoutcome|| | with no model involved (Table 4). 13 of 20 cells are inert, for three reasons. In two of four tasks the object is released above a surface and falls; the recorded outcome is its distance to a goal, a geometric constant. Neither mass nor friction affects a free fall. The rigidity axis set solref="-1000 1", which mixes MuJoCo’s two parameterisations; the simulator discarded the value and used its default, so every rigid-versus-deformable pair was the same scene, and the run completed with one warning per scene. Stacking was scored by the upper object’s final height, which geometry fixes once the stack settles, so the dynamics a perturbation does change (bounce, settling, toppling) were not measured. Table 4: Effect-size audit of a single-perturbation benchmark: mean |Δoutcome|| | as a percentage of the outcome scale. – marks cells where the perturbation moves the outcome by less than 10−410^-4, within integrator noise. Task Mass Friction Rigidity Size Height pick_and_place – – – – – push_to_goal 149% 70% – 16% – stack_two <<1% <<1% – 22% – insert_peg – – – 33% – None of these is visible from a results table. In the same codebase, two encoder wrappers substituted randomly initialised networks when a checkpoint failed to load, and one of those checkpoints did not exist, so a column reported as a physics-aware world model was random features. Our encoders raise on load failure. Appendix B Environment design details Collisions rather than pushes. A force F applied to mass m against sliding friction μ moves the object only if F/m>μgF/m>μ g, and displacement then grows as (F/m−μg)2(F/m-μ g)^2. Over a mass range wide enough to matter, a fixed force leaves roughly half the objects stationary and sends the rest out of frame; sampling forces and masses independently and requiring all three phases to land in a [1,30][1,30] cm window accepted under 2%2\% of draws. A collision has no threshold, recoil speed is bounded by impact speed, and the same window accepts nearly every draw. No rolling shapes. A sphere carries its recoil on rolling contact, so surface friction barely decelerates it. At matched physics spheres travelled 5.6×5.6× farther than boxes (median 0.560.56 versus 0.060.06 m) and corr(μ,displacement)corr(μ,displacement) was +0.10+0.10 for spheres against −0.39-0.39 for boxes, so friction would not be identifiable from the outcome. Targets are boxes of varying aspect ratio. Timing. Impact speeds and friction are chosen jointly so a slide lasts about 0.40.4 s rather than 0.150.15 s at the same displacement, spreading the motion over about 13 rendered frames instead of 5. Episodes whose displacement falls below one rendered pixel (<12<12 m) or leaves the workspace (>28>28 cm) are resampled; acceptance exceeds 70%70\%. Nuisance variation. Camera pose, lighting, floor tint, and two distractor objects are resampled per clip. Every draw is checked to keep the workspace in view, since a camera that crops the slide would make the episode unanswerable and register as a representation failure. Meeting that constraint requires a wider field of view than the nominal camera (scale 1.141.14–1.301.30). Readout details. Each clip’s features form one block. Blocks are centred and reduced by PCA to a shared component budget on the training rows, standardised, concatenated with the commanded speed, and fit by ridge regression with the penalty selected by inner cross-validation. Repeated 5-fold cross-validation (3 repeats) gives the reported means and fold spreads. Probes use the same pipeline with the latent property as target. Erasure follows iterative nullspace projection: probe directions are removed until a tuned probe’s R2R^2 for the property is at chance, and prediction is re-fit on the projected features; the control removes the same number of random directions. Speed selection uses the calibration-only readout (commanded speed plus the two calibration clips), since the query clip is rendered at the true query speed and cannot be re-rendered for each candidate; the speed is chosen on a 451-point grid over [0.40,0.85][0.40,0.85] m/s. Seed robustness. Repeating the evaluation with a different cross-validation seed moves every reported quantity by at most 0.010.01; for DINOv2 in the clean scene, prediction with calibration 0.983→0.9830.983→ 0.983, friction probe 0.345→0.3460.345→ 0.346, and the mass-erasure effect −0.227→−0.232-0.227→-0.232.