Paper deep dive
Do VLAs Understand and Adapt to the Objects They Handle, or Simply Replay Learned Behaviors?
Xinnuo Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper asks whether VLA generalization is grounded in a global understanding of objects' physical properties that enables policies to adapt their motion to unseen setups, or if policies simply replay the motions they've learnt that happen to succeed in new setups. The former reflects genuine generalization; the latter reflects incidental robustness. We first examine awareness of physical properties in seven VLAs by applying linear probing and representational similarity analysis (RSA) to their activations. We find that physical properties, including mass, fragility, deformability, friction and size are less decodable than non-physical properties such as semantic category, material, sound and price in nearly every modality stream. Compared with their base VLMs, robot pre-training weakens the linear encoding of physical properties in the language stream. Neither pre-training nor downstream fine-tuning strengthens the alignment between physical-property differences and activation distances. We then ask whether the weak physical information present in these activations shapes the actions a VLA generates. In a controlled LIBERO case study, we increase the mass of an in-domain object and signal the change through language or vision. Most VLAs use similar lifting behaviour for the heavier and original-mass objects, leading to task success declines. The few exceptions change their behaviour in response to lexical or visual cues rather than to mass itself. These results suggest that VLAs encode physical properties weakly and do not reliably use them to adapt their motion.
Tags
Links
- Source: https://arxiv.org/abs/2610.06078v1
- Canonical: https://arxiv.org/abs/2610.06078v1
Trouble viewing inline? Open PDF directly →
Full Text
46,975 characters extracted from source content.
Expand or collapse full text
Do VLAs Understand and Adapt to the Objects They Handle, or Simply Replay Learned Behaviors?Thanks: preprint Xinnuo Xu Email: xuxinnuo@yahoo.com Abstract This paper asks whether VLA generalization is grounded in a global understanding of objects’ physical properties that enables policies to adapt their motion to unseen setups, or if policies simply replay the motions they’ve learnt that happen to succeed in new setups. The former reflects genuine generalization; the latter reflects incidental robustness. We first examine awareness of physical properties in seven VLAs by applying linear probing and representational similarity analysis (RSA) to their activations. We find that physical properties, including mass, fragility, deformability, friction and size are less decodable than non-physical properties such as semantic category, material, sound and price in nearly every modality stream. Compared with their base VLMs, robot pre-training weakens the linear encoding of physical properties in the language stream. Neither pre-training nor downstream fine-tuning strengthens the alignment between physical-property differences and activation distances. We then ask whether the weak physical information present in these activations shapes the actions a VLA generates. In a controlled LIBERO case study, we increase the mass of an in-domain object and signal the change through language or vision. Most VLAs use similar lifting behaviour for the heavier and original-mass objects, leading to task success declines. The few exceptions change their behaviour in response to lexical or visual cues rather than to mass itself. These results suggest that VLAs encode physical properties weakly and do not reliably use them to adapt their motion. I Introduction Original Full “turn on the stove and put the moka pot on it” (Item mass: 68.2 g) “turn on the stove and put the moka pot full of coffee on it” (Item mass: 648.3 g) “put the white mug on the plate and put the chocolate pudding to the right of the plate” (Item mass: 16.5 g) “put the white mug on the plate and put the chocolate pudding to the right of the plate” (Item mass: 186.1 g) Fig. 1: Signalling a heavier object changes how VLAs act. Top: SmolVLA, moka pot; the instruction names the pot full of coffee and its mass is increased, and the pot ends tilted on the stove. Bottom: π0.5 _0.5, white mug; the instruction is unchanged, the mug is rendered full of coffee and its mass is increased, and the mug drops from the gripper. Left: original task, where both succeed. Vision Language Action models (VLAs) trained on large robot corpora show some robustness to unseen objects and environments [1, 2, 3, 4, 5, 6]. A fundamental question, however, remains: is this generalization grounded in a global understanding of object physical properties, e.g. mass and deformability, that lets the policy adapt its motion to new setups, or does it just replay the motions they’ve learnt in training, and assume they are robust in unseen setups? We examine two dimensions. Awareness: do VLAs represent the physical properties of the objects they manipulate in their latent representations? Following [7, 8, 9, 10, 11], we quantify awareness with two metrics. Decodability fits a linear probe to the model’s activations and asks whether a property can be read out of them; representation geometry uses representational similarity analysis (RSA) to ask whether pairwise distances between object activations track differences in that property, such that objects with similar property values are represented similarly. Across seven widely used VLAs on the OXE and DROID datasets, we find that all physical properties, including mass, fragility, deformability, friction and size, are less decodable than the other properties, such as category, material, sound and price, in almost every modality stream, and no stream’s pairwise activation distances align most strongly with differences in a physical property. Moreover, physical properties are less decodable from the language stream after robot-interaction pre-training and downstream fine-tuning than in the corresponding base VLMs, while neither stage strengthens the alignment between physical-property differences and activation distances (Sec. V). Behavioral adaptation: do VLAs adapt their actions to changes in objects’ physical properties? We answer this with a controlled case study of mass in LIBERO (Sec. VI). As shown in Fig. 1, we select manipulable containers from LIBERO, preserve their geometry while adding the mass of realistic contents, and signal the change through language (top row in Fig. 1) or vision (bottom row in Fig. 1). Six VLAs execute 2323 tasks with mass-amended and original objects under otherwise identical setups. Success falls for mass-amended objects, since for most VLAs the lifting-phase actions deviate no further from the original. Three policies are exceptions, but none tracks mass by exhibiting the human pattern of lifting heavier objects more slowly. Instead, the action divergence arises from either lexical rewording or pixel-level changes. As a conclusion, the results suggest that VLAs encode physical properties weakly and do not reliably use them to adapt their motion. I Related Work Probing VLA representations. Linear probes are widely used in LLMs, where affine readouts recover high-level concepts encoded as activation directions [12, 13]. Applied to VLAs, they reveal symbolic object and action states [7], value-like success structure [8], and observable end-effector variables such as yaw and gripper aperture [9]. Richer techniques expose complementary structure; for example, sparse autoencoders recover motion primitives, task semantics and episode-specific features [14]. We use linear probes for their simple, uniform readout across architectures. Prior work shows that robot training can degrade broad multimodal capabilities and pretrained representations, with retention depending on the action representation and training recipe [15, 16, 17]. We study this degradation through the lens of object representations, asking which physical properties are retained or lost; our analysis is therefore complementary rather than a rediscovery of the broader effect. Representational similarity analysis. Introduced in systems neuroscience [10], RSA relates two systems through their dissimilarity structures and is a standard tool for connecting neural and symbolic representations in language models [11]. We apply it to VLAs to ask what a probe cannot: whether a property organizes the representation. A property can be linearly decodable yet contribute nothing to the distances between object embeddings. Object physical properties. PhysObjects [18] annotates mass, fragility, deformability, material and other properties in household-object images, and uses them to ground a VLM that improves object selection by a high-level LLM planner. Human-norm resources provide reference labels for some properties, including measured mass in YCB [19] and size in THINGSplus [20]; we use these to validate our annotation (Sec. I). That line of work targets task planning, i.e. which object to pick for a goal, whereas we target motion execution: whether a policy’s low-level actions reflect the properties of the object it holds. I Dataset and Annotation In this paper we augment three widely-used datasets: OXE [21], DROID [22], and LIBERO [23], with commonsense object-property annotations elicited from GPT-4.1. We restrict our study to the pick-and-place task family, and from each dataset select the one-hand-manipulable objects: 200200 for OXE, 8888 for DROID, and 1313 for LIBERO (e.g. apple, mug, moka pot). For each object we annotate twelve properties spanning physical mechanics, semantic identity and everyday context: mass, fragility, deformability, friction, size, material, sound11 1 Whether the object is characteristically associated with producing a sound, e.g. a bell or blender vs. a towel., price, category22 2 The object’s superordinate semantic category, e.g. food, tool, location33 3 The location where the object is typically found, e.g. kitchen or office., colour44 4 The object’s most characteristic colour., and disposability55 5 Whether the object is typically single-use and discarded after use (e.g. a napkin) rather than reused (e.g. a mug).. We refer to the first five as physical properties: by common sense, they directly affect how an object is grasped, lifted and placed. Among these, mass, fragility, deformability, friction, size, sound and price are ordinal, and material, category, location, colour and disposability are categorical. For a categorical property we query GPT-4.1 one object at a time by name, e.g. “In what type of room is a object most typically found?” for location, and record its chosen label from a fixed label set. For an ordinal property we instead pose pairwise comparisons by name, “Which is more X, object1 or object2?”, over every within-dataset pair (each asked in both orders to cancel order bias). We then fit a Bradley-Terry (BT) model [24] to the resulting win matrix to obtain a per-object score and a ranking within each dataset. Annotation validity. We assess GPT-4.1’s annotations against human-labelled reference datasets. For the categorical properties we report classification accuracy and Cohen’s κ against a human gold label: category (0.830.83, κ 0.81κ\,0.81; THINGSplus [20]) and material (0.880.88, κ 0.85κ\,0.85; PhysObjects [18]). For the ordinal properties we report the balanced accuracy of GPT-4.1’s pairwise “which is more X?” judgments against the human-norm ordering: size 0.970.97 (THINGSplus [20]), sound 0.960.96 (Amsel et al. [25]), mass ∼0.84 \!0.84 (YCB measured mass [19]), and fragility 0.810.81 and deformability 0.790.79 against the PhysObjects [18]. Every accuracy above meets or exceeds the ≥70%≥\!70\% inter-human-annotator agreement that PhysObjects requires to accept a human gold label [18], so we treat the GPT-4.1 labels as trustworthy. The remaining properties, friction, price, location, colour and disposability, have no established human dataset and are annotated by GPT-4.1 without validation. Mass in LIBERO. LIBERO’s simulated mass does not reflect an object’s real-world weight. Every object is assigned a uniform density (100kgm−3100\,kg\,m^-3) with no explicit mass, so MuJoCo computes each object’s mass as density × collision-mesh volume, i.e. a pure size proxy that ignores material and contents. Because the LIBERO fine-tunes were trained under this physics, for LIBERO we discard the GPT-4.1 real-world mass annotation and label mass with the simulator’s density-derived value instead. IV Models and Methodology We study seven widely used VLA policies spanning the three dominant action-decoding paradigms: autoregressive discrete tokens (π0 _0-FAST [26], OpenVLA [27]), flow-matching action experts (π0 _0 [28], π0.5 _0.5 [29], SmolVLA [30], GR00T-N1.7 [31, 32]), and JEPA-style latent prediction (VLA-JEPA [33]). We ask whether these models internally represent object physical properties, rather than merely mapping pixels or text to actions. Borrowing from mechanistic interpretability in LLMs, we study the geometry of the frozen models’ layer-wise activations through linear probing and representational similarity analysis (RSA), asking where and how those latent representations encode each object property (property details in Sec. I). Fig. 2: The two VLA families we study, and how we extract their latent representation. Top: an autoregressive policy (OpenVLA, π0 _0-FAST) runs one transformer over a multimodal prefix and generates discrete action tokens one at a time. Bottom: a flow-matching policy (π0 _0, π0.5 _0.5, VLA-JEPA, SmolVLA, GR00T-N1.7) attends the VLM prefix, layer by layer, from a separate action expert that denoises a noisy action chunk over K flow steps. In both, we record the activations at every layer and mean-pool a modality’s token span into one embedding: per-view vision, the object head-noun text tokens, the optional state token, and the action tokens (generated tokens, or the expert’s tokens at the last flow step). The input modalities and their ordering vary across models. For clarity we omit VLA-JEPA’s world-model latent tokens and separate embodied-action tokens from this figure. See Sec. IV. Latent representation extraction. Fig. 2 shows how we extract frozen latent representations from the two mainstream VLA architecture classes, autoregressive (top) and flow-matching (bottom). Consider a frame t of a trajectory in which the robot manipulates object o. Given the task instruction, the camera views and the robot state at t, we run a single forward pass and record the activations at every layer ℓ . For each modality m, we then mean-pool the activations over that modality’s token span into one vector xo,ℓ(m)∈ℝdx^(m)_o, ^d: object pools the instruction’s object head-noun tokens, vision pools the patch tokens of each camera view, state takes the robot-state token where one exists, and action pools the generated action tokens (autoregressive) or the action-expert tokens at the last flow step (flow-matching). We call each pooled span a modality stream: object-text (the object’s head-noun tokens), the third-person camera and the wrist camera (each view is its own stream), robot state, and action tokens. Physical Property Decodability – Probing answers whether a physical property is linearly readable from a latent representation [12, 13, 9, 7, 8]. A probe is an affine map applied to a frozen representation. For a categorical property with classes c∈1,…,Kc∈\1,…,K\ (K≥2K≥ 2), we define the probe as a linear map g(xo,ℓ(m))=Wxo,ℓ(m)+bg(x^(m)_o, )=Wx^(m)_o, +b, with W∈ℝK×dW ^K× d and b∈ℝKb ^K, which predicts the class with the highest score, y^o=argmaxcg(xo,ℓ(m))c y_o= _cg(x^(m)_o, )_c. We train it as a multinomial logistic regression, minimising the ℓ2 _2-regularised cross-entropy over the training objects trO_tr with label yoy_o: ℒ=−∑o∈trlogsoftmax(g(xo,ℓ(m)))yo+γ∥W∥F2, =- _o _tr (g(x^(m)_o, ) )_y_o+γ W _F^2, (1) For an ordinal property, each object o is annotated with a continuous within-dataset Bradley-Terry (BT) score sos_o, so we probe order. For an object pair (i,j)(i,j) whose scores are separated by a margin, |si−sj|>ϵ|s_i-s_j|>ε with ϵ=0.05ε=0.05, we define the probe as a linear map on the embedding difference, hij=h(xi,ℓ(m),xj,ℓ(m))=w⊤(xi,ℓ(m)−xj,ℓ(m))+bh_ij=h(x^(m)_i, ,x^(m)_j, )=w (x^(m)_i, -x^(m)_j, )+b, with w∈ℝdw ^d and b∈ℝb , which predicts that i ranks above j when h>0h>0. With label zij=[si>sj]z_ij=1[s_i>s_j], we train it as a binary logistic regression, minimising the ℓ2 _2-regularised cross-entropy over the training pairs trP_tr: ℒ=−∑(i,j)∈tr[zijlogσ(hij)+(1−zij)log(1−σ(hij))]+γ∥w∥22, =-\!\! _(i,j) _tr\! [z_ij σ(h_ij)+(1-z_ij) (1-σ(h_ij) ) ]+γ w _2^2, (2) where σ is the logistic sigmoid. Every training pair enters in both orders, (i,j)(i,j) and (j,i)(j,i), so the two labels are balanced. The chance levels of the categorical and ordinal probes are 1/K1/K and 0.50.5, respectively. We evaluate each probe by its balanced accuracy on held-out test sets testO_test and testP_test, where tr∩test=∅O_tr _test= and tr∩test=∅P_tr _test= . A property is decodable from modality m at layer ℓ when this accuracy is above chance, and the closer it is to 11, the more strongly the property is linearly encoded. Representation’s geometry – RSA. Representational Similarity Analysis (RSA) reveals whether a property organizes a representation’s geometry. For the N objects of a dataset we build two Representational Dissimilarity Matrices (RDMs). The neural RDM uses the cosine distance between pooled embeddings, Dijrep=1−⟨xi,xj⟩∥xi∥∥xj∥, D^rep_ij=1- x_i,x_j x_i \, x_j , (3) and the property RDM uses the gap in BT score for ordinal properties, Dijprop=|si−sj|D^prop_ij=|s_i-s_j|, or label disagreement for categorical ones, Dijprop=[yi≠yj]D^prop_ij=1[y_i≠ y_j]. The RSA score is the Spearman rank correlation between the upper triangles of the two RDMs, RSA=ρs(Dijrepi<j,Dijpropi<j). = _s (\D^rep_ij\_i<j,\,\D^prop_ij\_i<j ). (4) RSA∈[−1,1]RSA∈[-1,1]. A high score means that, objects with similar property values have embeddings that are closer than objects far apart on the property. A score near 00 means embedding distances are unrelated to the property. A below 00 means objects with similar property values sit farther apart. Unlike the probe, RSA fits no parameters. a property can be decodable along one direction yet have RSA near 00, if other properties dominate the distances between embeddings. V Experiment Results This section reports results aggregated across models, while Sec. VI examines individual models in detail. Fig. 3: Physical properties by modality stream in pre-trained VLAs. (a) Decodability: the gap between probe accuracy and chance (1/K1/K for categorical properties, 0.50.5 for ordinal ones; Sec. V-A). (b) Representation geometry: RSA (Sec. V-B). Both metrics are defined in Sec. IV. Each polygon is one modality stream and each axis is one property. The dark-red boundary axes bracket the physical properties (mass to size). Values are averaged over the seven pre-trained VLAs (six for the wrist camera) and over OXE, DROID. Fig. 4: Base VLM vs. pre-trained VLA, by modality stream. Each radar overlays the base VLM (grey) and its pre-trained VLA (blue for the object-text stream, orange for the third-person camera stream) over the twelve properties, with the physical properties bracketed by the dark-red axes. (a, b) Decodability, i.e. probe accuracy minus chance; (c, d) Representation geometry, measured by RSA. Both are averaged over all VLA–VLM pairs and over OXE, DROID. See Sec. V-C for the discussion. V-A Property Decodability in Pre-trained VLAs In this section, we ask where physical properties are decodable in pre-trained VLAs. Setup. We probe the pre-trained checkpoint of each VLA: π0 _0 (pi0_base), π0.5 _0.5 (pi05_base), π0 _0-FAST (pi0fast-base), SmolVLA (smolvla_base), VLA-JEPA (VLA-JEPA-Pretrain), GR00T-N1.7-3B and OpenVLA-7B, on two real-robot demonstration datasets. From OXE, we sample one episode for each of the 200200 annotated objects. From DROID, we collect 410410 episodes covering the annotated objects, with a median of 4 episodes per object. From each episode, we take the frame at the early interaction phase (30%30\% of the way through) from every available camera: the third-person view only on OXE, and both the third-person and wrist views on DROID. The instruction names the object in a pick-and-place template, “pick up the ⟨ ⟩ ”. No robot state is used: π0 _0, SmolVLA, GR00T-N1.7 receive a zero state vector, and π0.5 _0.5 and π0 _0-FAST receive the instruction without state text. Each model then generates actions conditioned on the language instruction and the camera view(s). For each dataset, we hold out 20%20\% of the objects for testing. Ordinal test pairs are formed only among the held-out objects (780780 pairs on OXE and 105105 on DROID), and training pairs are subsampled to 400400 per dataset. We train an independent probe for every layer and every modality stream of a model, and report the layer that performs best under 2-fold cross-validation on the training objects. Findings. Fig. 3(a) reports decoding accuracy minus chance, averaged over datasets and then over the seven models; the dark-red boundary axes bracket the physical properties (mass to size). The wrist stream averages six models on DROID only, as OXE dataset and OpenVLA model take a single image. We find that (i) On average, every one of the 12 properties is decodable above chance from each of the four modality streams; (i) Object-text is the most decodable stream for 10 of the 12 properties, and action tokens are the least decodable stream for 8 of 12; and (i) The physical properties are not the most decodable: from object-text, mass (+0.19+0.19) and fragility (+0.19+0.19) rank 11th and 10th of 12, well below category (+0.41+0.41), material (+0.37+0.37), sound (+0.31+0.31) and price (+0.30+0.30). The same trend holds for the other modality streams. Fig. 5: RSA: LIBERO fine-tuning versus pre-training, one radar per modality stream. Each panel overlays the pre-trained VLA (grey) and its LIBERO fine-tune (in the stream’s colour) over the twelve properties (clockwise, Sec. I order; physical properties bracketed by the dark-red axes), averaged over the seven families with a LIBERO checkpoint. LIBERO mass is simulator mass; categorical RSA is defined on LIBERO, but its labels are nearly single-class (location: 14 of 15 objects kitchen). Discussed in Sec. V-D. V-B Representation Geometry of Pre-trained VLAs Here, we ask which properties organize the representation geometry of pre-trained VLAs. Setup. We compute RSA on the same early-frame representations, models and datasets as Sec. V-A. RSA fits no parameters, so it uses all annotated objects of a dataset without a train/test split; for each modality stream we report the peak over layers, which is an upper bound. Fig. 3(b) aggregates over all tested VLAs. Findings. (i) The geometry of object-text (blue) is organized most by two non-physical properties, location (0.290.29) and material (0.280.28), and less by any physical property (≤0.21≤ 0.21). (i) The two vision streams (orange and green) and the action stream (yellow) share a common profile: they are organized most by price, a non-physical property (third-person 0.280.28, wrist 0.310.31, action 0.170.17), and next by deformability, a physical one (0.180.18, 0.260.26, 0.140.14). Mass, friction and size contribute moderately, while fragility leaves no trace in their geometry (≤0≤ 0), even though it is decodable from every stream (Fig. 3(a)). Overall, non-physical properties are more prominent than physical ones in the representation geometry of every stream, and the strongest organizer of each stream is a non-physical property. V-C Base VLMs vs. Pre-trained VLAs Here, we ask how robot pre-training, i.e. further training a VLM on physical interaction data, changes the decodability and representation geometry of physical properties relative to the other object properties. Setup. We compare each VLA with the VLM it is built on: π0 _0, π0.5 _0.5 and π0 _0-FAST with PaliGemma-3B [34]; SmolVLA with SmolVLM2-500M [35]; VLA-JEPA with Qwen3-VL-2B [36]; GR00T-N1.7 with Cosmos-Reason2 [37]; and OpenVLA with Prismatic [38]. Each VLM receives the same pick-and-place instruction and third-person frame as its VLA, in a single forward pass without generation. Datasets, splits and layer selection follow Sec. V-A. Findings. Fig. 4 overlays each VLM and its VLA, averaged over all VLA–VLM pairs. (a) Robot pre-training lowers object-text decodability for 11 of the 12 properties, and physical properties are hit hardest: three of the four largest drops are fragility (−0.10-0.10), mass (−0.08-0.08) and deformability (−0.07-0.07). (b) Vision decodability changes little overall (+0.02+0.02 on average across properties), yet its largest gain is physical: mass rises by +0.08+0.08 and fragility by +0.03+0.03. (c, d) The properties that most strongly organize the representation stay largely the same after robot pre-training: the RSA ranking of the 12 properties correlates between VLMs and VLAs (Spearman ρ=0.91ρ=0.91 for object-text and 0.970.97 for vision). Object-text RSA falls for all 12 properties, but physical properties lose about half as much as the others (mean −0.02-0.02 vs. −0.05-0.05), whereas vision RSA changes by at most 0.0240.024. Training on physical interaction data thus weakens the linear encoding of physical properties in language and slightly strengthens mass and friction in vision, but it does not make physical properties more prominent in the representation geometry: their RSA does not increase in either stream, and non-physical properties such as location, material and price remain the strongest organizers. V-D Pre-trained VLAs vs. Downstream Dataset FT Here, we ask how downstream fine-tuning (FT), i.e. further training a pre-trained VLA on the physical interaction data of a downstream dataset, changes the decodability and representation geometry of physical properties relative to the other object properties, evaluated on the objects of that same dataset. Setup. We compare the pre-trained checkpoints of five VLAs from Sec. V-A—π0 _0-FAST, SmolVLA, VLA-JEPA, GR00T-N1.7 and OpenVLA—with their corresponding released LIBERO fine-tunes. We use 8 LIBERO demonstration episodes for each of the 1313 objects (104104 episodes in total). As in Sec. V-A, we take the frame 30%30\% of the way through each episode from both cameras, feed no robot state, and let each model generate its actions. To keep the input in-domain, the instruction is the episode’s own LIBERO instruction. With only 1313 objects, probe estimates are too noisy, so we report only RSA. Each object’s embedding is averaged over its 8 episodes before computing RSA, and layer selection follows Sec. V-B. Findings. Fig. 5 shows that in object-text (blue), RSA falls much less for physical than for non-physical properties (mean −0.01-0.01 vs. −0.07-0.07), with the largest drops in material (−0.14-0.14), sound (−0.09-0.09) and price (−0.09-0.09). In the vision streams (red and green), physical properties instead lose RSA while the others gain (third-person −0.04-0.04 vs. +0.03+0.03; wrist −0.03-0.03 vs. +0.03+0.03). In the action stream (yellow), 7 of the 12 properties gain RSA and the largest gain is non-physical (sound, +0.13+0.13). As with robot pre-training, fine-tuning on LIBERO fails to make physical properties more prominent in the representation geometry. VI LIBERO Case Study: Mass (a) Language-based cue (b) Vision-based cue Fig. 6: Task success rate under mass-relevant cues. Success rate averaged over tasks (13 with the language-based cue, 10 with the vision-based cue; 20 episodes per task and setup), with error bars showing the standard error across tasks. (a) The four language-based setups are listed in Table VI-A. (b) The two vision-based setups are illustrated in the bottom row of Fig. 1. Setups are detailed in Sec. VI-A and results discussed in Sec. VI-B. Sec. V-A showed that mass is linearly decodable from every modality stream of every VLA we study. Decodability, however, does not mean the policy uses the information. In this section, we ask whether a cue that signals the change of object mass in language or vision, actually changes how a VLA acts. VI-A Controlled Experimental Setup LIBERO assigns every object the same default density of 100100 kg/m3, so an object’s simulated mass is proportional to its collision volume and reflects its size rather than its weight. For containers, this default mass (55–6868 g) is close to that of an empty container. To isolate mass, we keep each object’s geometry and initial pose fixed and change only its mass and the cue that signals it. We take the containers that appear in the training data of the four LIBERO suites and split them by whether their contents are visible, so that mass is signalled to the policy through language alone for one group and through vision alone for the other. Language-based cue. This group holds sealed, opaque containers: alphabet soup, cream cheese, salad dressing, BBQ sauce, ketchup, tomato sauce, butter, milk, chocolate pudding and orange juice (LIBERO-Object), the moka pot (LIBERO-10) and the wine bottle (LIBERO-Goal, in two tasks), 13 tasks in total. Filling these containers changes no pixel, so mass can be signalled only through the instruction. Table VI-A shows the setups derived from each Original LIBERO instruction. Empty names the container’s default state explicitly (“empty carton of milk”) at the default mass. Full names the filled state (“full carton of milk”) and adds the mass of the contents, computed from their density and 85%85\% of the object’s collision volume, with the rendering unchanged. As a control, Rephrase uses the same restructured name without the mass-indication word (“carton of milk”). It shows whether a change in the policy’s behavior comes from rewording the object name or from the mass the word implies. Vision-based cue. This group holds open containers: two types of mugs (LIBERO-10, three tasks) and a bowl (LIBERO-Goal and LIBERO-Spatial, seven tasks), 10 tasks in total. The instruction is never changed; Full setup renders visible contents (coffee or soup) inside the container and adds their mass, computed from their density and rendered volume. The contents have no collision. This group has only the original and full conditions. The bottom row in Fig 1 shows an example. Full masses range from 4848 to 648648 g, 88–25×25× the default. Setup Instruction Mass Orig pick up the milk and place it in the basket 31.731.7 g Reph pick up the carton of milk and place it in the basket 31.731.7 g Empty pick up the empty carton of milk and place it in the basket 31.731.7 g Full pick up the full carton of milk and place it in the basket 308.9308.9 g Table I: Setups of the language-based cue, shown for the milk task. Bold: the object phrase, the only part of the instruction that changes. Original, Rephrase and Empty keep the carton’s default LIBERO mass; Full adds 277.2277.2 g of milk content, computed from its density (1.031.03 g/cm3) and 85%85\% of the carton’s collision volume. Setup details are in Sec. VI-A. π0 _0 π0.5 _0.5 π0 _0-FAST SmolVLA VLA-JEPA OpenVLA Language-based cue, z action Rephrase 1.01 0.97 1.02 1.40 0.99 1.02 Empty 1.00 0.98 1.03 1.35 1.01 1.06 Full 1.01 0.99 1.03 1.38 1.01 1.05 Language-based cue, full action space (7-D) Rephrase 1.01 0.98 1.01 1.27 1.00 1.01 Empty 1.00 0.99 1.02 1.19 1.02 1.04 Full 1.01 0.99 1.03 1.21 1.01 1.03 Vision-based cue, z action Full 0.98 1.07 1.33 0.98 1.15 1.03 Vision-based cue, full action space (7-D) Full 0.99 1.01 1.25 0.98 1.10 1.02 Table I: Agreement of commanded actions right after the grasp. R compares episodes generated under Original setup with those generated under the setup named in the row: the mean distance between the two setups divided by the mean distance within each setup. R≈1R≈ 1 means the mass cue moves the commanded actions no further than run-to-run variation within Original. Bold: the episodes from the two setups differ in at least half of the tasks. Discussed in Sec. VI-C. VI-B Findings: Task Success Each model executes every task for 2020 episodes per setup, starting from the same initial states in every task. An episode counts as successful when LIBERO’s built-in goal check is satisfied. Fig. 6 reports the success rate averaged over tasks (13 language-based, 10 vision-based), with error bars showing the standard error across tasks. With the language-based cue (Fig. 6a), Full lowers success for every model, most for SmolVLA (0.71→0.120.71→ 0.12) and OpenVLA (0.81→0.490.81→ 0.49). However, rewording the object name at the default mass, in either Rephrase or Empty, costs nearly as much, and adding the mass in Full lowers success by at most a further 0.080.08 relative to Empty for any model. π0.5 _0.5 and VLA-JEPA stay within 0.050.05 of Original in every language setup. With the vision-based cue (Fig. 6b), where the instruction is unchanged, Full again lowers success for every model, most for π0 _0-FAST (0.93→0.740.93→ 0.74) and π0.5 _0.5 (0.90→0.770.90→ 0.77). π0.5 _0.5, π0 _0-FAST and VLA-JEPA lose more success when the heavier object is signalled only through vision than through language, whereas π0 _0, SmolVLA and OpenVLA lose less. VI-C Episode Analysis Success rate measures how robust a model is to a mass change signalled through language or vision; VLA-JEPA, for example, loses less than 0.100.10 under both cues. Success alone, however, cannot tell whether a model stays robust because it adjusts its actions to the heavier object, or because it issues the same actions as for the original object and the added mass is simply not enough to make them fail. We therefore analyse the episodes the models generate. Human prehension studies show that mass shows up in how an object is lifted rather than in how it is approached [39]. Heavier objects are lifted later and with a lower lift velocity [40]. A policy that uses mass should likewise reveal it right after the grasp, when the object first loads the arm. We therefore locate the grasp point in each episode as the step at which the gripper opening first drops below 90%90\% of its initial width.66 6 When an episode closes the gripper more than once, we take the closure that immediately precedes the object’s rise. We then take the 1010 consecutive commanded actions starting at the grasp point in every episode,77 7 For the models that plan and execute 5050-step action chunks (π0 _0, π0.5 _0.5 and SmolVLA), we specifically keep only episodes whose 1010 steps lie inside the chunk that contains the grasp point. A heavier object makes the arm track its commands less closely, so a chunk planned after the grasp is conditioned on different camera views and robot state; a difference in its actions could then reflect this changed input rather than the model’s knowledge of mass. and test whether VLAs command a different lift in this window than they do for the original object. Let at(e)∈ℝDa^(e)_t ^D be the action commanded at step t of episode window e, where D is the number of action dimensions. In LIBERO, D=7D=7. The distance between two windows e and e′e is the root mean square (RMS) difference of their actions, d(e,e′)=1TD∑t=1T∥at(e)−at(e′)∥22. d(e,e )= 1TD _t=1^T a^(e)_t-a^(e )_t _2^2. (5) where T=10T=10 is the window length. We use RMS rather than a scale-invariant measure such as cosine similarity or correlation because the magnitude of an action is exactly what we want to compare: lifting twice as fast doubles the action while leaving its direction unchanged, so cosine and correlation stay the same, whereas the RMS distance doubles. For a task, let ℰOE_O and ℰXE_X be the episode windows of Original and of a changed setup X. We average d over all pairs within ℰOE_O (d¯OO d_O), within ℰXE_X (d¯XX d_X) and across the two sets (d¯OX d_OX), and compare the cross-setup distance with the run-to-run variation within each setup: R=d¯OX12(d¯OO+d¯XX)R= d_OX 12 ( d_O+ d_X ). We average R over tasks, and report the results in Table VI-A. We report R over the full action space (D=7D=7) and over the vertical component Δz z of the commanded end-effector displacement alone (D=1D=1), the direction in which the load acts. Both give the same picture. Most entries sit at R≈1R≈ 1: changing the setup moves a model’s actions no further than simply repeating Original does. These models never adapt their lift to the cue, and instead rely on one policy being robust enough across masses. Three models are exceptions. However, none of them adapts in a way that tracks mass. SmolVLA departs from Original under every language setup (R=1.40R=1.40, 1.351.35 and 1.381.38 on Δz z for Rephrase, Empty and Full), commanding a slower lift each time (−6.9-6.9, −9.3-9.3 and −9.2-9.2 m/step, slower in 10/1310/13, 11/1211/12 and 11/1311/13 tasks). Since Rephrase and Empty leave the mass untouched and Empty even asserts an empty container, this shift follows the reworded object name rather than the mass it implies. Under the vision-based cue, π0 _0-FAST and VLA-JEPA depart from Original (R=1.33R=1.33 and 1.151.15), but they command a faster lift for the heavier object (+7.3+7.3 and +2.7+2.7 m/step, faster in 10/1010/10 tasks), the opposite of the human pattern of lifting heavier objects more slowly [40]. Visible contents therefore change what these policies do without making them handle the added mass more carefully. VII Conclusion and Discussion We asked whether VLAs are aware of the physical properties of the objects they manipulate, and whether they act on them. Across seven policies, three datasets and twelve annotated properties, the answer to the first question is a qualified yes: mass, fragility, deformability, friction and size are decodable from every modality stream, but they are among the least prominent properties in the representation, and neither robot pre-training nor downstream fine-tuning promotes them. The answer to the second is no: when the same object is made heavier and the change is signalled in language or in vision, the actions a policy commands for lifting move no further than run-to-run variation on the original mass, except where the surface form of the cue – a reworded name or visible contents – moves them in a direction unrelated to the mass. The two halves of our study point to the same conclusion: physical properties are weakly represented, and they are not connected to motion generation. One likely reason is the training signal. Teleoperated demonstrations primarily capture whether a task succeeds, rather than whether the motion handles each object safely and robustly. Consequently, the collected data may favour a single motion strategy across many objects, providing no signal for adapting motion to different physical properties. Closing this gap requires data or objectives in which a physical property itself, rather than the wording or appearance that accompanies it, changes the required motion. References [1] X. Zhou, Y. Xu, G. Tie, Y. Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun (2025) LIBERO-PRO: towards robust and fair evaluation of vision-language-action models beyond memorization. arXiv preprint arXiv:2510.03827. External Links: Document, Link Cited by: §I. [2] S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu (2026) LIBERO-Plus: a progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 38574–38583. External Links: Link Cited by: §I. [3] G. Wang, C. Zhang, Q. Liu, J. Zhang, J. Cai, J. Liu, and X. Liu (2026) LIBERO-X: robustness litmus for vision-language-action models. In Proceedings of Robotics: Science and Systems, External Links: Document, Link Cited by: §I. [4] C. Kim, M. Kim, M. Kang, H. Kim, and D. Jung (2026) LIBERO-Para: a diagnostic benchmark and metrics for paraphrase robustness in VLA models. arXiv preprint arXiv:2603.28301. Note: Accepted to EMNLP 2026 External Links: Document, Link Cited by: §I. [5] A. Mete, H. Xue, A. Wilcox, Y. Chen, and A. Garg (2024) QueST: self-supervised skill abstractions for learning continuous control. In Advances in Neural Information Processing Systems, Vol. 37, p. 4062–4089. External Links: Document, Link Cited by: §I. [6] X. Zhou, Y. Xu, G. Tie, C. Hu, B. Tao, X. Zhao, X. Xiang, P. Zhou, L. Sun, and Y. Chen (2026) Dismantling the illusion of vision-language-action models competence via explicit distributional shifts. In Proceedings of the 43rd International Conference on Machine Learning, External Links: Link Cited by: §I. [7] H. Lu, H. Li, P. S. Shahani, S. Herbers, and M. Scheutz (2025) Probing a vision-language-action model for symbolic states and integration into a cognitive architecture. arXiv preprint arXiv:2502.04558. Cited by: §I, §I, §IV. [8] J. Zhang, J. Nie, J. Lao, W. Cheng, C. Liu, J. Jiang, and S. Huang (2026) What frozen vlas already know about success: a probing study of value-like structure in foundation robot policies. arXiv preprint arXiv:2605.28527. Cited by: §I, §I, §IV. [9] H. Buurmeijer, C. Amo Alonso, A. Swann, and M. Pavone (2026) Observing and controlling features in vision-language-action models. arXiv preprint arXiv:2603.05487. Cited by: §I, §I, §IV. [10] N. Kriegeskorte, M. Mur, and P. Bandettini (2008) Representational similarity analysis — connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience 2. Cited by: §I, §I. [11] G. Chrupała and A. Alishahi (2019) Correlating neural and symbolic representations of language. In Proc. Association for Computational Linguistics (ACL), Cited by: §I, §I. [12] K. Park, Y. J. Choe, and V. Veitch (2024) The linear representation hypothesis and the geometry of large language models. In Int. Conf. on Machine Learning (ICML), p. 39643–39666. Cited by: §I, §IV. [13] K. Park, Y. J. Choe, Y. Jiang, and V. Veitch (2025) The geometry of categorical and hierarchical concepts in large language models. In Int. Conf. on Learning Representations (ICLR), Cited by: §I, §IV. [14] A. Swann, L. McGranahan, H. Buurmeijer, M. Kennedy I, and M. Schwager (2026) Sparse autoencoders reveal interpretable and steerable features in VLA models. arXiv preprint arXiv:2603.19183. Cited by: §I. [15] A. J. Hancock, X. Wu, L. Zha, O. Russakovsky, and A. Majumdar (2026) Actions as language: fine-tuning VLMs into VLAs without catastrophic forgetting. In International Conference on Learning Representations (ICLR), Cited by: §I. [16] S. Grover, A. Gopalkrishnan, B. Ai, H. I. Christensen, H. Su, and X. Li (2025) Enhancing generalization in vision-language-action models by preserving pretrained representations. arXiv preprint arXiv:2509.11417. Cited by: §I. [17] H. Shi, X. Ren, Y. Zhang, Q. Zhang, J. Hu, H. Shan, H. Dong, J. Lu, Y. Chen, Y. Zhang, Y. Dai, and X. Ju (2026) VLA-Trace: diagnosing vision-language-action models through representation and behavior tracing. arXiv preprint arXiv:2605.30117. Cited by: §I. [18] J. Gao, B. Sarkar, F. Xia, et al. (2024) Physically grounded vision-language models for robotic manipulation. In IEEE Int. Conf. on Robotics and Automation (ICRA), Cited by: §I, §I. [19] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar (2015) The YCB object and model set: towards common benchmarks for manipulation research. In Int. Conf. on Advanced Robotics (ICAR), Cited by: §I, §I. [20] L. M. Stoinski, J. Perkuhn, and M. N. Hebart (2024) THINGSplus: new norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images. Behavior Research Methods 56, p. 1583–1603. Cited by: §I, §I. [21] Open X-Embodiment Collaboration et al. (2024) Open X-embodiment: robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. External Links: Document, Link Cited by: §I. [22] A. Khazatsky K. Pertsch et al. (2024) DROID: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. Cited by: §I. [23] B. Liu, Y. Zhu, C. Gao, et al. (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [24] R. A. Bradley and M. E. Terry (1952) Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §I. [25] B. D. Amsel, T. P. Urbach, and M. Kutas (2012) Perceptual and motor attribute ratings for 559 object concepts. Behavior Research Methods 44 (4), p. 1028–1041. Cited by: §I. [26] K. Pertsch, K. Stachowicz, B. Ichter, et al. (2025) FAST: efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747. Cited by: §IV. [27] M. J. Kim, K. Pertsch, S. Karamcheti, et al. (2024) OpenVLA: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §IV. [28] K. Black, N. Brown, D. Driess, et al. (2024) π0 _0: a vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: §IV. [29] Physical Intelligence, K. Black, N. Brown, et al. (2025) π0.5 _0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054. Cited by: §IV. [30] M. Shukor, D. Aubakirova, F. Capuano, et al. (2025) SmolVLA: a vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844. Cited by: §IV. [31] NVIDIA, J. Bjorck, F. Castañeda, et al. (2025) GR00T N1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734. External Links: Link Cited by: §IV. [32] NVIDIA (2026) GR00T-N1.7-LIBERO. Note: https://huggingface.co/nvidia/GR00T-N1.7-LIBEROOfficial model card, accessed September 2026 Cited by: §IV. [33] J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen (2026) VLA-JEPA: enhancing vision-language-action model with latent world model. arXiv preprint arXiv:2602.10098. External Links: Link Cited by: §IV. [34] L. Beyer, A. Steiner, A. S. Pinto, et al. (2024) PaliGemma: a versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726. Cited by: §V-C. [35] A. Marafioti, O. Zohar, M. Farré, et al. (2025) SmolVLM: redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299. Cited by: §V-C. [36] S. Bai et al. (2025) Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §V-C. [37] NVIDIA (2025) Cosmos-Reason2: physical AI common sense and embodied reasoning models. Note: https://huggingface.co/nvidia/Cosmos-Reason2-2BModel release, accessed September 2026 Cited by: §V-C. [38] S. Karamcheti, S. Nair, A. Balakrishna, et al. (2024) Prismatic VLMs: investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865. Cited by: §V-C. [39] P. L. Weir, C. L. MacKenzie, R. G. Marteniuk, S. L. Cargoe, and M. B. Frazer (1991) The effects of object weight on the kinematics of prehension. Journal of Motor Behavior 23 (3), p. 192–204. External Links: Document, Link Cited by: §VI-C. [40] D. Eastough and M. G. Edwards (2007) Movement kinematics in prehension are affected by grasping objects of different mass. Experimental Brain Research 176, p. 193–198. External Links: Document Cited by: §VI-C, §VI-C.