Paper deep dive
PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos
Siyao Yan, Bo Han, Jisheng Dang, Bimei Wang, Shude Wang, Hong Peng, Yulan Guo, Jianhuang Lai, Bin Hu, Tat-SengChua
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear this http URL likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.24574v1
- Canonical: https://arxiv.org/abs/2608.24574v1
Trouble viewing inline? Open PDF directly →
Full Text
52,368 characters extracted from source content.
Expand or collapse full text
SCIENCE CHINA Information Sciences . RESEARCH PAPER . PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos Siyao Yan 1 , Bo Han 1 , Jisheng Dang 1* , Bimei Wang 1 , Shude Wang 1 , Hong Peng 1* , Yulan Guo 2 , Jianhuang Lai 2 , Bin Hu 1* & Tat-SengChua 3 1 School of Information Science and Engineering, Lanzhou University, Lanzhou, 730000, China 2 School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 510006, China 3 School of Computing, National University of Singapore, Singapore, 119077, Singapore Abstract Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear nearby.One likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at https://github.com/tusu-code/20260121-icml2026-2.git. Keywords Language-guided video segmentation, spatiotemporal consistency, representation distillation, knowledge distillation, prior injection, parameter-efficient fine-tuning Citation Title for citation. Sci China Inf Sci, for review 1 Introduction Video multimodal large language models, or Video MLLMs, have recently shown strong semantic grounding through large scale image text alignment and instruction tuning. When combined with strong mask priors and lightweight decoders, they provide a practical backbone for language guided video segmentation [1, 2]. However, the main training signals give limited direct guidance for keeping predictions consistent over time. As a result, masks that look reasonable in each frame can still become unstable across frames, especially under occlusion, distractors, and appearance changes. Figure 1 shows a key pattern. Many failures in video segmentation come from unstable feature changes over time rather than from isolated errors in the mask decoder. This observation motivates a training strategy that guides representation dynamics toward stable object centered features, instead of relying only on stronger heads or test time rules. We focus on three common failure patterns in practice. They include temporal jitter, drifting in identity or shape, and identity switches after occlusion. These issues reflect missing spatial priors, which weakens spatial object permanence and spatial continuity across frames. To make this failure mode more concrete, we further examine the official Sa2VA-InternVL3-2B baseline on the ReVOS validation split. Its overall J&F is 54.61, but the score drops to 33.65 on small-target cases and 31.29 on occlusion or disappearance cases. These decreases of 20.96 and 23.32 points show that the main difficulty is not only frame-level mask prediction, but also the preservation of stable object identity and spatial continuity under challenging video conditions. This observation motivates a * Corresponding author (email: dangjisheng@lzu.edu.cn, pengh@lzu.edu.cn, bh@lzu.edu.cn) arXiv:2608.24574v1 [cs.AI] 25 Aug 2026 Sci China Inf Sci 2 Current V-MLLMs: Temporal Inconsistency & Unstable Representations Zig-zag feature Trajectory (unstable) Temporal jitter, drift, 1D switch t+1 t+2 t-1 t-2 t t-1 t Root Cause t-1 t-2 t-1 t+1 Teacher prior Shaping towards stable cluster (physics-inspired prior) Distillation Loss warmup→ramp→hold Training - time ONLY PhysMLLMs Student （Trainabl) Frozen DINOv2 Teacher REPA-Global (Align) REPA-Global (Align) Video frames Text prompt Student tokens Our PhysMLLMs: Spatial-Prior Supervision (Training-Only) Teacher Cache (Offline) Teache rglobal embes Distillation Loss Global embeddings (Teacher prior) Video frames Output: Consistent & Stable Segmentation with Zero Overhead Quantitative Gains (J&F) ReVOS: +1.35% MeVIS u : +1.20% λ( t) Stable identity, reduced jitter, robust to occlusion Zero inference-time overhead (Teacher removed) t=0 t=5t=10 t=0 t=5t=10 distillsegtotal LtLL+=)( ）（ distill L Figure 1 Motivation and overview of spatial prior injection for video segmentation. Unstable representations can lead to temporally inconsis- tent masks (left). We inject a continuity prior by distilling global visual representations from a frozen DINOv2 teacher using an offline teacher cache and scheduled distillation weight (middle). The resulting model yields more consistent segmentations with zero inference-time overhead, since the teacher is removed at test time (right). training-time prior that regularizes visual representations toward temporally coherent object-centric features. We do not rely on explicit physics-based simulation or hand-crafted physical equations. Instead, we use a physics-inspired spatial consistency prior: once an object is identified, its representation should remain coherent over time unless the video provides clear evidence of a real physical change. This prior reflects object permanence, spatial continuity, and temporal coherence in the physical world. This perspective suggests that higher per frame accuracy alone is often not enough. Memory based association is a common remedy in video segmentation [3–5], but it may not fully fix instability when upstream Video MLLM features are not well regularized. In our experiments, unstable masks can remain even with memory, which suggests that the main bottleneck often lies in the representation space. We propose a training-stage prior injection strategy that transfers object centered and geometry aware priors from a strong self supervised teacher, DINOv2 [6], into a Video MLLM through representation distillation [7]. Our core mechanism, REPA-Global, aligns the student global visual representation with cached DINOv2 global embeddings. This provides a representation-level regularizer that is expected to reduce abrupt cross-frame shifts while keeping inference unchanged. To make distillation scalable, we precompute teacher embeddings for each frame offline and distill from the cache during training, which avoids repeated teacher forward passes. To reduce interference with general multimodal capability and to stabilize training, we combine a scheduled distillation weight with calibrated parameter efficient fine tuning, including Vision LoRA [8] and MaskDecoder only calibration. This design routes most injected gradients into the visual and decoding pathway while keeping the core vision language alignment largely intact. Our contributions are summarized below. • We propose PhysMLLMs, a training-stage prior injection strategy that injects physics-inspired spatial continu- ity priors into Video MLLMs to improve temporal stability for language-guided video segmentation and support more stable video reasoning behavior, while preserving the general grounding ability of the underlying vision-language model. • We develop REPA-Global, which aligns the student global embedding with a frozen DINOv2 teacher to inject a physics-inspired spatial continuity prior that improves cross-frame consistency with no added inference-time cost. • We make the training strategy practical at scale by combining an offline teacher cache, a scheduled distillation plan, and a calibrated parameter efficient fine tuning recipe, which improves efficiency and reduces the risk of general capability drop during specialization. • Extensive experiments show that PhysMLLMs consistently improves language-guided video segmentation on ReVOS, MeVIS, and Ref-DAVIS17, especially in challenging video scenarios with small targets, fast motion, occlu- sion, distractors, and reasoning queries. Meanwhile, it preserves competitive single-frame referring segmentation Sci China Inf Sci 3 performance on RefCOCO, RefCOCO+, and RefCOCOg, and shows no clear degradation on representative general VLM benchmarks. 2 Related Work 2.1 Referring Video Object Segmentation Referring video object segmentation extends video object segmentation by adding language guidance. It requires accurate grounding of free form expressions and masks that remain consistent across frames. Compared with class agnostic tracking, the model must resolve language ambiguity, handle multiple similar instances, and keep the target identity stable over time. Common evaluation suites include DAVIS [9] and YouTube VOS [10], as well as language guided datasets such as Ref YouTube VOS [11] and MeVIS [12]. These benchmarks cover diverse scenes and motion patterns, and they often stress occlusion, viewpoint change, and appearance variation, which can expose long range instability. Earlier systems often improve temporal association with memory based designs, e.g., STM [3], STCN [4], and XMem [5]. Memory helps reuse past cues and reduce short term flicker, but it can still struggle when the upstream representation drifts or when the query target is visually similar to distractors. More recently, strong mask priors and large vision language models have become widely used. Segment Anything provides open world mask priors [2], and vision language models, e.g., LLaVA, improve semantic grounding [1]. In practice, these components can improve per frame mask quality and language alignment, yet long range stability remains difficult under occlusion and appearance changes. Predictions may show jitter, drift, and identity switches over time. Our PhysMLLMs architecture addresses these consistency problems at the representation level and uses a training stage prior injection strategy to improve temporal stability in video segmentation. Classical memory-based video object segmentation methods, such as STCN and XMem, provide strong temporal association when an object mask or reliable object cue is available. However, they are not directly comparable to language-guided Video-MLLM systems because they do not resolve free-form referring expressions or generate masks from open-ended language queries. We therefore discuss them as representative temporal association methods, while the main quantitative comparison focuses on language-guided Video-MLLM systems that share a closer input-output protocol. 2.2 Self-Supervised Spatial Priors and Geometric Structure Self supervised pretraining can learn transferable visual priors from large scale unlabeled data. These priors are useful as teacher signals when dense video mask labels are limited and when the training distribution is diverse. Contrastive and bootstrap methods learn invariances and object centered cues [13–16]. Masked prediction methods provide complementary supervision for dense representations [17–20]. DINO style self distillation learns strong semantic features and often improves sensitivity to shape and geometry [21]. DINOv2 further strengthens these properties through scaling and data curation [6]. These properties make such teachers attractive for transferring stable global structure without requiring extra manual labels. For video, masked autoencoding, e.g., VideoMAE, naturally uses temporal structure and can capture motion related cues [22]. Other self supervised video objectives also encourage temporal invariance, which can support smoother feature evolution even when the pixel content changes quickly. These priors provide a practical way to encourage temporally coherent features beyond frame by frame segmentation supervision. They also relate to spatial awareness because stable object identity over time depends on spatial continuity priors that standard segmentation losses do not always enforce, especially when the target becomes small, blurred, or partly hidden. 2.3 Consistency Oriented Distillation and Representation Alignment Knowledge distillation has moved beyond logit matching and now often uses feature level alignment and geometry preserving objectives. In this setting, the teacher provides structural guidance for student representations [23]. Supervision on intermediate features can provide steadier signals and improve spatial selectivity [24]. Relation aware and contrastive variants can also preserve the geometry of the feature space [25]. These methods are often useful when the task needs stable structure and when the student model must retain general ability while specializing. For dense prediction, structured distillation can be more reliable than direct pixel matching when global layout and long range dependencies matter [26]. Global alignment can also reduce sensitivity to local noise, which is important when distractors trigger large false masks. These findings motivate our REPA-Global design within the PhysMLLMs architecture. We align global visual representations to transfer object centered priors that support persistent identity and coherent geometry over time [27]. Because overly strong alignment may suppress valid Sci China Inf Sci 4 Video frames Text prompt e.g.,"The blue car on the left." Frame SamplingTokenizer Vision Encoder LLM SAM2 Memory SAM2 Image SAM2 Mask Decoder MeanPool Patch Tokens Student Video-MLLM Backbone SAM2 Segmentation Module Text Embeddings Visual tokens Offline Cache A B Masks Z t TRAINING (Teacher-assisted) Vision Images Prompt Tokens REPA-Global Distillation (Training-only) Projector P Global Pooling Z s REPA-Global Distillation (Training-only) Projector P Global Pooling Z s Segmentation Loss L seg Distill Loss REPA-Global: Total Objective Warm up ramp hold Total Objective Warm up ramp hold Total Loss Computation INFERENCE (Student-only) Prompt Tokens Video frames Text prompt e.g.,"The blue car on the left." B Frame sampling Tokenizer Student SAM2 LLM Image Outputs Video Outputs Text Decoder Image Caption Video Caption Image QA Image QA A Images DINOv2 (teacher) Vision Encoder L = L seg + (t)·L distill L distill = 1 cos(P(z s ),z t ) Figure 2 PhysMLLMs overview. During training, REPA-Global injects a physics-inspired spatial continuity prior by aligning the student global visual embedding z s with a frozen DINOv2 teacher embedding z t retrieved from an offline cache. Text embeddings condition the LLM and downstream mask prediction, but they do not enter the global visual pooling branch before teacher alignment. At inference, the teacher branch and cache are removed. changes, we treat the distillation weight as a training control variable rather than a fixed constant [28]. This design supports physics-inspired spatial continuity priors, such as object permanence and temporal coherence, while avoiding an overly rigid representation. 2.4 Parameter Efficient Fine Tuning and Capability Preservation Parameter efficient fine tuning adapts pretrained models while limiting representational drift, which helps preserve multimodal abilities [29]. This is important for Video MLLMs because the same model is often expected to support both segmentation and general language based reasoning. Adapter based methods and composition strategies can reduce forgetting and support safer task updates [30]. Low rank and prompt based tuning further constrains the learnable space and often improves efficiency with reduced interference [8, 31, 32]. These approaches also make it easier to control where the model changes, which matters when adding new training signals. In our setting, Vision LoRA and MaskDecoder only calibration act as practical guardrails. They direct most injected gradients into the visual and decoding pathways and reduce unnecessary changes to vision language align- ment. Capacity remains an important factor. If the parameter efficient budget is too small, the model may not absorb the injected priors well, especially when the training data contains long occlusions or fast motion. This makes module placement and rank important for performance. 3 Method 3.1 Overview We present PhysMLLMs, a transferable prior injection architecture for large video segmentation models. PhysM- LLMs uses a training stage strategy that injects spatially grounded consistency through representation distillation. Our goal is not to rely on test time heuristics, e.g., optical flow or post processing. Instead, we guide how visual representations evolve over time during training. This design supports deployment because the teacher is used only during training and is removed at inference, so runtime cost remains unchanged, as shown in Fig.2. In this work, the term physics-inspired prior refers to a soft constraint derived from object permanence, spatial continuity, and temporal coherence, rather than an explicit physical simulator. We implement this prior through representation-level distillation, so that the student representation is encouraged to evolve smoothly across frames while still allowing real appearance and pose changes. Given a training sample, the student predicts segmentation masks and we optimize it with the standard segmentation lossL seg . We further add a representation-level distillation Sci China Inf Sci 5 term L distill to encode spatial priors, e.g., spatial persistence and geometric continuity of the target. We observe that temporally implausible outputs, e.g., jitter, drift, and identity switches after occlusion, can happen even when each frame looks locally correct. This often correlates with unstable feature changes across frames. We therefore impose a consistency constraint in representation space rather than using pixel level smoothing, which can depend strongly on the decoder design. The overall objective is L =L seg + λ(t)L distill ,(1) where t denotes the training step and λ(t) controls the strength of the injected prior. We treat λ(t) as a training control signal rather than a fixed scalar. If the constraint becomes too strong too early, it can weaken grounding and localization and may lead to over smoothing. We therefore use a progressive schedule as part of the training strategy, described in Sec. 3.4. We implement the student on Sa2VA [33]. A pretrained multimodal backbone produces visual tokens, and a segmentation decoder predicts masks conditioned on the language query. For each frame x, we extract student visual tokens S(x) from the visual encoder, pool them into a student global embedding in Eq. 5, and map it through a lightweight projector for teacher alignment in Sec. 3.2. We compute the distillation loss per frame and aggregate it over frames within a batch. This provides a representation level regularizer that complements the mask loss. 3.2 REPA-Global Distillation We instantiate L distill with a global prior alignment term, denoted as REPA-Global. We use DINOv2 [6] with ViT B 14 as the frozen vision teacher. For an input frame x, the teacher produces patch tokens T(x)∈ R N×D , D = 768,(2) and we compute the teacher global embedding by mean pooling z t (x) = MeanPool(T(x))∈ R D .(3) The student provides visual tokens S(x)∈ R M×C ,(4) and we obtain a student global embedding by pooling z s (x) = Pool(S(x))∈ R C .(5) The student embedding z s (x) is extracted from visual tokens before text-conditioned mask prediction. Text condi- tioning occurs later through the LLM-generated segmentation token and the language-conditioned mask decoder. Thus, REPA-Global injects a training-time visual prior rather than distilling a text-conditioned multimodal hidden state. We use global alignment because the targeted failures, such as jitter, drift, and identity switches, often reflect unstable object-level trajectories rather than isolated patch errors. Dense token-level imitation may also over-constrain query-irrelevant regions, so we treat token-level alignment as an ablation. Our default choice is mean pooling over the visual tokens, and we treat CLS pooling as an optional variant. Since C ̸= D, we use a lightweight projector P (·), implemented as a linear layer, to map the student embedding to the teacher dimension. We then align normalized embeddings using cosine distance: L global distill = E x h 1− cos \ P (z s (x)), [ z t (x) i ,(6) where both P (z s (x)) and z t (x) are L2-normalized before computing the cosine distance. We use cosine alignment for scale robustness, which often improves optimization stability. As an optional variant, we also evaluate MSE on normalized embeddings. REPA-Global provides a training-time representation-level regularization signal that encourages physics-inspired object-centered features without tying the training strategy to a specific mask head. The novelty is not the use of DINOv2 alone, but its integration with a cache-enabled training workflow, scheduled prior injection, and calibrated PEFT. This makes the teacher a training-only physics-inspired prior provider while keeping the inference graph identical to the original student. Sci China Inf Sci 6 Algorithm 1 End-to-end workflow for cache-enabled prior injection Require: Training frames x; frozen teacher T; student S θ ; schedule λ(t); max weight λ max Ensure: Trained student parameters θ for deployment 1: Cache generation (offline) 2: for all training frame x do 3:compute teacher global embedding z t (x) 4:store z t (x) in the teacher cache 5: end for 6: Warmup 7: for training step t < t w do 8:set λ(t)← 0 9:update θ by minimizing L seg 10: end for 11: Ramp and hold 12: for training step t⩾ t w do 13:read cached z t (x) for frames in the current batch 14:compute L distill 15:optimize L seg + λ(t)L distill 16:update only calibrated PEFT parameters 17: end for 18: Deployment 19: discard the teacher and cache; run inference with the student only 3.3 Calibrated PEFT for Prior Injection Injecting priors through distillation can interact with multimodal alignment. To limit unwanted drift and keep deployment simple, we use a parameter efficient tuning strategy. We freeze the language model and most pretrained modules, and we update only two parts. We enable Vision LoRA [8] in the visual backbone and we calibrate only the mask decoder on the segmentation side. For a selected linear weight matrix W 0 , LoRA parameterizes the update as W = W 0 + ∆W,∆W = BA.(7) In this setup, the base matrix W 0 stays frozen and the model learns only the low rank factors A and B. This constrained update helps preserve semantic grounding while allowing the injected prior to shape visual features that affect temporal stability. A practical limitation is capacity. If the parameter efficient budget is too small, the model may not absorb the spatial priors well, so module placement and rank become important hyperparameters. 3.4 Overall Training Workflow for Cache, Scheduling, and Deployment Stable prior injection requires consistent teacher supervision and controlled regularization strength. As summarized in Algorithm 1, we first precompute the teacher global embedding z t (x) for each training frame and store it in an offline cache. During training, the student reads cached teacher embeddings to compute L distill , which avoids repeated teacher forward passes and keeps the distillation target deterministic. We treat λ(t) as a control variable and use a warmup-ramp-hold schedule: λ(t) = 0,t < t w , λ max · t−t w t r −t w , t w ⩽ t < t r , λ max ,t⩾ t r , (8) where t w and t r denote the end of warmup and ramp. The warmup stage optimizes only the segmentation objective, while the scheduled stage introduces the physics-inspired prior through L distill and updates only calibrated PEFT parameters. At inference, the teacher and cache are removed, and only the student model is used. Sci China Inf Sci 7 Table 1 Comparison with recent Video-MLLM systems on language-guided video segmentation. We report DAVIS-style J&F on MeVIS U, ReVOS, and Ref-DAVIS17 because most published Video-MLLM papers report aggregate J&F but do not consistently provide separate J and F values across all datasets. Higher is better. “–” means not reported or not available under a comparable protocol. MethodMeVIS UReVOSRef-DAVIS17 PG-Video-LLaVA [34]18.9– GLaMM + SAM2 [35]38.7– VideoGLaMM-3.8B [35]45.2– VISA-13B [36]44.550.970.4 VideoLISA-3B [37]44.4–68.8 HyperSeg-3B [38]–55.771.2 InstructSeg [39]–54.571.1 ViLLa-InternVideo2-6B [40]49.457.0 74.3 GLUS-7B/GLUS-ED [41]51.354.9– Sa2VA-8B [33]46.253.673.8 Sa2VA-InternVL3-2B [33]53.9 56.274.5 PhysMLLMs dw01255.157.476.0 4 Experiments We evaluate whether PhysMLLMs improves language-guided video segmentation while preserving image-level grounding and general VLM capability. We first clarify the setup and evaluation protocol, then compare with recent Video-MLLMs, analyze the effect of prior injection, and summarize capability-preservation results. 4.1 Experimental Setup and Evaluation Protocol Unless otherwise stated, PhysMLLMs is built on Sa2VA-InternVL3-2B, which combines an InternVL3-2B multi- modal backbone with the SAM2 segmentation module. The full student model contains approximately 2B param- eters. All controlled comparisons and ablations use this same 2B backbone, so the observed differences come from REPA-Global rather than model scale. Table 1 includes Sa2VA-8B only as an external reference. We use a frozen DINOv2 ViT-B/14 teacher and precompute frame-level teacher embeddings into an offline cache. During inference, the teacher and cache are removed. To preserve vision-language alignment, we freeze the language model and most pretrained modules, and update only Vision-LoRA and mask-decoder calibration parameters. The settings dw008, dw012, and dw020 denote peak distillation weights λ max = 0.008, 0.012, and 0.020 under the same warmup-ramp-hold schedule. For video segmentation, we report DAVIS-styleJ ,F , andJ &F when available, and useJ &F as the main metric for cross-method comparison because many published Video-MLLM systems do not report separate J and F . For image grounding, we report RefCOCO-series accuracy as a non-degradation check. For general VLM capability, we evaluate MMBench, MME, POPE, and TextVQA, with detailed results provided in the appendix. 4.2 Main Comparison with Recent Video-MLLM Systems Table 1 positions PhysMLLMs relative to representative recent Video-MLLM systems on MeVIS U, ReVOS, and Ref-DAVIS17, when the corresponding numbers are available. To reflect recent progress, we include recently reported Video-MLLM segmentation systems such as HyperSeg-3B and InstructSeg when their published results are available. The comparison covers a range of model sizes and training strategies, and it includes the Sa2VA baseline that our method builds upon. PhysMLLMs improves consistently across the reported benchmarks, indicating that representation-level prior injection is not tied to a single dataset or evaluation setup. It is worth noting that Table 1 contains models with different backbone capacities, including Sa2VA-8B and Sa2VA-InternVL3-2B. Therefore, this table is used to position PhysMLLMs among recent Video-MLLM systems, rather than to serve as the only evidence for a controlled capacity-matched comparison. The controlled comparisons are conducted under the same Sa2VA-InternVL3-2B + SAM2 backbone in the following ablation and capability- check analyses.Some compared methods report results on only one or two datasets. We keep these entries as “–” rather than re-estimating them under unmatched settings, because differences in model size, training data, and evaluation protocol may otherwise confound the comparison. This reporting follows the published availability of each method and avoids introducing non-comparable numbers. We use the aggregate J&F score in Table 1 to maximize coverage across published Video-MLLM systems. Separate J and F values are reported in controlled internal analyses when they are available, such as Table 2. Sci China Inf Sci 8 Table 2 Ablation of global and token-level DINOv2 alignment on ReVOS under the same Sa2VA-InternVL3-2B + SAM2 backbone. Global- only REPA-Global outperforms token-only and global+token variants, suggesting that dense patch-level constraints may interfere with language- conditioned target selection. SettingReVOS JReVOS FReVOS J &F Global-only dw01254.3160.5157.41 DINOv2 token-only53.32 59.2156.26 DINOv2 global+token53.0159.1256.07 Table 3 Ablation of teacher-prior injection under the same Sa2VA-InternVL3-2B + SAM2 backbone. The symbols dw008, dw012, and dw020 denote peak distillation weights λ max = 0.008, 0.012, and 0.020 under the same warmup-ramp-hold schedule. Higher J&F is better. SettingKDReVOS J&FMeVIS U J&F baselinew/o56.0653.92 dw008w/56.8854.51 dw008w/o56.4654.42 dw012w/57.4155.12 dw012w/o56.7654.45 dw020w/56.8755.24 dw020w/o56.6654.32 Table 4 Complexity-stratified analysis on ReVOS. Here, n denotes the number of ReVOS validation expressions in each stratum, rather than videos or frames. We report the official Sa2VA-InternVL3-2B result, the PhysMLLMs dw012 result, and the absolute J&F improvement. “Hard union” includes expressions satisfying at least one challenging condition. Stratumn Official J&F Ours dw012∆ All582254.6155.91+1.30 Small target145933.6535.91 +2.26 Fast motion145749.5750.94+1.37 Occlusion or disappearance 145931.2933.22+1.92 Distractor-heavy146055.9557.97+2.02 Reasoning query247551.7453.50+1.76 Hard union469950.4752.07+1.59 4.3 Effect of Prior Injection Teacher-prior injection is the dominant source of gain. Table 3 isolates the effect of the teacher spatial prior by toggling distillation while keeping the rest of the training recipe unchanged. Across the tested settings, enabling distillation improves segmentation on ReVOS and transfers to MeVIS U. This pattern suggests that the gains are driven by the injected physics-inspired spatial prior rather than incidental optimization effects. The improvements on MeVIS U are particularly informative because this split differs from the training distribution, indicating that the continuity constraint strengthens generalization in addition to improving in-domain performance. The gains are concentrated in difficult video scenarios. To better understand why the gains are more pronounced in video segmentation than in single-frame referring tasks, we conduct a complexity-stratified analysis on ReVOS. As shown in Table 4, PhysMLLMs improves over the official Sa2VA-InternVL3-2B model by +2.26 J&F on small targets, +1.92 on occlusion or disappearance cases, +2.02 on distractor-heavy cases, +1.37 on fast-motion cases, +1.76 on reasoning queries, and +1.59 on the hard union subset. These scenarios directly stress temporal identity preservation, cross-frame spatial continuity, and robustness to visual distractors. In contrast, RefCOCO is a single-frame benchmark where the official Sa2VA-InternVL3-2B baseline is already strong and no temporal identity maintenance is required. We therefore use the RefCOCO-series results mainly as non-degradation checks for image-level grounding, rather than as the primary target of the proposed video-oriented prior. The remaining gap on small-target and occlusion-heavy cases also suggests that extremely small or persistently occluded objects remain challenging, which clarifies the current applicability boundary of PhysMLLMs. Global alignment is more stable than token-level alignment. We further evaluate whether DINOv2 token-level teacher support can improve fine-grained video segmentation. As shown in Table 2, token-only alignment achieves 56.26 J&F and global+token alignment achieves 56.07 J&F, both lower than the global-only REPA-Global objective with 57.41 J&F. Although DINOv2 provides strong patch-level semantic features, directly imposing dense token-level constraints may over-regularize the student visual tokens and interfere with language-conditioned target selection. Therefore, we adopt global representation alignment as a more stable training-time prior. The current evidence supports a mechanism-oriented interpretation. Although we do not include a separate feature-trajectory probe, the current evidence is consistent with the intended representation-level mecha- nism. The KD on/off comparison shows that the gain comes from the teacher-prior term rather than only from the training recipe. The global-versus-token alignment ablation further shows that a compact global constraint is more Sci China Inf Sci 9 Is the stock price index ofthe banking industry rising or falling? Is the stock price index ofthe banking industry rising or falling? The banking sector's share price index rose 2.92% The banking sector's share price index rose 2.92% Yes, they can. And they also can dive.Yes, they can. And they also can dive. Can the critters in the video swim?Can the critters in the video swim? 15. 15. What number is printed on the man in red's shirt? What number is printed on the man in red's shirt? 15. What number is printed on the man in red's shirt? What did the suspect show in his pinned tweet? What did the suspect show in his pinned tweet? A picture of a burning rainbow flag.A picture of a burning rainbow flag. What did the suspect show in his pinned tweet? A picture of a burning rainbow flag. Figure 3 Qualitative visual video QA examples. Each panel shows the input video, the question, and the model response, illustrating that the injected priors do not noticeably degrade multimodal reasoning on representative cases. Table 5 Unified evaluation across image grounding, video segmentation, and MMBench under the InternVL3-2B + SAM2 backbone. We report RefCOCO accuracy, ReVOS and MeVIS U J&F, and MMBench score. Higher is better. Run IDRefCOCOReVOSMeVIS UMMBench baseline81.42%56.0653.920.7876 dw01281.61% 57.4155.120.7878 dw00881.73%56.8854.510.7878 dw02081.68%56.8755.240.7878 effective than dense token-level imitation for language-conditioned video segmentation. Together with the temporal stability proxy in the appendix, these results suggest that REPA-Global acts as a representation-level temporal regularizer, rather than only as a mask-head adjustment. 4.4 Default Configuration and General Grounding Table 5 summarizes the default configuration across image grounding, video segmentation, and MMBench. The default dw012 setting gives the strongest ReVOS performance and competitive MeVIS U transfer, while RefCOCO and MMBench remain close to the baseline. Additional MME, POPE, TextVQA, RefCOCO-series, and distillation- weight sensitivity results are reported in the appendix. These results indicate that the physics-inspired prior improves video consistency without causing a clear collapse in image-level grounding or representative general VLM capability. 4.5 Limitations and Failure Cases PhysMLLMs still has several limitations. As shown in the appendix failure cases, dense similar distractors, persistent occlusion, and extremely small targets can still cause incorrect localization or identity ambiguity. Although Table 4 shows gains on small-target and occlusion/disappearance strata, their absolute J&F scores remain much lower than the full validation average. This indicates that these scenarios are not fully solved. The limitation mainly comes from the global nature of REPA-Global. The injected physics-inspired prior encour- ages object permanence, spatial continuity, and temporal coherence through a soft representation-level regularizer, but it does not explicitly model physical forces, 3D dynamics, object interactions, or query-specific target iden- tity. Direct feature-trajectory verification also remains future work. These limitations suggest that query-aware or mask-aware prior injection may further improve fine-grained video grounding. Sci China Inf Sci 10 Table 6 General VLM capability check on MME, POPE, and TextVQA. All rows use the same Sa2VA-InternVL3-2B backbone and compare checkpoints from different calibration stages rather than different model backbones. Higher is better. Checkpoint / training stageMME perception MME reasoning MME total POPE F1 TextVQA val acc Sa2VA-InternVL3-2B official1630.838520.3572151.19587.42176.808 RefCOCO-ft init1626.126536.0712162.19887.40576.580 ReVOS calib it2001610.827522.5002133.327 87.57576.876 ReVOS calib it5001602.206527.8572130.06487.54276.796 ReVOS calib it500 + KD dw0121622.166530.714 2152.880– 5 Conclusion PhysMLLMs is a training-stage prior injection strategy for Video-MLLM-based language-guided video segmentation. It targets common temporal failures, including jitter, drift, and identity switches that often appear under occlusion and distractors. The key idea is to inject a physics-inspired spatial continuity prior by aligning the student global visual representation with a frozen DINOv2 teacher through REPA-Global. Teacher supervision is delivered through an offline embedding cache, which makes distillation scalable and keeps inference unchanged because the teacher is removed at test time. We further combine scheduled distillation weighting with calibrated parameter-efficient fine-tuning, which improves training stability and helps preserve general grounding ability while improving mask quality and cross-frame consistency. The complexity-stratified ReVOS analysis further shows that the gains are concentrated in difficult video scenar- ios, including small targets, fast motion, occlusion or disappearance, distractor-heavy scenes, and reasoning queries. This supports our motivation that the injected prior mainly addresses temporal identity and spatial continuity. Meanwhile, the RefCOCO-series and general VLM results indicate that the proposed training-time prior preserves image-level grounding and representative multimodal capability. At the same time, the current evidence mainly supports the proposed mechanism through controlled ablations, challenging-scenario analysis, and temporal stability proxies. Direct feature-trajectory analysis remains future work, especially for query-specific cases with persistent occlusion, extremely small targets, or dense similar distractors. Acknowledgements This work was supported by the National Natural Science Foundation of China (Grants No. 62227807 and U24B20186). This work was also supported by the Supercomputing Center of Lanzhou University. Appendix A. Additional Experimental Results This appendix provides supplementary quantitative results that support the main experimental conclusions, includ- ing general VLM capability checks, RefCOCO-series referring image segmentation comparisons, and distillation- weight sensitivity analysis. These results are moved from the main text to keep the core experimental section concise while preserving the detailed evidence for capability preservation and hyperparameter sensitivity. A.1 General VLM Capability Check All rows in Table 6 are based on the same Sa2VA-InternVL3-2B backbone. The listed entries denote checkpoints obtained at different calibration stages, including the official checkpoint, the RefCOCO fine-tuned initialization, ReVOS-calibrated checkpoints, and the ReVOS-calibrated checkpoint with REPA-Global KD. Table 6 reports additional general VLM capability checks on MME, POPE, and TextVQA. The ReVOS calib it500 + KD dw012 checkpoint achieves an MME total score of 2152.880, which is comparable to the official Sa2VA-InternVL3-2B score of 2151.195. The nearby ReVOS-calibrated checkpoints also remain stable on POPE and TextVQA. These results suggest that the video-oriented calibration path and REPA-Global injection do not cause an obvious collapse in representative general VLM capability. A.2 RefCOCO-Series Referring Image Segmentation Table 7 reports the RefCOCO-series comparison with representative fine-tuned models. PhysMLLMs remains competitive on RefCOCO, RefCOCO+, and RefCOCOg. Since the proposed spatial prior is designed mainly for video consistency, these results are used as non-degradation checks for image-level referring capability. Sci China Inf Sci 11 Table 7 Referring image segmentation comparison with representative fine-tuned models. We report validation accuracy on RefCOCO, RefCOCO+, and RefCOCOg. Higher is better. ModelRefCOCORefCOCO+RefCOCOg LAVT [42]72.762.161.2 GLaMM-7B [43]79.572.674.2 OMG-LLaVA-7B [44]78.069.172.9 F-LMM-7B [45]76.165.268.5 Sa2VA-InternVL3-2B [33]81.4 75.780.3 PhysMLLMs81.976.380.4 Table 8 Sensitivity to the peak distillation weight under a fixed warmup-ramp-hold schedule. The column “Peak weight λ max ” denotes the maximum scheduled KD weight. Higher tIoU mean and lower tIoU variance indicate stronger temporal stability. Setting Peak weight λ max ReVOS J&F ↑ tIoU mean ↑ tIoU var ↓ dw00020.00256.640.64610.0841 dw0040.00456.530.64520.0843 dw0060.00656.590.63960.0854 dw0080.00856.880.64760.0839 dw0100.01056.660.65140.0824 dw0120.01257.410.64190.0854 dw0150.01556.820.64510.0868 dw0200.02056.870.65050.0817 A.3 Distillation-Weight Sensitivity Table 8 reports the sensitivity to the peak distillation weight under the same warmup–ramp–hold schedule. Moderate distillation gives the best ReVOS J&F, while stronger distillation tends to improve the temporal stability proxy with a small cost in mask quality. This supports using dw012 as the default setting in the main experiments. A.4 Qualitative Success Cases Figure 4 provides additional qualitative comparisons on LV-VIS. These examples cover human-object interaction, multiple similar instances, small or thin objects, and reasoning-style language queries. The baseline often produces unstable masks, such as over-segmented regions under distractors, missing small objects, or identity drift after partial occlusion. PhysMLLMs produces more coherent masks across frames and more accurate target localization in these examples. This supports the quantitative finding that the physics-inspired prior mainly improves temporal consistency and target identity preservation. A.5 Failure Cases Figure 5 shows representative failure cases of PhysMLLMs. Although the proposed physics-inspired prior improves temporal consistency in many challenging scenarios, it can still fail under dense similar distractors, persistent occlusion, and extremely small targets. Sci China Inf Sci 12 (a) BASELINE OURS (a)(b)(c)(d) (e)(f) Figure 4 Qualitative comparison on LV-VIS for language-guided video segmentation. The top three rows show the baseline results and the bottom three rows show PhysMLLMs results. Red boxes highlight baseline failures, and blue boxes highlight improved predictions by PhysMLLMs. The segmentation prompts are: (a) “Please segment a person holding a mop.” (b) “Which creatures are primates? Please answer using segmentation masks.” (c) “Please segment the object(s) carrying ammunition.” (d) “Which gray mat(s) have patterns? Please respond with a segmentation mask.” (e) “Please segment the object fanning its wings on the koala’s snout.” (f) “Please segment the aircraft that is most likely to be in a state of lacking fuel.” Overall, PhysMLLMs improves mask quality and cross-frame consistency under occlusion, distractors, and reasoning-style referring expressions. Sci China Inf Sci 13 Red = false positive, purple = false negative. Figure 5 Representative failure cases of PhysMLLMs. Each row shows the input image, ground truth, baseline prediction, and our prediction. The three rows correspond to dense similar distractors, persistent occlusion, and small target cases. Red regions indicate false positives, and purple regions indicate false negatives. These examples show that global prior alignment improves temporal stability but may still be insufficient for query-specific identity disambiguation under highly ambiguous visual conditions. Sci China Inf Sci 14 References 1 Liu H, Li C, Wu Q, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023 2 Kirillov A, Mintun E, Ravi N, et al. Segment anything. In: Proceedings of IEEE/CVF International Conference on Computer Vision, 2023 3 Oh S W, Lee J Y, Xu N, et al. Video object segmentation using space-time memory networks. In: Proceedings of IEEE/CVF International Conference on Computer Vision, 2019 4 Cheng H K, Tai Y W, Tang C K. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. In: Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 5 Cheng H K, Oh S W, Price B, et al. XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In: Proceedings of European Conference on Computer Vision, 2022 6 Oquab M, Darcet T, Moutakanni T, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023 7 Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015 8 Hu E J, Shen Y, Wallis P, et al. LoRA: Low-rank adaptation of large language models. In: Proceedings of International Conference on Learning Representations, 2022 9 Pont-Tuset J, Perazzi F, Caelles S, et al. The 2017 DAVIS challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017 10 Xu N, Yang L, Fan Y, et al. YouTube-VOS: A large-scale video object segmentation benchmark. In: Proceedings of European Conference on Computer Vision, 2018 11 Seo S, Lee J Y, Han B. Ref-YouTube-VOS: A large-scale video object segmentation benchmark for referring expressions. arXiv preprint, 2020 12 Ding H, et al. MeVIS: A large-scale benchmark for video segmentation with motion expressions. arXiv preprint arXiv:2308.03772, 2023 13 Chen T, Kornblith S, Norouzi M, et al.A simple framework for contrastive learning of visual representations.In: Proceedings of International Conference on Machine Learning, 2020 14 He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020 15 Grill J B, Strub F, Altche F, et al. Bootstrap your own latent: A new approach to self-supervised learning. In: Proceedings of Advances in Neural Information Processing Systems, 2020 16 Caron M, Misra I, Mairal J, et al. Unsupervised learning of visual features by contrasting cluster assignments. In: Proceedings of Advances in Neural Information Processing Systems, 2020 17 He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022 18 Bao H, Dong L, Piao S, et al. BEiT: BERT pre-training of image transformers. In: Proceedings of International Conference on Learning Representations, 2022 19 Zhou J, Wei C, Wang H, et al. iBOT: Image BERT pre-training with online tokenizer. In: Proceedings of International Conference on Learning Representations, 2022 20 Wei C, Xie C, Kong T, et al. Masked feature prediction for self-supervised visual pre-training. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022 21 Caron M, Touvron H, Misra I, et al. Emerging properties in self-supervised vision transformers. In: Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 22 Tong Z, Song Y, Wang J, et al. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602, 2022 23 Gou J, Yu B, Maybank S J, et al. Knowledge distillation: A survey. International Journal of Computer Vision, 2021, 129: 1789–1819 24 Romero A, Ballas N, Kahou S E, et al. FitNets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2015 25 Park W, Kim D, Lu Y, et al. Relational knowledge distillation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019 26 Liu Y, Shun C, Wang J, et al. Structured knowledge distillation for dense prediction. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019 27 Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers and distillation through attention. In: Proceedings of International Conference on Machine Learning, 2021. 10347–10357 28 Han Z, Gao C, Liu J, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024 29 Houlsby N, Giurgiu A, Jastrzebski S, et al. Parameter-efficient transfer learning for NLP. In: Proceedings of Chaudhuri K, Salakhutdinov R, editors, Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019. 2790–2799 30 Pfeiffer J, Kamath A, R ̈uckl ́e A, et al. Adapterfusion: Non-destructive task composition for transfer learning, 2021 31 Li X L, Liang P. Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of Zong C, Xia F, Li W, et al., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online: Association for Computational Linguistics, 2021. 4582–4597 32 Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. In: Proceedings of Moens M F, Huang X, Specia L, et al., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. 3045–3059 33 Yuan H, Li X, Zhang T, et al. Sa2VA: Marrying SAM2 with LLaVA for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025 34 Munasinghe S, Thushara R, Maaz M, et al.PG-Video-LLaVA: Pixel grounding large video-language models.arXiv preprint arXiv:2311.13435, 2023 35 Munasinghe S, Gani H, Zhu W, et al. VideoGLaMM: A large multimodal model for pixel-level visual grounding in videos. arXiv preprint arXiv:2411.04923, 2024 36 Yan C, Wang H, Yan S, et al. VISA: Reasoning video object segmentation via large language models. In: Proceedings of European Conference on Computer Vision, 2024 37 Bai Z, He T, Mei H, et al. One token to seg them all: Language instructed reasoning segmentation in videos. In: Proceedings of Advances in Neural Information Processing Systems, 2024 38 Wei C, Zhong Y, Tan H, et al. HyperSeg: Towards universal visual segmentation with large language model. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025 39 Wei C, Zhong Y, Tan H, et al. InstructSeg: Unifying instructed visual segmentation with multi-modal large language models. arXiv preprint arXiv:2412.14006, 2024 40 Zheng R, Qi L, Chen X, et al. Villa: Video reasoning segmentation with large language model. In: Proceedings of IEEE/CVF International Conference on Computer Vision, 2025. 23667–23677 41 Lin L, Yu X, Pang Z, et al. Glus: Global-local reasoning unified into a single large language model for video segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 8658–8667 42 Yang Z, Wang J, Tang Y, et al. LAVT: Language-aware vision transformer for referring image segmentation. In: Proceedings of IEEE/CVF Sci China Inf Sci 15 Conference on Computer Vision and Pattern Recognition, 2022 43 Rasheed H, Maaz M, Mullappilly S S, et al. GLaMM: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356, 2024 44 Zhang T, Li X, Fei H, et al. OMG-LLaVA: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406.19389, 2024 45 Wu S, Jin S, Zhang W, et al. F-LMM: Grounding frozen large multimodal models. arXiv preprint arXiv:2406.05821, 2025