Paper deep dive
SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning
Chaoyue Li, Shoudong Han, Haoyang Wu, Boxue Yang, Sijia Chen, Yufan Chen, Ruping Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/14/2026, 9:55:54 AM
Summary
The paper introduces SceneForesee, a benchmark and Scene Evolution Understanding (SEU) task for predicting and interpreting how persistent entities evolve in video scenes. It proposes EvoSee, a spatiotemporal reasoning framework that learns future geometry and event semantics through shared candidate states and joint assignment. EvoSee outperforms baselines like Qwen3.5-9B on entity continuity and scene evolution reasoning metrics.
Entities (8)
Relation Signals (6)
SceneForesee → contains → 144.57K temporal pairs
confidence 95% · It contains 144.57K temporal pairs, 1.05M entity correspondences, and 272.49K Events paired with observed evidence.
EvoSee → outperforms → Qwen3.5-9B
confidence 95% · On SceneForesee, EvoSee outperforms Qwen3.5-9B by 10.45 points in Combined Point-F1 and 17.24 points in Event Global F1.
EvoSee → improves → Dyn-Bench
confidence 92% · On the 4D reasoning benchmark Dyn-Bench, it improves question answering accuracy and grounding J&F by 4.8 and 3.58 points over their respective base systems.
SceneForesee → derivedfrom → APIDIS
confidence 90% · SceneForesee pairs adjacent, equally long observed and future intervals from APIDIS, OVT-B, and TAO
SceneForesee → derivedfrom → TAO
confidence 90% · SceneForesee pairs adjacent, equally long observed and future intervals from APIDIS, OVT-B, and TAO
SceneForesee → derivedfrom → OVT-B
confidence 90% · SceneForesee pairs adjacent, equally long observed and future intervals from APIDIS, OVT-B, and TAO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large multimodal models (LMMs) need to understand how entities move and interact as a scene develops. This requires following the same entities over time and relating their observed motion to subsequent changes. We formulate Scene Evolution Understanding (SEU), a task for predicting and interpreting this change. Given observed video, a model estimates entity histories and predicts one scene continuation through future trajectories and Events, accompanied by Visual Evidence from observed motion and a scene description. We construct SceneForesee, a benchmark for evaluating entity continuity and scene evolution reasoning through paired observed and future scenes with shared entity identities. It contains 144.57K temporal pairs, 1.05M entity correspondences, and 272.49K Events paired with observed evidence. We propose a spatiotemporal reasoning framework, EvoSee, that learns future geometry and Event semantics through shared candidate states and joint assignment. It selects one continuation per entity from observation and uses the selected changes for entity refinement, retention, and scene representation. On SceneForesee, EvoSee outperforms Qwen3.5-9B by 10.45 points in Combined Point-F1 and 17.24 points in Event Global F1. On the 4D reasoning benchmark Dyn-Bench, it improves question answering accuracy and grounding J&F by 4.8 and 3.58 points over their respective base systems. These results support scene evolution as both a prediction target and a dynamic representation that brings predicted entity changes into spatiotemporal this http URL project is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.14439v1
- Canonical: https://arxiv.org/abs/2610.14439v1
Trouble viewing inline? Open PDF directly →
Full Text
105,635 characters extracted from source content.
Expand or collapse full text
SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning Chaoyue Li 1 , Shoudong Han 1 , Haoyang Wu 1 , Boxue Yang 2 Sijia Chen 1 , Yufan Chen 1 , Ruping Zhang 1 1 Huazhong University of Science and Technology 2 Shanghai Jiao Tong University Large multimodal models (LMMs) need to understand how entities move and interact as a scene develops. This requires following the same entities over time and relating their observed motion to subsequent changes. We formulate Scene Evolution Understanding (SEU), a task for predicting and interpreting this change. Given observed video, a model estimates entity histories and predicts one scene continuation through future trajectories and Events, accompanied by Visual Evidence from observed motion and a scene description. We construct SceneForesee, a benchmark for evaluating entity continuity and scene evolution reasoning through paired observed and future scenes with shared entity identities. It contains 144.57K temporal pairs, 1.05M entity correspondences, and 272.49K Events paired with observed evidence. We propose a spatiotemporal reasoning framework, EvoSee, that learns future geometry and Event semantics through shared candidate states and joint assignment. It selects one continuation per entity from observation and uses the selected changes for entity refinement, retention, and scene representation. On SceneForesee, EvoSee outperforms Qwen3.5-9B by 10.45 points in Combined Point-F1 and 17.24 points in Event Global F1. On the 4D reasoning benchmark Dyn-Bench, it improves question answering accuracy and grounding J&F by 4.8 and 3.58 points over their respective base systems. These results support scene evolution as both a prediction target and a dynamic representation that brings predicted entity changes into spatiotemporal reasoning. Project: https://github.com/mikubaka88/SceneForesee-EvoSee 1 Introduction Large multimodal models (LMMs) are moving from visual recognition toward reasoning about motion and interaction in videos (Bai et al., 2025a; Huang et al., 2026). An entity moving behind another retains its identity even as its position, visibility, and spatial relations change. This continuity makes the changes attributable to the same entity, while its observed motion informs what follows. Scene evolution concerns the development of persistent entities and their relations over time, placing continuity at the center of dynamic video reasoning. Continuity and change appear across several lines of dynamic video research. Video language models interpret extended visual content (Bai et al., 2025b; Wang et al., 2025), while motion grounding and evidence localization relate answers to dynamic observations (Deng et al., 2025; Gao et al., 2026; Wei et al., 2026). Tracking and forecasting establish entity identities and predict their spatial continuation (Meinhardt et al., 2022; Dendorfer et al., 2022; Ashutosh et al., 2025). Event prediction and predictive representations address the changes that follow observation (Wang et al., 2026; Assran et al., 2025). At the intersection of these capabilities, we study how spatial and semantic predictions can refer to the same observed entities and jointly inform scene reasoning. Relating spatial and semantic change requires knowing which observations belong to the same entity. Persistent identity supplies this reference when observation ends and future prediction begins. In Figure 1(a,b), following the console and the person’s hands makes their changing positions and interaction attributable to the same participants. A trajectory describes where an entity moves, while an Event summarizes its state or relations. These spatial and semantic descriptions concern the same continuation and can be learned and evaluated with a common entity reference. Visual Evidence records the observed motion associated with that continuation, such as the changing grip and console orientation in the example. Together, these descriptions relate predicted change to the entities and motion from which it is inferred, providing context for scene reasoning. 1 arXiv:2610.14439v1 [cs.CV] 10 Oct 2026 T001 T002 Large Language Model Decoder Vision Encoder (1) Observed Scene Description Qwen3.5-9B EvoSee A panda lies among bamboo leaves. A reclining panda shifts among scattered bamboo, moving its torso and paws. (2) Scene Evolution Prediction Qwen3.5-9B EvoSee The spotted seal rises toward the darker seal. Both seals rise as the spotted one curves slightly right beneath the other. (3) Evidence with Entity Trajectory Tracking Qwen3.5-9B EvoSee The person holds the console steady while adjusting their grip. The console moves left and turns over as the person shifts their grip.” (a) LLaVA-OneVision(b) EvoSee for SEU Text Tokenizer Instruction Video Instruction Video Entity Visual Evidence Encoding Text Tokenizer EntityEvidence Text TextVision Geometry Event Alignment Reasoning Evolution Conditioned Decoding (c) Performance Radar Describe the main evolution process in this video. Describe the main evolution process in this video. A person’s hands are holding a handheld console above a green mat, adjusting their grip. A person changed the way the game console<trk1> was held by hands<trk2>, tilting it to the left and preparing to flip it over. t (d) Three Aspects of Scene Evolution Understanding Observed Future Evidence T001T001 T001 T002 T001 T002 T001 T002 T001 T002 <trk> 10 20 30 30.54 31.10 30.13 26.53 27.4726.77 28.61 29.32 28.51 20 40 60 49.08 57.33 49.62 47.09 48.9150.11 54.45 57.77 57.93 Entity Continuity Observed Future Combined Scene Evolution Reasoning Entity Event Visual Evidence Point-F1RecallMeanIoU TrackFormerQwen3.5-9B MeMOTRSambaMOTR EvoSee Global F1Scene-F1GPR LTrackQwen3.5-9B Qwen3-VL-8BLLaVA-OV2 EvoSee Figure 1. Scene Evolution Understanding. (a,b) Description and entity continuation from observation. (c) Eighteen continuity and reasoning metrics, plotted as percentage scores; labels give EvoSee values. (d) Observed movement, anticipated interaction, and evidence. Future frames show the evaluation reference. Tracking baselines denote task adaptations. We formulate Scene Evolution Understanding (SEU) as a task for predicting and interpreting how persistent entities develop. From observed video, a model estimates entity histories and infers one scene continuation through future trajectories and Events. Its output also identifies associated Visual Evidence in observation and describes the evolving scene. We construct SceneForesee, a benchmark for evaluating scene evolution understanding through paired observed and future intervals with shared entity identities. Its 144.57K temporal pairs contain 1.05M entity correspondences and 272.49K Events paired with observed evidence. This organization supports complementary evaluations of entity continuity and scene evolution reasoning on the same scenes. To learn and use scene evolution, we propose a spatiotemporal reasoning framework, EvoSee. EvoSee treats the observed entity as the reference for both its future motion and the Events in which it participates. For each entity, it learns candidate continuations whose geometric and Event predictions share future states and joint assignment during training. It selects one candidate using the observed entity state and evidence. The selected change then refines the entity state and contributes to entity retention and scene representation. EvoSee outperforms the evaluated baselines in entity continuity and Event category recognition on SceneForesee. Against Qwen3.5-9B, it also improves complete Event prediction, where the category and all participants must match. Ablations relate these gains to joint assignment, continuation selection, and the use of predicted change. The value of this change extends to question answering and grounding on Dyn-Bench (Huang et al., 2026), where the learned scene representation supports 4D reasoning. Our contributions are summarized as follows. •We formulate a new task, Scene Evolution Understanding, which aims to predict and interpret how persistent entities evolve over time. The task represents one scene continuation through future entity motion and Events, together with associated visual evidence from observation. •We introduce SceneForesee, a benchmark for evaluating entity continuity and scene evolution reasoning. Shared identities associate observed motion, future trajectories, and Events, supporting complementary evaluations of continuity and change on the same scenes. 2 •We propose EvoSee, an entity-centric spatiotemporal reasoning framework that models future geometry and Event semantics as complementary aspects of scene evolution. Shared future states and joint assignment align geometric and Event predictions for each entity, and the selected changes update entity and scene representations. •We demonstrate experimentally that modeling scene evolution improves future prediction and observed entity estimation. The learned representations also improve dynamic question answering and grounding on the 4D reasoning benchmark Dyn-Bench. 2 Related Work Following entities through time. Tracking establishes correspondence between observations of the same entity. Object query trackers maintain this correspondence across frames (Meinhardt et al., 2022; Gao & Wang, 2023), while other formulations predict identities (Gao et al., 2025) or associate tracked entities with descriptions and referring expressions (Li et al., 2024; Wu et al., 2023). These entity histories support forecasting of subsequent locations and interactions beyond observation (Dendorfer et al., 2022; Lu et al., 2026; Liang et al., 2019; Ashutosh et al., 2025; Nguyen et al., 2025). SEU builds on this identity correspondence to attribute future motion and Events to the same observed entities. Predicting scene development. Beyond future locations, prediction concerns changes in entity states and relations. FutureBench and FeVOS study future events (Wang et al., 2026; Lan et al., 2026), while scene graph forecasting represents changes through entities and their relations (Pistilli et al., 2026). Anticipatory detection and predictive representations learn expected future states (Papais et al., 2025; Assran et al., 2025). SEU uses persistent identities to relate future trajectories, Events, and observed Visual Evidence within the same scene continuation. Reasoning with motion and visual evidence. Video language models use dynamic visual content to produce descriptions and answers (Bai et al., 2025b; Wang et al., 2025; Zhang et al., 2025). Motion grounding and evidence localization identify observations relevant to these outputs (Deng et al., 2025; Galoaa et al., 2026; Wei et al., 2026). DynTrace represents observed motion through traces, and Dyn-Bench evaluates dynamic question answering and grounding (Gao et al., 2026; Huang et al., 2026). EvoSee uses predicted entity continuations to enrich the scene representation for such reasoning. 3 Task and Benchmark 3.1 Scene Evolution Understanding Scene Evolution Understanding requires a model to estimate entity histories from observed video and predict one scene continuation through future trajectories and Events with explicit participants. The observed entities provide the reference for these predictions. The output also identifies Visual Evidence in observation and describes the scene evolution. Figure 1(d) illustrates observed motion, anticipated interaction, and visual evidence through example scenes. Future frames supply training and evaluation targets; Appendix A specifies sampling and output format. 3.2 Benchmark Construction SceneForesee pairs adjacent, equally long observed and future intervals from APIDIS, OVT-B, and TAO, covering sports interactions, open category entities, and varied natural motion (De Vleeschouwer et al., 2008; Liang & Han, 2024; Dave et al., 2020). Timestamped boxes preserve shared identities between observed and future intervals, with at least one continuing entity per sample. These identities anchor the annotations in Figure 2(c). The four Event categories in Figure 2(b) summarize trajectory configurations, providing discrete targets for predicting evolution under shared identities. Approach uses decreasing separation. Occlusion uses box overlap throughout the image at the transition from observation to prediction. Leave uses proximity to the image edge at the future endpoint, while relation change covers remaining configurations. Each Event records participants and a future interval, paired with observed Visual Evidence of individual trajectory trends or pairwise relation changes. Figure 2(c) pairs the console’s anticipated occlusion with the entities’ observed relation. A scene description expresses their motion and development. 3 (a) SceneForesee statistics(b) Events and evidence(c) Benchmark construction (d) Evaluation targets and metrics 82.96% Multi-entity samples 74.19% Events with two entities 2,876 source videos 2 Visual Evidence categories 1 Temporal pairing APIDIS · TAO · OVT-B / source trajectories Observed inputFuture reference (GT) T001 T002 s + Hs + 2H 2 Event and evidence annotation 3 Sample assembly and validation Events and evidence share entity identities Trajectory trendRelation change T001 · consoleT002 · person Future Event: OcclusionObserved Visual Evidence Relation change ObservedFuture GT ObservedFuture GT t t′ 12 34 t t′ Trajectories T001T002 EventEvidence Event–Evidence Observed Future Evidence Scene description Entity Continuity 9 Metrics Point-F1MeanIoU Scene Evolution Reasoning 9 Metrics Global F1Scene-F1 GPR GT Pred. Observed Future Entity IDs GT Pred. T001T002 T001T002 Event categories GT Pred. Evidence categories GT Pred. Box overlap (IoU) GT Pred. Point-Recall Relation change Approach Occlusion Leave 4 Event types Human Verify LLM as a Judge Figure 2. SceneForesee. (a) Corpus statistics. (b) Event and Visual Evidence categories. (c) Temporal pairing, annotation, and sample validation. (d) Entity Continuity and Scene Evolution Reasoning metrics. Ring labels give counts; sector angles are schematic. Common annotation criteria apply to all sources, with LLM assessment and manual inspection of generated annotations. Data splits follow source videos and disjoint APIDIS episodes, keeping overlapping windows together. Figure 2(a) summarizes 2,876 videos, with multiple entities in 82.96% of samples and two participants in 74.19% of Events. Appendix A.2 details construction and splits. 3.3 Evaluation Metrics Figure 2(d) presents complementary evaluations of entity continuity and scene evolution reasoning on the same temporal pairs. Entity Continuity. Identity matching from observed trajectories remains fixed across all intervals. Point-F1@0.5 and Point-Recall@0.5 measure recovery of entity locations under this correspondence, while MeanIoU measures spatial overlap. Combined scores weight observed and future scores by their annotated point counts. Scene Evolution Reasoning. Entity scores evaluate identities referenced in descriptions; Event and Visual Evidence scores evaluate category occurrences. Global F1 and the geometric mean of global precision and recall (GPR) pool matches across scenes; Scene-F1 weights scenes equally. Complete Event micro-F1 additionally requires the category and all participants to match. BERTScore F1 measures similarity between complete descriptions (Zhang et al., 2020). Appendix A defines matching and aggregation. 4 Method EvoSee learns entity continuations from observed video and uses the predicted changes for scene reasoning. Figure 3 shows how observed entity states and evidence condition shared geometric and Event predictions. A selector uses the observed representations to choose one continuation per entity. The selected change refines the entity state and contributes to retention and scene representation. 4 Candidate continuations f i 1 Event f i k Event f i K Event Shared future state Future state Vision Encoder Entity stateObserved evidence Observation Guided Selection State Observed video Directed Attention Observed Entity Event Text Tokenizer “ Describe the main evolution process in this video. Instruction Entity Encoder Visual Evidence Encoder Entity Visual Evidence Encoding Entity Retention Evolution Refinement Consistency Geometry Event Alignment Reasoning Future event Occlusion Observed evidence Relation change T001 Console T002 Person “ The console turns over as fingers move across the screen. Dynamic QAVideo grounding ? Observed 0.95 s Future 1.15 s Future 1.95 s T001 T002 T001 T002 T001 T002 x y t Scene reasoning Scene representation Evolution Conditioned Decoding t Figure 3.EvoSee. Entity state and evidence condition shared geometric and Event continuations. Selected changes inform refinement, retention, and scene representation. Entity in directed attention denotes geometric state. 4.1 Entity Visual Evidence Encoding Entity Visual Evidence Encoding (EVE) uses a shared learned queryq i to encode entity state and observed motion. The query attends to observed visual memoryX obs for statea i at the observation boundary and to frame featuresX t for temporal statesh t i (Meinhardt et al., 2022; Vaswani et al., 2017). Normalized relevance scores aggregate these temporal states into evidencee i . Evidence intervals supervise temporal relevance, and evidence categories supervise the aggregated representation. Appendix B.1 gives the encoding equations. 4.2 Geometry Event Alignment Reasoning Geometry Event Alignment Reasoning (GEAR) infers future geometry and Event semantics from entity state and evidence. Their sumu a i = a i +e i initializes geometric and semantic statesu g i ,u s i . Stackingu a i ,u g i ,u s i in this order gives U i . Each attention headhprojects normalizedU i into queriesQ h i , keysK h i , and valuesV h i of widthd h , computing output A h i under maskM. A h i = softmax Q h i (K h i ) ⊤ √ d h +M V h i , M = 0 −∞ −∞ 00 −∞ 000 .(1) The mask implements the dependencies in Figure 3, allowing each state to attend only to itself and preceding states. Concatenated head outputs are projected for a residual update of U i . Each updated state independently weights feedforward transformations, preserving these dependencies. The resulting geometric and semantic states condition a future sequencef i ∈R L×d withLtime steps and feature widthd. Each ofK candidate continuations uses embeddingd k ∈R d and transitionφ k (Rupprecht et al., 2017). The vector1 L broadcasts 5 the embedding across time. As depicted in Figure 3, decodersD g andD s share candidate statef k i to predict geometry g k i and Event logits s k i , with task context embedding c informing semantics. f k i = f i + φ k (f i + 1 L d ⊤ k ), g k i = D g (f k i ), s k i = D s (f k i ;c).(2) Joint assignment selects one candidate for training against annotated future geometryg ⋆ i and Event targetss ⋆ i , using their combined prediction error. b k i ∈ arg min k ℓ geo (g k i ,g ⋆ i ) + λ s ℓ sem (s k i ,s ⋆ i ) .(3) Hereℓ geo measures box error,ℓ sem measures Event classification error, andλ s = 0.20. Each Event supplies its category target to every participant, including both identities for pairwise Events. All future supervision uses the assigned index, coupling geometric and Event learning within the same candidate. The assigned index also supervises selection from observation. A selectorRmaps entity state, evidence, and observed motion embeddingo i into candidate probabilitiesρ i . It learns b k i through negative log likelihood. At inference, the most probable candidatek ∗ i determines future boxes, visibility, persistence, and Event scores, giving one continuation per retained entity. Appendix B.2 details selection and the endpoint separation regularizer used during training. 4.3 Evolution Conditioned Decoding Evolution Conditioned Decoding (ECD) refines the observed entity state using predicted change. Projecting temporally averaged future boxes and the sigmoid of temporally averaged Event logits gives summariesg i ands i . The summaries use probability weighted decoded predictions during training and the selected candidate at inference. MapΦcombines them with observed state and evidence into evolution representation m i . A sigmoid gate from Γ weights the projected evolution representation P f (m i ). m i = Φ([a i ;g i ;s i ;e i ]), a + i = a i + σ(Γ([a i ;m i ]))⊙ P f (m i ).(4) Hereσis sigmoid and⊙denotes elementwise multiplication. The residual keepsa i as a reference, while the updated state supports observed trajectory estimation and entity confidence. Entity retention combines confidenceℓ obj i from updated statea + i with agreement among observed and predicted features. Six cosine distances measure relations among evidence, entity state, geometry, semantics, evolution representation, and task context. Their meanB i enters retention scoreκ i = ℓ obj i − λ cons B i , with nonnegative weightλ cons . A margin ranking loss favors lower distances for queries matched to annotations than for unmatched queries. The highest retention scores define setI ∗ . Appendix B.3 specifies the relations and supervision. Retained entities supply trajectories and Event scores from their selected continuations. Complete Events specify a category and all participants using predicted entity IDs, which evaluation maps through the observation correspondence. For scene reasoning, EvoSee averages entity statesa i , evidencee i , and evolution representationsm i over all queries. Their concatenation with task contextcis projected into scene representationh scene (Appendix B.3). This representation supplies observed and predicted context for description, dynamic question answering, and grounding (Deng et al., 2025; Gao et al., 2026). 5 Experiments SceneForesee comparisons share test samples, observation intervals, temporal extents, and metrics. Qwen3.5-9B and EvoSee predict trajectories; other language models use shared predictions from the TrackFormer-style task adaptation (Meinhardt et al., 2022). Tables 1–2 report eighteen metrics as percentages; ablation and difficulty tables average each metric triplet. Ablations share training and evaluation conditions. Dyn-Bench evaluates the learned scene representation for 4D reasoning. Appendix C.1 details both protocols. 5.1 Comparison with Baselines Entity Continuity. Table 1 shows EvoSee leading all nine continuity metrics. Combined Point-F1 exceeds Qwen3.5-9B by 10.45 points, while Future Point-F1 exceeds SambaMOTR by 10.92 points. Recall and overlap gains indicate more complete recovery of entity locations and more accurate spatial continuation. 6 Table 1. Entity Continuity. P-F1 and P-R denote Point-F1@0.5 and Point-Recall@0.5. Underlined best baselines define red gains; bold marks best results. Tracking adaptations are trained for SceneForesee. ObservedFutureCombined Method P-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU TrackFormer-style15.5913.9417.3814.4613.1117.0815.0513.5317.24 MeMOTR-style15.6213.7116.5515.0013.4416.4315.3513.6016.50 LTrack-style14.5913.1916.1312.7811.6015.5813.7212.4215.87 SambaMOTR17.4614.9818.9415.61 14.1616.6115.2814.1617.84 Qwen3.5-9B20.5519.7221.5815.4914.7817.5518.1617.3519.65 EvoSee (Ours) 30.54 (+9.99) 31.10 (+11.38) 30.13 (+8.55) 26.53 (+10.92) 27.47 (+12.69) 26.77 (+9.22) 28.61 (+10.45) 29.32 (+11.97) 28.51 (+8.86) Scene Evolution Reasoning. Table 2 shows Global F1 gains of 8.62 points over Qwen2.5-VL-7B for Entity, 17.24 over Qwen3.5-9B for Event, and 22.22 over Qwen3-VL-8B for Visual Evidence. Scene-F1 gains also hold when scenes receive equal weight. Figure 1(c) places these improvements alongside entity continuity across all eighteen metrics. Table 2. Scene Evolution Reasoning. Event and Evidence match category occurrences. Underlined best baselines define red gains; bold marks best results. OV1.5 and OV2 are 8B LLaVA-OneVision models. EntityEventVisual Evidence Method GF1SF1GPRGF1SF1GPRGF1SF1GPR Tracking and entity continuity methods TrackFormer-style29.5940.4237.1819.5616.9728.3312.1710.5917.63 MeMOTR-style32.4542.6340.3121.0318.1328.7214.1813.2919.37 LTrack-style30.1841.0637.6022.3020.3229.8715.0314.0420.12 SambaMOTR33.9142.2543.0221.8720.0628.1016.4613.4621.93 Multimodal and video language models Qwen3.5-9B37.4540.7340.1429.8528.2029.8729.8234.2329.84 Qwen3-VL-8B30.0425.0330.3919.9013.7823.5632.2323.9038.17 InternVL3.5-8B32.9230.5333.9215.8612.5617.2030.1327.3232.66 Qwen2.5-VL-7B40.4651.2143.2411.6712.0511.9327.4834.2928.10 LLaVA-Video-7B35.8438.1136.1610.168.4910.6720.4322.1021.47 LLaVA-OV1.533.6934.8134.6312.799.5813.4224.1121.0425.30 LLaVA-OV233.5434.1433.759.217.869.8115.9014.7916.94 Proposed method EvoSee (Ours) 49.08 (+8.62) 57.33 (+6.12) 49.62 (+6.38) 47.09 (+17.24) 48.91 (+20.71) 50.11 (+20.24) 54.45 (+22.22) 57.77 (+23.48) 57.93 (+19.76) Table 3. Description similarity and complete Events. Method BERTScore F1 Micro-F1 Qwen3.5-9B95.9819.94 EvoSee (Ours)96.2640.76 Table 3 evaluates complete Events and scene descrip- tions.EvoSee raises micro-F1 from Qwen3.5-9B’s 19.94 to 40.76 when category and all participants must match together. Although the two methods have similar BERTScore F1, EvoSee is more accurate when Event categories and participants are evaluated jointly. Ap- pendix C.6 gives present, future, and evidence description scores. 5.2 Ablation Study and Discussion Table 4 tests continuation learning and the use of predicted change. Appendix C gives complete metrics and additional variants. Role of joint assignment. Rows 1 and 8 of Table 4 compare geometric and joint assignment. With Event supervision retained, assignment from geometric error alone reduces Future and Event means by 5.03 and 7.43 points. Using both errors in Eq. 3 improves geometric and Event predictions. How does predicted change inform refinement? Observation-only Refinement (row 3) restores the update removed in row 2 and improves future prediction. EvoSee (row 8) uses predicted change, adding 4.32, 4.18, and 4.31 points over 7 Table 4. Ablations on SceneForesee. Columns report dimension means. TrajectoryReasoning IDVariant ObservedFutureCombinedEntityEventEvidence 1Geometry-only Assignment27.8221.8924.0344.8341.2743.15 2w/o Evolution Refinement29.2218.2722.1148.5942.7150.12 3Observation-only Refinement29.1022.6025.9349.3644.5252.41 4w/o Continuation Selection25.3718.8922.0045.6138.7043.65 5Single Hypothesis27.8621.8824.1646.9840.6746.47 6Full Attention28.4924.5225.9749.9044.1350.07 7w/o Visual Evidence27.0219.7322.5746.6839.8936.90 8EvoSee (Ours)30.5926.9228.8152.0148.7056.72 row 3 in Future, Event, and Evidence, and 1.49 in Observed. The Observed gain supports refining entity estimates with predicted change. Choice of continuation. Against EvoSee (row 8), Single Hypothesis (row 5) reduces Future and Event means, while removing Continuation Selection (row 4) gives the lowest Combined mean. Both candidate diversity and selection from observation contribute to entity continuation. Direction of attention. Full Attention (row 6) reduces Future and Event means by 2.41 and 4.58 points relative to EvoSee (row 8). Directed attention in Eq. 1 conditions geometry on observation and Event semantics on observed and geometric states. Contribution of observed evidence. Removing Visual Evidence (row 7 vs. row 8) reduces the Evidence mean by 19.82 points, the largest drop in this dimension, and lowers Future and Event means. Observed motion cues thus support both evidence recognition and entity continuation. 5.3 Further Analysis Table 5. Results by observed entity count. TrajectoryReasoning Method CountObs.Fut.AllEnt.Evt.Evd. Qwen3.5-9B 1–328.35 21.90 25.26 58.06 28.30 50.72 4–622.99 17.30 20.31 53.46 28.28 24.95 ≥ 79.367.498.47 19.68 31.39 14.81 1–344.0638.9641.5771.5052.9665.12 4–629.0826.9028.0369.5948.3054.00 EvoSee (Ours) ≥ 713.6811.1812.4930.9445.5050.70 Effect of entity count. Table 5 shows EvoSee gains in all six dimensions for every entity count group. With at least seven entities, its Entity and Event means ex- ceed Qwen3.5-9B by 11.27 and 14.11 points, alongside improved future continuity. Evidence recognition also improves by 35.88 points. The gains persist as more iden- tities and interactions must be distinguished. Its Event scores vary less than its trajectory scores across groups. Table 6. Results across temporal extents H . TrajectoryReasoning Method H (s)Obs.Fut.AllEnt.Evt.Evd. Qwen3.5-9B 1–321.97 17.29 19.76 41.35 27.20 31.89 4–615.78 11.75 13.82 32.50 37.21 29.41 ≥ 7 13.64 7.95 10.78 31.15 38.76 28.15 1–330.4627.4729.0453.4549.0257.26 4–631.2926.3728.8546.7748.1254.45 EvoSee (Ours) ≥ 730.8121.8226.3246.1746.2554.46 Effect of temporal extent. Table 6 compares equal ob- servation and future durations, varying both the available motion history and the extent of prediction. EvoSee leads all six dimensions in every group. Its Future gain over Qwen3.5-9B is 10.18 points in the shortest group and 13.87 in the longest. Evidence gains exceed 25 points throughout, accompanying improved future localization across temporal scales. Combined scores also decline less than Qwen3.5-9B’s. 8 Table 7. EvoSee dimension means by source. TrajectoryReasoning Source Obs.Fut.AllEnt.Evt.Evd. APIDIS24.405.2314.5350.5165.6381.25 OVT-B32.2729.4530.8850.4150.2657.18 TAO23.3716.2520.0559.9241.4854.36 Variation across video sources. Table 7 shows distinct source profiles. OVT-B leads in trajectory means, TAO in Entity, and APIDIS in Event and Evidence. The source ranking thus depends on whether evaluation concerns spatial continuity, entity references, or semantic change. Appendix C.5 gives scores and sample counts for each source. Qualitative Scene Evolution. Figure 4 shows EvoSee following both seals through changing overlap and capturing the spotted seal’s curved continuation. Its description relates this movement to the other seal and their observed upward motion, connecting the predicted interaction with observed evidence. Appendix D presents six further scenes. Observed TrajectoriesFuture Trajectories GT Qwen3.5-9B EvoSee (Ours) T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 Seal 1Seal 2ObservedFuture T001 T002 T001 T002 T001 T002 T001 T002 0.15s0.95s1.15s1.95s Closer future localization as the animals interact. Combined Point-F1 +35.0 pts Combined MeanIoU +13.6 pts Scene Evolution Understanding ObservedFuture 0.15 s0.95 s1.95 s Qwen3.5-9B During 1 second of observation, a spotted seal (T001) approaches a darker seal (T002). Over the next 1 second, T001 should continue upward to make contact with T002. This forecast is supported by its observed upward trajectory toward the other seal. EvoSee (Ours) During 1 second of observation, a spotted seal (T001) faces upward beneath a darker seal (T002) in shallow water, with their bodies overlapping in the view. Over the next 1 second, both rise as the spotted seal curves slightly right and the darker seal remains above it. This forecast is supported by their upward movement, the spotted seal’s changing body angle and the darker seal’s position near the surface. Figure 4. Seal interaction at matched timestamps. Identity colors distinguish the seals; solid and dashed paths show observed and future centers. EvoSee retains both through changing overlap. 5.4 Scene Evolution for 4D Reasoning EvoSee’s scene representation augments Qwen3.5-9B for VQA and Sa2VA-InternVL2.5-8B for grounding on Dyn- Bench (Huang et al., 2026; Yuan et al., 2026). Observation-only Refinement retains each base system and substitutes observed summaries, testing the value of predicted change for these tasks. Table 8. Dyn-Bench grounding J&F. Red gains use the Sa2VA- InternVL2.5-8B base. MethodOverallInter.ObjectCam. UniPixel-7B65.2366.0071.1058.60 Sa2VA-Qwen2.5-VL-7B72.8073.4075.9069.10 Sa2VA-InternVL2.5-8B75.7076.8080.2070.10 Observation-only Refinement 77.0277.1280.2173.73 EvoSee (Ours) 79.28 (+3.58) 78.03 (+1.23) 80.28 (+0.08) 79.54 (+9.44) Table 9.Dyn-Bench VQA accuracy. Red gains use the Qwen3.5-9B base. Overall aggregates all questions. MethodOverallInter.ObjectCam. Qwen3-VL-8B61.459.0076.2056.00 DynTrace65.866.3380.5358.63 Qwen3.5-9B63.559.2079.3558.87 Observation-only Refinement 66.767.4080.6260.83 EvoSee (Ours) 68.3 (+4.8) 69.80 (+10.60) 80.85 (+1.50) 61.50 (+2.63) Dynamic grounding. Table 8 shows overall J&F gains of 3.58 points over Sa2VA-InternVL2.5-8B and 2.26 over Observation-only Refinement. Camera motion shows the largest gain over this control, 5.81 points, supporting predicted change as context for grounding under changing viewpoints. Dynamic question answering. Table 9 shows overall gains of 4.8 points over Qwen3.5-9B and 1.6 over Observation- only Refinement, with a 2.40 point interaction gain over this control. Across VQA and grounding, predicted change most benefits interactions and changing viewpoints, while object motion scores remain close. 9 6 Conclusion SceneForesee evaluates how persistent entities move and how their relations develop through continuity and change in the same scenes. EvoSee learns entity continuations through shared future states and joint assignment, then uses predicted change to update entity and scene representations. Results on SceneForesee and Dyn-Bench support scene evolution as both a prediction target and a dynamic representation for spatiotemporal reasoning. SceneForesee currently represents motion in the image plane and evaluates four Event categories. Extending this formulation to 3D trajectories and a broader Event vocabulary is a direction for future work. AI Use Statement Generative AI tools assisted with synthetic data generation and language polishing. The authors take responsibility for the final data and manuscript. References Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. LLaVA-OneVision-1.5: Fully open framework for democratized multimodal training, 2025. URL https://arxiv.org/abs/2509.23661. Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Changrui Chen, Xiuwei Zhao, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Kaichen Zhang, Wenkang Zhang, Zheng Cheng, Nansen Zhang, Chunsheng Wu, Chunjiang Ge, Zimin Ran, Dehua Song, Chunyuan Li, Shikun Feng, Ming Hu, Zhangquan Chen, Junbo Niu, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, and Jiankang Deng. LLaVA-OneVision-2: Towards next-generation perceptual intelligence, 2026. URL https://arxiv.org/abs/2605.25979. Kumar Ashutosh, Georgios Pavlakos, and Kristen Grauman.FIction: 4D future interaction prediction from video.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17613–17625. IEEE, 2025. URLhttps://openaccess.thecvf.com/content/CVPR2025/html/Ashutosh_FIction_4D_ Future_Interaction_Prediction_from_Video_CVPR_2025_paper.html. Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning, 2025. URL https://arxiv.org/abs/2506.09985. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025a. URLhttps://arxiv.org/abs/2511. 21631. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report, 2025b. URL https://arxiv.org/abs/2502.13923. Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. TAO: A large-scale benchmark for tracking any object. In European Conference on Computer Vision, p. 436–454. Springer, 2020. Christophe De Vleeschouwer, Fan Chen, Damien Delannay, Christophe Parisot, Christophe Chaudy, Eric Martrou, and Andrea Cavallaro. Distributed video acquisition and annotation for sport-event summarization. In NEM Summit, 2008. URLhttps: //ispgroup.gitlab.io/code/apidis/. Patrick Dendorfer, Vladimir Yugay, Aljosa Osep, and Laura Leal-Taixé.Quo Vadis:Is trajectory forecasting the key towards long-term multi-object tracking?In Advances in Neural Information Processing Systems, vol- ume 35, p. 15657–15671, 2022. URLhttps://proceedings.neurips.c/paper_files/paper/2022/hash/ 647dc4a76b3efdd676f50f32949299a8-Abstract-Conference.html. Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, and Chen Chen.Motion-grounded video reasoning: Understanding and perceiving motion at pixel level.In 10 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8625–8636. IEEE, 2025. URLhttps://openaccess.thecvf.com/content/CVPR2025/html/Deng_Motion-Grounded_Video_ Reasoning_Understanding_and_Perceiving_Motion_at_Pixel_Level_CVPR_2025_paper.html. Bishoy Galoaa, Shayda Moezzi, Xiangyu Bai, and Sarah Ostadabbas. Motion-o: Trajectory-grounded video reasoning. arXiv preprint arXiv:2603.18856, 2026. URL https://arxiv.org/abs/2603.18856. Rongxin Gao, Yuzhi Huang, Dongxuan Liu, Chu Li, Zhenye Wang, Jie Wu, Shuzhao Xie, Jingyan Jiang, Xinghao Ding, Xiaotong Tu, et al. DynTrace: Tracking dynamic object evidence for 4D spatio-temporal reasoning in MLLMs. arXiv preprint arXiv:2607.12503, 2026. URL https://arxiv.org/abs/2607.12503. Ruopeng Gao and Limin Wang.MeMOTR: Long-term memory-augmented transformer for multi-object track- ing.In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9901– 9910, 2023. URLhttps://openaccess.thecvf.com/content/ICCV2023/html/Gao_MeMOTR_Long-Term_ Memory-Augmented_Transformer_for_Multi-Object_Tracking_ICCV_2023_paper.html. Ruopeng Gao, Ji Qi, and Limin Wang. Multiple object tracking as ID prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 27883–27893, 2025. Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jie Wu, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, et al. Thinking in dynamics: How multimodal large language models perceive, track, and reason dynamics in phys- ical 4D world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 33446–33456, 2026. URLhttps://openaccess.thecvf.com/content/CVPR2026/html/Huang_Thinking_ in_Dynamics_How_Multimodal_Large_Language_Models_Perceive_Track_CVPR_2026_paper.html. Pere-Lluís Huguet Cabot and Roberto Navigli. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, p. 2370–2381, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.204. URLhttps://aclanthology. org/2021.findings-emnlp.204/. Kehan Lan, Kaining Ying, and Henghui Ding. FeVOS: Foresight expression video object segmentation, 2026. URLhttps: //arxiv.org/abs/2606.25585. Yunhao Li, Qin Li, Hao Wang, Xue Ma, Jiali Yao, Shaohua Dong, Heng Fan, and Libo Zhang. Beyond MOT: Semantic multi-object tracking. In European Conference on Computer Vision, p. 276–293. Springer, 2024. Haiji Liang and Ruize Han. OVT-B: A new large-scale benchmark for open-vocabulary multi-object tracking. Advances in Neural Information Processing Systems, 37:14849–14863, 2024. URLhttps://proceedings.neurips.c/paper_files/ paper/2024/hash/1adeeac24ce6168e20bcee85645720e9-Abstract-Datasets_and_Benchmarks_ Track.html. Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G Hauptmann, and Li Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5725–5734, 2019. Ye Liu, Zongyang Ma, Junfu Pu, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. UniPixel: Unified object referring and segmentation for pixel-level visual reasoning. In Advances in Neural Information Processing Systems, 2025. URLhttps: //arxiv.org/abs/2509.18094. Xuanchen Lu, Ang Cao, Chao Feng, and Andrew Owens.Generative point tracking and forecasting.In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 28167–28178, 2026. URLhttps://openaccess.thecvf.com/content/CVPR2026/html/Lu_Generative_Point_Tracking_ and_Forecasting_CVPR_2026_paper.html. Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. TrackFormer: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8844– 8854, 2022. URLhttps://openaccess.thecvf.com/content/CVPR2022/html/Meinhardt_TrackFormer_ Multi-Object_Tracking_With_Transformers_CVPR_2022_paper.html. Trong-Thuan Nguyen, Pha Nguyen, Jackson Cothren, Alper Yilmaz, and Khoa Luu. HyperGLM: HyperGraph for video scene graph generation and anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 29150–29160, 2025. URLhttps://openaccess.thecvf.com/content/CVPR2025/html/Nguyen_HyperGLM_ HyperGraph_for_Video_Scene_Graph_Generation_and_Anticipation_CVPR_2025_paper.html. Sandro Papais, Letian Wang, Brian Cheong, and Steven L Waslander. ForeSight: Multi-view streaming joint object detection and tra- jectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 25474–25484, 2025. 11 URLhttps://openaccess.thecvf.com/content/ICCV2025/html/Papais_ForeSight_Multi-View_ Streaming_Joint_Object_Detection_and_Trajectory_Forecasting_ICCV_2025_paper.html. Francesca Pistilli, Simone Alberto Peirone, and Giuseppe Averta. Learning to evolve scenes: Reasoning about human activities with scene graphs, 2026. URL https://arxiv.org/abs/2607.02425. Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5, 2026. Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D. Hager. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 3591–3600, 2017. URLhttps://openaccess.thecvf.com/content_iccv_2017/ html/Rupprecht_Learning_in_an_ICCV_2017_paper.html. Mattia Segu, Luigi Piccinelli, Siyuan Li, Yung-Hsu Yang, Luc Van Gool, and Bernt Schiele.Samba:Synchro- nized set-of-sequences modeling for multiple object tracking.In International Conference on Learning Represen- tations, p. 30057–30070, 2025.URLhttps://proceedings.iclr.c/paper_files/paper/2025/hash/ 4a32a646254d2e37fc74a38d65796552-Abstract-Conference.html. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URLhttps:// proceedings.neurips.c/paper/2017/hash/3f5e243547dee91fbd053c1c4a845a-Abstract.html. Haonan Wang, Hongfu Liu, Xiangyan Liu, Chao Du, Kenji Kawaguchi, Ye Wang, and Tianyu Pang. Fostering video reasoning via next-event prediction. In International Conference on Learning Representations, 2026. URLhttps://openreview.net/ forum?id=8nUgzuvskm. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. URL https://arxiv.org/abs/2508.18265. Zhaoyang Wei, Guoliang Wang, Guohua Gao, Yanchao Hao, Mingda Li, Wenchao Ding, Xi Chen, Shizhu He, and Xuehui Yu. Seeing is believing: Grounding long-video understanding in spatio-temporal visual evidence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p. 10584–10592, 2026. doi: 10.1609/aaai.v40i13.38031. URLhttps: //doi.org/10.1609/aaai.v40i13.38031. Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14633–14642, 2023. En Yu, Songtao Liu, Zhuoling Li, Jinrong Yang, Zeming Li, Shoudong Han, and Wenbing Tao. Generalizing multiple object tracking to unseen domains by introducing natural language representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, p. 3304–3312, 2023. doi: 10.1609/aaai.v37i3.25437. URLhttps://ojs.aaai.org/index.php/AAAI/ article/view/25437. Haobo Yuan, Xiangtai Li, Tao Zhang, Yueyi Sun, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2VA: Marrying SAM2 with MLLM for dense grounded understanding of images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. doi: 10.1109/TPAMI.2026.3720379. URLhttps://arxiv. org/abs/2501.04001v4. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations, 2020. URL https://arxiv.org/abs/1904.09675. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. LLaVA-Video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URLhttps://openreview.net/ forum?id=EElFGvt39K. 12 Appendix A Benchmark Construction and Evaluation A.1 Temporal Sampling and Annotations The observation boundary separates the video available for inference from the future continuation used for training and evaluation. For videoVand a window beginning ats, durationHdefines the boundaryt b and the paired observed and future intervals. t b = s + H, V obs = V [s,s + H], V fut = V (s + H,s + 2H], H ∈1,..., 10 s.(A.1) Videos are normalized to 5 frames per second, and candidate windows advance byH/4. Each observed interval supplies four uniformly sampled keyframes. The model receives the observed video and keyframes, withHspecified by the task context. Annotations from this interval supervise entity states and Visual Evidence, while future annotations supervise trajectories and Events. Each sample calls for one scene evolution prediction comprising observed and future trajectories, Events, Visual Evidence, and a scene description. Entity states. Each entity annotation includes an identity within the sample, timestamped boxes in the image plane, visibility, and view. Entities that continue across the observation boundary retain this identity in their future persistence and state annotations. Every sample contains at least one continuing entity. Counting shared identities over all temporal pairs gives 1.05M entity correspondences. Counting observed and future trajectory records separately gives 2.14M records. Both totals accumulate over samples. Events and Visual Evidence.The four Event categories, relation change, approach, occlusion, and leave, summarize entity configurations using observed motion and states at the transition to prediction or the future endpoint. Each Event records its participating identities and the requested future interval. Its paired Visual Evidence records an individual trajectory trend or pairwise relation change, together with the participating identities, observed keyframes, and an observed interval. The paired evidence annotations record associated motion in observation and supervise evidence localization and category prediction from observed frames. All 272.49K Events have paired evidence, and each of the 144.57K samples has a scene description. A.2 Annotation Construction and Data Splits SceneForesee constructs annotations around identities shared across each temporal pair. These identities associate timestamped boxes and visibility with continuing entities, whose trajectories determine the Event annotations. The same construction criteria apply to APIDIS, OVT-B, and TAO. Together, these sources cover sports interactions, open category entities, and varied natural motion (De Vleeschouwer et al., 2008; Liang & Han, 2024; Dave et al., 2020). Event assignment describes relative motion, spatial overlap, and position in the field of view under these shared identities. The categories provide discrete targets for predicting the configurations represented by continuous trajectories. Approach requires the distance between entity centers to decrease by more than 10% during observation and remain at most 1.10 times the observation endpoint distance at the future endpoint. Occlusion uses box overlap at the temporal transition from observation to prediction, with pairwise IoU above 0.05 at the last observed or first future state. This criterion applies throughout the image, including its interior. Leave uses position relative to the image edge at the future endpoint, with the box center within the outer 10% of either image axis. Relation change records the remaining configurations when none of these rules applies and is assigned to the first continuing identity in source order. The geometric criteria define Event supervision for the same entities and temporal pairs used in trajectory prediction. Pairwise Events are enumerated in source identity order, with approach preceding occlusion for each pair, followed by individual leave Events. Retaining the first three records and their paired evidence defines the subset of configurations covered by Event evaluation. A sample is retained when it contains an entity annotated across the observation boundary, an Event assigned to the requested continuation, and its evidence in observation. Visual Evidence associates each Event with observed motion. Its category follows the Event’s participant structure, assigning relation change to pairwise Events and trajectory trend to individual Events. The evidence description uses 13 Table A1. Scale and Event distribution of SceneForesee. Entity correspondences count identities shared across paired windows within each sample. Trajectory records count observed and future windows separately. Each Event has one associated Visual Evidence annotation. Val denotes validation. StatisticCount Train / Val / Test116,681 / 14,259 / 13,626 Total samples144,566 Source videos2,876 Entity correspondences1,046,165 Observed trajectory records1,098,141 Future trajectory records1,046,165 Entity trajectory records2,144,306 Event and evidence pairs272,487 Scene descriptions144,566 Event categories4 Visual Evidence categories2 SourceSamples Relation change ApproachOcclusionLeaveEvents APIDIS3174144131135828 TAO26,6447,91012,87919,9836,18846,960 OVT-B117,60543,10058,499110,03913,061224,699 All sources144,56651,05171,819130,33319,284272,487 the first observed motion segment containing a participating identity. If no segment shares an Event participant, it uses the first available observed segment as scene context. If no segment is available, the observed interval supplies the description and time span. The resulting evidence record carries the Event participants and observed keyframes, with its temporal extent confined to observation. Construction included LLM assessment and manual inspection of the generated annotations. The train, validation, and test sets contain 116,681, 14,259, and 13,626 samples, respectively. Partitions use source videos, with disjoint chronological episodes defining the APIDIS partitions, so overlapping windows remain together. Each source contributes the windows whose continuing entities and Events meet the common construction criteria. Table A1 reports the resulting scale, counting units, and Event distribution. Table A1 shows the distribution of scene evolution in the retained annotations. Occlusion and approach are the most frequent Events, capturing spatial overlap and relative motion, while relation change and leave cover residual configurations and proximity to the image edge. Separate counts for observed trajectories and continuing identities distinguish entities present in observation from those followed into the future. The retained scenes also vary in entity count and Event composition. Observed windows contain a mean of 7.60 entities and a median of 3, ranging from 1 to 89. Multiple Events occur in 47.72% of samples and multiple Event categories in 27.34%. Two entities participate in 74.19% of Events. These distributions support analysis of continuity and change across scene densities. A.3 Entity Continuity LetPandGdenote all predicted and annotated identities in one sample. Trajectory overlap in the observed interval determines matched subsetsP m ⊆ PandG m ⊆ G, with a bijectionπ : P m → G m . This partial matching remains fixed for future evaluation. Unmatched predictions remain in the prediction counts, and unmatched annotations remain in the reference counts. For interval W , letA W contain point pairs aligned by π, frame, and view, and define m W = X (g,p)∈A W 1[IoU(g,p)≥ 0.5].(A.2) 14 LetN gt W andN pred W count annotated and predicted points in intervalW. Point-Recall and Point-F1 normalize the matched count by these totals. PointRecall W = m W N gt W ,PointF1 W = 2m W N gt W + N pred W .(A.3) MeanIoU averages spatial overlap over the same annotated points. MeanIoU W = 1 N gt W X (g,p)∈A W IoU(g,p).(A.4) Unmatched annotated points contribute zero to MeanIoU. The observed interval measures entity state estimation, and the future interval measures trajectory prediction under the sameπ. Combined scores are formed within each sample. For any of the three interval scores M , the paired result is M comb = N gt obs M obs + N gt fut M fut N gt obs + N gt fut .(A.5) The weights are the annotated point counts of that sample’s observed and future intervals. The reported score is the arithmetic mean over test samplesS . M W = 1 |S| X s∈S M s,W , W ∈obs, fut, comb.(A.6) Thus annotated point counts weight intervals within a sample, while samples receive equal weight in the final result. A score with a zero denominator is assigned zero. A.4 Scene Evolution Reasoning Scene Evolution Reasoning evaluates entity references and the categories of Events and Visual Evidence. Entity scoring extracts the identities referenced in predicted and reference descriptions, maps predictions throughπ, and compares the resulting sets. Event and Visual Evidence scoring compares category multisets within each scene, retaining repeated occurrences. For dimensiondin scenes, letP d s andG d s denote predicted and annotated collections, andt d s their intersection count, with multiplicity for Event and Visual Evidence. Global precision and recall pool these counts across scenes. P d g = P s t d s P s |P d s | , R d g = P s t d s P s |G d s | .(A.7) Global F1 summarizes matching counts across the test set. GlobalF1 d = 2 P s t d s P s |P d s | + P s |G d s | .(A.8) Scene-F1 assigns equal weight to each scene. SceneF1 d = 1 |S| X s∈S 2t d s |P d s | +|G d s | .(A.9) GPR is the geometric mean of global precision and recall. GPR d = q P d g R d g .(A.10) A zero denominator gives zero. Entity, Event, and Visual Evidence collections are matched independently, measuring identity references and category multiplicities. Reported scores multiply the ratios above by 100. Visual Evidence scores compare individual trajectory trend and pairwise relation change categories from the observed context defined in Appendix A.2. They measure category recognition, without testing temporal localization or the support a segment provides for a particular Event. Complete Event matching separately tests the predicted category and its participants. 15 Exact matching of complete Events. Complete Event micro-F1 follows the structured matching principle used in relation extraction (Huguet Cabot & Navigli, 2021). A correct Event must match its category and complete participant record. The score is F 1,micro = 100× 2TP 2TP + FP + FN .(A.11) TP counts matched Events, FP unmatched predictions, and FN unmatched references, pooled across test scenes. Complete Event records contain a category and all participating identities. These identities are part of the model output before evaluation and refer to its predicted entities. Participant roles or ordering follow the annotation schema. Evaluation maps the predicted identities through the observation correspondenceπused for trajectory evaluation. An Event with any unmapped or incorrect participant cannot match a reference and remains in the prediction count. Within each scene, maximum one-to-one exact matching retains multiplicities. Each predicted and annotated record can be used at most once. Unmatched predictions and references contribute FP and FN, respectively. Counts are pooled over all test scenes before computing micro-F1; a zero denominator gives zero. The matching key comprises the Event category and all participant identities, with no additional temporal localization condition. Thus the score tests a category and its participants jointly, while Event category scores compare category occurrences alone. A.5 Dimension Means and Group Aggregation The main ablation and difficulty tables summarize six dimensions. Observed, Future, and Combined each average Point-F1, Point-Recall, and MeanIoU for the corresponding interval. Entity, Event, and Visual Evidence each average Global F1, Scene-F1, and GPR. These arithmetic means give equal weight to the three metrics within a dimension. Table 4 averages the corresponding scores reported in Section C and rounds the resulting means to two decimal places. Differences between dimension means are calculated before this final rounding. Event micro-F1 with exact matching is reported separately and does not enter these means. For a sample group, trajectory scores and Scene-F1 are averaged across its samples. Global F1 and GPR are calculated from the group’s pooled true positive, prediction, and reference counts. Combining groups therefore requires pooling these counts before computing the global semantic metrics. Overall results use the complete test population, and source results apply the same definitions to each source’s retained samples. The dimension mean is taken after the three constituent scores have been computed for the group. A.6 Scene Description Similarity BERTScore F1 with RoBERTa-large evaluates similarity between the predicted and reference scene descriptions (Zhang et al., 2020). The complete description covers observed motion, predicted development, and evidence. We also score the present, future, and evidence parts separately to examine which content contributes to the overall result. Appendix C.6 reports these component scores. B Method Details EvoSee encodes observed entity states and Visual Evidence, then uses these representations to learn candidate continu- ations. Each candidate shares future states between geometry and Event semantics, and joint assignment determines which candidate receives future supervision. A selector learns this assignment from observation. The predicted change then enters entity refinement, consistency ranking, and scene representation. B.1 Entity Visual Evidence Encoding Visual features from the observed video and its keyframes form memoryX obs . A learned queryq i attends to this memory to obtain entity statea i at the observation boundary. The same query attends to frame featuresX t at each valid observed time t∈O i , producing temporal entity state h t i . These two uses of cross attention are a i = CrossAttn(q i ,X obs ,X obs ), h t i = CrossAttn(q i ,X t ,X t ).(B.1) 16 A learned mapψ e assigns each temporal state a relevance logitr t i . Normalizing its sigmoid response over observed times gives weights α t i for aggregating the states into evidence e i . α t i = σ(r t i ) P ℓ∈O i σ(r ℓ i ) , e i = X t∈O i α t i h t i .(B.2) Herer t i = ψ e (h t i ), andσis the sigmoid function. Evidence intervals supervise temporal relevance, while evidence categories supervise the aggregated features. These two objectives train the representation to identify the observed motion associated with each entity. B.2 Geometry Event Alignment Reasoning GEAR uses the entity and evidence representations to initialize its geometric and semantic states. Their sumu a i = a i +e i is projected intou g i ,u s i ∈R d . Stacking these states as observation, geometry, and semantics givesU i ∈R 3×d . For attention head h of width d h , layer normalization followed by learned projections gives queries, keys, and values. Q h i = LN(U i )W h Q , K h i = LN(U i )W h K , V h i = LN(U i )W h V .(B.3) The mask in Eq. 1 permits a receiving state to attend to itself and the states preceding it in this order. ThusM rs = 0 fors≤ randM rs =−∞fors > r. The resulting head outputsA h i return to the state space through concatenation, learned projection W O , and a residual update. e U i = U i + Concat h softmax Q h i (K h i ) ⊤ √ d h +M V h i W O .(B.4) The observed state receives information only from observation. The geometric state additionally uses this observed context, and the semantic state uses both observation and geometry. The updated states use separate transformation weights to preserve these dependencies. With normalized states Z i = LN( e U i ), learned matrixW R and biasb R assign weights overEfeedforward transformations;1 3 broadcasts the bias across the three states. Π i = softmax Z i W R + 1 3 b ⊤ R .(B.5) Softmax acts over theEtransformations independently for each state. WritingΠ i,:,m for the weights of transformation FFN m , the update is U ′ i = e U i + E X m=1 diag(Π i,:,m ) FFN m (Z i ).(B.6) Each transformation acts separately on the three states, preserving the dependencies defined by the attention mask. The refined geometric and semantic states jointly condition the future sequence from which candidate continuations are learned. ForLfuture steps, this sequence isf i ∈R L×d . Each ofKcandidate continuations transforms it with embedding d k ∈R d and transitionφ k , as in Eq. 2. The vector1 L broadcasts the embedding across future time steps. Both decoders use the resulting statef k i . The Event decoder also receives embeddingc, which represents task context in the entity feature space. One candidate index therefore determines future boxes, visibility, persistence, view, and Event semantics. Joint assignment compares each candidate with future annotations using box error and Event classification error. LetF i contain valid annotated future steps, b k i,t ∈R 4 be a predicted box, and b ⋆ i,t its reference. The geometric error is ℓ i,k geo = 1 4|F i | X t∈F i b k i,t − b ⋆ i,t 1 .(B.7) The Event classification error uses logits averaged over theLfuture steps, ̄s k i,j = L −1 P L t=1 s k i,t,j . For each reference Event of categoryj, every recorded participantireceivesy i,j = 1, so both identities in a pairwise Event carry its category target. These labels over J output categories define the semantic error. ℓ i,k sem =− 1 J J X j=1 y i,j logσ( ̄s k i,j ) + (1− y i,j ) log 1− σ( ̄s k i,j ) .(B.8) 17 The assigned candidate minimizesℓ i,k geo + 0.20ℓ i,k sem , and its index b k i is shared across all future supervision terms. This index also supplies the target for selecting a continuation from observation. The selectorRcombines entity statea i , evidencee i , and embeddingo i of the entity’s observed trajectory history. ItsK output logits define candidate probabilitiesρ i and the most probable candidate k ∗ i . ρ i = softmax(R(a i + e i + o i )), k ∗ i ∈ arg max k ρ k i .(B.9) The selector learns the assigned candidate by minimizing L prior =− 1 |I| X i∈I logρ b k i i ,(B.10) whereIcontains matched entities with valid future annotations. Future annotations determine the training target, while the selection distribution is inferred from observation. Trajectory diversity is encouraged through the separation of candidate endpoints. For final predicted centerp k i , the regularizer is L diversity = 1 |I|K(K− 1) X i∈I X k̸=ℓ m−∥p k i − p ℓ i ∥ 1 + , m = 0.05.(B.11) Here[x] + = max(0,x), and coordinates are normalized to the image extent. This expression applies forK > 1and nonemptyI. WithK = 1, the diversity term is zero. The regularizer acts on endpoint separation while candidate assignment uses the complete valid trajectory and its Event prediction. The candidate probabilities also define a weighted future state. e f i = K X k=1 ρ k i f k i .(B.12) During training, ECD uses weighted predictions formed from the boxes and Event logits decoded for each candidate, as defined below. Inference usesk ∗ i ∈ arg max k ρ k i and e f i = f k ∗ i i . The same index determines future boxes, visibility, persistence, and Event predictions, preserving one geometric and semantic continuation for each entity. The selected Event scores express the categories associated with that entity. Complete Event records additionally specify all participants and are evaluated by joint category and identity matching in Appendix A. WithK = 1, its sole future state directly defines the continuation. B.3 Evolution Conditioned Decoding ECD summarizes geometric and Event predictions as features for entity refinement. At inference, it uses the selected continuation, whose box at future timetisb k ∗ i i,t ∈R 4 and whose Event logits ares k ∗ i i,t . Temporal averaging and projection give the geometric and semantic summaries. g i = P g 1 L L X t=1 b k ∗ i i,t ! , s i = P s σ 1 L L X t=1 s k ∗ i i,t !! .(B.13) HereP g andP s project intoR d . During training, the corresponding boxes and logits are weighted by selection probabilities. b i,t = X k ρ k i b k i,t , s i,t = X k ρ k i s k i,t .(B.14) These weighted predictions undergo the same temporal averaging and projection as the selected predictions at inference. Equation 4 combines the resulting summaries with entity state and observed evidence into evolution representationm i . The map Γ in Eq. 4 uses entity state and evolution representation to compute a vector gate. γ i = σ(W γ [a i ;m i ] + b γ ),γ i ∈ (0, 1) d .(B.15) 18 Normalization and projection give P f (m i ) = W f LN(m i ) + b f . The update is a + i = a i +γ i ⊙ P f (m i ).(B.16) The gate weights the contribution of predicted change to each feature dimension of the entity state. The updated state supports observed trajectory estimation and produces entity confidence ℓ obj i . Entity retention uses feature agreement alongside this confidence. Six ordered relation pairs specify the features compared by Consistency. R i = (e i ,a i ), (a i ,g i ), (g i ,s i ), (s i ,e i ), (g i ,m i ), (s i ,c) .(B.17) The pairs connect evidence to entity state, entity state to geometry, geometry to semantics, and semantics to evidence. They also compare geometry with the evolution representation and semantics with task context. Prediction losses supervise the predicted content, while relation supervision distinguishes queries matched to annotated entities from unmatched queries. For relation r, let (u i,r ,v i,r ) denote the corresponding pair inR i . Its cosine distance is b i,r = 1− u ⊤ i,r v i,r max(∥u i,r ∥ 2 ,ε) max(∥v i,r ∥ 2 ,ε) , ε = 10 −6 .(B.18) The stabilizer ε keeps the distance defined for zero norm features. Averaging the six distances gives B i . B i = 1 6 6 X r=1 b i,r .(B.19) A smallerB i indicates closer feature agreement. Its geometric and semantic inputs come from weighted predictions during training and the selected candidate at inference. The same mean distance contributes to relation supervision and entity retention. Relation supervision. Annotation matching divides entity queries into matched setI m and unmatched setI u . Let ̄ B m and ̄ B u be their respective mean distances. The ranking loss is L rank = m + ̄ B m − ̄ B u + , m = 0.25.(B.20) Training applies this loss to the aggregate distance and to each individual relation. It favors lower distances for entities associated with annotated trajectories. The term is evaluated when both sets are nonempty and is zero otherwise. Geometric, semantic, and evidence supervision jointly train the representations on which the relation distances depend. Supervision and continuation learning. Observed annotations supervise entity estimation, while the evidence intervals and categories supervise temporal relevance and the evidence representation. Future box and Event losses determine the shared candidate assignment and supervise its geometric and semantic outputs. The assigned index trains the selector throughL prior , andL diversity encourages separation among candidate endpoints. Relation ranking encourages lower feature distances for matched queries than for unmatched queries. At inference, observed video and task context determine candidate selection, entity retention, and scene representation. Entity retention combines entity confidence with the mean relation distance. κ i = ℓ obj i − λ cons B i .(B.21) The entities with the highest scores form the retained setI ∗ . Smaller relation distances raise an entity’s rank at fixed confidence, linking the final entity set to both observed support and predicted continuation. The evolution representations also contribute to scene reasoning. Together with entity states and evidence, they are averaged over all N q queries. ̄a = 1 N q X i a i , ̄e = 1 N q X i e i , ̄m = 1 N q X i m i .(B.22) 19 Concatenating these averages with task context c and projecting them gives the scene representation. h scene = P scene ([ ̄a; ̄e; ̄m;c]).(B.23) The scene representation combines observed states with predicted geometry and Event semantics across all queries. Alongside this representation, the retained setI ∗ supplies structured trajectories and Events with one continuation per entity, whilea + i supplies observed trajectory estimates and entity confidence. SceneForesee evaluates these spatial and semantic predictions directly. Dyn-Bench evaluates the scene representation in question answering and grounding, testing the value of learned scene evolution for 4D reasoning. C Experimental Protocols and Additional Results The additional results examine entity continuity and recognition of change across model variants, entity counts, temporal extents, and video sources. Each of the six evaluation dimensions contains three metrics. P-F1 and P-R denote Point- F1@0.5 and Point-Recall@0.5. GF1, SF1, and GPR denote Global F1, Scene-F1, and the geometric mean of global precision and recall. Tables report scores as percentages and place group sizeNbefore the metrics. Bold indicates the leading value within each comparison. The main ablation and difficulty tables summarize these dimensions using Obs., Fut., All, Ent., Evt., and Evd. for Observed, Future, Combined, Entity, Event, and Visual Evidence. Each dimension mean averages its three constituent metrics. Complete Event micro-F1 and description BERTScore F1 are reported separately. C.1 Comparison Protocols SceneForesee comparisons.SceneForesee comparisons use the same test samples, observed video intervals, temporal extents, and metrics, with one scene prediction per method and sample. The TrackFormer-style, MeMOTR-style, LTrack-style, and SambaMOTR adaptations (Meinhardt et al., 2022; Gao & Wang, 2023; Yu et al., 2023; Segu et al., 2025) implement the cited tracking principles and are trained for this task rather than using the released checkpoints. Their outputs include trajectories and categories for both evaluations. Qwen3.5-9B (Qwen Team, 2026) and EvoSee predict trajectories and scene descriptions from observation. The remaining multimodal and video language models use shared trajectories from the TrackFormer-style adaptation as geometric context, including LLaVA-Video (Zhang et al., 2025), LLaVA-OneVision-1.5 (An et al., 2025), and LLaVA-OneVision-2 (An et al., 2026). For each method, matching its observed trajectories to annotations establishes the correspondence used for future continuity and Entity scoring. Ablation variants. Nine ablations use the full model’s training and evaluation conditions. Single Hypothesis sets K = 1to test the role of candidate continuations. Removing Consistency, Continuation Selection, Evolution Refinement, Entity Retention, or Visual Evidence tests feature agreement, selection from observation, refinement using predicted change, retention of continuing entities, or observed evidence, respectively. Every variant produces one scene prediction. Observation-only Refinement retains the projection and gated residual update in ECD. It replaces the predicted geometric and Event inputs to the evolution representation with observed entity, motion, and evidence summaries of the same input dimensions. The resulting representation enters both the entity update andh scene , replacing predicted change in both paths shown in Figure 3. The candidate continuations and their prediction supervision remain in the model. This comparison tests the information used for refinement while retaining the update structure; w/o Evolution Refinement removes the update itself. Geometry-only Assignment chooses b k i using geometric error while retaining Event classification supervision with its original loss weight. Both geometric and Event predictions use this assigned index for supervision, and the same index trains the selector. Full Attention instead removes the directional constraint by setting every entry ofMto zero, allowing observation, geometry, and semantics to attend to one another. Both variants retain EvoSee’s candidate continuations and learning conditions. Dyn-Bench transfer.On Dyn-Bench, scene representationh scene carries predicted scene evolution into 4D reasoning. It augments Qwen3.5-9B for question answering and the Sa2VA-InternVL2.5-8B mask decoder for grounding. Each gain is measured against the corresponding base system. Additional comparisons include Qwen3-VL-8B and DynTrace for question answering, and UniPixel and Sa2VA for grounding (Huang et al., 2026; Gao et al., 2026; Liu et al., 2025; 20 Yuan et al., 2026). DynTrace uses Qwen3-VL-8B, and the published Qwen3-VL and grounding reference results follow Dyn-Bench. Inter., Object, and Cam. denote interaction, object motion, and camera motion. VQA Overall is accuracy across all questions, while each dimension averages the accuracies of its three constituent tasks. Grounding Overall averages the three dimension J&F scores. Gains use the corresponding overall scores of the matched base systems. Observation-only Refinement retains these respective base systems and uses the observed summaries defined above. Comparing this control with EvoSee tests the value of predicted change in the scene representation for VQA and grounding. These tasks evaluate 4D reasoning downstream of SceneForesee, whose trajectories describe the image plane over time. C.2 Complete Ablation Results Table A2 examines how continuation learning and refinement affect entity continuity. EvoSee leads every trajectory metric. Geometry-only Assignment retains Event supervision but reduces all future metrics, supporting the contribution of semantic error to joint assignment. Full Attention also lowers every future metric relative to directed attention. Removing Evolution Refinement produces the largest loss in each future metric and in Combined Point-F1; Future Point-F1 falls by 8.44 points. Restoring the update with observed information recovers 4.62 points, and using predicted change adds a further 3.82 points. This additional gain extends to all three future metrics. EvoSee also improves the Observed mean to 30.59, exceeding Observation-only Refinement at 29.10 and the variant without refinement at 29.22 by 1.49 and 1.37 points. These gains support using the predicted continuation to refine entity estimates within the observed interval. Continuation Selection affects a different aspect of the prediction, with its removal producing the largest declines in Combined Point-Recall and MeanIoU. Single Hypothesis also lowers future accuracy. These comparisons distinguish the contributions of predicted change, selection from observation, and candidate continuations. Table A2. Complete ablation results for Entity Continuity. ObservedFutureCombined VariantP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU Qwen3.5-9B20.5519.7221.5815.4914.7817.5518.1617.3519.65 Geometry-only Assignment26.9328.4828.0522.3921.2422.0524.0423.0924.96 w/o Evolution Refinement29.7728.6729.2318.0917.0319.6821.4421.6323.25 Observation-only Refinement29.3128.9629.0222.7122.3622.7425.7425.8826.17 w/o Continuation Selection25.4925.4625.1718.7417.2420.6922.6321.2222.16 Single Hypothesis28.6526.5628.3821.3921.6822.5824.5522.8525.08 Full Attention28.0029.5127.9524.3224.0325.2025.0726.3626.49 w/o Visual Evidence26.1027.9726.9818.8819.7820.5422.8521.4623.39 w/o Consistency27.4324.7327.3622.0122.1421.8024.0122.2624.23 w/o Entity Retention28.5828.1327.7722.2322.3423.6622.9624.8224.23 EvoSee (Ours)30.5431.1030.1326.5327.4726.7728.6129.3228.51 Table A3 shows that the same ablations also affect Event prediction. Geometry-only Assignment lowers all Event scores, including Global F1 by 6.16 points. The concurrent trajectory and Event declines support using both errors in joint assignment. Full Attention also reduces all three Event metrics. Removing Entity Retention produces the largest decline in all Entity metrics. Removing Consistency most affects Event Global F1 and GPR, whereas removing Continuation Selection most affects Event Scene-F1. Removing Visual Evidence produces the largest losses in evidence category recognition, including 19.42 points in Global F1, and also lowers Event scores. Observation-only Refinement improves Event and Visual Evidence scores over removing the update. Using predicted change further improves their Global F1 scores by 4.23 and 4.14 points, respectively. EvoSee leads all nine semantic metrics. 21 Table A3. Complete ablation results for Scene Evolution Reasoning. EntityEventVisual Evidence VariantGF1SF1GPRGF1SF1GPRGF1SF1GPR Qwen3.5-9B37.4540.7340.1429.8528.2029.8729.8234.2329.84 Geometry-only Assignment41.3848.4244.6940.9340.2642.6341.8443.8443.78 w/o Evolution Refinement45.5053.6146.6740.6342.1045.4048.4350.3151.62 Observation-only Refinement46.8254.1147.1542.8645.0345.6750.3153.1853.74 w/o Continuation Selection43.4649.0044.3638.5436.9640.5941.3145.4344.20 Single Hypothesis43.8849.8847.1840.1538.7943.0645.1747.4146.83 Full Attention46.3354.9748.3942.9245.5943.8749.1649.8151.24 w/o Visual Evidence43.3150.3346.3938.7438.4142.5335.0338.6537.01 w/o Consistency44.4550.6045.4636.0137.8737.8346.3250.2248.32 w/o Entity Retention39.9344.1541.7440.3343.3241.4946.2350.4349.35 EvoSee (Ours)49.0857.3349.6247.0948.9150.1154.4557.7757.93 C.3 Entity Count Table A4 shows that EvoSee improves every trajectory metric in each entity count group. Its trajectory scores decrease across the three groups, while Future MeanIoU retains a 5.90 point advantage over Qwen3.5-9B in the highest group. Gains in both point recovery and overlap show that this continuity advantage holds across scene densities. Table A4. Entity Continuity by observed entity count. ObservedFutureCombined EntitiesMethodNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU 1–3Qwen3.5-9B6,40026.4127.6531.0019.8120.5825.3123.2824.2528.26 1–3EvoSee (Ours)6,40043.9446.6541.5838.3241.1837.3841.2143.9539.55 4–6Qwen3.5-9B2,33525.0821.5622.3218.3515.7517.8121.9218.8120.19 4–6EvoSee (Ours)2,33529.6928.8928.6527.3027.2326.1628.5528.0927.45 ≥ 7Qwen3.5-9B4,89110.718.478.918.486.737.279.657.648.12 ≥ 7EvoSee (Ours)4,89113.4111.7915.8410.749.6313.1712.1410.7714.57 Table A5 shows gains in all semantic metrics for every entity count group. EvoSee’s Entity scores decrease more than its Event and Visual Evidence scores as entity count rises. Its Visual Evidence Global F1 margin over Qwen3.5-9B grows to 33.09 points in the highest group, accompanied by Scene-F1 gains. Evidence category recognition thus retains an advantage even where entity references become less accurate. Table A5. Scene Evolution Reasoning by observed entity count. EntityEventVisual Evidence EntitiesMethodNGF1SF1GPRGF1SF1GPRGF1SF1GPR 1–3Qwen3.5-9B6,40059.7754.5459.8628.7626.5029.6548.7453.1650.25 1–3EvoSee (Ours)6,40070.8272.6871.0152.4453.7252.7364.2866.4464.64 4–6Qwen3.5-9B2,33553.6350.2956.4627.9028.6828.2624.6125.3224.93 4–6EvoSee (Ours)2,33569.3369.6869.7546.6647.5750.6651.8153.9356.26 ≥ 7Qwen3.5-9B4,89118.5718.1122.3531.5730.1932.4215.1613.7115.57 ≥ 7EvoSee (Ours)4,89130.1031.3731.3643.3343.2549.9348.2548.2555.59 Individual entity counts. Individual counts resolve variation within the broad density groups. Counts above 30 are pooled into> 30, with trajectory scores and Scene-F1 averaged over samples and Global F1 and GPR recomputed from pooled matching counts. 22 Table A6 shows that EvoSee leads every trajectory metric for one to four entities and for reported counts from sixteen onward, including> 30. Qwen3.5-9B has higher Point-F1 in several intermediate groups, including five to eleven entities. The advantage therefore varies within the broad entity count groups. Table A6. Entity Continuity at each observed entity count. ObservedFutureCombined EntitiesMethodNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU 1Qwen3.5-9B94720.6826.6333.2812.7916.3524.7716.8821.6829.18 1EvoSee (Ours)94764.7370.9256.0653.5757.7447.5459.2964.4751.89 2Qwen3.5-9B3,45830.8231.3934.2223.4423.6827.9027.2827.6331.16 2EvoSee (Ours)3,45846.3046.9639.8839.4140.3735.4842.9343.6937.74 3Qwen3.5-9B1,99521.5021.6524.3316.8517.2121.0819.3819.6022.81 3EvoSee (Ours)1,99529.9734.6037.6629.1934.7235.8629.6534.6736.82 4Qwen3.5-9B1,35120.1617.9620.6715.2813.6817.3817.9015.9519.11 4EvoSee (Ours)1,35136.3435.0733.2534.6434.1331.4135.5434.6332.36 5Qwen3.5-9B49232.6827.0025.0822.6918.3318.3228.0322.9521.93 5EvoSee (Ours)49222.5622.9124.0818.8019.4520.1720.8121.2922.25 6Qwen3.5-9B49231.0026.0224.0822.4318.8318.4926.8722.5321.40 6EvoSee (Ours)49218.5417.9220.5815.6716.0417.7417.1216.9419.18 7Qwen3.5-9B57830.3424.1922.8824.8319.8618.8427.7222.1220.95 7EvoSee (Ours)57824.2921.4620.9922.2819.9518.9223.3320.7520.03 8Qwen3.5-9B39527.9323.6622.4722.2318.9618.7325.2821.4520.71 8EvoSee (Ours)39516.9216.8022.4814.4514.6519.6115.8215.8521.21 9Qwen3.5-9B37828.0623.6921.9621.5518.0917.6524.9521.0019.89 9EvoSee (Ours)37816.9816.9721.9213.5613.6919.0915.3615.4220.59 10Qwen3.5-9B19615.0112.4313.6510.368.519.6012.8610.5711.69 10EvoSee (Ours)19612.7812.8017.348.808.9312.9010.8910.9515.20 11Qwen3.5-9B10018.4613.1913.5316.4111.6311.3417.4712.4312.46 11EvoSee (Ours)10012.7212.0416.749.529.0112.9511.1810.5814.92 12Qwen3.5-9B816.484.115.345.013.063.715.863.654.50 12EvoSee (Ours)8110.579.9214.758.618.4812.809.699.2913.72 13Qwen3.5-9B8810.466.936.959.045.995.709.776.476.33 13EvoSee (Ours)886.716.3210.376.636.699.036.666.489.68 14Qwen3.5-9B1215.253.394.374.292.743.234.793.083.82 14EvoSee (Ours)1217.525.919.904.883.897.256.244.938.61 15Qwen3.5-9B1215.813.574.174.913.033.215.383.313.71 15EvoSee (Ours)1215.715.1010.103.773.676.504.804.438.41 16Qwen3.5-9B1024.192.583.313.352.062.293.812.342.84 16EvoSee (Ours)1026.625.3510.413.903.096.115.334.288.41 17Qwen3.5-9B621.741.081.901.240.730.691.520.931.35 17EvoSee (Ours)625.494.849.252.021.804.953.873.417.23 18Qwen3.5-9B650.220.141.030.030.020.320.120.080.68 18EvoSee (Ours)655.744.829.052.692.225.044.283.577.12 19Qwen3.5-9B581.721.162.821.200.781.941.460.972.38 19EvoSee (Ours)587.076.4213.125.294.858.656.215.6610.97 20Qwen3.5-9B723.962.915.593.152.535.133.622.745.38 20EvoSee (Ours)7214.6813.1320.7312.3811.3818.1513.5712.2719.51 21Qwen3.5-9B673.422.465.733.162.295.123.292.385.44 21EvoSee (Ours)6715.2213.0817.9511.279.9215.7113.3311.5816.93 Continued on next page 23 Table A6 (continued) ObservedFutureCombined EntitiesMethodNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU 22Qwen3.5-9B1165.844.777.945.254.377.665.564.587.81 22EvoSee (Ours)11619.3316.4923.2815.4913.5122.1317.4715.0522.73 23Qwen3.5-9B1533.992.756.683.162.226.263.602.496.47 23EvoSee (Ours)15322.4219.2724.4917.2215.0422.0919.8817.2123.33 24Qwen3.5-9B1302.551.685.152.151.434.682.351.564.92 24EvoSee (Ours)13025.8522.0724.4522.0219.1022.4123.9820.6223.45 25Qwen3.5-9B1663.262.015.552.771.745.213.031.885.38 25EvoSee (Ours)16626.5522.8325.5023.4920.2222.9225.0621.5524.23 26Qwen3.5-9B513.652.436.622.531.896.043.112.166.33 26EvoSee (Ours)5122.8519.4321.9816.5514.5418.3919.8917.1320.28 27Qwen3.5-9B760.840.541.060.780.510.850.810.530.96 27EvoSee (Ours)766.254.777.345.033.956.365.674.386.87 28Qwen3.5-9B681.040.581.140.760.430.700.900.500.91 28EvoSee (Ours)6811.749.0212.699.327.3210.9210.578.2011.81 29Qwen3.5-9B1300.830.480.990.700.390.810.770.440.91 29EvoSee (Ours)13012.799.7116.6411.148.6415.0712.039.2215.89 30Qwen3.5-9B1301.280.711.130.740.410.851.030.570.99 30EvoSee (Ours)13011.258.6614.637.495.8711.449.467.3313.11 > 30Qwen3.5-9B1,3871.220.670.930.740.410.660.990.540.80 > 30EvoSee (Ours)1,3876.314.748.974.113.116.355.273.977.71 Table A7 shows that EvoSee improves Visual Evidence Global F1, Scene-F1, and GPR at every reported count. Its Entity and Event results have local reversals, including Entity GPR and Event F1. For> 30entities, it leads all nine metrics, with a 39.15 point Visual Evidence Global F1 gain. The advantage in evidence category recognition therefore covers more individual counts than the advantages in Entity and Event recognition. Table A7. Scene Evolution Reasoning at each observed entity count. EntityEventVisual Evidence EntitiesMethodNGF1SF1GPRGF1SF1GPRGF1SF1GPR 1Qwen3.5-9B94758.6262.8461.0836.0531.4937.5771.5079.1374.51 1EvoSee (Ours)94768.6279.8671.0581.7381.7381.7384.6984.6984.69 2Qwen3.5-9B3,45858.8651.4659.1229.5626.6331.0451.5754.7554.15 2EvoSee (Ours)3,45869.5570.7169.6149.8350.3849.8764.4465.2864.49 3Qwen3.5-9B1,99561.0955.9461.3725.0023.8925.2536.5838.0736.94 3EvoSee (Ours)1,99572.6772.6772.8645.3846.2346.6656.6259.7958.21 4Qwen3.5-9B1,35150.4446.9154.2132.1734.2132.5422.9223.8123.18 4EvoSee (Ours)1,35176.6376.5676.8852.3553.8955.7256.4359.1560.07 5Qwen3.5-9B49257.9556.3759.7623.4821.0823.9029.6130.4930.14 5EvoSee (Ours)49261.5060.9861.9739.7538.2844.2247.5248.0452.86 6Qwen3.5-9B49256.0653.4658.1922.9521.1023.2323.4924.3123.77 6EvoSee (Ours)49259.6359.4960.4440.3639.5045.1945.3945.4950.82 7Qwen3.5-9B57841.0236.6544.7518.2216.4118.9317.2316.4617.91 7EvoSee (Ours)57848.8848.9749.9942.5742.2148.7546.2146.3152.92 8Qwen3.5-9B39535.3831.5340.2823.6421.7624.2521.1721.0021.71 8EvoSee (Ours)39549.2349.3750.0843.0842.7049.2449.2249.2856.26 Continued on next page 24 Table A7 (continued) EntityEventVisual Evidence EntitiesMethodNGF1SF1GPRGF1SF1GPRGF1SF1GPR 9Qwen3.5-9B37833.6931.2736.8623.8021.6024.1621.5221.1621.84 9EvoSee (Ours)37848.5248.3949.4846.1446.0853.1948.5448.4655.95 10Qwen3.5-9B19628.9927.1233.7635.0632.8735.8119.5017.8719.92 10EvoSee (Ours)19637.9638.1339.1141.7541.9248.0546.9147.0253.98 11Qwen3.5-9B10038.3236.5343.4329.3028.0329.7421.4819.9021.81 11EvoSee (Ours)10038.9140.0040.1743.0043.0049.6546.0046.0053.12 12Qwen3.5-9B8129.0128.3135.9230.6927.7831.9214.2913.6614.86 12EvoSee (Ours)8132.1832.4333.0542.8642.5949.3945.3445.0652.25 13Qwen3.5-9B8827.8825.9231.2329.4626.7829.9520.0917.9520.42 13EvoSee (Ours)8826.4326.5627.0445.4545.4552.4948.8648.8656.42 14Qwen3.5-9B12117.1914.5323.0541.8240.6642.8811.139.7211.41 14EvoSee (Ours)12129.4129.4830.5640.9941.0547.3048.4548.4855.90 15Qwen3.5-9B12121.7119.8829.8536.7734.0537.856.195.906.37 15EvoSee (Ours)12130.3930.6031.5444.0743.9450.7946.1546.0153.18 16Qwen3.5-9B10212.5811.4718.6344.8443.0745.877.947.258.12 16EvoSee (Ours)10231.9732.1533.7139.8039.7145.9243.7343.6350.46 17Qwen3.5-9B627.696.8612.6844.3043.3945.338.477.808.67 17EvoSee (Ours)6229.5429.3731.2741.1341.1347.4945.9745.9753.08 18Qwen3.5-9B654.735.588.3738.4137.0339.124.274.364.35 18EvoSee (Ours)6533.3334.1435.1535.3835.3840.8645.3845.3852.41 19Qwen3.5-9B586.956.1312.4638.6437.6440.756.066.036.39 19EvoSee (Ours)5824.6523.2826.2446.5546.5553.7547.4147.4154.75 20Qwen3.5-9B728.457.3212.6336.0032.7837.0316.0013.8016.46 20EvoSee (Ours)7225.5425.3526.4948.6148.6156.1348.6148.6156.13 21Qwen3.5-9B6712.2111.2917.2530.0027.2631.909.338.019.92 21EvoSee (Ours)6730.3230.2531.9747.0147.0154.2950.0050.0057.74 22Qwen3.5-9B1167.496.9210.8136.1431.8737.2717.8915.7818.45 22EvoSee (Ours)11623.7323.7324.4549.1449.1456.7449.5749.5757.24 23Qwen3.5-9B1535.474.728.2934.9030.7035.8820.6717.6321.25 23EvoSee (Ours)15319.6319.5320.5349.0249.0256.6049.3549.3556.98 24Qwen3.5-9B1304.573.797.0518.7816.3119.5918.4516.2619.25 24EvoSee (Ours)13019.5919.6220.6547.3147.3154.6348.4648.4655.96 25Qwen3.5-9B1663.553.125.4230.6727.6331.4618.4016.1218.88 25EvoSee (Ours)16618.8618.8820.1445.7845.7852.8749.4049.4057.04 26Qwen3.5-9B518.227.4410.8430.0428.0430.7217.3914.3817.79 26EvoSee (Ours)5120.1320.0521.4738.2438.2444.1545.1045.1052.07 27Qwen3.5-9B764.113.865.5133.3334.1234.8116.9514.6517.70 27EvoSee (Ours)7628.0728.4029.7742.1142.1148.6246.7146.7153.94 28Qwen3.5-9B6810.5911.6813.4430.4331.4731.5811.189.0211.60 28EvoSee (Ours)6825.0625.4026.5038.9738.9745.0044.8544.8551.79 29Qwen3.5-9B1308.187.989.7131.2432.1031.5518.6916.5618.87 29EvoSee (Ours)13026.6426.7727.5842.6942.6949.3048.0848.0855.51 30Qwen3.5-9B1307.397.068.9930.3430.5131.0215.1713.1015.51 30EvoSee (Ours)13021.6821.9022.6943.0843.0849.7448.4648.4655.96 > 30Qwen3.5-9B1,38711.3011.4213.2738.0138.3939.0710.718.7711.01 > 30EvoSee (Ours)1,38720.0120.4921.2342.1442.1448.6649.8649.8657.57 25 C.4 Temporal Extent Equal observation and future durations define each temporal group. Table A8 shows that EvoSee leads all trajectory metrics in every group. Its observed accuracy stays close across groups, while future accuracy decreases with longer intervals. Its Future Point-F1 advantage over Qwen3.5-9B reaches 15.14 points in the longest group, accompanied by recall and overlap gains. Table A8. Entity Continuity across temporal extents. ObservedFutureCombined ExtentMethodNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU 1–3 sQwen3.5-9B11,00122.2821.3022.3217.1116.3118.4519.8618.9320.48 1–3 sEvoSee (Ours)11,00130.2531.0230.1226.9928.2127.2228.7029.6728.74 4–6 sQwen3.5-9B1,61614.0813.9519.3210.219.9015.1512.2111.9617.29 4–6 sEvoSee (Ours)1,61631.9031.6930.2826.5826.5126.0329.2829.1028.18 ≥ 7 sQwen3.5-9B1,00911.9711.8017.156.345.9211.599.148.8114.40 ≥ 7 sEvoSee (Ours)1,00931.4730.9929.9621.4820.9023.0826.4925.9226.54 Table A9 shows that EvoSee also improves all semantic metrics in every temporal group. Its Event Global F1 margins narrow with longer intervals, while Visual Evidence Global F1 margins remain above 24 points. The accompanying Scene-F1 gains show that the advantage also holds when scenes receive equal weight. Table A9. Scene Evolution Reasoning across temporal extents. EntityEventVisual Evidence ExtentMethodNGF1SF1GPRGF1SF1GPRGF1SF1GPR 1–3 sQwen3.5-9B11,00139.9041.9742.1727.6826.2127.7130.4934.6630.51 1–3 sEvoSee (Ours)11,00151.0057.9851.3847.3849.2750.4054.9358.4158.44 4–6 sQwen3.5-9B1,61628.8035.7932.9037.8235.9837.8227.7132.8027.71 4–6 sEvoSee (Ours)1,61642.4054.4443.4646.5848.2649.5152.4655.1355.77 ≥ 7 sQwen3.5-9B1,00926.7935.2031.4739.4737.3539.4726.3331.7826.33 ≥ 7 sEvoSee (Ours)1,00941.0054.9142.6044.8546.0247.8752.3855.0755.92 Individual temporal extents.The individual temporal extents reveal variation within these groups. Table A10 shows that EvoSee leads all trajectory metrics at all ten extents. Its observed scores vary less than its future scores, which decrease over longer intervals. Identity matching remains fixed from observation, so these gains concern continuation of the same entities at each temporal extent. Table A10. Entity Continuity at each temporal extent. ObservedFutureCombined ExtentMethodNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU 1 sQwen3.5-9B7,29427.1225.9126.8520.9920.0022.3524.2423.1024.70 1 sEvoSee (Ours)7,29430.5931.5030.4926.8328.4827.6028.7930.0429.11 2 sQwen3.5-9B2,49818.0117.2518.9313.4312.8115.2315.8815.1717.19 2 sEvoSee (Ours)2,49829.5330.1629.4327.4127.7926.6028.5629.0328.12 3 sQwen3.5-9B1,2091.921.842.021.271.251.541.611.551.79 3 sEvoSee (Ours)1,20929.7129.9129.3127.0827.4526.1528.4628.7127.81 4 sQwen3.5-9B69014.8414.8120.1110.9010.7316.1712.9412.8118.20 4 sEvoSee (Ours)69031.7531.6430.5128.1128.1526.9929.9829.9228.79 5 sQwen3.5-9B52014.2513.9919.1110.6010.2015.0012.4712.1217.09 5 sEvoSee (Ours)52031.7331.3829.9026.3126.1025.6129.0428.7227.77 Continued on next page 26 Table A10 (continued) ObservedFutureCombined ExtentMethodNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU 6 sQwen3.5-9B40612.5812.4518.278.558.1013.6210.6510.3015.99 6 sEvoSee (Ours)40632.3632.1630.3924.3324.2524.9428.3828.2127.69 7 sQwen3.5-9B32913.3313.0718.677.637.0312.9910.5010.0215.86 7 sEvoSee (Ours)32932.8932.4930.2722.9122.4523.9027.9027.4427.11 8 sQwen3.5-9B26811.8011.6516.906.546.0611.609.138.7814.28 8 sEvoSee (Ours)26830.8430.2529.6621.7220.9323.1126.3325.5826.41 9 sQwen3.5-9B22210.3410.2615.634.624.4510.397.427.2713.02 9 sEvoSee (Ours)22231.3730.7729.8521.3620.7322.6126.3325.6526.22 10 sQwen3.5-9B19011.7511.6116.635.815.5410.538.808.5713.64 10 sEvoSee (Ours)19030.0329.7229.9818.8118.4022.1524.4724.0626.13 Table A11 shows that EvoSee also leads every semantic metric at every temporal extent. Its Event and Visual Evidence Global F1 scores vary less than Entity Global F1, and Scene-F1 gains accompany the pooled improvements. Both continuity and semantic evaluations therefore show gains across the individual temporal settings. Table A11. Scene Evolution Reasoning at each temporal extent. EntityEventVisual Evidence ExtentMethodNGF1SF1GPRGF1SF1GPRGF1SF1GPR 1 sQwen3.5-9B7,29447.4349.4949.7625.4024.1125.4430.4934.4530.52 1 sEvoSee (Ours)7,29452.3258.6252.6247.5649.5250.6454.7058.1558.25 2 sQwen3.5-9B2,49835.3937.7137.7530.2828.5030.3030.2634.8530.28 2 sEvoSee (Ours)2,49849.5357.1049.9947.2549.1050.2255.6259.2659.11 3 sQwen3.5-9B1,2095.375.385.8735.9334.2035.9330.9535.5430.95 3 sEvoSee (Ours)1,20946.4455.9847.1646.5548.1349.3654.8958.1658.20 4 sQwen3.5-9B69028.2534.5632.2736.8734.6936.8728.9333.4928.93 4 sEvoSee (Ours)69042.7254.3043.6446.6948.4349.5952.4554.9055.71 5 sQwen3.5-9B52029.5737.1433.5738.5536.9338.5526.9232.4826.93 5 sEvoSee (Ours)52042.2554.3643.3546.5148.2449.4452.9956.0956.33 6 sQwen3.5-9B40628.7036.1433.1038.5136.9538.5126.6332.0326.64 6 sEvoSee (Ours)40642.0854.7743.3346.4847.9949.4751.8254.2755.16 7 sQwen3.5-9B32928.9037.5533.4039.5237.6539.5325.9531.6725.95 7 sEvoSee (Ours)32941.0054.5142.4044.0045.2446.9451.5453.8554.97 8 sQwen3.5-9B26825.9634.8130.5436.3333.9436.3326.6232.8226.62 8 sEvoSee (Ours)26841.1054.7942.6745.8747.0148.9652.6755.5356.22 9 sQwen3.5-9B22226.1134.3530.7241.5539.5041.5827.7032.0327.72 9 sEvoSee (Ours)22240.6154.7942.3444.8046.1747.8352.4255.2655.95 10 sQwen3.5-9B19025.2432.6830.3841.2639.1241.2624.9130.1924.91 10 sEvoSee (Ours)19041.2855.9043.1644.9045.7948.0053.4056.3257.09 C.5 Video Source The source analysis covers 32 APIDIS, 11,048 OVT-B, and 2,546 TAO test windows. Table A12 separates localization in observation from continuation into the future. OVT-B has the highest future and combined trajectory scores. APIDIS has the highest observed MeanIoU alongside lower future accuracy, and TAO also shows a decline from observation to future. Spatial continuation therefore varies across sources even when observed localization is strong. 27 Table A12. Entity Continuity across video sources for EvoSee. ObservedFutureCombined SourceNP-F1P-RmIoUP-F1P-RmIoUP-F1P-RmIoU APIDIS3213.3825.7934.042.093.4410.157.5914.3121.70 OVT-B11,04832.9732.7431.0929.5530.0928.7031.2931.4229.92 TAO2,54620.1924.0425.8813.7716.3918.6017.2420.4222.49 Table A13 reveals a different source profile for semantic prediction. TAO has the highest Entity scores, while APIDIS leads all Event and Visual Evidence metrics. On APIDIS, strong category recognition coexists with lower future geometric accuracy. Evaluating both predictions on the same source exposes this difference between recognition of change and spatial continuation. Table A13. Scene Evolution Reasoning across video sources for EvoSee. EntityEventVisual Evidence SourceNGF1SF1GPRGF1SF1GPRGF1SF1GPR APIDIS3248.7549.9052.8865.6365.6365.6381.2581.2581.25 OVT-B11,04846.9956.5547.6848.4750.5251.7954.8258.1558.58 TAO2,54659.4560.8159.4940.4241.7242.2952.4355.8154.85 C.6 Scene Description Results Table 3 reports BERTScore F1 for complete scene descriptions. Table A14 separates the present, future, and evidence content of those descriptions. EvoSee improves all three parts and gains 0.28 points overall over Qwen3.5-9B, with the largest gain in the present description. Table A14. BERTScore F1 for the complete scene description and its present, future, and evidence parts. MethodOverallPresentFutureEvidence Qwen3.5-9B95.9895.5996.6596.31 EvoSee (Ours) 96.2696.0696.8196.52 D Qualitative Comparisons The examples compare ground truth (GT), Qwen3.5-9B, and EvoSee under the same observation and future durations. The common question is “Given the observedH-second video window, its fixed keyframes, and the camera view, what scene evolution follows within the nextHseconds?” The accompanying instruction requests observed trajectories, future trajectories, and a description relating the forecast to observed evidence. Reference close-ups from observation and future appear in the right panel. Each row uses the same four timestamps. For each entity, the displayed centers accumulate from one to four points, with solid connections in observation and dashed connections into the future. Colors follow the fixed observed identity correspondence, and labels retain each method’s output identities. The right panel places reference close-ups above the two descriptions. Gains displayed in each figure are EvoSee minus Qwen3.5-9B on that sample, measured in percentage points. 28 Observed trajectoriesFuture trajectories GT Qwen3.5-9B EvoSee (Ours) T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 CubePersonObservedFuture T001 T002 T001 T002 T001 T002 T001 T002 0.05s0.85s1.05s1.85s More precise localization as the cube turns. Combined Point-F1 +30.9 pts Combined MeanIoU +14.8 pts Scene Evolution Understanding ObservedFuture 0.05 s0.85 s1.85 s Qwen3.5-9B During 1 second of observation, a person turns a small cube (T002) between both hands (T001). Over the next 1 second, the hands turn the cube through green, orange, then yellow faces. This forecast is supported by the cube's successive face changes during the observed turns. EvoSee (Ours) During 1 second of observation, a person (T001) turns a 1×1 Rubik’s Cube (T002). Over the next 1 second, the cube turns and shifts slightly downward between the fingers, revealing different faces. This forecast is supported by its observed orientation changes and sustained contact with the person’s hands. Figure A1. Cube rotation. EvoSee follows the changing spatial extent of the handheld cube, improving Combined Point-F1 by 30.9 points and MeanIoU by 14.8 points. Observed trajectoriesFuture trajectories GT Qwen3.5-9B EvoSee (Ours) T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 T001 T002 ConsolePersonObservedFuture T001 T002 T001 T002 T001 T002 T001 T002 0.15s0.95s1.15s1.95s Closer trajectory agreement during device reorientation. Combined Point-F1 +55.0 pts Combined MeanIoU +26.2 pts Scene Evolution Understanding ObservedFuture 0.15 s0.95 s1.95 s Qwen3.5-9B During 1 second of observation, a person (T001) holds a console (T002). Over the next 1 second, the console should remain steady, with possible grip adjustments or button interaction. This forecast is supported by their consistent observed positions. EvoSee (Ours) During 1 second of observation, a person (T001) holds a teal handheld console (T002) above a green mat, adjusting the grip around its edges. Over the next 1 second, the console shifts left and turns over, bringing its rear casing into view as the fingers pass across the screen. This forecast is supported by the changing hand placement and the console’s movement with the supporting hands. Figure A2. Device manipulation. Observed and future tracks follow a person and handheld console during reorientation. Combined Point-F1 and MeanIoU gains are 55.0 and 26.2 points. 29 Observed trajectoriesFuture trajectories GT Qwen3.5-9B EvoSee (Ours) T001T002 T001 T002 T001 T002 T001 T002 T001T002T001T002T001T002T001T002 CicadaPersonObservedFuture T001 T002 T001 T002 T001 T002 T001 T002 0.05s0.85s1.05s1.85s More accurate localization of the hand and insect. Combined Point-F1 +26.4 pts Combined MeanIoU +27.3 pts Scene Evolution Understanding ObservedFuture 0.05 s0.85 s1.85 s Qwen3.5-9B During 1 second of observation, a person (T001) holds a cicada (T002). Over the next 1 second, the grip should remain steady, with possible small adjustments. This forecast is supported by the consistent positions of the hand and insect. EvoSee (Ours) During 1 second of observation, a person (T001) holds a cicada (T002) beside a tree, with its folded wings extending below the fingertips. Over the next 1 second, the hand and insect move upward and left together as the fingers adjust around the insect’s body, partly covering its sides. This forecast is supported by the insect’s displacement relative to the tree and its maintained contact with the fingers. Figure A3. Hand and insect motion. The comparison shows entity localization at different spatial scales. Combined Point-F1 and MeanIoU improve by 26.4 and 27.3 points. Observed trajectoriesFuture trajectories GT Qwen3.5-9B EvoSee (Ours) T001 T001T001 T001 T001 T002 T001 T002 T001 T002 T001 T002 HippoExtra trackObservedFuture T001 T001T001 T001 0.10s0.90s1.10s1.90s Closer future localization in low visibility. Combined Point-F1 +35.5 pts Combined MeanIoU +3.7 pts Scene Evolution Understanding ObservedFuture 0.10 s0.90 s1.90 s Qwen3.5-9B During 1 second of observation, a marine mammal (T001) swims through cloudy water. Over the next 1 second, it should continue forward near a spherical object (T002). This forecast is supported by its consistent observed position and orientation. EvoSee (Ours) During 1 second of observation, a hippopotamus (T001) moves through cloudy water on the right, its head and broad body becoming more distinct as it draws nearer. Over the next 1 second, it remains on the right while rising slightly in the view and occupying a larger area. This forecast is supported by the upward movement of its body outline and its increasing apparent size across the observed frames. Figure A4. Underwater motion. EvoSee more closely localizes the animal through low visibility, with Combined Point-F1 and MeanIoU gains of 35.5 and 3.7 points. 30 Observed trajectoriesFuture trajectories GT Qwen3.5-9B EvoSee (Ours) T001T001T001 T001 T001 T002 T001 T002 T001 T002 T001 T002 PandaExtra trackObservedFuture T001T001T001 T001 0.15s4.95s5.15s9.95s Closer tracking agreement through changes in body position. Combined Point-F1 +36.7 pts Combined MeanIoU +8.9 pts Scene Evolution Understanding ObservedFuture 0.15 s4.95 s9.95 s Qwen3.5-9B During 5 seconds of observation, a panda (T001) lies among bamboo leaves (T002). Over the next 5 seconds, it should roll or shift while handling the leaves. This forecast is supported by observed movements of its head and paws. EvoSee (Ours) During 5 seconds of observation, a reclining panda (T001) shifts among scattered bamboo, with its rounded back toward the camera and its limbs moving beside its body. Over the next 5 seconds, it rolls slightly and settles lower and left, changing its body outline while remaining near the bamboo. This forecast is supported by the observed tilt of its torso, the changing limb positions and its gradual displacement across the ground. Figure A5. Extended body motion. Five seconds of observation precede a forecast of five seconds. Combined Point-F1 and MeanIoU gains are 36.7 and 8.9 points. Observed trajectoriesFuture trajectories GT Qwen3.5-9B EvoSee (Ours) T001T001T001T001 T001T001T001T001 BearObservedFuture T001T001T001T001 0.00s4.00s4.20s8.00s More reference points recovered during the observed interval. Combined Point-F1 +19.5 pts Combined MeanIoU +5.0 pts Scene Evolution Understanding ObservedFuture 0.00 s4.00 s8.00 s Qwen3.5-9B During 4 seconds of observation, a bear (T001) leans over the water. Over the next 4 seconds, it should stay near the ledge, possibly lifting and lowering its head. This forecast is supported by its consistent observed posture and position. EvoSee (Ours) During 4 seconds of observation, a black bear (T001) stands on a narrow ledge beside a pool, lowering its muzzle toward the water while keeping its paws on the bank. Over the next 4 seconds, it adjusts its stance along the ledge, with a slight upward and leftward shift in the view. This forecast is supported by its changing shoulder position, small posture adjustments and continued footing beside the water. Figure A6. Motion near the water edge. Intervals of four seconds show localization around a stationary scene structure. Combined Point-F1 and MeanIoU improve by 19.5 and 5.0 points. 31