Paper deep dive
Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning
Xiaoyu Zhu, Xinke Deng, Suresh Taddewadikar, Arnab Kumar Mondal, Zhongyu Jiang, Ian Fasel, Joerg Liebelt
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/22/2026, 3:12:57 AM
Summary
The paper introduces Internalized Visual Thinking (IVT), a post-training framework for multimodal large language models (MLLMs) that enables proactive video reasoning by predicting latent representations of future frames during training, rather than generating explicit visual chain-of-thought (Visual CoT) images at inference. IVT jointly optimizes textual prediction and next-embedding prediction over unlabeled videos, allowing the model to capture motion, object transitions, and latent intent internally. Experiments on datasets like Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100 show that IVT achieves comparable or better performance than explicit Visual CoT (e.g., Zebra-CoT) while reducing end-to-end latency by more than 5x. The study highlights that predictive world modeling can be internalized to create efficient, accurate multimodal reasoners without the computational overhead of pixel-space generation at inference time.
Entities (13)
Relation Signals (10)
Internalized Visual Thinking → evaluatedon → Ego-Exo4D
confidence 95% · We train and evaluate on Ego-Exo4D (Grauman et al. 2024)
Internalized Visual Thinking → evaluatedon → Ego4D
confidence 95% · We train and evaluate on Ego-Exo4D (Grauman et al. 2024), Ego4D (Grauman et al. 2022)
Internalized Visual Thinking → evaluatedon → EPIC-KITCHENS-100
confidence 95% · We train and evaluate on Ego-Exo4D (Grauman et al. 2024), Ego4D (Grauman et al. 2022), and EPIC-KITCHENS-100 (Damen et al. 2022)
Internalized Visual Thinking → reduceslatencyby → 5x
confidence 95% · Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x.
Internalized Visual Thinking → improvesover → Visual Chain-of-Thought
confidence 92% · Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x.
Zebra-CoT → builton → BAGEL-7B
confidence 90% · Zebra-CoT (Li et al. 2025a), a representative Visual-CoT baseline built on BAGEL-7B
Zebra-CoT → isbaselinefor → Visual Chain-of-Thought
confidence 90% · Zebra-CoT (Li et al. 2025a), a representative Visual-CoT baseline
Internalized Visual Thinking → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-embedding prediction over unlabeled videos. Given a partially observed video, IVT predicts latent representations of future frames together with the target textual answer, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames. We conduct controlled studies across target representations, decoder designs, prediction horizons, data mixtures, training curricula, and predictive objectives. IVT improves over direct-answer fine-tuning on all six evaluation settings while retaining the same inference pathway. Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x. Together, our findings suggest that explicit pixel-space generation at inference time, as used in visual chain-of-thought, may not be necessary for effective proactive video reasoning. Predictive world modeling can be internalized during training to produce multimodal reasoners that are both more accurate and substantially more efficient.
Tags
Links
- Source: https://arxiv.org/abs/2608.15869v1
- Canonical: https://arxiv.org/abs/2608.15869v1
Trouble viewing inline? Open PDF directly →
Full Text
80,635 characters extracted from source content.
Expand or collapse full text
Xiaoyu Zhu: xiaoyu_zhu2@apple.com Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning Xiaoyu Zhu Xinke Deng Suresh Taddewadikar Arnab Kumar Mondal Zhongyu Jiang Ian Fasel Joerg Liebelt Affiliation: Apple August 16, 2026 Abstract Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-embedding prediction over unlabeled videos. Given a partially observed video, IVT predicts latent representations of future frames together with the target textual answer, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames. We conduct controlled studies across target representations, decoder designs, prediction horizons, data mixtures, training curricula, and predictive objectives. IVT improves over text-only post-training across all six evaluation settings while retaining the same efficient inference pathway. Compared with Visual CoT, IVT achieves comparable or better performance and reduces end-to-end latency by more than 5×5×. Our findings suggest that explicit pixel-space generation at inference time, as used in Visual CoT, may not be necessary for proactive video reasoning. Predictive world modeling can be internalized during training to produce multimodal reasoners that are both more accurate and substantially more efficient. 1 Introduction Intelligent systems operating in the visual world must do more than recognize what is currently visible. They must infer how an unfolding event will evolve and anticipate what is likely to happen next. Chain-of-thought (CoT) has substantially improved the reasoning ability of multimodal large language models (MLLMs) to solve problems by decomposing them into intermediate steps before producing an answer (Wei et al. 2022; Zhang et al. 2023). Most existing MLLMs instantiate this process as textual CoT, in which intermediate computation is represented as a sequence of natural-language tokens. Although effective, language is an imperfect reasoning representation for visual dynamics. Textual rationales can be unnecessarily long or redundant, increasing autoregressive decoding cost without consistently improving accuracy (Wu et al. 2025b; Xu et al. 2025b). More fundamentally, generated rationales are not guaranteed to faithfully reflect the computation that determines the final prediction (Zhao et al. 2026; Turpin et al. 2023; Lanham et al. 2023). In multimodal settings, this problem is compounded by a modality mismatch: text-based reasoning can rely excessively on linguistic priors, hallucinate visual details, or fail to remain grounded in the spatial and temporal evidence contained in the input (Wu et al. 2026; Kancheti et al. 2026). These limitations have motivated a growing shift from reasoning in words toward reasoning in images. Existing approaches construct visual reasoning by extracting relevant regions, annotating or transforming the input, drawing intermediate sketches, or generating new images as part of the reasoning process (Zhou et al. 2024; Hu et al. 2024; Chern et al. 2025). Such visual chains of thought can preserve geometry, object states, motion, and other information that is difficult to express precisely in language. In temporal and embodied settings, a particularly compelling formulation is predictive Visual CoT: the model first generates an image of a plausible future or goal state and subsequently conditions its answer or action on that visual prediction (Zhao et al. 2025; Li et al. 2025a). By externalizing the model’s thinking of the future, predictive Visual CoT provides an intuitive mechanism for visual foresight and has demonstrated strong results on spatial reasoning, planning, and robotic manipulation. However, explicitly rendering visual thoughts transfers a substantial computational burden to inference. Unlike textual reasoning, which autoregressively decodes discrete tokens, Visual CoT requires decoding dense visual representations into an intermediate image and subsequently re-encoding that image into visual tokens for downstream reasoning, resulting in substantially higher computational cost. This inference cost is particularly problematic for latency-critical applications in proactive reasoning, where a prediction loses value if it arrives after the event has already occurred. Figure 1: Three post-training paradigms for proactive video reasoning. Answer-Only SFT maps the observed input directly to the predicted textual state. Visual CoT first generates an explicit future visual state in pixel space and then conditions on it to produce the answer, so it pays for the image generation cost at inference. In contrast, our Internalized Visual Thinking (IVT) predicts the future visual state as a latent embedding target only during training; at inference it generates the answer directly, without synthesizing an intermediate image, and therefore executes the same computation graph as Answer-Only SFT. Following the definition in Zhao and Wildes 2021, we consider two primary tasks for proactive video reasoning. Early-event prediction requires recognizing an ongoing action before it is completed, whereas next-event prediction requires anticipating an action that has not yet begun and may occur several seconds into the future (Zhao and Wildes 2021; Sadegh Aliakbarian et al. 2017). Unlike offline video question answering, these tasks cannot rely on evidence that is already fully observable. Instead, models must infer latent intent, causal dynamics, and plausible future states from incomplete observations in a timely manner. This motivates our central question: Can MLLMs learn to think visually during training, yet reason efficiently at inference without explicitly generating their visual thoughts? We answer this question through Internalized Visual Thinking (IVT), a post-training framework that learns predictive world representations from unlabeled videos. Rather than generating a future image at inference time, IVT jointly trains the model to (i) predict the latent embeddings of future frames from the observed video and (i) generate the corresponding textual prediction, as shown in Figure 1. The future embeddings provide dense supervision that requires the model to capture object transitions, motion, interactions, and scene dynamics. After post-training, the model directly produces an answer from the observed video. In this sense, IVT reformulates visual foresight from an explicit inference-time operation into the model’s internal representations. The formulation’s effectiveness depends on several nontrivial design choices. We conduct controlled experiments across multiple model architectures, target representations, prediction horizons, multimodal data-mixture ratios, loss functions, batching strategies, and curriculum designs. Beyond reporting the best configuration, we systematically analyze both successful and unsuccessful training recipes, which demonstrates when future prediction improves reasoning and when it instead interferes with language generation. Our experiments yield three principal findings. First, next-embedding prediction can substantially improve proactive video reasoning over text-only post-training. This result is notable because recent studies of unified generation and understanding suggest that adding generative capabilities to MLLMs does not consistently improve and can sometimes degrade the visual question answering performance (Wen et al. 2026; Cheng et al. 2026). Our results demonstrate that the value of generation lies not necessarily in synthesizing high-quality images, but in learning predictive representations that benefit proactive reasoning tasks. Second, internalized visual thinking is not merely a cheaper approximation of explicit Visual CoT. Under matched base models, training data, and evaluation protocols, IVT surpasses strong Visual CoT counterparts on 4 of 6 benchmarks. These results indicate that Visual CoT may not be necessary for the proactive video reasoning task. Predictive world modeling can be internalized during training to produce efficient multimodal reasoners. Third, the gains do not emerge automatically from adding an auxiliary prediction loss. We find that sharing the language-model decoder, carefully controlling the ratio between visual-prediction and text-generation for data batching, selecting an appropriate predictive horizon, and introducing the objectives through a suitable curriculum are all critical. These findings turn IVT from an isolated model improvement into a set of practical principles for building efficient MLLMs. Together, our results establish a training-time alternative to Visual CoT: rather than asking a model to render what it imagines, we teach it to encode predictive visual knowledge in the representations used to reason. This produces MLLMs that retain the foresight benefits of visual world modeling while preserving the inference efficiency required by proactive reasoning tasks. 2 Internalized Visual Thinking 2.1 Method Overview Let X≤t=(I1,…,It)X_≤ t=(I_1,…,I_t) denote the observed video prefix up to the observation boundary t, let q denote the textual prompt, and let y=(y1,…,yT)y=(y_1,…,y_T) denote the target textual response. A standard supervised fine-tuning (SFT) objective minimizes the token-averaged negative log-likelihood ℒtext=−1T∑j=1Tlogpθ(yj∣y<j,X≤t,q),L_text=- 1T _j=1^T p_θ (y_j y_<j,X_≤ t,q ), (2.1) where θ denotes the trainable MLLM parameters. Although this objective teaches the model to map partial observations to a target response, it does not explicitly constrain the learned representation to capture how the visual state may evolve beyond the observation boundary. IVT augments language supervision with an auxiliary objective that predicts latent representations of future video frames. Let ℋH denote a set of future temporal offsets and let K=|ℋ|K=|H|. For each h∈ℋh , the corresponding future frame is It+hI_t+h. A target encoder EtarE_tar maps this frame to a sequence of M latent tokens, Zh=Etar(It+h)=(zh,1,…,zh,M)∈ℝM×d,Z_h=E_tar(I_t+h)=(z_h,1,…,z_h,M) ^M× d, (2.2) where d denotes the target-feature dimension. Conditioned only on the observed context (X≤t,q)(X_≤ t,q), the MLLM produces predictive hidden states at designated future-prediction positions. A projection head PϕP_φ maps these hidden states into the target representation space, Z^h=Pϕ(Fθ(X≤t,q,h))∈ℝM×d, Z_h=P_φ (F_θ(X_≤ t,q;h) ) ^M× d, (2.3) where Fθ(⋅,h)F_θ(·;h) denotes the predictive hidden states associated with future offset h, and ϕφ denotes the projection-head parameters. We optimize the predicted future representations using a predictive objective ℒpredL_pred. The overall IVT objective is ℒIVT=ℒtext+λpredℒpred,L_IVT=L_text+ _predL_pred, (2.4) where we set λpred=1 _pred=1 in all experiments. 2.2 Design Choices Target representations. We compare four future-state target spaces: reconstruction-oriented Flux-VAE latents (Black Forest Labs 2024), semantic features from DINOv2 (Oquab et al. 2023), and SigLIP2-based features (Tschannen et al. 2025) with either an adaptive target encoder jointly optimized with the text-generation branch or a frozen target encoder. These target spaces differ primarily in two respects. First, reconstruction-oriented latents preserve fine-grained appearance and spatial information required for image reconstruction, whereas semantic representations emphasize higher-level visual concepts and invariances. Second, adaptive targets can evolve together with the model during post-training, while frozen targets provide a stationary prediction space. For all target representations, the target tensor used by ℒpredL_pred is detached through sg(⋅)sg(·), such that gradients from the predictive objective do not propagate through the target branch. For frozen encoders, the target-encoder parameters remain fixed throughout training. For the adaptive SigLIP2 variant, the encoder can still be updated through the text-generation objective, allowing the target space to adapt during joint optimization. Model architectures. We study two architectures that differ in how strongly future-state prediction is coupled with language reasoning. In the Dense design, understanding and predictive tokens are processed by the same decoder parameters, so gradients from both ℒtextL_text and ℒpredL_pred directly update the shared decoder. In the Mixture-of-Experts (MoE) design, understanding and predictive tokens use separate feed-forward experts within each transformer layer. Expert selection is deterministic according to token type rather than controlled by a learned router. Specifically, understanding tokens are processed by the understanding expert, whereas predictive tokens are processed by a separate prediction expert, while the remaining transformer components are shared. This comparison examines whether future-state supervision is most effective when it directly updates the parameters used for language reasoning or when the predictive pathway is partially separated from the understanding pathway. Prediction horizon. We vary the prediction horizon to study the effect of supervising future states at increasing temporal offsets. For a maximum prediction horizon H, we define the set of future temporal offsets as ℋH=1,…,HH_H=\1,…,H\. Frames are sampled at 1fps1 \ fps, so an offset h∈ℋHh _H corresponds to the future frame It+hI_t+h, occurring h seconds after the final observed frame. Thus, a maximum horizon H supervises the sequence of future states at +1,…,+H+1,…,+H seconds relative to the observation boundary. Data mixture. We control the relative amount of language and future-state supervision by varying the mixture of understanding and future-prediction examples packed into each training batch. We compare 1:11:1, 3:13:1, and 5:15:1 understanding-to-prediction mixtures, where larger ratios place greater emphasis on answer-generation supervision. Training curriculum. We compare joint optimization with a two-stage training curriculum. In the two-stage setting, the model is first optimized for future-state prediction and is subsequently fine-tuned using only the downstream understanding objective. This comparison evaluates whether future-state prediction is most effective when learned jointly with language supervision or when introduced as a separate training stage. Predictive objectives. We study two objectives for learning future-state representations. Direct feature regression directly predicts the target representation ZhZ_h from the observed context. For K=|ℋ|K=|H| prediction horizons, we optimize ℒreg=1KMd∑h∈ℋ‖Z^h−sg(Zh)‖F2,L_reg= 1KMd _h \| Z_h-sg(Z_h) \|_F^2, (2.5) where ∥⋅∥F\|·\|_F denotes the Frobenius norm and sg(⋅)sg(·) denotes stop-gradient. The objective therefore corresponds to the mean squared error across prediction horizons, predictive tokens, and feature dimensions. Rectified-flow matching instead learns a conditional velocity field between a noise sample and the target future representation. For each target, we define Zh(1)=sg(Zh)Z_h^(1)=sg(Z_h), sample Zh(0)∼(0,I)Z_h^(0) (0,I) and τ∼(0,1)τ (0,1), construct Zh(τ)=(1−τ)Zh(0)+τZh(1)Z_h^(τ)=(1-τ)Z_h^(0)+τ Z_h^(1), and define the target velocity as Vh⋆=Zh(1)−Zh(0)V_h =Z_h^(1)-Z_h^(0). Conditioned on the observed context, the model predicts V^h=Vθ,ϕ(Zh(τ),τ∣X≤t,q) V_h=V_θ,φ(Z_h^(τ),τ X_≤ t,q) and minimizes ℒFM=1KMd∑h∈ℋ‖V^h−Vh⋆‖F2.L_FM= 1KMd _h \| V_h-V_h \|_F^2. (2.6) Inference. At inference time, the model receives only the observed video prefix and textual prompt (X≤t,q)(X_≤ t,q) and autoregressively generates the textual response according to pθ(y∣X≤t,q)p_θ(y X_≤ t,q). IVT uses the same inference path as the corresponding Answer-Only SFT model and introduces no additional image generation. 3 Experiments 3.1 Experimental Protocol Tasks. Given an observed video prefix X≤tX_≤ t and a textual prompt q, the model generates an open-ended textual prediction y about an ongoing or future event. In early-event prediction (EEP), the observation ends within the annotated action interval [ts,te][t_s,t_e], such that ts<t<tet_s<t<t_e. We evaluate four observation ratios, 0.1,0.3,0.5,0.7\0.1,0.3,0.5,0.7\, measured relative to the duration of the target action (Stergiou and Damen 2023). In next-event prediction (NEP), the observed prefix ends before the subsequent event begins, satisfying t≤tnext−τat≤ t_next- _a, where the anticipation gap is fixed to τa=1 _a=1 second (Assran et al. 2025; Mittal et al. 2024). We focus on short-horizon anticipation, where immediate motion, object interactions, and local visual dynamics provide informative predictive signals. Long-horizon anticipation depends more strongly on semantic goals, procedural knowledge, and language priors, and is therefore left for future work. Datasets. We train and evaluate on Ego-Exo4D (Grauman et al. 2024), Ego4D (Grauman et al. 2022), and EPIC-KITCHENS-100 (Damen et al. 2022), using their official splits. After preprocessing, the EEP training sets contain 224,058224,058, 83,81683,816, and 268,750268,750 examples, respectively, while the corresponding NEP training sets contain 55,94355,943, 23,12723,127, and 67,21767,217 examples. The EEP validation sets contain 56,95456,954, 45,69945,699, and 38,62138,621 examples, and the NEP validation sets contain 15,91015,910, 11,30511,305, and 9,6689,668 examples. Unless otherwise noted, ablations use a randomly sampled subset of 200 Ego-Exo4D validation examples for early-event prediction. We retain each dataset’s native textual supervision: free-form keystep descriptions for Ego-Exo4D, verb–noun action labels for Ego4D, and narrations for EPIC-KITCHENS-100. Metrics. Following prior work (Tan et al. 2025; Liang et al. 2022; Chang et al. 2026), we evaluate open-ended predictions using four metrics: ROUGE-L, METEOR, CIDEr, and BERTScore (Zhang et al. 2020). For next-event prediction, we report Top-5 recall to account for the inherent multiplicity of plausible future outcomes, following Damen et al. 2022; Assran et al. 2025; Mittal et al. 2024. Implementation details. We optimize all models with AdamW using β1=0.9 _1=0.9, β2=0.95 _2=0.95, ϵ=10−15ε=10^-15, and zero weight decay. The learning rate is linearly warmed up for 2,0002,000 steps to 2×10−52× 10^-5 and then held constant. Gradients are clipped to a maximum norm of 1.01.0, and training uses bfloat16 mixed precision. Frames are sampled at 1fps1 \ fps and resized to resolutions between 224224 and 448448 pixels. All models are trained for 20,00020,000 optimization steps. To ensure a controlled comparison with Zebra-CoT (Li et al. 2025a), a representative Visual-CoT baseline built on BAGEL-7B (Deng et al. 2025), we initialize the LLM decoder used for visual understanding and future-state prediction from the corresponding pretrained BAGEL checkpoint. BAGEL adopts a Mixture-of-Transformers architecture with separate LLM decoders for visual understanding and generation, totaling approximately 1414B parameters. Our framework instead uses the Dense and MoE architectures described in Section 2.2. To isolate the effect of individual design choices, the studies in Sections 3.3, 3.4, 3.5, and 3.6 vary one component at a time from a shared configuration. This configuration uses the MoE decoder with prediction horizon H=1H=1, a Flux-VAE target with a flow-matching objective, and a 1:11:1 understanding-to-prediction mixture. All other factors are held fixed within each study. Inference efficiency measurement. We use end-to-end wall-clock latency per sample as the primary efficiency metric. Normalizing latency solely by the number of generated text tokens would not provide a controlled comparison because explicit Visual CoT additionally incurs future-frame generation, image decoding, and generated-frame re-encoding. These operations are not reflected in text-decoding throughput. We profile each method on a single NVIDIA B200 GPU. Measurements use a fixed 200200-example validation subset from each benchmark. Video frames are loaded before timing; model loading, compilation, and the first warm-up example are excluded. We hold the number and resolution of input frames, prompt format, maximum answer length, and text-decoding configuration fixed across methods. We report both mean and p95 latency, since the latter captures tail behavior that is particularly important for time-sensitive prediction. 3.2 Motivation: When Does Explicit Visual CoT Help, and at What Cost? We adopt Zebra-CoT (Li et al. 2025a), a representative Visual CoT baseline, for comparison. Zebra-CoT first generates a plausible future frame as an explicit visual thought and then conditions the final textual prediction on the generated frame. Table 1 compares this pipeline with Answer-Only SFT under a controlled setting using the same backbone and fine-tuning data. To disentangle the value of future visual context from errors introduced by visual generation, we additionally evaluate an oracle-frame variant: at the same model checkpoint, the generated frame is replaced with the corresponding ground-truth future frame. Although this oracle is not available in practice, it provides a diagnostic upper bound on how effectively the model can exploit accurate future visual evidence. Table 1: Accuracy–efficiency trade-off of Visual CoT. For each task, we report performance averaged across the validation subsets of Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100. Conditioning on a generated future frame improves early-event prediction but provides limited benefit for next-event prediction, while substantially increasing mean inference latency. Replacing the generated frame with the ground-truth future frame improves both tasks, revealing a considerably higher upper bound when accurate future visual context is available. This gap suggests that the effectiveness of Visual CoT depends strongly on the fidelity of the generated future state, as intermediate images that fail to preserve plausible scene dynamics or task-relevant visual information may provide weak or misleading conditioning signals. Latency (↓ ) Answer Quality (↑ ) Task Method Mean P95 ROUGE-L BERTScore METEOR CIDEr Early Event Pred. Answer-Only SFT 1.07 1.79 34.4 41.3 24.3 1.43 Visual CoT (generated frame) 6.56 (6.2×) 8.72 (4.9×) 36.4 (++5.9%) 43.2 (++4.5%) 29.8 (++22.4%) 1.69 (++17.9%) Visual CoT (oracle frame) — — 41.9 (++21.6%) 48.3 (++17.0%) 34.2 (++40.6%) 2.05 (++43.4%) Next Event Pred. Answer-Only SFT 1.32 2.05 47.2 50.2 35.3 2.14 Visual CoT (generated frame) 6.55 (5.0×) 7.12 (3.5×) 46.8 (−-0.8%) 49.9 (−-0.7%) 36.9 (++4.3%) 2.13 (−-0.6%) Visual CoT (oracle frame) — — 50.4 (++6.8%) 52.5 (++4.6%) 40.4 (++14.2%) 2.38 (++11.3%) Observation 1. For early-event prediction, Visual CoT improves all four metrics, including ROUGE-L by 5.9%5.9\% and CIDEr by 17.9%17.9\%. For next-event prediction, however, its effect is negligible or negative on three of the four metrics, with only METEOR improving by 4.3%4.3\%. These modest and inconsistent gains come with substantial computational overhead. Specifically, mean latency increases by 6.2×6.2× for early-event prediction and 5.0×5.0× for next-event prediction. In contrast, conditioning the same model on the ground-truth future frame improves every metric on both tasks, yielding ROUGE-L gains of 21.6%21.6\% for early-event prediction and 6.8%6.8\% for next-event prediction. Takeaway 1. The primary bottleneck of explicit Visual CoT is not the value of future visual context, but the cost and fidelity of externalizing that context as pixels. Accurate future frames substantially improve both early- and next-event prediction, confirming that future visual state contains useful evidence for proactive reasoning. However, self-generated frames recover only a fraction of this benefit for early-event prediction and provide no consistent gain for next-event prediction, where the target event lies entirely beyond the observed video prefix. At the same time, generating and reprocessing the intermediate frame increases mean inference latency by approximately 55–6×6×. Those experiments identify a promising alternative by using future-state supervision as a training signal while removing pixel-space visual generation from the inference path. 3.3 Which Future Representation Should the Model Predict? We first investigate what the model should predict as its future-state supervision. Figure 2 compares four target representations: reconstruction-oriented latents from Flux-VAE; semantic features from DINOv2; and features from a SigLIP2-style ViT encoder that is either jointly optimized with the text-generation branch (SigLIP2-adaptive) or kept frozen (SigLIP2-frozen). We compare all variants with Answer-Only SFT, which receives no future-embedding supervision. Results are averaged across the three benchmarks for both early-event and next-event prediction. Figure 2: Comparison of different target representations. We report ROUGE-L, BERTScore, METEOR, and CIDEr, averaged across the three benchmarks for early-event and next-event prediction. The dashed line denotes the Answer-Only SFT baseline. Predicting a Flux-VAE latent consistently achieves the strongest performance across all metrics and both tasks, whereas semantic targets provide smaller and less consistent gains. Observation 2. The choice of target representation has a substantial effect on whether future-state prediction benefits downstream reasoning. Flux-VAE brings substantial performance improvements on both tasks. In contrast, the semantic targets yield smaller and less consistent gains. DINOv2 improves three of the four early-event metrics and all four next-event metrics, but remains consistently below Flux-VAE. The adaptive SigLIP2 target generally outperforms its frozen counterpart. Takeaway 2. The effectiveness of future-state supervision depends on the choice of target representation. Both Flux-VAE and DINOv2 targets improve over Answer-Only SFT, while Flux-VAE delivers the strongest and most consistent gains across tasks and metrics. This suggests that the benefit of future-state prediction depends not simply on adding an auxiliary visual objective, but on whether the target representation exposes structure that the model can use to anticipate how the scene will evolve. DINOv2 features remain beneficial, indicating that high-level semantic structure alone can still provide useful predictive supervision. 3.4 How Much Predictive Supervision Is Needed? We next examine how frequently future-prediction examples should be interleaved with understanding examples during joint training. Figure 3 compares three understanding-to-prediction sampling ratios: 1:11:1, 3:13:1, and 5:15:1, where larger ratios allocate a greater fraction of updates to the understanding objective. This ablation tests whether next-embedding prediction is effective as occasional auxiliary supervision or instead requires sustained exposure throughout training. Figure 3: Effect of the understanding-to-prediction data-mixture ratio on Ego-Exo4D early-event prediction. We report performance throughout training for three sampling ratios. Understanding-heavy mixtures (3:13:1 and 5:15:1) improve more rapidly at early checkpoints but subsequently plateau or decline. In contrast, the balanced 1:11:1 mixture improves steadily and achieves the strongest result on all four metrics at 20k steps. Observation 3. The data-mixture ratio changes not only final performance but also the optimization trajectory. The understanding-heavy 3:13:1 and 5:15:1 mixtures perform better at the earliest checkpoints, suggesting that allocating more updates to answer supervision accelerates initial task adaptation. However, both configurations converge after approximately 12–18k steps and subsequently fluctuate or decline. Their final performance remains close to, or below, their early peaks. The balanced 1:11:1 mixture exhibits the opposite pattern. Although it begins below the understanding-heavy variants, it continues to improve across later checkpoints and achieves the best result on every metric at 20k steps, including ROUGE-L of 34.534.5, METEOR of 27.427.4, CIDEr of 2.022.02, and BERTScore of 36.536.5. Takeaway 3. Future-state prediction should be treated as a primary training objective rather than a sparsely applied auxiliary regularizer. Oversampling understanding examples yields faster early progress, but this advantage is transient. Reducing the frequency of predictive updates causes performance to saturate well below that of the balanced mixture. The 1:11:1 result suggests that sustained optimization of both objectives is necessary for the predictive signal to influence the representations used for proactive reasoning, rather than being overwritten by task-specific answer supervision. We therefore adopt a balanced 1:11:1 understanding-to-prediction ratio in the remaining experiments. 3.5 How Should Future Representations Be Predicted? Having identified the target representation and data-mixture ratio, we next examine how future-state prediction should be supervised. We compare two predictive objectives. The first is rectified-flow matching, in which the prediction head estimates the velocity field from a noisy intermediate state toward the target representation at a sampled flow timestep. The second is direct feature regression, in which the model deterministically predicts the clean target representation. Figure 4 compares these objectives for the VAE and DINOv2 targets. Figure 4: Comparison of predictive objectives on Ego-Exo4D early-event prediction. We compare rectified-flow matching and direct feature regression for VAE and DINOv2 future-state targets. Direct regression substantially outperforms flow matching for DINOv2 and is also competitive with flow matching for VAE across most checkpoints and metrics. Observation 4. The preferred predictive objective depends strongly on the target representation. For DINOv2, direct regression provides a consistent advantage over flow matching across all four metrics. For the VAE target, the difference is considerably smaller. Direct regression leads throughout training on ROUGE-L and CIDEr and at most checkpoints on BERTScore and METEOR. Flow matching becomes competitive at the final checkpoint, slightly surpassing regression on BERTScore (36.536.5 vs. 34.734.7) and METEOR (27.427.4 vs. 27.227.2). These results reveal an interaction between the representation and the predictive objective. Direct regression is decisively preferable for DINOv2, whereas VAE latents can be learned effectively with either objective. Takeaway 4. The target representation and predictive objective should be designed jointly rather than optimized in isolation. Direct regression substantially outperforms flow matching for DINOv2 and achieves similar performance to flow matching for VAE. Notably, although BAGEL performs visual generation in VAE latent space using rectified flow, direct regression achieves similar downstream reasoning performance. This suggests that the most suitable predictive objective is determined less by the formulation originally used to train the visual-generation branch and more by the properties of the target representation itself, such as its scale, normalization, dimensionality, and predictability from the observed context. 3.6 Which Training Curriculum Best Integrates Predictive Supervision? We next examine whether future-state prediction should be introduced as a separate training stage or optimized jointly with the downstream task. We compare three curricula on Ego-Exo4D early-event prediction: (i) Answer-Only SFT, which optimizes only the textual answer objective; (i) Two-Stage Training, which first performs next-embedding prediction for 10k steps and then fine-tunes exclusively on the understanding task; and (i) Joint Training, which maintains both the next-embedding-prediction and answer-generation objectives throughout post-training. Table 2: Joint optimization is more effective than sequential training. We report the results on Ego-Exo4D early-event prediction. Relative changes are computed against the Answer-Only SFT baseline. Two-stage training underperforms Answer-Only SFT on every metric, whereas joint training achieves the strongest result across all metrics. Best value in each column is shown in bold. Method ROUGE-L BERTScore METEOR CIDEr Answer-Only SFT 30.6 32.5 22.6 1.60 Two-Stage Training 28.6 (−-6.5%) 30.7 (−-5.6%) 21.3 (−-5.5%) 1.54 (−-3.5%) Joint Training 34.5 (++12.6%) 36.5 (++12.4%) 27.4 (++21.4%) 2.02 (++26.3%) Observation 5. Joint training achieves the strongest performance on every metric, improving over Answer-Only SFT by 12.6%12.6\% in ROUGE-L and 26.3%26.3\% in CIDEr. In contrast, two-stage training underperforms Answer-Only SFT on every metric by 6.5%6.5\% in ROUGE-L, 5.6%5.6\% in BERTScore, 5.5%5.5\% in METEOR and 3.5%3.5\% in CIDEr. Therefore, predictive training followed by task-only fine-tuning is worse than no predictive training. Takeaway 5. Next visual state prediction is most effective when it is jointly trained with text generation throughout post-training. Joint optimization allows predictive representations to remain directly aligned with the downstream language objective, whereas separating the objectives into sequential stages substantially weakens their transfer. Thus, the benefit of next-embedding prediction should not be interpreted as a generic predictive training stage that can simply precede task adaptation. It arises from continuously coupling visual-dynamics supervision with the representations used to generate the answer. 3.7 How Should Future Prediction Be Structured? We finally examine whether future-state prediction and language understanding should be optimized through shared or task-specific decoder parameters. Figure 5 compares two architectures on Ego-Exo4D early-event prediction: a Dense model, in which both objectives update a single shared decoder, and a Mixture-of-Experts (MoE) model, in which predictive and language objectives are assigned to separate experts. We evaluate both architectures across different prediction horizons to determine how parameter sharing interacts with the amount of future-state supervision. Figure 5: Interaction between decoder architecture and prediction horizon on Ego-Exo4D early-event prediction. We report the experimental results for each prediction horizon H, defined as the number of predicted future frames. The dashed line denotes the Answer-Only SFT baseline. Both architectures outperform it across all horizons. The shared Dense decoder performs best at short horizons, whereas the MoE architecture is comparatively less sensitive to horizon and reaches its strongest results around H=3H=3. Observation 6. Both architectures outperform Answer-Only SFT across all four metrics and every evaluated horizon. However, the magnitude of this benefit depends strongly on the architecture. At short horizons (H=1,2H=1,2), the shared Dense model consistently outperforms MoE on all four metrics. For example, at H=2H=2, Dense achieves 37.437.4 ROUGE-L, compared with 33.833.8 for MoE. The two architectures also exhibit different sensitivity to prediction horizon. Dense performs best with one or two predicted frames and subsequently declines as H increases. MoE is comparatively flatter: its performance generally peaks around H=3H=3 and changes only modestly between H=3H=3 and H=4H=4. This interaction suggests that dense parameter sharing is particularly effective when the predictive target is sufficiently constrained, but becomes less advantageous as the prediction task grows more uncertain. Takeaway 6. The dense architecture most effectively transfers predictive supervision to language reasoning when the future target is short and well constrained. A shared decoder allows gradients from next-embedding prediction to directly shape the representations used for answer generation, which is consistent with its clear advantage at H=1H=1 and H=2H=2. As the horizon increases, however, future prediction becomes more ambiguous, and tightly coupling this objective to the language decoder may introduce less task-aligned supervision. Separating the objectives through MoE appears to reduce this sensitivity, although it does not match the strongest short-horizon Dense configuration. 3.8 Main Results: Quality and Efficiency Table 3: Results on the full Ego-Exo4D validation set. We evaluate early-event prediction with top-1 decoding and next-event prediction with top-5 decoding using ROUGE-L, BERTScore, METEOR, and CIDEr. Answer-Only SFT uses the same matched backbone and training setup as IVT but removes future-state prediction, retaining only the textual prediction objective. External models are included as reference points, while bold and underlined values denote the best and second-best results within the controlled BAGEL-7B-based block. Early Event Pred. (top-1) Next Event Pred. (top-5) Method ROUGE-L BERTScore METEOR CIDEr ROUGE-L BERTScore METEOR CIDEr External models fine-tuned on the same data LLaVA-NeXT-Video-7B 24.3 5.5 17.4 1.11 44.2 42.3 33.7 2.40 VideoLLaMA3-7B 14.6 17.4 7.7 0.31 30.2 26.3 19.6 1.30 Qwen2.5-VL-7B 34.9 36.0 26.6 1.75 48.0 45.9 37.9 2.77 Qwen3-VL-8B 37.0 38.1 28.5 1.90 52.1 49.5 42.0 3.09 BAGEL-7B-based methods Answer-Only SFT 28.8 29.8 21.3 1.40 46.3 45.0 35.6 2.59 Text-CoT 24.6 26.1 17.1 1.10 32.6 33.0 22.0 1.48 Visual-CoT 32.7 33.2 25.4 1.69 42.6 41.1 31.6 2.22 IVT (Ours, MoE) 33.3 34.2 25.7 1.74 48.5 46.6 37.9 2.75 IVT (Ours, Dense) 33.2 34.0 25.5 1.72 48.2 46.8 37.8 2.76 Table 4: Results on the full Ego4D validation set. We evaluate early-event prediction with top-1 decoding and next-event prediction with top-5 decoding using ROUGE-L, BERTScore, METEOR, and CIDEr. Answer-Only SFT uses the same matched backbone and training setup as IVT but removes future-state prediction, retaining only the textual prediction objective. External models are included as reference points, while bold and underlined values denote the best and second-best results within the controlled BAGEL-7B-based block. Early Event Pred. (top-1) Next Event Pred. (top-5) Method ROUGE-L BERTScore METEOR CIDEr ROUGE-L BERTScore METEOR CIDEr External models fine-tuned on the same data LLaVA-NeXT-Video-7B 32.9 33.4 22.3 0.98 52.6 63.9 39.5 1.91 VideoLLaMA3-7B 25.5 41.5 15.3 0.57 41.7 53.1 28.5 1.27 Qwen2.5-VL-7B 33.1 49.5 22.3 0.97 51.1 62.5 37.8 1.80 Qwen3-VL-8B 34.3 50.5 23.7 1.06 52.5 62.1 38.9 1.83 BAGEL-7B-based methods Answer-Only SFT 33.3 49.2 22.6 0.99 51.9 63.7 39.0 1.87 Text-CoT 31.7 50.6 21.3 0.98 40.9 55.6 27.8 1.29 Visual-CoT 36.1 52.6 25.1 1.13 52.2 63.0 39.3 1.88 IVT (Ours, MoE) 34.0 50.4 23.2 1.03 52.5 64.4 39.5 1.90 IVT (Ours, Dense) 34.8 50.8 23.8 1.07 52.7 64.5 39.7 1.90 Table 5: Results on the full EPIC-KITCHENS-100 validation set. We evaluate early-event prediction with top-1 decoding and next-event prediction with top-5 decoding using ROUGE-L, BERTScore, METEOR, and CIDEr. Answer-Only SFT uses the same matched backbone and training setup as IVT but removes future-state prediction, retaining only the textual prediction objective. External models are included as reference points, while bold and underlined values denote the best and second-best results within the controlled BAGEL-7B-based block. Early Event Pred. (top-1) Next Event Pred. (top-5) Method ROUGE-L BERTScore METEOR CIDEr ROUGE-L BERTScore METEOR CIDEr External models fine-tuned on the same data LLaVA-NeXT-Video-7B 36.8 30.3 27.2 1.49 46.3 46.3 33.1 1.98 VideoLLaMA3-7B 15.6 19.9 9.8 0.44 24.3 22.2 13.3 0.63 Qwen2.5-VL-7B 35.8 38.6 26.0 1.39 49.2 47.9 35.6 2.12 Qwen3-VL-8B 39.5 41.9 29.1 1.59 52.9 51.8 39.7 2.37 BAGEL-7B-based methods Answer-Only SFT 36.5 40.0 26.7 1.45 45.5 46.4 32.5 1.90 Text-CoT 26.8 31.2 19.0 0.92 32.3 33.7 20.6 1.12 Visual-CoT 40.2 42.4 29.8 1.67 42.8 44.0 30.0 1.75 IVT (Ours, MoE) 39.0 42.3 29.0 1.59 49.1 48.9 35.5 2.08 IVT (Ours, Dense) 39.0 41.9 28.9 1.59 48.3 48.4 34.5 2.01 Tables 4–5 compare IVT with four representative open-source video MLLMs: LLaVA-NeXT-Video-7B (Zhang et al. 2024), VideoLLaMA3-7B (Zhang et al. 2025), Qwen2.5-VL-7B (Bai et al. 2025b), and Qwen3-VL-8B (Bai et al. 2025a). The Qwen models provide strong general-purpose vision-language foundation-model baselines, while LLaVA-NeXT-Video and VideoLLaMA3 represent video-specialized MLLMs. All models are fine-tuned on the corresponding target dataset. Because the external models differ in backbone, pretraining data, and optimization recipe, we treat them as reference points rather than as strictly controlled comparisons. Within the BAGEL-7B-based block, Answer-Only SFT denotes the matched backbone trained only with the textual prediction objective, without future-state supervision. Text CoT, Visual CoT, and IVT share the same initialization for the understanding branch. Accordingly, bold and underlined values denote the best and second-best results only within this controlled BAGEL-7B-based comparison. For the Text CoT baseline, we use Qwen3-VL-30B-A3B-Thinking (Bai et al. 2025a) as a teacher to generate intermediate reasoning traces conditioned on the input frames and ground-truth event descriptions. These traces are then used as supervision for post-training the BAGEL-7B-based Text CoT model. For IVT, we use the configurations selected from the architecture–horizon study in Figure 5, with H=1H=1 for Dense and H=3H=3 for MoE. We use Flux-VAE as the target representation, optimize it with the flow-matching objective, and set the understanding-to-prediction data-mixture ratio to 1:11:1. These configurations are fixed across all six dataset–task settings. Observation 7. Relative to the Answer-Only SFT baseline, Visual CoT improves four of the six dataset–task settings, while Text CoT generally degrades performance. In contrast, IVT improves all four metrics across all six settings, showing consistent gains for both partially observed actions and future events that have not yet begun. Compared with Visual CoT, IVT performs better in four of the six settings and wins on all three next-event prediction benchmarks. Visual CoT remains stronger on two early-event prediction benchmarks, exceeding IVT by 1.31.3 ROUGE-L points on Ego4D and 1.21.2 points on EPIC-KITCHENS-100. Compared with external baselines, IVT is competitive with state-of-the-art models at the 7B scale, while still trailing the larger Qwen3-VL-8B model. Takeaway 7. IVT improves the quality–efficiency frontier by moving future-state prediction from inference-time computation to training-time supervision. Unlike explicit Visual CoT, IVT retains the Answer-Only SFT inference pathway while achieving stronger overall performance. The advantage is particularly clear for next-event prediction, where IVT improves the average ROUGE-L from 45.945.9 to 49.749.7 over Visual CoT across the three benchmarks. These results suggest that explicit future-frame generation is not necessary to obtain the benefits of visual foresight. Instead, predictive world modeling can be internalized during training to support more accurate and efficient proactive reasoning. 4 Related Work Proactive Video Reasoning. Proactive video reasoning seeks to infer ongoing or future events, underlying goals, and plausible outcomes from incomplete visual observations. Following the definition in Zhao and Wildes 2021, proactive video reasoning comprises two primary tasks. Early-event prediction requires recognizing an ongoing action before it is completed (Sadegh Aliakbarian et al. 2017; Wang et al. 2019; Stergiou and Damen 2023), whereas next-event prediction requires anticipating an action that has not yet begun and may occur several seconds into the future (Furnari and Farinella 2020; Girdhar and Grauman 2021; Gong et al. 2022). More recent approaches incorporate the semantic and commonsense priors of large language models to generate plausible long-horizon futures (Mittal et al. 2024; Zhao et al. 2024). Beyond predicting what will happen, ProactiveVideoQA (Wang et al. 2025b) evaluates whether video MLLMs can determine when to respond during streaming video, extending evaluation from response correctness to temporally appropriate interaction. Wang et al. 2025a formulate next-event prediction as a self-supervised video-to-text objective in which an MLLM observes the first part of a video and predicts a textual description of its unseen continuation. Their analysis shows that supervised fine-tuning provides a strong and efficient baseline, whereas reinforcement learning substantially improves performance on the targeted future-reasoning benchmark but can reduce general video-understanding performance. Video-as-Answer (Cheng et al. 2026) further extends next-event prediction from textual descriptions to generated future videos, directly visualizing the predicted continuation. These methods demonstrate the value of predictive supervision but express the anticipated future through language or computationally expensive visual outputs. In contrast, our method learns future dynamics internally by jointly predicting the next latent visual representation and the future textual state. Visual Chain-of-Thought. Chain-of-thought reasoning (Wei et al. 2022) has been extended beyond language by introducing explicit visual intermediates into the reasoning trajectory. Visual CoT (Shao et al. 2024) identifies and revisits question-relevant regions through intermediate bounding boxes; Visual Sketchpad (Hu et al. 2024) enables models to draw auxiliary visual marks and invoke specialist vision tools; and Visualization-of-Thought methods externalize intermediate spatial states as textual or generated visualizations (Wu et al. 2024; Li et al. 2025c). For video reasoning, Video-of-Thought (Fei et al. 2024) structures reasoning from fine-grained spatiotemporal perception to high-level cognition using scene-graph representations. Zebra-CoT (Li et al. 2025a) provides large-scale interleaved image–text reasoning trajectories for training unified multimodal models, while CoT-VLA (Zhao et al. 2025) predicts future images as visual subgoals before decoding robotic actions. Although explicit visual reasoning can improve spatial, temporal, and embodied reasoning, it introduces substantial inference overhead through image-token decoding, repeated visual-model calls, or both. Our approach addresses this limitation in the temporal domain. Rather than generating images, sketches, or other explicit visual thoughts, IVT internalizes predictive visual reasoning within the model’s latent computation, retaining the benefits of visual imagination while substantially reducing the inference cost. Latent Visual Reasoning and Next-Embedding Prediction. Joint-embedding predictive architectures demonstrate that predicting semantic representations in latent space can provide a more abstract learning signal than reconstructing raw pixels (Assran et al. 2023; Bardes et al. 2024; Assran et al. 2025; Xu et al. 2025a; Li et al. 2025b; Chen et al. 2025; Zhu et al. 2023; Wang et al. 2025a; Tong et al. 2026). I-JEPA (Assran et al. 2023) predicts masked image representations from visible context, while V-JEPA (Bardes et al. 2024) extends this principle to video by predicting masked spatiotemporal representations. V-JEPA 2 (Assran et al. 2025) further scales latent video prediction to large-scale video understanding, action anticipation, and action-conditioned planning, demonstrating that predictive representations can encode physical dynamics without explicitly synthesizing future pixels. Related ideas have recently been incorporated into vision and vision-language models. NEPA (Xu et al. 2025a) trains causal vision transformers solely through next-embedding prediction. Collectively, these studies show that embedding-space prediction can support visual representation learning, multimodal reasoning, and efficient vision-language inference. Our approach differs from these methods by jointly predicting the next latent visual representation and the next textual event state during MLLM post-training, explicitly coupling visual dynamics with semantic anticipation. Unified Understanding and Generation Models. Unified multimodal models seek to support visual understanding and generation within a shared architecture (Sun et al. 2024; Chameleon Team 2024; Xie et al. 2025; Zhu et al. 2024; Zhou et al. 2025; Wu et al. 2025a; Tian et al. 2026; Tian et al. 2025). Emu (Sun et al. 2024) and Chameleon (Chameleon Team 2024) autoregressively model interleaved image and text sequences, while Show-o (Xie et al. 2025) combines autoregressive text modeling with discrete visual generation in a single transformer. Transfusion (Zhou et al. 2025) instead combines next-token prediction for language with diffusion-based prediction for continuous visual representations. Janus (Wu et al. 2025a) further shows that separating the visual encoding pathways for understanding and generation alleviates representational conflicts while retaining a shared multimodal transformer. These models primarily employ their generative pathways to produce observable visual outputs. Our objective is fundamentally different: we use the visual generation pathway as an internal predictive mechanism. Specifically, IVT forecasts future visual representations from partial video evidence and uses predictive supervision to improve anticipatory language reasoning, without decoding them into images at inference time. 5 Conclusion We introduced Internalized Visual Thinking (IVT), a post-training framework for proactive video reasoning that learns to predict future visual states during training without explicitly generating them at inference time. By jointly supervising future latent representations and textual predictions, IVT transfers visual foresight from an expensive intermediate generation process into the model’s internal representations. Across early-event and next-event prediction benchmarks, IVT consistently improves over text-only post-training and, under matched settings, outperforms Visual CoT on four of six splits while reducing average end-to-end inference latency by more than 5×5×. These results show that the benefits of visual reasoning do not necessarily require rendering visual thoughts explicitly. Our analysis further shows that future-state supervision is not a universally beneficial objective. Its effectiveness depends critically on how predictive learning is integrated into the model, including the target representation, decoder architecture, prediction horizon, balance between understanding and prediction data, predictive objective, and training curriculum. These findings suggest that the key role of visual generation for reasoning may lie less in producing visually realistic intermediate images and more in providing structured predictive supervision that encourages the model to represent how the visual world may evolve. More broadly, IVT points toward a different way of designing multimodal reasoning systems. Rather than externalizing every intermediate computation into language or pixels, models can acquire rich predictive structure during training and internalize that structure for efficient inference. For proactive reasoning, where both foresight and latency are essential, this work provides a promising path toward multimodal systems that are simultaneously predictive, grounded, and efficient. Acknowledgment We sincerely thank Mingfei Gao, Anshul Shah, Jiqi Yang, and Rosie Zhao for the helpful discussions. References Assran et al. (2023) Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15619–15629, 2023. Assran et al. (2025) Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. Bai et al. (2025a) Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a. Bai et al. (2025b) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025b. URL https://arxiv.org/abs/2502.13923. Bardes et al. (2024) Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024. Black Forest Labs (2024) Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024. Chameleon Team (2024) Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. Chang et al. (2026) Boyu Chang, Qi Wang, Xi Guo, Zhixiong Nan, Yazhou Yao, and Tianfei Zhou. Abductivemllm: Boosting visual abductive reasoning within mllms. arXiv preprint arXiv:2601.02771, 2026. Chen et al. (2025) Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Yejin Bang, Allen Bolourchi, Yann LeCun, and Pascale Fung. Vl-jepa: Joint embedding predictive architecture for vision-language. arXiv preprint arXiv:2512.10942, 2025. Cheng et al. (2026) Junhao Cheng, Liang Hou, Xin Tao, and Jing Liao. Video-as-answer: Predict and generate next video event with joint-grpo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 38915–38925, 2026. Chern et al. (2025) Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu. Thinking with generated images. arXiv preprint arXiv:2505.22525, 2025. Damen et al. (2022) Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, 130(1):33–55, 2022. Deng et al. (2025) Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025. Fei et al. (2024) Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint arXiv:2501.03230, 2024. Furnari and Farinella (2020) Antonino Furnari and Giovanni Maria Farinella. Rolling-unrolling lstms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence, 43(11):4021–4036, 2020. Girdhar and Grauman (2021) Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13505–13515, 2021. Gong et al. (2022) Dayoung Gong, Joonseok Lee, Manjin Kim, Seong Jong Ha, and Minsu Cho. Future transformer for long-term action anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3052–3061, 2022. Grauman et al. (2022) Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022. Grauman et al. (2024) Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024. Hu et al. (2024) Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. Advances in Neural Information Processing Systems, 37:139348–139379, 2024. Kancheti et al. (2026) Sai Srinivas Kancheti, Aditya Sanjiv Kanade, Vineeth N Balasubramanian, and Tanuja Ganu. Chain-of-thought degrades visual spatial reasoning capabilities of multimodal llms. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 862–876, 2026. Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. Li et al. (2025a) Ang Li, Charles Wang, Deqing Fu, Kaiyu Yue, Zikui Cai, Wang Bill Zhu, Ollie Liu, Peng Guo, Willie Neiswanger, Furong Huang, et al. Zebra-cot: A dataset for interleaved vision language reasoning. arXiv preprint arXiv:2507.16746, 2025a. Li et al. (2025b) Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen, and Zicheng Liu. Latent visual reasoning. arXiv preprint arXiv:2509.24251, 2025b. Li et al. (2025c) Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542, 2025c. Liang et al. (2022) Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. Visual abductive reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15565–15575, 2022. Mittal et al. (2024) Himangi Mittal, Nakul Agarwal, Shao-Yuan Lo, and Kwonjoon Lee. Can’t make an omelette without breaking some eggs: Plausible action anticipation using large video-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18580–18590, 2024. Oquab et al. (2023) Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. Sadegh Aliakbarian et al. (2017) Mohammad Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, and Lars Andersson. Encouraging lstms to anticipate actions very early. In Proceedings of the IEEE international conference on computer vision, pages 280–289, 2017. Shao et al. (2024) Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612–8642, 2024. Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European conference on computer vision, pages 510–526. Springer, 2016. Stergiou and Damen (2023) Alexandros Stergiou and Dima Damen. The wisdom of crowds: Temporal progressive attention for early action prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14709–14719, 2023. Sun et al. (2024) Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In International conference on learning representations, volume 2024, pages 12352–12380, 2024. Tan et al. (2025) Kuo Tan, Zhaobo Qi, Jianping Zhong, Yuanrong Xu, and Weigang Zhang. Kn-vlm: Knowledge-guided vision-and-language model for visual abductive reasoning. Multimedia Systems, 31(2):146, 2025. Tian et al. (2025) Rui Tian, Mingfei Gao, Haiming Gang, Jiasen Lu, Zhe Gan, Yinfei Yang, Zuxuan Wu, and Afshin Dehghan. Unigen-1.5: Enhancing image generation and editing through reward unification in reinforcement learning. arXiv preprint arXiv:2511.14760, 2025. Tian et al. (2026) Rui Tian, Mingfei Gao, Mingze Xu, Jiaming Hu, Jiasen Lu, Zuxuan Wu, Yinfei Yang, and Afshin Dehghan. Unigen: Enhanced training & test-time strategies for unified multimodal understanding and generation. Advances in Neural Information Processing Systems, 38:152386–152415, 2026. Tong et al. (2026) Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, et al. Beyond language modeling: An exploration of multimodal pretraining. arXiv preprint arXiv:2603.03276, 2026. Tschannen et al. (2025) Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Wang et al. (2025a) Haonan Wang, Hongfu Liu, Xiangyan Liu, Chao Du, Kenji Kawaguchi, Ye Wang, and Tianyu Pang. Fostering video reasoning via next-event prediction. arXiv preprint arXiv:2505.22457, 2025a. Wang et al. (2019) Xionghui Wang, Jian-Fang Hu, Jian-Huang Lai, Jianguo Zhang, and Wei-Shi Zheng. Progressive teacher-student learning for early action prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3556–3565, 2019. Wang et al. (2025b) Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, and Dongyan Zhao. Proactivevideoqa: A comprehensive benchmark evaluating proactive interactions in video large language models. arXiv preprint arXiv:2507.09313, 2025b. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Wen et al. (2026) Zimo Wen, Boxiu Li, Wanbo Zhang, Junxiang Lei, Xiaoyu Chen, Yijia Fan, Qi Zhang, Yujiang Wang, Lili Qiu, Bo Li, et al. Unig2u-bench: Do unified models advance multimodal understanding? arXiv preprint arXiv:2603.03241, 2026. Wu et al. (2025a) Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12966–12977, 2025a. Wu et al. (2026) Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Rongrong Ji. Grounded chain-of-thought for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 33577–33587, 2026. Wu et al. (2024) Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large language models. Advances in Neural Information Processing Systems, 37:90277–90317, 2024. Wu et al. (2025b) Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266, 2025b. Xie et al. (2025) Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In International Conference on Learning Representations, volume 2025, pages 28240–28264, 2025. Xu et al. (2025a) Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, Weiyang Jin, Joyce Chai, Saining Xie, and Stella X Yu. Next-embedding prediction makes strong vision learners. arXiv preprint arXiv:2512.16922, 2025a. Xu et al. (2025b) Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less, 2025. URL https://arxiv. org/abs/2502.18600, 2025b. Zhang et al. (2025) Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. Zhang et al. (2024) Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. URL https://llava-vl.github.io/blog/2024-04-30-llava-next-video/. Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. Zhao and Wildes (2021) He Zhao and Richard P Wildes. Review of video predictive understanding: Early action recognition and future action prediction. arXiv preprint arXiv:2107.05140, 2021. Zhao et al. (2024) Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-term action anticipation from videos? In International Conference on Learning Representations, volume 2024, pages 56677–56697, 2024. Zhao et al. (2025) Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1702–1713, 2025. Zhao et al. (2026) Rosie Zhao, Anshul Shah, Xiaoyu Zhu, Xinke Deng, Zhongyu Jiang, Yang Yang, Joerg Liebelt, and Arnab Kumar Mondal. On robustness and chain-of-thought consistency of rl-finetuned vlms. In Forty-third International Conference on Machine Learning, 2026. Zhou et al. (2025) Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. In International Conference on Learning Representations, volume 2025, pages 6446–6469, 2025. Zhou et al. (2024) Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. Image-of-thought prompting for visual reasoning refinement in multimodal large language models. arXiv preprint arXiv:2405.13872, 2024. Zhu et al. (2023) Xiaoyu Zhu, Po-Yao Huang, Junwei Liang, Celso M De Melo, and Alexander G Hauptmann. Stmt: A spatial-temporal mesh transformer for mocap-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1526–1536, 2023. Zhu et al. (2024) Xiaoyu Zhu, Hao Zhou, Pengfei Xing, Long Zhao, Hao Xu, Junwei Liang, Alexander Hauptmann, Ting Liu, and Andrew Gallagher. Open-vocabulary 3d semantic segmentation with text-to-image diffusion models. In European Conference on Computer Vision, pages 357–375. Springer, 2024. Appendix A Training Algorithms Algorithms 1 and 2 detail the training procedures for IVT and the two-stage baseline, respectively. For IVT, the textual and predictive objectives are computed within the same forward pass, with disjoint supervision masks identifying answer and future-prediction positions. The data-mixture ratio controls the relative frequency of understanding and prediction examples in the packed training data. Predictive targets are detached through sg(⋅)sg(·), so gradients from ℒpredL_pred do not propagate through the target branch. Frozen target encoders, including Flux-VAE, remain fixed throughout training. Algorithm 1 Joint training for Internalized Visual Thinking 1: Understanding data textD_text, prediction data predD_pred, model parameters θ, target encoder EtarE_tar, predictive-loss weight λpred=1 _pred=1, understanding-to-prediction mixture 1:11:1 2: for training step s=1,…,Ss=1,…,S do 3: Construct a packed batch according to the prescribed data mixture 4: Encode the observed frames and prompts into an interleaved multimodal sequence 5: Perform a single forward pass to obtain answer logits and predictive hidden states 6: Compute the token-normalized text loss ℒtextL_text 7: if the batch contains future-prediction positions then 8: Compute detached targets Z←sg(Etar(If))Z (E_tar(I_f)) 9: Compute ℒpredL_pred using feature regression or rectified-flow matching 10: else 11: ℒpred←0L_pred← 0 12: end if 13: ℒ←ℒtext+λpredℒpredL _text+ _predL_pred 14: Backpropagate ℒL and update θ using AdamW 15: Update the exponential-moving-average (EMA) weights 16: end for 17: return EMA model parameters The two stages use independent optimizer states. Stage 2 initializes the model from the EMA weights obtained in Stage 1 but uses a fresh AdamW optimizer and a newly initialized learning-rate schedule. Algorithm 2 Two-stage predictive training followed by task fine-tuning 1: Prediction data predD_pred, understanding data textD_text 2: Initialize from the same BAGEL-7B-MoT checkpoint used by the other methods 3: for s=1,…,Spred=10ks=1,…,S_pred=10k do 4: Optimize only the predictive objective ℒpredL_pred 5: end for 6: Initialize Stage 2 from the Stage-1 EMA weights 7: Reinitialize AdamW and reset the learning-rate schedule 8: for s=1,…,Stext=20ks=1,…,S_text=20k do 9: Optimize only the textual objective ℒtextL_text 10: end for 11: return final model Appendix B Dataset Construction and Preprocessing For each task family, we jointly train on the native training splits of Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100. We inherit the official train/validation partitions of each dataset. The three training datasets are mutually disjoint by construction. Algorithm 3 summarizes the construction of proactive video reasoning examples. Algorithm 3 Constructing proactive video reasoning examples 1: Annotated video V, task type r, observation ratios ℛ=0.1,0.3,0.5,0.7R=\0.1,0.3,0.5,0.7\, anticipation gap τa=1 _a=1 s, prediction offsets ℋH 2: Observed video prefix X≤tX_≤ t, textual target y, future-frame targets It+hh∈ℋ\I_t+h\_h 3: if r=EarlyEventr= EarlyEvent then 4: Select an action segment (ts,te,a)(t_s,t_e,a) 5: for each ρ∈ℛρ do 6: Set observation boundary t←ts+ρ(te−ts)t← t_s+ρ(t_e-t_s) 7: Sample X≤tX_≤ t through t at 11 fps, using at most 1616 frames 8: Set y to the native action description of a 9: Extract future-frame targets It+hh∈ℋ\I_t+h\_h 10: end for 11: else if r=NextEventr= NextEvent then 12: Select context ending at t and identify the next action anexta_next with onset tnextt_next 13: Enforce t≤tnext−τat≤ t_next- _a 14: Sample X≤tX_≤ t through t at 11 fps 15: Set y to the native description of anexta_next 16: Extract future-frame targets It+hh∈ℋ\I_t+h\_h 17: end if 18: return X≤t,y,It+hh∈ℋX_≤ t,y,\I_t+h\_h Appendix C Implementation Details Table 6 summarizes the training configuration used for the target-representation ablations and the main IVT experiments. Table 6: Training hyperparameters. Default configuration used for the target-representation ablations and main IVT experiments. Architecture and curriculum variants modify only the corresponding design choices described in the main paper. Configuration Value Input resolution 224224–448448 px Optimizer AdamW (β1=0.9 _1=0.9, β2=0.95 _2=0.95, ϵ=10−15ε=10^-15, weight decay 00) Peak learning rate 2×10−52× 10^-5 Warm-up steps 2,0002,000 Learning-rate schedule Constant after warm-up Gradient clipping 1.01.0 Packed tokens / step 36,86436,864 (VAE); 73,72873,728 (DINOv2 / SigLIP2) Gradient accumulation None Precision bfloat16 (autocast) Predictive-loss weight λpred _pred 11 (=mse_weight=ce_weight= mse\_weight= ce\_weight) EMA decay 0.99990.9999 Appendix D Additional Experimental Results D.1 Out-of-Domain Transfer to Charades Table 7: Out-of-domain transfer from Ego-Exo4D to Charades. For IVT, we keep the configurations used in the main results (Tables 4–5), namely H=1H=1 for Dense and H=3H=3 for MoE. Method Acc. (%) External methods Qwen2.5-VL-7B 65.3 Qwen3-VL-8B 64.6 LLaVA-NeXT-Video-7B 69.6 VideoLLaMA3-7B 70.1 BAGEL-7B-based methods Answer-Only SFT 71.7 Text-CoT 71.1 Visual-CoT 58.2 IVT (Ours, MoE, H=3H=3) 71.9 IVT (Ours, Dense, H=1H=1) 73.2 Table 8: Charades transfer across prediction horizons. Out-of-domain accuracy of Dense and MoE IVT across prediction horizons H, with Answer-Only SFT shown as a reference. Dense IVT exceeds Answer-Only SFT across all evaluated horizons and consistently outperforms the corresponding MoE variants. The main experiments evaluate models in-domain following existing practice (Sadegh Aliakbarian et al. 2017; Stergiou and Damen 2023; Mittal et al. 2024), where post-training and evaluation use the same underlying dataset distribution. We additionally examine whether representations learned through future-state prediction remain useful under distribution shift. We hold out Charades (Sigurdsson et al. 2016) entirely from training and directly evaluate models post-trained on Ego-Exo4D early-event prediction. This setting introduces shifts in both visual content and task format. Ego-Exo4D uses open-ended descriptions of partially observed actions, whereas Charades requires selecting an answer option. Performance therefore reflects both transfer to an unseen video distribution and the preservation of instruction-following and output-format capabilities after source-domain post-training. Table 8 shows that IVT achieves the strongest out-of-domain performance among the evaluated BAGEL-7B-based methods. Figure 8 further shows that the transfer advantage of Dense IVT persists across the prediction horizons. D.2 Additional Inference-Efficiency Analysis Algorithm 4 End-to-end inference-latency measurement 1: Model M, evaluation examples ℰE, warm-up count W=1W=1 2: Fix hardware to 1×1×B200, bf16 precision, batch size 11, decoding configuration, and maximum output length 3: Run and discard W warm-up examples 4: for each example e∈ℰe do 5: Load video frames from disk ⊳ excluded from timed region 6: torch.cuda.synchronize() 7: Start wall-clock timer 8: Execute the complete method-specific inference pipeline 9: torch.cuda.synchronize() 10: Record elapsed time ℓe _e 11: end for 12: return mean(ℓe)mean(\ _e\) and P95(ℓe)P95(\ _e\) Our primary efficiency metric is end-to-end wall-clock latency per sample at batch size 11, measured using the protocol in Algorithm 4. All methods are evaluated on identical hardware and precision, with matched input-frame sampling and decoding limits. Table 9 reports the per-benchmark measurements underlying the aggregate results in Table 1 and Section 3.1. Across the six dataset–task settings, Visual CoT requires an average of 6.566.56 s per sample, compared with 1.201.20 s for Answer-Only SFT and 1.221.22 s for IVT. This corresponds to approximately 5.5×5.5× and 5.4×5.4× higher mean latency, respectively. The difference is also substantial in the latency tail: Visual CoT reaches a mean P95 latency of 7.927.92 s, compared with 1.921.92 s for Answer-Only SFT and 1.771.77 s for IVT. In contrast, Answer-Only SFT and IVT exhibit closely matched latency profiles. For Visual CoT, the timed region includes future-frame generation, VAE decoding, visual re-encoding, and final answer generation. For Answer-Only SFT and IVT, it includes input-frame encoding and answer generation. The reported measurements therefore capture the complete method-specific inference pipeline rather than an isolated model forward pass. For next-event prediction, five candidates are generated in a single deterministic beam-search call with num_beams=5 num\_beams=5 and do_sample=False do\_sample= False, so the reported latency covers generation of the complete candidate set. Early-event prediction uses greedy decoding. Table 9: End-to-end inference latency. Per-sample latency is measured on a single B200 GPU with batch size 11 and bf16 precision over 200200 validation examples per benchmark. Mean and P95 latency are reported in seconds. IVT retains an inference profile comparable to Answer-Only SFT, whereas Visual CoT incurs substantially higher latency due to intermediate future-frame generation and visual re-encoding. Answer-Only SFT Visual CoT IVT (Ours) Task Benchmark Mean P95 Mean P95 Mean P95 Early Event Prediction Ego-Exo4D 1.42 2.45 7.08 10.54 1.52 2.34 Ego4D 0.83 1.27 6.38 7.28 0.81 1.12 EPIC-KITCHENS-100 0.95 1.64 6.24 8.34 0.87 1.30 Next Event Prediction Ego-Exo4D 1.37 2.40 7.70 8.62 1.25 1.91 Ego4D 0.92 1.49 3.72 4.01 0.82 0.98 EPIC-KITCHENS-100 1.68 2.26 8.24 8.74 2.07 2.96