Paper deep dive
Mid-Training Language Models on Raw Video
Jaedong Hwang, Xiaoqian Shen, Ernie Chang, Changsheng Zhao, Chong Zhou, Saksham Suri, Qi Qian, Zechun Liu, Lemeng Wu, Qinsi Wang, Raghuraman Krishnamoorthi, Wei Wen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:12:20 AM
Summary
This paper investigates using raw web video as mid-training data for pretrained language models without captions or text loss. The authors mid-train Qwen3-1.7B on 16-frame clips from YT-Temporal-1B by predicting continuous visual tokens. This approach improves video understanding by 2.9 points and image understanding by 5.1 points on average benchmarks, while preserving text performance. The method eliminates the need for expensive captioning pipelines.
Entities (7)
Relation Signals (5)
Qwen3-1.7B → developedby → Meta AI
confidence 95% · Authors affiliated with Meta AI developed the model
Qwen3-1.7B → midtrainedon → YT-Temporal-1B
confidence 95% · We mid-train Qwen3-1.7B on raw clips from YT-Temporal-1B
Qwen3-1.7B → usesvisionbackbone → SigLIP SO400M/14
confidence 92% · We instantiate the visual backbone with SigLIP SO400M/14
Video Mid-Training → improvesperformanceon → EgoSchema
confidence 90% · 4.8-point gain on EgoSchema
Video Mid-Training → improvesperformanceon → ChartQA
confidence 88% · largest gains on visually dense tasks such as ChartQA (+8.2 points)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models learn mostly from paired image-text data or annotated video, and raw web video is rarely used to further train an existing language model. We study whether raw video, with no captions and no text loss, can serve as mid-training data for a pretrained language model. Frames are encoded into continuous visual tokens, and the language model learns to predict the next visual token. We mid-train Qwen3-1.7B on raw clips from YT-Temporal-1B and then apply the same image-text instruction tuning to it and to the model without mid-training, so that the two differ only in mid-training. The mid-trained model scores 2.9 points higher on average across four video benchmarks and 5.1 points higher across ten image benchmarks, spanning perception, document, and chart tasks. Text performance is preserved even though mid-training includes no text, with an average of 48.9 across 14 text benchmarks compared with 48.0 for the model without mid-training. Analyses across training show that the image and video gains emerge within 30% of training and plateau thereafter, varying by less than 0.5 points. Predicting captions fails to outperform next-visual-token prediction, demonstrating that video mid-training can remain purely self-supervised without the computational overhead or labeling noise of automated captioning.
Tags
Links
- Source: https://arxiv.org/abs/2610.11019v1
- Canonical: https://arxiv.org/abs/2610.11019v1
Trouble viewing inline? Open PDF directly →
Full Text
41,630 characters extracted from source content.
Expand or collapse full text
Mid-Training Language Models on Raw Video Jaedong Hwang Affiliation: Meta AI Affiliation: Massachusetts Institute of Technology Work done at Meta Xiaoqian Shen Affiliation: Meta AI Affiliation: KAUST Work done at Meta Ernie Chang Affiliation: Meta AI Changsheng Zhao Affiliation: Meta AI Chong Zhou Affiliation: Meta AI Saksham Suri Affiliation: Meta AI Qi Qian Affiliation: Meta AI Zechun Liu Affiliation: Meta AI Lemeng Wu Affiliation: Meta AI Qinsi Wang Affiliation: Meta AI Affiliation: Duke University Work done at Meta Raghuraman Krishnamoorthi Affiliation: Meta AI Wei Wen Affiliation: Meta AI Abstract Multimodal large language models learn mostly from paired image-text data or annotated video, and raw web video is rarely used to further train an existing language model. We study whether raw video, with no captions and no text loss, can serve as mid-training data for a pretrained language model. Frames are encoded into continuous visual tokens, and the language model learns to predict the next visual token. We mid-train Qwen3-1.7B on raw clips from YT-Temporal-1B and then apply the same image-text instruction tuning to it and to the model without mid-training, so that the two differ only in mid-training. The mid-trained model scores 2.9 points higher on average across four video benchmarks and 5.1 points higher across ten image benchmarks, spanning perception, document, and chart tasks. Text performance is preserved even though mid-training includes no text, with an average of 48.9 across 14 text benchmarks compared with 48.0 for the model without mid-training. Analyses across training show that the image and video gains emerge within 30% of training and plateau thereafter, varying by less than 0.5 points. Predicting captions fails to outperform next-visual-token prediction, demonstrating that video mid-training can remain purely self-supervised without the computational overhead or labeling noise of automated captioning. †date: October 7, 2026†correspondence: Wei Wen at wewen@meta.com†Project Page: https://jd730.github.io/projects/RawVideoMidTraining 1 Introduction Large language models (LLMs) have advanced rapidly across reasoning, coding, and knowledge-intensive tasks (Achiam et al., 2023; Brown et al., 2020; Gemma Team et al., 2025; Yang et al., 2025). These capabilities have been driven in large part by data scaling, with recent models trained on tens of trillions of tokens, well beyond compute-optimal allocations (Hoffmann et al., 2022; Yang et al., 2025). However, training set growth already outpaces human text generation, and the stock of public human text is projected to be fully used in a decade (Villalobos et al., 2024). Scaling the text pretraining corpus, a central driver of progress under neural scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), is therefore approaching fundamental limits. Repeated text yields diminishing returns beyond a few epochs (Muennighoff et al., 2023), and recursive training on synthetic text degrades model quality (Shumailov et al., 2024). Sustaining scaling trends necessitates drawing upon non-text modalities. Among non-text modalities, video is the largest source of data. Its stock is estimated at roughly 1350 trillion text-token equivalents, about 2.6 times the indexed web (Villalobos et al., 2024). Video also suits language model training, since its frames arrive in temporal order like tokens in text and show objects, actions, and scenes that text describes only indirectly. Yet current multimodal models use video mainly through text, pairing sampled frames with captions or question-answer annotations (An et al., 2026; Bai et al., 2025). Captioning web-scale video frame by frame is expensive, and the captions inherit the errors and omissions of the captioning model. This raises the question of whether a pretrained language model can learn from raw video directly, with no captions, by predicting the next visual token. While world models trained on large-scale video show that predicting future observations, in pixel space or in a learned representation space, supports simulation, planning, and control (Assran et al., 2025; Bruce et al., 2024; NVIDIA, 2026), integrating raw video into language model pretraining remains challenging. Prior multimodal efforts either train unified transformers from scratch on text, image, and video tokens (Wang et al., 2026) or add latent future-frame prediction heads over curated video clips (Yang et al., 2026). Neither directly addresses the most practical regime for scaling, namely continued pretraining of an existing, pretrained LLM on raw web video. In this work, we mid-train a pretrained language model on raw video with next-token prediction over visual tokens. Each frame is encoded by a pretrained vision encoder and projected into the language model’s embedding space, turning a clip into a continuous sequence of visual tokens. The language model learns to predict each next visual token from its hidden state, with no captions or discrete visual tokenizer. We train Qwen3-1.7B (Yang et al., 2025) this way on 16-frame clips from YT-Temporal-1B (Zellers et al., 2022) and compare it with a baseline that receives the same instruction tuning without mid-training. Video mid-training raises the average over four video benchmarks by 2.9 points, with a 4.8-point gain on EgoSchema, and the average over ten image benchmarks by 5.1 points. Although mid-training includes no text, the mid-trained model preserves text performance, raising the average text score from 48.0 to 48.9 and keeping every knowledge and commonsense score within 1.4 points of the model without mid-training. Replacing the visual target with a caption of the next frame gives similar video and image scores but lowers the average text score by 1.5 points, and a caption of the current frame leaves the image average at 48.8, within 0.1 points of the baseline. Video mid-training therefore needs no frame-level captions. Our primary contributions are as follows: • We propose mid-training a pretrained language model on raw video by predicting the next visual token, which requires no captions or annotations. • We show that video mid-training improves video understanding by 2.9 points on average (4.8 points on EgoSchema) and image understanding by 5.1 points, while maintaining baseline text performance despite using no text supervision. • We demonstrate that video mid-training eliminates the need for expensive captioning pipelines, as predicting current- or next-frame captions provides no advantage over purely self-supervised next-visual-token prediction. 2 Related Work 2.1 Learning from Video without Language Self-supervised video models are trained by reconstructing masked spatiotemporal patches (Tong et al., 2022), autoregressively generating discrete visual tokens (Yan et al., 2021), or predicting representations in a latent feature space (Assran et al., 2023; Assran et al., 2025). When scaled to internet video, predictive objectives yield rich representations that support physical planning (Assran et al., 2025) as well as generative world models that simulate interactive environments (Bruce et al., 2024; NVIDIA, 2026). However, these models are trained strictly from scratch in the visual domain and typically incorporate language only post hoc through downstream probing or fine-tuning. In contrast, rather than training a standalone video backbone, our objective is to investigate what an already-competent language model acquires when trained directly on continuous visual streams. 2.2 Video as Training Data for Language Models An emerging line of research explores video as pretraining data for language architectures. Tong et al. (2026) pretrain a transformer from scratch on a joint mixture of text and raw video, finding that adding video keeps text perplexity close to text-only pretraining, lower on in-distribution web text but higher on an out-of-distribution corpus. However, their setup trains jointly from scratch with an explicit text language modeling loss. Similarly, Emu3 (Wang et al., 2026) pretrains a single transformer from scratch across interleaved discrete image, video, and text tokens. Cambrian-S (Yang et al., 2026) adds a latent frame prediction head to a vision-language model and trains it jointly with spatial instruction tuning on a curated video corpus. Other recent efforts cast diverse multimodal tasks as video frame prediction (Hudson et al., 2025) or train vision-language models to anticipate and describe future environmental states (Chen et al., 2025). Unlike these methods, we continue training an existing pretrained language model instead of training from scratch, and we use raw web video with no captions and no text loss. The model learns only to predict the next continuous visual token, so video extends its autoregressive pretraining rather than serving as a task format or an instruction-tuning signal. 2.3 Vision-Language Models and Mid-Training Data Standard vision-language models connect a pretrained vision encoder (Zhai et al., 2023) to a causal language model (Yang et al., 2025) via a learned projection layer, followed by supervised instruction tuning on multimodal data (Tong et al., 2024; Li et al., 2025). Under this conventional recipe, the language model encounters visual tokens primarily during instruction tuning, and video is reduced to a sparse subset of sampled frames. To bridge the gap between static image pretraining and downstream tasks, recent open models add a large-scale multimodal mid-training stage in which video is supervised mainly through synthesized clip captions (An et al., 2026; Bai et al., 2025). Our ablations show that next-frame captions give no gain over predicting continuous visual features directly (see Section 4.4). Bypassing caption generation lets video mid-training use large web collections such as YT-Temporal-1B (Zellers et al., 2022) without any annotation. 3 Next-Token Prediction on Visual Tokens Figure 1: Overview of continuous video mid-training. Ordered video frames x1:Tx_1:T are encoded and projected into continuous token representations. Spatial patch grids are serialized with row-delimiter tokens and concatenated into a unified sequence V. A causal language model processes the continuous visual stream without captions or text supervision, and a prediction head rωr_ω regresses the subsequent visual representation vi+1v_i+1 from the hidden state hih_i. A stop-gradient operator sg(⋅)sg(·) is applied to the regression target so that gradients do not flow through it. We formulate video mid-training as autoregressive next-token prediction over continuous visual representations, keeping the causal decoder-only architecture of the language model. Rather than quantizing video into discrete codebooks (Yan et al., 2021) or relying on generated text captions (An et al., 2026), our approach encodes raw video frames into continuous feature vectors that directly populate the input space of the language model backbone. The model is trained on this serialized visual stream via self-supervised feature regression, without paired captions or any language loss (see Figure 1). This enables seamless data scaling by bypassing the cost and complexity of manual annotation or synthetic pseudo-captioning pipelines. Architecture. Given an ordered sequence of T video frames x1:T=[x1,…,xT]x_1:T=[x_1,…,x_T], a vision encoder gϕg_φ and projector pψp_ψ map each frame xtx_t into the language model’s hidden dimension d: zt=pψ(gϕ(xt))∈ℝN×d,z_t=p_ψ(g_φ(x_t)) ^N× d, (1) where N denotes the number of visual tokens per frame. Each grid ztz_t is serialized in row-major order with a learned newline embedding enl∈ℝde_nl ^d after every row, which keeps the two-dimensional layout recoverable from the flat sequence. The per-frame sequences are temporally concatenated into a unified continuous token stream V=[v1,…,vL]∈ℝL×dV=[v_1,…,v_L] ^L× d, where L is the total sequence length including the newline embeddings. These embeddings are directly fed to the causal Transformer blocks of fθf_θ in the language model. Objective. Standard language model pretraining maximizes the log-likelihood of discrete token sequences under causal masking. For continuous visual representations where no discrete vocabulary exists, we frame next-token prediction as self-supervised feature regression in the embedding space (i.e., token space), similar to V-JEPA 2 (Assran et al., 2025), which regresses the embeddings of masked video regions. At each position i∈1,…,L−1i∈\1,…,L-1\, the language model produces a contextual hidden state hi=fθ(v1:i)h_i=f_θ(v_1:i) conditioned strictly on the preceding visual context. A two-layer MLP prediction head rω:ℝd→ℝdr_ω:R^d ^d maps hih_i to a predicted feature vector v^i+1=rω(hi) v_i+1=r_ω(h_i). The model minimizes the distance between the predicted representation and the ground-truth visual token vi+1v_i+1 at every position whose next token is a visual patch, ℒ=1|L−1|∑i=1L−1(v^i+1,sg(vi+1)),L= 1|L-1| _i=1^L-1D ( v_i+1,sg(v_i+1) ), (2) where (u,v)D(u,v) and sg(⋅)sg(·) denote a distance function and the stop-gradient operator. In this work, we use cosine distance (u,v)=1−u⊤v‖u‖2‖v‖2D(u,v)=1- u v\|u\|_2\|v\|_2. Applying sg(⋅)sg(·) to the target representation vi+1v_i+1 keeps gradients from flowing through the target. This ensures that gradients propagate solely through the causal predictive pathway from the vision encoder to the prediction head (ϕ,ψ,θ,ωφ,ψ,θ,ω), while the subsequent visual feature acts as a fixed regression target during each optimization step. Under standard causal self-attention over the serialized sequence, this single objective unifies spatial next-patch modeling within frames and temporal prediction across frame boundaries. It adds only a small prediction head used during mid-training and requires no discrete codebook or language supervision. 4 Experiments We first outline the experimental setup, then report downstream evaluations across video, image, and text benchmarks in Tables 1 to 3, and finally analyze cross-modal training dynamics and the mid-training objective through ablation experiments. 4.1 Experimental Details 4.1.1 Architecture Following Cambrian-S (Yang et al., 2026), our model connects a pretrained vision encoder to a causal language model through a two-layer GELU (Hendrycks and Gimpel, 2016) MLP projector. We instantiate the visual backbone with SigLIP SO400M/14 at 384×384384× 384 resolution (Zhai et al., 2023) and the language model with Qwen3-1.7B (Yang et al., 2025). Visual patch tokens are projected directly into the 2,048-dimensional language embedding space. The auxiliary prediction head rωr_ω used during mid-training is a lightweight two-layer GELU MLP with 2,048 hidden units. 4.1.2 Dataset We draw mid-training video from YT-Temporal-1B (YT-1B) (Zellers et al., 2022), a collection of approximately 20 million public YouTube videos. After excluding videos that are no longer accessible online, our source pool comprises 17 million videos. Frames are sampled at 0.2 fps (one frame every five seconds). From videos containing between 16 and 63 sampled frames, we extract contiguous 16-frame clips starting at random temporal offsets. The composed dataset contains 6.99M clips with 64.4B visual tokens in our implementation. For subsequent supervised fine-tuning, we incorporate LLaVA-OneVision-Data (Li et al., 2025) (3.9M images, 3.2B tokens). 4.1.3 Captioning Mid-Training Data for Baselines For the caption baselines in Section 4.4, frames are captioned by Qwen3-VL-30B-A3B (Bai et al., 2025) in consecutive pairs with a progress-aware prompt adapted from Xue et al. (2025). The captioning model sees both frames of a pair together and writes one caption per frame that describes the action in that frame without referring to the other frame, so the two captions differ where the action has progressed. The caption of frame t is the target of current-caption prediction, and the caption of frame t+1t+1 is the target of next-caption prediction from frame t. The system prompt instructs the model to describe only what is visible and never to invent content absent from the frames. The user turn presents the two frames in order and closes with the instruction reproduced in Figure 2. System. You are a precise video-frame captioner. You are shown an ordered sequence of keyframes from a single video. Describe what is VISIBLE in each frame -- people, objects, actions, setting, camera. Never invent things that are not in the frame. User. Frame 1: <image> Frame 2: <image> These 2 frames are consecutive moments of one action, in order. Write one caption per frame. Requirements: (1) Ensure each frame’s description is specific to the corresponding frame, not referencing other frames (no ’continues’, ’now’, ’still’, ’again’, ’then’, ’previous’, ’next’). (2) The description should focus on the specific action being performed, capturing the progression of the action. There is no need to comment on other elements, such as the background or unrelated objects. - 1-3 concrete sentences; keep every caption distinct. - Write a plain natural-language sentence for each frame. Do NOT output bounding boxes, coordinates, object labels, or detection JSON. - Output ONLY a JSON array of exactly 2 plain strings (one per frame, in order), e.g. ["caption 1", "caption 2", ...] -- no objects, no extra text. Figure 2: Captioning prompt for the caption baselines. Qwen3-VL-30B-A3B receives two consecutive frames and writes one caption per frame, following a prompt adapted from the progress-aware captioning of Xue et al. (2025). Each caption describes the action in its own frame without referring to the other frame, and each <image> is replaced by the visual tokens of that frame. 4.1.4 Training All model parameters (ϕ,ψ,θ,ωφ,ψ,θ,ω) are trained end-to-end for one epoch (27,300 steps) across 128 NVIDIA H100 GPUs. We train with an effective batch size of 256 clips and a context window of 16k tokens. We use AdamW (Loshchilov and Hutter, 2019) with a learning rate of 0.00001 for the language model and projector and 0.000002 for the vision encoder, with cosine decay and a 3% linear warmup. Following mid-training, we instruction-tune mid-trained models for one epoch on LLaVA-OneVision data (Li et al., 2025) using the same training parameters used in mid-training. 4.1.5 Benchmarks We evaluate instruction-tuned models across video, static image, and text suites. Video understanding. We group the video benchmarks by the ability they test, namely long-form egocentric understanding with EgoSchema (Mangalam et al., 2023), general video question answering with NExT-QA (Xiao et al., 2021) and Video-MME (Fu et al., 2025), and fine-grained temporal reasoning with TempCompass (Liu et al., 2024b). Image understanding. We group the image benchmarks by task type, namely text-rich image understanding (Masry et al., 2022; Mathew et al., 2021; Mathew et al., 2022; Singh et al., 2019), general perception and diagram understanding (Kembhavi et al., 2016; Liu et al., 2024a; Li et al., 2024), text recognition (Liu et al., 2024c), and visual reasoning (Chen et al., 2024; Lu et al., 2024b). Text understanding and reasoning. To measure how well the model retains its language ability, we evaluate commonsense reasoning and world knowledge (Bisk et al., 2020; Clark et al., 2018; Clark et al., 2019; Hendrycks et al., 2021a; Mihaylov et al., 2018; Sakaguchi et al., 2021; Sap et al., 2019; Zellers et al., 2019), mathematical, scientific, and multi-step reasoning (Cobbe et al., 2021; Hendrycks et al., 2021b; Lightman et al., 2024; Rein et al., 2024; Suzgun et al., 2023), and code generation (Austin et al., 2021). 4.1.6 Evaluation Protocol Image and video benchmarks are evaluated via lmms-eval (Zhang et al., 2025), where multiple-choice questions are scored by option-letter log-likelihood and open-ended queries are decoded greedily. Multiple-choice and short-answer text benchmarks are evaluated zero-shot by ranking candidate-answer log-likelihoods. Free-form text benchmarks use greedy decoding with standard few-shot exemplars: eight for GSM8K, and three for MBPP and BBH. Finally, GPQA is evaluated via lighteval (Habib et al., 2023) in non-reasoning mode with an 8,192-token generation budget, a presence penalty of 1.5, and top-k=20k=20. Table 1: Effect of video mid-training on video benchmarks. No Mid-Training denotes the language model instruction-tuned without video mid-training, and Video Mid-Training denotes the model mid-trained for one epoch (64.4B visual tokens) and then given the same instruction tuning. All scores are accuracies in percent, and the best value in each row is in bold. Benchmark No Mid-Training Video Mid-Training Text-Rich Image Understanding ChartQA 43.76 51.96 DocVQA 32.13 38.57 InfographicVQA 22.31 25.28 TextVQA 45.49 51.05 General Perception & Diagrams AI2D 66.39 72.51 MMBench 76.00 79.46 SEED-Bench 63.00 69.10 Character Recognition OCRBench 38.40 41.60 Complex Visual Reasoning MathVista 58.15 62.59 MMStar 41.60 45.93 Average 48.72 53.80 Table 2: Effect of video mid-training on image benchmarks. Columns follow Table 1. All scores are accuracies in percent, with OCRBench divided by ten, and the best value in each row is in bold. Benchmark No Mid-Training Video Mid-Training NExT-QA 59.04 62.18 VideoMME 42.85 45.15 TempCompass 51.58 53.04 EgoSchema 45.20 50.00 Average 49.67 52.59 Table 3: Effect of video mid-training on text benchmarks. Qwen3-1.7B is the language model before any multimodal training. All scores are accuracies in percent, and the best value in each row is in bold. Benchmark Qwen3-1.7B No Mid-Training Video Mid-Training Commonsense & General Knowledge OpenBookQA 25.99 26.79 26.59 WinoGrande 60.53 61.87 61.64 HellaSwag 45.18 45.26 45.68 SIQA 44.03 44.18 44.49 PIQA 71.47 72.34 72.17 ARC-Easy 71.58 73.31 73.61 ARC-Challenge 39.81 41.35 42.72 BoolQ 79.46 80.59 81.54 MMLU 62.35 61.16 60.75 Mathematical & Complex Reasoning MATH500 45.63 27.98 29.17 GSM8K 73.41 49.92 54.77 BBH 38.89 23.92 29.40 GPQA 18.00 19.50 17.50 Code Generation MBPP 43.06 44.25 44.44 Average 51.38 48.03 48.89 Figure 3: Progress over mid-training. Each point corresponds to an intermediate mid-training checkpoint followed by identical instruction tuning (0.3, 0.5, and 1.0 epochs), plotted against cumulative visual tokens processed during mid-training. The point at zero represents the baseline instruction-tuned model without video mid-training. Figure 4 reports individual benchmark trajectories. 4.2 Video Mid-Training Improves Video and Image Understanding Tables 1, 2, and 3 report downstream evaluation results across video, image, and text benchmarks, respectively. Compared with the model instruction-tuned without mid-training, our video mid-trained model improves every video benchmark (+2.9 points on average), with the largest gain of 4.8 points on EgoSchema. These representations transfer to static image perception, improving all ten image benchmarks (+5.1 points on average), with the largest gains on visually dense tasks such as ChartQA (+8.2 points), DocVQA (+6.4 points), and AI2D (+6.1 points). On text benchmarks, multimodal instruction tuning lowers performance on mathematical and multi-step reasoning, dropping 23.5 points on GSM8K, 17.7 points on MATH500, and 15.0 points on BBH relative to the original Qwen3-1.7B, while leaving knowledge and commonsense benchmarks largely unaffected. Prior vision-language models report the same loss and reduce it by mixing text-only data into training (Lu et al., 2024a), whereas our instruction-tuning set contains only image-text examples. The mid-trained model shows the same pattern with a smaller drop, 18.6 points on GSM8K, 16.5 points on MATH500, and 9.5 points on BBH. One possible reason is that mid-training has already adapted the language model to visual tokens, so instruction tuning needs to change it less. Figure 4: Performance trajectory across mid-training on each benchmark. Each point corresponds to an intermediate mid-training checkpoint after subsequent instruction tuning, plotted against cumulative visual tokens processed during mid-training. The point at zero represents the baseline model without mid-training. In text benchmark panels, the dashed line indicates the original Qwen3-1.7B before multimodal training. 4.3 Image and Video Gains Appear Early in Mid-Training Figure 3 illustrates average video, image, and text performance across mid-training, evaluated at 0.3, 0.5, and 1.0 epochs. At each training stage, we perform supervised fine-tuning on LLaVA-OneVision-Data (Li et al., 2025) to evaluate downstream multimodal capabilities. Average image and video scores rise by 4.9 and 3.1 points within the first 0.3 epochs (19.5B visual tokens) and vary by less than 0.5 points over the remaining 0.7 epochs. The average text score stays within 0.9 points of the model without mid-training across all evaluated training steps. Figure 4 breaks down individual benchmark trajectories. After full mid-training, the largest video gains are on long-form egocentric understanding (EgoSchema) and general video question answering (NExT-QA, VideoMME), while fine-grained temporal reasoning (TempCompass) improves by 1.5 points. Averaged by task type, the largest image gains at the end of mid-training are on text-rich image understanding (+5.8 points) and general perception and diagram understanding (+5.2 points), followed by visual reasoning (+4.4 points) and text recognition (+3.2 points). Across all evaluation intervals, scores on commonsense, knowledge, coding, and GPQA remain within 3 points of the baseline. In contrast, reasoning benchmarks (GSM8K, MATH500, BBH) fluctuate by up to 9 points. 4.4 Visual Next-Token Prediction Is Sufficient Without Annotations Table 4 demonstrates that purely self-supervised visual next-token prediction is sufficient for effective mid-training, matching or outperforming objectives that supervise with frame-level dense captions. We compare our approach against two caption-based targets, namely current-caption and next-caption prediction, where the continuous visual target is replaced by a text caption of the current or next frame (see Section 4.1.3). Predicting next-frame captions yields no empirical advantage over predicting visual tokens directly. It yields similar video and image scores, but degrades language performance by 1.5 points. Even when controlling for training budget by comparing against our 0.5-epoch visual model (52.37 video, 53.76 image, 48.32 text), visual next-token prediction matches vision gains within 0.3 points while better preserving text capability. Current-caption prediction likewise fails to offer a compelling alternative, falling 5.0 points behind visual tokens on image understanding and showing inconsistent transfer across tasks. These comparisons establish that raw visual token prediction is sufficient on its own, allowing video mid-training to remain fully self-supervised while avoiding the computational overhead and label noise of caption generation. Table 4: Comparison of mid-training objectives on YT-1B. All models are instruction-tuned on LLaVA-OneVision-Data, and No mid-training is the same model instruction-tuned without mid-training. The best value in each column is in bold. Objective Video Image Text No mid-training 49.67 48.72 48.03 Current caption 51.28 48.77 47.44 Next caption 52.51 53.54 47.40 Visual next token 52.59 53.80 48.89 5 Discussion We propose autoregressive next-token prediction on raw web video as a mid-training objective without text supervision that bridges language pretraining and multimodal instruction tuning. Despite seeing only continuous video frames, the model gains 2.9 points on video-language and 5.1 points on image-language benchmarks, with the greatest improvements in text-rich image understanding and general perception. Predicting next-frame captions offers no advantage over predicting raw visual tokens, demonstrating that video mid-training can remain entirely self-supervised without the expense or label noise of captioning pipelines. Although mid-training has no text loss, the mid-trained model averages 48.9 on text benchmarks after instruction tuning, compared with 48.0 for the model without mid-training. Both models score below the original language model on GSM8K, MATH500, and BBH, and for the model without mid-training this drop comes from instruction tuning alone. While the mid-trained model degrades less, its 5.5-point margin on BBH is smaller than the 11.7-point discrepancy between instruction-tuning runs from the same initial state. Downstream multimodal gains appear within the first 0.3 epochs, after which scores change by less than 0.5 points across the remainder of training. Whether denser temporal sampling or longer sequence horizons can yield further improvements remains an open question. In addition, determining whether these gains stem from temporal structure or simply from additional exposure to visual tokens requires future frame-shuffling experiments. Evaluating these dynamics across larger language models will establish whether uncaptioned video mid-training can serve as a standard precursor to multimodal instruction tuning. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. An et al. (2026) X. An, Y. Xie, F. Tang, Y. Yan, H. Tan, D. Zhu, C. Chen, X. Zhao, B. Qin, K. Yang, et al. LLaVA-OneVision-2: towards next-generation perceptual intelligence. arXiv preprint arXiv:2605.25979. Cited by: §1, §2.3, §3. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, Cited by: §2.1. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. V-JEPA 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: §1, §2.1, §3. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §4.1.5. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §2.3, §4.1.3. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. In AAAI, Cited by: §4.1.5. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurIPS, Cited by: §1. Bruce et al. (2024) J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: generative interactive environments. In ICML, Cited by: §1, §2.1. Chen et al. (2025) D. Chen, T. Moutakanni, W. Chung, Y. Bang, Z. Ji, A. Bolourchi, and P. Fung Planning with reasoning using vision language world model. External Links: 2509.02722 Cited by: §2.2. Chen et al. (2024) L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, and F. Zhao Are we on the right way for evaluating large vision-language models?. In NeurIPS, Cited by: §4.1.5. Clark et al. (2019) C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova BoolQ: exploring the surprising difficulty of natural yes/no questions. In NAACL, Cited by: §4.1.5. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §4.1.5. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.1.5. Fu et al. (2025) C. Fu, Y. Dai, Y. Luo, et al. Video-MME: the first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. In CVPR, Cited by: §4.1.5. Gemma Team et al. (2025) Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: §1. Habib et al. (2023) N. Habib, C. Fourrier, H. Kydlíček, T. Wolf, and L. Tunstall LightEval: a lightweight framework for LLM evaluation. External Links: Link Cited by: §4.1.6. Hendrycks et al. (2021a) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In ICLR, Cited by: §4.1.5. Hendrycks et al. (2021b) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In NeurIPS, Cited by: §4.1.5. Hendrycks and Gimpel (2016) D. Hendrycks and K. Gimpel Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415. Cited by: §4.1.1. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. In NeurIPS, Cited by: §1. Hudson et al. (2025) G. T. Hudson, D. Slack, T. Winterbottom, J. Sterling, C. Xiao, J. Shentu, and N. Al Moubayed Everything is a video: unifying modalities through next-frame prediction. In ICCV, External Links: 2411.10503 Cited by: §2.2. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §1. Kembhavi et al. (2016) A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi A diagram is worth a dozen images. In ECCV, p. 235–251. Cited by: §4.1.5. Li et al. (2025) B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li LLaVA-OneVision: easy visual task transfer. Transactions on Machine Learning Research. Cited by: §2.3, §4.1.2, §4.1.4, §4.3. Li et al. (2024) B. Li, Y. Ge, Y. Ge, G. Wang, R. Wang, R. Zhang, and Y. Shan SEED-Bench: benchmarking multimodal large language models. In CVPR, Cited by: §4.1.5. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In ICLR, Cited by: §4.1.5. Liu et al. (2024a) Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin MMBench: is your multi-modal model an all-around player?. In ECCV, Cited by: §4.1.5. Liu et al. (2024b) Y. Liu, S. Li, Y. Liu, et al. TempCompass: do video LLMs really understand videos?. In Findings of ACL, Cited by: §4.1.5. Liu et al. (2024c) Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences 67 (12), p. 220102. Cited by: §4.1.5. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In ICLR, Cited by: §4.1.4. Lu et al. (2024a) H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, et al. DeepSeek-VL: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525. Cited by: §4.2. Lu et al. (2024b) P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, Cited by: §4.1.5. Mangalam et al. (2023) K. Mangalam, R. Akshulakov, and J. Malik EgoSchema: a diagnostic benchmark for very long-form video language understanding. In NeurIPS, Cited by: §4.1.5. Masry et al. (2022) A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, Cited by: §4.1.5. Mathew et al. (2022) M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar InfographicVQA. In WACV, Cited by: §4.1.5. Mathew et al. (2021) M. Mathew, D. Karatzas, and C. V. Jawahar DocVQA: a dataset for VQA on document images. In WACV, Cited by: §4.1.5. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, Cited by: §4.1.5. Muennighoff et al. (2023) N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel Scaling data-constrained language models. In NeurIPS, Cited by: §1. NVIDIA (2026) NVIDIA Cosmos 3: omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800. Cited by: §1, §2.1. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level Google-proof Q&A benchmark. In COLM, Cited by: §4.1.5. Sakaguchi et al. (2021) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM 64 (9), p. 99–106. Cited by: §4.1.5. Sap et al. (2019) M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi Social IQa: commonsense reasoning about social interactions. In EMNLP, Cited by: §4.1.5. Shumailov et al. (2024) I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal AI models collapse when trained on recursively generated data. Nature 631 (8022), p. 755–759. Cited by: §1. Singh et al. (2019) A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach Towards VQA models that can read. In CVPR, Cited by: §4.1.5. Suzgun et al. (2023) M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. H. Chi, D. Zhou, and J. Wei Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of ACL, Cited by: §4.1.5. Tong et al. (2024) S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, et al. Cambrian-1: a fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, Cited by: §2.3. Tong et al. (2026) S. Tong, D. Fan, J. Nguyen, E. Brown, G. Zhou, S. Qian, B. Zheng, T. Vallaeys, J. Han, R. Fergus, et al. Beyond language modeling: an exploration of multimodal pretraining. In ICML, Cited by: §2.2. Tong et al. (2022) Z. Tong, Y. Song, J. Wang, and L. Wang VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, Cited by: §2.1. Villalobos et al. (2024) P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, and M. Hobbhahn Position: will we run out of data? Limits of LLM scaling based on human-generated data. In ICML, Cited by: §1, §1. Wang et al. (2026) X. Wang, Y. Cui, J. Wang, F. Zhang, Y. Wang, X. Zhang, Z. Luo, Q. Sun, Z. Li, Y. Wang, et al. Multimodal learning with next-token prediction for large multimodal models. Nature 650 (8101), p. 327–333. Cited by: §1, §2.2. Xiao et al. (2021) J. Xiao, X. Shang, A. Yao, and T. Chua NExT-QA: next phase of question-answering to explaining temporal actions. In CVPR, Cited by: §4.1.5. Xue et al. (2025) Z. Xue, J. An, X. Yang, and K. Grauman Progress-aware video frame captioning. In CVPR, Cited by: Figure 2, §4.1.3. Yan et al. (2021) W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas VideoGPT: video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104.10157. Cited by: §2.1, §3. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1, §1, §2.3, §4.1.1. Yang et al. (2026) S. Yang, J. Yang, P. Huang, E. Brown, Z. Yang, Y. Yu, S. Tong, Z. Zheng, Y. Xu, M. Wang, et al. Cambrian-S: towards spatial supersensing in video. In ICLR, External Links: 2511.04670 Cited by: §1, §2.2, §4.1.1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In ACL, Cited by: §4.1.5. Zellers et al. (2022) R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi MERLOT Reserve: neural script knowledge through vision and language and sound. In CVPR, Cited by: §1, §2.3, §4.1.2. Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training. In ICCV, Cited by: §2.3, §4.1.1. Zhang et al. (2025) K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li, and Z. Liu LMMs-Eval: reality check on the evaluation of large multimodal models. In Findings of NAACL, Cited by: §4.1.6.