Paper deep dive
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/29/2026, 2:59:51 AM
Summary
The paper introduces LeVJEPA, a video pretraining method that eliminates architectural asymmetries (like target encoders and stop-gradients) by using the LeJEPA collapse-free objective with SIGReg regularization. It demonstrates that LeVJEPA achieves superior efficiency (5.6x to 20.8x less compute than V-JEPA 2) and accuracy on motion-centric benchmarks by utilizing uniform random token dropping and block-causal attention, making video a viable substrate for general-purpose visual pretraining.
Entities (11)
Relation Signals (8)
LeVJEPA → achieveshighaccuracyon → Kinetics-400
confidence 95% · attains the highest Kinetics-400 accuracy
LeVJEPA → achieveshighaccuracyon → ImageNet-1K
confidence 95% · exceeds the strongest video baseline by 7.6 points on ImageNet-1K
LeVJEPA → employs → Uniform Random Token Dropping
confidence 95% · uniform random token dropping renders this number small while simultaneously improving downstream accuracy
LeVJEPA → employs → Block-Causal Attention
confidence 95% · the encoder can be trained with block-causal attention at no measurable accuracy cost
LeVJEPA → uses → SIGReg
confidence 95% · regularized by SIGReg, which excludes collapse with a provable guarantee
LeVJEPA → uses → LeJEPA
confidence 95% · LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective
LeVJEPA → outperforms → V-JEPA 2
confidence 90% · LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute
LeVJEPA → outperforms → DINOv2
confidence 90% · LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.
Tags
Links
- Source: https://arxiv.org/abs/2608.27395v1
- Canonical: https://arxiv.org/abs/2608.27395v1
Trouble viewing inline? Open PDF directly →
Full Text
56,037 characters extracted from source content.
Expand or collapse full text
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics Lukas Kuhn1,2 Lucas Maes4,5 Giuseppe Serra1,2 Quentin Le Lidec8 Yann LeCun7,8 Randall Balestriero*,6,8 Florian Buettner*,1,2,3 1German Cancer Research Center 2German Cancer Consortium 3Goethe University Frankfurt 4Mila 5Université de Montréal 6Brown University 7Courant Institute, New York University 8Advanced Machine Intelligence (AMI Labs) *Equal advising. Correspondence: lukas.kuhn@dkfz-heidelberg.de Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space with a dedicated decoder. We introduce LeVJEPA, the first video encoder trained under LeJEPA’s collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The trainable architecture consequently reduces to an encoder and a projector, and the objective simplifies to a single hyperparameter. This formulation admits two properties that we examine in this paper. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small and, remarkably, simultaneously improves downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.65.6 to 20.8×20.8× less total pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.67.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost, such that the representation of every frame is a function of past observations alone: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining. Figure 1: LeVJEPA Training. From each clip we construct one global view (H×W×TH× W× T) and several local views (h×w×T×Vh× w× T× V), all sharing the same temporal window; local views are additionally cropped and photometrically augmented. Within every view, 95%95\% of the patch tokens are dropped uniformly at random (grey), and only the retained tokens enter the token sequence (L×dL× d). Both views are processed by the same shared block-causal encoder EθE_θ, and from the output tokens (L×dL× d) the loss reads only the [cls] embedding of each view. The objective combines a mean squared error that drives each local [cls] embedding toward the global one, with gradients flowing through both branches, and SIGReg [5] (right), which projects the batch of embeddings onto random directions (a) and penalizes the deviation of each projection from a standard Gaussian via a normality test (b), constraining the embedding distribution to an isotropic Gaussian and thereby provably excluding collapse. No target encoder, predictor, stop-gradient, or masked-token reconstruction is required. 1 Introduction Video is the natural substrate for learning representations of the physical world: it is abundant, requires no annotation, and carries the temporal structure (motion, causality, object permanence) that static images cannot supply [26, 24]. Yet video has remained the expensive path to visual representations. A single clip comprises an order of magnitude more tokens than an image, and the methods that learn from it have compounded this cost with architectural machinery: leading video joint-embedding methods train two encoders and a predictor to avoid representation collapse [16, 6], and inherit masking schemes designed around that machinery rather than around video itself [28]. In this work, we train a video encoder with SIGReg [5], a statistically principled regularizer that excludes collapse with a provable guarantee. This makes previous collapse-prevention, and the design conventions built around it, unnecessary, and we explore in this paper the substantial simplifications and efficiency gains this enables. Under this formulation, pretraining reduces to a single encoder trained with a single loss: an invariance term between a global view and several local views of a clip, and SIGReg applied to their embeddings. Where prior methods devote a large share of each training step to components that exist only to stabilize learning — a target-encoder forward pass over the full clip, a predictor over masked queries [6] — for us every operation contributes directly to the objective, and the cost of a step is governed only by the number of tokens the encoder observes. This admits two consequences that we develop in this paper. First, the observed token set becomes a free parameter of the method rather than a component of a prediction task, and can be made extremely sparse; the resulting encoder attains accuracy comparable to state-of-the-art video joint-embedding methods at a fraction of their pretraining FLOPs. Second, because no asymmetry between branches is required, the attention topology of the encoder is unconstrained, and we exploit this freedom to train with block-causal attention, bidirectional within a frame and causal across frames. Causality aligns the encoder with the predictive feature principle [24], which posits that representations of temporally adjacent stimuli be predictive of one another and therefore requires frame representations computable from past observations alone. It is equally consequential at inference: because past frames need not be re-encoded when new frames arrive, the representation of a video can be extended frame by frame at constant incremental cost, a property required by autoregressive world models and streaming settings that bidirectional encoders can only approximate by re-encoding or by fitting a separate temporal model after pretraining [32, 3, 20]. We instantiate this formulation as LeVJEPA11 1 Code and models are available at https://levjepa.github.io/, the first video encoder trained under LeJEPA’s collapse-free objective, and evaluate it under frozen probing against video and image pretraining baselines retrained on identical data, in both epoch-matched and FLOP-matched regimes. Our experiments support three findings: • FLOP-efficient video pretraining. At matched epochs on identical data, LeVJEPA attains accuracy comparable to or exceeding V-JEPA 2 across ViT-S/B/L at 5.65.6 to 20.8×20.8× less total pretraining compute. Granted an equal total FLOP budget, it leads the strongest video baseline by 7.67.6 points on ImageNet-1K, attains the highest Kinetics-400 accuracy, and remains competitive on Something-Something-v2. Against a DINOv2 baseline trained on frames of the same videos at equal compute, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. The efficiency extends to accessibility: a ViT-Tiny pretrained for 1212 hours on a single consumer GPU on unlabeled walking videos attains non-trivial ImageNet accuracy. • Causal frame representations at no accuracy cost. Block-causal attention matches fully bidirectional attention under frozen probing, so temporal ordering becomes a property of the encoder itself: frame representations are computable from past observations alone and extend to incoming frames without re-encoding, as autoregressive world modeling and streaming inference require, without a separate temporal model fitted after pretraining. • Significant simplification of the video pretraining recipe. The trainable architecture reduces to an encoder and a small projector, with no predictor network, target encoder, or stop-gradient, and the objective carries a single hyperparameter that is fixed to its published default in every experiment. The remaining design choices simplify with it: uniform random token dropping replaces structured masking and acts as an augmentation rather than an approximation, raising ImageNet accuracy monotonically from 33.9%33.9\% when every token is processed to 47.6%47.6\% when 95%95\% are discarded; temporal patch aggregation at the input is unnecessary, so tokenization is per frame; and dense, semantically organized patch representations emerge although only the clip-level token is supervised, without the auxiliary patch-level objectives of prior work. Figure 2: Accuracy against total pretraining compute at matched epochs. ImageNet-1K attentive-probing accuracy of ViT-S, ViT-B, and ViT-L encoders pretrained for 240240 epochs on the identical 20%20\% subsample of K710 (marker size indicates model size; the horizontal axis is logarithmic and reversed). Figure 3: Patch-token PCA across methods. The three leading principal components of the patch-token representations, visualized as RGB, for the same input image. LeVJEPA yields a decomposition that cleanly separates the animal from the surrounding furniture and background, comparable to that of V-JEPA 2.1, which obtains its dense structure through an explicitly introduced auxiliary patch-level loss [21]. V-JEPA 2, trained without such a loss, exhibits no comparable token-level organization. In LeVJEPA, this structure emerges although the training objective supervises only the [cls] token and no patch-level supervision is applied. 2 Related Work Self-supervised learning from images. Joint-embedding methods map augmented views of the same image to nearby representations, and differ principally in how they exclude the constant solution. BYOL [16] showed that an asymmetry between an online branch and an EMA target branch, with a stop-gradient and a predictor head, suffices without negative pairs. DINO [11] adopts the same teacher–student arrangement with centering and sharpening, and introduces the multi-crop scheme in which several aggressively cropped local views are matched to a small number of global views. DINOv2 [23] combines this with masked-image modelling [33] and scales it on curated data, while I-JEPA [2] replaces hand-designed augmentations with prediction between masked regions in representation space. LeVJEPA adopts the global–local view construction of this line, but none of its collapse-prevention machinery. Explicit constraints on the embedding distribution. A second line replaces architectural asymmetries with a criterion applied to the embeddings themselves. VICReg [7] and Barlow Twins [31] pair an invariance term with variance and covariance criteria, requiring neither target network nor stop-gradient, but constraining only second-order statistics and saying nothing about which embedding distribution is preferable. LeJEPA [5] supplies that characterization: under mild assumptions the isotropic Gaussian uniquely minimizes worst-case downstream probing risk, and SIGReg enforces it by reducing the high-dimensional constraint, via the Cramér–Wold theorem, to univariate goodness-of-fit tests along random directions at linear cost. The recipe uses a single network and a single loss, and has been validated on static images. Pixel reconstruction from video. Masked autoencoding reconstructs held-out content in pixel space and admits no trivial solution, so it needs no collapse prevention. VideoMAE [28] observes that the temporal redundancy of natural video makes naive masking too easy, since content masked at one instant can be copied from adjacent frames; tube masking, which occludes the same spatial region in every frame, removes this shortcut and permits masking ratios of 9090–95%95\%. VideoMAEv2 [30] scales the approach with an additional decoder-side mask. The tube pattern is therefore a property of the imputation task rather than of video itself, a distinction that becomes visible when the objective imputes nothing (Section 4). Feature prediction and video world models. V-JEPA [6] predicts representations rather than pixels, training an encoder jointly with a narrow predictor against targets from an exponential-moving-average copy of the encoder; following VideoMAE, its multi-block masks span the full temporal extent of the clip. V-JEPA 2 [3] scales this along data, model size, schedule, and resolution, and reports state-of-the-art frozen performance on motion understanding and action anticipation. To use these representations for planning, V-JEPA 2-AC freezes the encoder, applies it per frame, and trains a separate block-causal predictor conditioned on actions and end-effector states on robot interaction data, supporting zero-shot goal-conditioned manipulation through model-predictive control. Other work learns action-conditioned dynamics in pixel space through video generation [17, 9], at substantially higher planning cost. In each case the temporal model is a separate, action-supervised stage fitted after pretraining; LeVJEPA instead places the temporal constraint in the encoder during pretraining. 3 Methodology: LeVJEPA Our goal is to learn visual representations from video with a single network and a single loss, discarding the collapse-prevention heuristics that video joint-embedding methods have so far relied on. LeVJEPA transfers the LeJEPA recipe [5], namely the invariance loss paired with SIGReg as the sole mechanism for preventing representation collapse, to video, on top of a V-JEPA-style video transformer backbone [6]. Given a video, we sample a clip of 1616 frames and construct V+1V+1 views of it: one global view x0x_0 at full resolution and V local views x1,…,xVx_1,…,x_V obtained by aggressive spatial cropping and photometric augmentation. All views share the identical temporal window and differ only spatially and photometrically. Each view is processed by the same encoder Eθ(⋅)E_θ(·); a learnable [cls] token provides a clip-level readout, which a small projector hϕ(⋅)h_φ(·) maps to an embedding zv=hϕ(Eθ(xv)[cls])∈ℝKz_v=h_φ\! (E_θ(x_v)_ [cls] ) ^K. This projection is necessary because the final encoder layer applies layer normalization [4], which constrains the [cls] representation to a sphere and thereby prevents the SIGReg objective from being optimized effectively in the encoder’s output space [19]. The training loss is applied in this K-dimensional space; the projector is discarded after pretraining and downstream tasks operate on the encoder’s representations. Joint-embedding methods for video have thus far relied on architectural asymmetries to prevent representation collapse: an exponential-moving-average target encoder, a stop-gradient on the target branch, and a predictor network conditioned on masked-token queries [6, 16]. While empirically effective, these mechanisms introduce additional networks and schedules whose influence on the learning dynamics is difficult to characterize, and for which no analytical guarantee against collapse is available. In LeVJEPA, collapse is instead excluded by an explicit distributional constraint. A single encoder is trained with the objective ℒ=ℒinv+λℒSIGReg,L\;=\;L_inv\;+\;λ\,L_SIGReg, (1) where ℒinvL_inv enforces that the embeddings of local views be predictive of the embedding of the global view, and ℒSIGRegL_SIGReg constrains the distribution of the embeddings such that collapsed solutions are provably excluded [5]. The trade-off weight λ balances the two terms and constitutes the objective’s only hyperparameter; following Balestriero and LeCun [5], it is fixed to λ=0.02λ=0.02 in all trainings and is not tuned for any experiment in this paper. 3.1 Training Objective Invariance. The encoder is trained to satisfy the constraint that the embedding of any local view of a clip be predictive of the embedding of the global view of the same clip. As the global view is the only view that remains photometrically unaltered and covers the largest spatial extent, it constitutes the prediction target by construction of the views alone. The invariance term is the mean squared error ℒinv=1V+1∑v=0V‖z0−zv‖22.L_inv\;=\; 1V+1 _v=0^V z_0-z_v _2^2. (2) Gradients propagate through both variables in Equation 2: the target embedding z0z_0 is produced by the same encoder, in the same forward pass, as the local embeddings; no stop-gradient operation or target network is employed. Minimizing Equation 2 in isolation therefore admits a trivial solution in which the encoder outputs a constant embedding regardless of its input. Rather than excluding this solution through the architectural asymmetries discussed above, LeVJEPA excludes it through an explicit constraint on the embedding distribution, described next. Regularization. SIGReg [5] constrains the embedding distribution to match an isotropic Gaussian — the distribution shown by Balestriero and LeCun [5] to minimize worst-case downstream probing risk, and from which any collapsed solution, having zero variance along some direction, is maximally distant. By the Cramér–Wold theorem, the embeddings match (0,IK)N(0,I_K) if and only if every one-dimensional projection matches (0,1)N(0,1), reducing the high-dimensional constraint to univariate goodness-of-fit tests. At each step, M directions a1,…,aMa_1,…,a_M are sampled uniformly on K−1S^K-1, and the deviation of each projected batch ⟨zi,am⟩i=1n\ z_i,a_m \_i=1^n from the standard Gaussian is penalized via the Epps–Pulley statistic [14], ℒSIGReg=1M∑m=1M∫|1n∑i=1neit⟨zi,am⟩−e−t2/2|2e−t2/2dt,L_SIGReg\;=\; 1M _m=1^M 1n _i=1^ne^\,i\,t z_i,a_m -e^-t^2/2 ^2\,e^-t^2/2\,dt, (3) computed per view and approximated by quadrature. The empirical characteristic function underlying Equation 3 is bounded with bounded gradients, making the loss robust to outliers and free of whitening or centering operations, and permits distributed evaluation on the full global batch at negligible communication cost; implementation details are provided in Appendix A. 3.2 Architecture The encoder is a Vision Transformer adapted for video [13, 1]. A clip of 1616 frames is tokenized by a convolutional patch embedding of spatial extent 16×1616× 16 and temporal extent τ; by default τ=1τ=1, such that each token corresponds to a patch of a single frame and no temporal aggregation is imposed at the input (3,1363,136 tokens for a 2242224^2 global view, 576576 for a 96296^2 local view). Temporal aggregation (τ=2τ=2) [1, 28, 6] is supported and compared against the per-frame default in Section 4. During pretraining, a fraction ρ=0.95ρ=0.95 of the patch tokens of each view is discarded uniformly at random after patch embedding; the retained tokens constitute a sparse observation of the clip, and this token dropping, which determines the computational cost of pretraining, is analyzed in Section 4. A learnable [cls] token prepended to the sequence serves as the clip-level readout, receives the training objective exclusively, and is never dropped; patch tokens receive no direct supervision. Self-attention follows a block-causal pattern: patch tokens attend bidirectionally within their frame and causally to preceding frames, while the [cls] token attends to all tokens but is not attended to. The representation of a frame is therefore a function of the current and past frames only; this is the default for all results in this paper, and its consequences are examined in Sections 4. Positional information is provided by factorized three-dimensional rotary embeddings [27, 3], which encode relative position and thereby let the same encoder process both view resolutions without interpolation. The [cls] representation is mapped to the K-dimensional embedding space by a small projector shared across views. The trainable architecture comprises the encoder and projector only: no predictor network is instantiated and no target encoder is maintained during training; a Polyak average of the encoder weights is retained solely as the evaluation checkpoint [10] and plays no role in the objective. Architectural details are provided in Appendix B. 3.3 Pretraining Data and Evaluation Setup Pretraining. Unless stated otherwise, models are pretrained on a class-balanced subsample comprising 20%20\% of K710, the union of the Kinetics-400/600/700 training sets [18] with validation overlap removed, following Bardes et al. [6]. Restricting the default pretraining set keeps the cost of controlled experiments low and, as we show in Section 5, already suffices for competitive downstream performance. In addition, we pretrain a ViT-L/16 on the combination of K710, Something-Something-v2 [15], Walking Tours [29], and the PE Video Dataset released with Perception Encoder [8]. Walking Tours consists of a small number of hours-long egocentric walking videos; the PE Video Dataset contributes a large corpus of diverse, curated video. Together these sources span curated action-recognition clips, long uncurated egocentric footage, and web-scale general video. Evaluation. Pretrained encoders are evaluated frozen, following the attentive-probing protocol of Bardes et al. [6] exactly: a lightweight cross-attention block with a learnable query pools the encoder’s output tokens, and a linear classifier is trained jointly with the probe while the encoder parameters remain fixed. We evaluate action recognition on Kinetics-400 [18], motion classification on Something-Something-v2 [15], and object recognition on ImageNet-1K [25], thereby covering appearance-based video understanding, temporal understanding, and static image understanding, respectively. Probing hyperparameters, view sampling at test time, and the adaptation of the video encoder to static images are taken from Bardes et al. [6] without modification. Non-linear pooling is appropriate here because the pretraining objective provides no guarantee that the frozen token representations are linearly separable for a given downstream task [12, 6]; a probe with a learnable attention-based readout evaluates the information content of the representation rather than its linear geometry. On Kinetics-400, whose training set is considerably larger than those of the other benchmarks, we report linear probing on mean-pooled tokens instead, as training the attentive probe at this scale is computationally disproportionate to its purpose; since mean pooling followed by a linear classifier is a strictly weaker adaptation than the attentive probe, the reported Kinetics-400 accuracies constitute a conservative estimate. 4 What Matters for Efficient Video Pretraining? Having specified the method, we examine its design space. The purpose of this analysis is twofold: to determine which choices materially affect the quality of the learned representations, and to characterize the behavior of the objective when components that are standard in video pretraining — temporal patch aggregation, structured masking, and bidirectional attention — are removed. Unless stated otherwise, all experiments in this section pretrain a ViT-B/16 on the 20%20\% subsample of K710 described in Section 3.3, and report the top-1 accuracy of a frozen attentive probe on ImageNet-1K, which we found to be the most discriminative single indicator of representation quality among our evaluations. 4.1 Token Dropping Improves Representations (a) Masking ratio (b) Number of local views Figure 4: Ablations on view construction. (4(a)) Effect of the masking ratio and (4(b)) effect of the number of local views on ImageNet linear probing accuracy. We first vary the dropping ratio ρ (Figure 44(a)). If token dropping constituted merely an approximation adopted for efficiency, downstream accuracy would be expected to degrade as ρ increases. We observe the opposite: ImageNet accuracy increases monotonically with the dropping ratio, from 33.9%33.9\% when all tokens are processed to 47.6%47.6\% at ρ=0.95ρ=0.95. Token dropping therefore serves a dual role. It reduces the cost of each forward pass by up to a factor of (1−ρ)−1(1-ρ)^-1 in the feed-forward layers, and it simultaneously acts as a stochastic augmentation that requires the clip-level embedding to be inferable from sparse, randomly located observations of the clip. The comparison between ρ=0.9ρ=0.9 and ρ=0.95ρ=0.95 is of particular practical relevance: halving the number of processed tokens leaves accuracy unchanged within the observed variability (47.4%47.4\% vs. 47.6%47.6\%), so the most aggressive dropping ratio considered is simultaneously the most computationally economical configuration. The benefit of aggressive dropping is, however, not uniform across evaluations. On Something-Something-v2, which primarily probes motion understanding, accuracy declines for dropping ratios beyond 0.30.3, in contrast to the monotonic improvement observed on ImageNet. We find that this degradation is mitigated by extending the training schedule: with longer training, higher dropping ratios recover the accuracy of lower ones on Something-Something-v2 while retaining their lower per-iteration cost, such that aggressive dropping remains the more efficient configuration in terms of total pretraining compute. A plausible interpretation is that sparse random observations render motion cues, which depend on correspondences across frames, less frequently recoverable within a single view, and that additional iterations compensate for the reduced per-sample signal; a characterization of this interaction, and dropping schemes that preserve motion information at high sparsity, are left to future work. The spatial structure of the retained token set is similarly consequential. We compare uniform random dropping against a structured tube variant that retains identical spatial locations in every frame, mirroring the space-time masks employed in masked video modeling [28, 6]. Tube dropping reduces accuracy substantially, from 50.7%50.7\% to 39.6%39.6\% on ImageNet, and the same ordering holds on Something-Something-v2 (28.8%28.8\% against 26.4%26.4\%, under the τ=2τ=2 configuration of Section 4.3). We attribute this reversal of the established finding to the difference in objectives. In masked prediction, structured masks are necessary to render the imputation task non-trivial, as randomly distributed masks permit missing content to be interpolated from spatial neighbors. In the present setting, no content is imputed: the retained tokens constitute the encoder’s sole observation of the clip. A tube pattern permanently occludes the majority of the scene across all frames, whereas uniform random dropping yields a spatio-temporally distributed sample from which the content of the clip remains identifiable. 4.2 Local View Budget We next vary the number of local views V (Figure 44(b)). Accuracy improves consistently from 47.6%47.6\% at V=4V=4 to 50.2%50.2\% at V=10V=10, and saturates thereafter (49.8%49.8\% at V=12V=12). Each additional local view introduces one further prediction constraint into Equation 2 at a marginal cost of approximately 2929 processed tokens, corresponding to less than one fifth of the cost of the global view. We nevertheless retain V=4V=4 for all comparisons in this paper if not specified otherwise, and report this sweep to establish that the results presented in Section 5 do not exhaust the method: additional accuracy is available at a modest increase in pretraining cost by enlarging the view budget alone. 4.3 Temporal Patch Aggregation Is Not Required Video transformers conventionally aggregate pairs of consecutive frames at the input (τ=2τ=2), halving the token count prior to the transformer blocks [1, 28, 6]. We assess whether this aggregation contributes to representation quality through a comparison in which both the pretraining cost and the evaluation conditions are matched. During pretraining, both configurations process identical 1616-frame clips, and the doubled token count of per-frame patching is offset by a correspondingly halved retention rate (1−ρ=0.051-ρ=0.05 against 0.10.1), such that both encoders retain an equal number of tokens per view; the two dropping ratios are indistinguishable in downstream accuracy (Figure 44(a)), so the matching itself does not favor either configuration. Table 1: Temporal patch aggregation at a matched token budget. Top-1 accuracy of a frozen attentive probe. Both configurations retain an equal number of tokens per view during pretraining and are evaluated on sequences of 88 temporal slots. Patch embedding IN1K SSv2 τ=2τ=2, ρ=0.90ρ=0.90 47.4 28.8 τ=1τ=1, ρ=0.95ρ=0.95 50.7 30.4 At evaluation, where no dropping is applied, the sequence length is matched by adjusting the temporal extent of the input to 88 frames for τ=1τ=1 and 1616 frames for τ=2τ=2, yielding 88 temporal slots in both cases; for ImageNet, whose static images are repeated along the temporal axis following the protocol of Bardes et al. [6], this corresponds to 88 and 1616 repetitions, respectively. As reported in Table 1, the per-frame configuration attains higher accuracy on both benchmarks. The result on Something-Something-v2 is of particular note: temporal aggregation at the input is commonly motivated as a means of capturing short-range motion, yet its removal does not degrade performance on the benchmark most dependent on motion understanding. We conclude that the objective does not depend on temporal aggregation at the input, and that the choice of τ may consequently be governed by downstream requirements. 4.4 Causal Attention Incurs No Accuracy Penalty Finally, we compare fully bidirectional attention against the block-causal topology defined in Section 3.2, in which tokens attend bidirectionally within a frame and causally across frames. Causal attention constrains the representation of each frame to depend exclusively on the current and preceding frames, a property that bidirectional video encoders lack. Since causal masking removes future tokens from the receptive field of every token, a reduction in representation quality could reasonably be anticipated. Table 2 shows that no such reduction occurs: the block-causal encoder matches its bidirectional counterpart (51.2%51.2\% against 50.7%50.7\%). Temporal causality is thus obtained at no measurable cost to downstream accuracy, and we adopt block-causal attention as the default for all results reported in this paper. Table 2: Effect of the attention topology. ImageNet-1K top-1 accuracy of a frozen attentive probe; both configurations use τ=1τ=1, ρ=0.95ρ=0.95, V=4V=4, and uniform random dropping. Attention IN1K top-1 Bidirectional 50.7 Block-causal 51.2 5 Comparison with Prior Work 5.1 Comparison to Video Models Comparisons between self-supervised video models are frequently confounded by differences in pretraining data, schedule, and compute. To remove these confounds, we retrain all baselines on the identical 20%20\% subsample of K710 used throughout this paper, using their official implementations and recommended hyperparameters, for the same number of epochs (240240) and at the same effective batch size (3,0723,072) as our models. All encoders are evaluated with the frozen attentive-probing protocol of Bardes et al. [6]; since our encoders operate on per-frame tokens, the evaluation sequence length is equalized across methods by adjusting the temporal extent of the probe input, as described in Section 4.3, such that every method is probed on the same number of tokens. (a) Cosine similarity (b) Patch-token PCA Figure 5: Further patch-token visualizations. (5(a)) Cosine similarity between a query patch placed on the object and all patch tokens of the image: high similarity is confined sharply to the object, indicating that the token representations are not only semantically organized but spatially precise. (5(b)) Patch-token PCA as in Figure 3, for a different scene: the decomposition again groups tokens by semantic region, separating the animal from furniture, floor, and background, indicating highly semantic patch tokens. Figure 2 reports ImageNet accuracy against total pretraining compute for three encoder sizes under this epoch-matched protocol. Across all sizes, LeVJEPA attains accuracy comparable to V-JEPA 2 at a small fraction of the total pretraining compute, with the efficiency advantage ranging from 5.6×5.6× at ViT-L to 20.8×20.8× at ViT-S. At ViT-B, the two methods are separated by less than one accuracy point while LeVJEPA requires 4.84.8 rather than 36.436.4 ExaFLOPs; at ViT-L, LeVJEPA is no longer merely comparable but surpasses V-JEPA 2 by 1.91.9 points at 5.6×5.6× lower cost — notably, its ViT-L consumes less than half the compute of the V-JEPA 2 ViT-S. This efficiency follows directly from the mechanisms of Section 4: no full-length token sequence is processed at any point during pretraining, and no target-encoder or predictor forward passes are performed. VideoMAEv2 occupies an intermediate position in Figure 2, below both methods in accuracy at a compute cost between them. Table 3: FLOP-matched comparison on identical pretraining data. ViT-B encoders pretrained on the 20%20\% subsample of K710 at equal total pretraining FLOPs, evaluated frozen; IN1K and SSv2 report attentive-probing top-1 accuracy, K400 reports linear-probing top-1 accuracy. Method IN1K SSv2 K400 VideoMAEv2 53.4 43.6 37.4 V-JEPA 2 51.6 42.5 40.7 LeVJEPA 61.0 40.4 44.6 FLOP-matched comparison. The epoch-matched protocol above holds the number of training samples fixed and lets total compute vary; we now instead hold total pretraining compute fixed. All methods train ViT-B encoders at an equal total FLOP budget; since LeVJEPA processes each sample at a fraction of the baselines cost, the equal budget grants it a correspondingly longer schedule of 1,0851,085 epochs with V=10V=10 local views. Table 3 reports the outcome on all three benchmarks. Under equal total compute, LeVJEPA attains the highest ImageNet accuracy by a margin of 7.67.6 points and the highest K400 linear-probing accuracy, while being close in the motion based evaluation, reaching within 3.23.2 points of the strongest baseline on Something-Something-v2 (40.4%40.4\% against 43.6%43.6\%). 5.2 Comparison to Image Models Image-based self-supervised learning has so far constituted the stronger pretraining paradigm for appearance-centric transfer: video-pretrained encoders have consistently trailed their image-pretrained counterparts on static image benchmarks, such that video pretraining had to be motivated by motion understanding alone [29, 6]. We revisit this comparison under matched compute by training DINOv2 [22] with its official implementation on individual frames drawn from the identical video data (11.711.7M frame samples over 11,40011,400 optimizer steps) at the same total pretraining FLOPs as the 240240-epoch LeVJEPA ViT-B. As reported in Table 4, the image-pretrained encoder retains an advantage of 3.13.1 points on ImageNet, while the video-pretrained encoder attains nearly twice its accuracy on Something-Something-v2 (30.4%30.4\% against 16.9%16.9\%). Table 4: Comparison to image-based pretraining at matched compute. ViT-B encoders pretrained at equal total FLOPs on [identical source data]; top-1 accuracy of frozen attentive probes. Method IN1K SSv2 DINOv2 53.8 16.9 LeVJEPA 50.7 30.4 To our knowledge, this is the first FLOP-matched comparison in which video pretraining reaches near-parity with a state-of-the-art image method on appearance-centric evaluation while retaining a decisive advantage on motion-centric evaluation. The implication extends beyond the present method: if the appearance cost of pretraining on video can be reduced to a few points at equal compute then video, which supplies temporal structure that static images cannot, becomes the more FLOP-efficient pretraining substrate for general-purpose visual representations. 5.3 Pretraining on consumer hardware The low per-sample cost of the method extends to what hardware suffices for pretraining. We train a ViT-Tiny for 1212 hours on a single consumer GPU (RTX 5080, 1616 GB) on eight videos of the Walking Tours dataset [29], i.e., unlabeled, uncurated egocentric footage totalling approximately 620620k frames, from which the model processes roughly 55M clips. ImageNet top-1 accuracy of the frozen encoder improves from 8.9%8.9\% at initialization to 25.2%25.2\%. Beyond the reduced FLOPs, the small memory footprint of sparse token sequences is what makes such training practical: on the same 1616 GB device, LeVJEPA trains at batch size 128128 in under 88 GB, whereas a V-JEPA configuration with an identically sized encoder saturates the card at batch size 2828. Video pretraining with the present method is thus feasible not only at reduced cluster budgets but on commodity hardware. 5.4 Scaling the Pretraining Data The comparisons above deliberately restrict pretraining to the 20%20\% subsample of K710 to permit controlled, retrained baselines. We now remove this restriction and ask whether the method continues to improve as the pretraining corpus grows. We pretrain a ViT-L/16 for 100100 epochs on the combined corpus of Section 3.3, comprising K710, Something-Something-v2, Walking Tours, and the PE Video Dataset [8]. The resulting encoder reaches 69.5%69.5\% top-1 accuracy on ImageNet-1K and 55.0%55.0\% on Something-Something-v2 under frozen attentive probing, improving over the ViT-L trained on the 20%20\% subsample (Figure 2) by 9.59.5 points on ImageNet within a shorter schedule. The method thus benefits from additional data without any adjustment of the objective or its single hyperparameter, and the corpus used here remains orders of magnitude below the internet-scale collections of V-JEPA 2 [3], indicating headroom rather than saturation. 5.5 Emergent Token-Level Structure The training objective of LeVJEPA supervises a single clip-level token; the patch tokens receive no loss at any point during pretraining. Figure 3 shows that these unsupervised tokens nevertheless acquire semantically organized representations: the three leading principal components of the patch tokens, visualized for a single image, group by semantic region and separate the object from its surroundings, a degree of visible organization that V-JEPA 2 does not exhibit and that V-JEPA 2.1 obtains through an explicitly introduced auxiliary patch-level objective [21], both evaluated from their publicly released checkpoints. Figure 5 extends this picture: the decomposition is consistent across scenes, and the cosine similarity to a query patch is confined sharply to the object, indicating that the representations are spatially precise in addition to semantically organized. 6 Discussion This work indicates that self-supervised learning from video can be both simpler and substantially more compute-efficient than previously assumed. A single encoder, trained with an invariance loss and a distributional regularizer whose single hyperparameter is not tuned, attains accuracy comparable to or exceeding that of established video joint-embedding methods at a fraction of their total pretraining compute; the target encoder, predictor network, and associated schedules that these methods employ for stability are not required under the present objective. The analysis of Section 4 extends this simplification to individual design choices: masking structure, temporal patch aggregation, and attention topology become free parameters that can be selected for efficiency or for downstream requirements, and the resulting configuration, sparse random observation of per-frame tokens under block-causal attention, is simultaneously the cheapest and among the most accurate that we evaluate. The comparison to image-based pretraining suggests a broader implication. Image pretraining has constituted the standard source of general-purpose visual representations, with video regarded as a specialized complement for motion understanding, justified where its additional cost is warranted. Under the present objective, this division is no longer imposed by efficiency: at equal total compute on identical source data, the video-pretrained encoder approaches a DINOv2 baseline on appearance-centric evaluation while attaining nearly twice its accuracy on motion-centric evaluation. Since video strictly contains the appearance information of its constituent frames while additionally supplying temporal structure, these results indicate that video is a viable substrate for general-purpose visual pretraining rather than a specialized one, and that the choice between the two paradigms may increasingly be governed by data availability rather than by computational cost. The data-scaling experiment of Section 5.4 is consistent with this trajectory, with both appearance-centric and motion-centric accuracy improving under a growing corpus; a controlled comparison at internet scale remains an open question. Block-causal attention opens a corresponding perspective for temporal modeling. Current systems obtain causal dynamics by fitting a temporal model over the frozen outputs of a pretrained encoder, an arrangement that has proven effective for planning and control [32, 3, 20] but that separates representation learning from temporal structure. Our results show that the two need not be separated: causality can be established during pretraining, at no measurable cost to downstream accuracy, such that the encoder itself provides per-frame state that respects temporal ordering and extends to incoming frames without re-encoding. This positions a single pretrained encoder as a foundation for streaming perception and autoregressive world modeling. Several directions remain open. Motion-centric performance is the axis with the most remaining headroom: aggressive token dropping reduces Something-Something-v2 accuracy at short schedules, and although longer training largely recovers the difference within the same compute budget, dropping schemes that preserve temporal correspondences at high sparsity constitute a natural refinement of the method. Our controlled comparisons are conducted on a restricted corpus at up to ViT-L scale; the behavior of the objective at the model and data scales of recent video foundation models, and the interaction of SIGReg with very large batch and model regimes, remain to be characterized, and the favorable scaling observed in Section 5.4 makes this a promising rather than merely open question. The training signal is applied to a single clip-level token, and although semantically organized patch representations emerge without dense supervision, their sufficiency for dense prediction tasks such as segmentation and tracking has not yet been evaluated. None of these directions requires revisiting the formulation itself: each concerns extending a fixed, simple objective to broader regimes, which we regard as the principal advantage of the approach. In summary, LeVJEPA demonstrates that a single encoder, a single loss, and one fixed hyperparameter suffice for competitive self-supervised learning from video, and that the resulting formulation is efficient enough to pretrain on commodity hardware, causal by construction, and responsive to growing data. We view these properties as complementary: efficiency determines who can train such models, causality determines what they can be used for, and simplicity determines how reliably the results transfer to new regimes. Together they suggest that video, long the most information-rich but least accessible source of visual supervision, is becoming a practical foundation for general-purpose representation learning. Acknowledgments We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003). This work was co-funded by the European Union (ERC, TAIPO, 101088594 to F.B.) grant. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or ERC. Neither the European Union nor the granting authority can be held responsible for them. References [1] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid (2021) Vivit: a video vision transformer. In 2021 IEEE/CVF international conference on computer vision (ICCV), p. 6816–6826. Cited by: §3.2, §4.3. [2] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas (2023) Self-supervised learning from images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15619–15629. Cited by: §2. [3] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. (2025) V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: §1, §2, §3.2, §5.4, §6. [4] J. L. Ba, J. R. Kiros, and G. E. Hinton (2016) Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: §3. [5] R. Balestriero and Y. LeCun (2025) Lejepa: provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544. Cited by: Figure 1, §1, §2, §3.1.2, §3, §3. [6] A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas (2024) V-JEPA: latent video prediction for visual representation learning. External Links: Link Cited by: Appendix B, Appendix C, §1, §1, §2, §3.2, §3.3, §3.3, §3.3, §3, §3, §4.1, §4.3, §4.3, §5.1, §5.2. [7] A. Bardes, J. Ponce, and Y. LeCun (2021) Vicreg: variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906. Cited by: Appendix A, §2. [8] D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Bangalath, et al. (2026) Perception encoder: the best visual embeddings are not at the output of the network. Advances in Neural Information Processing Systems 38, p. 60884–60937. Cited by: §3.3, §5.4. [9] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. (2024) Genie: generative interactive environments. In Forty-first international conference on machine learning, Cited by: §2. [10] D. Busbridge, A. Shidani, F. Weers, J. Ramapuram, E. Littwin, and R. Webb (2025) Distillation scaling laws. arXiv preprint arXiv:2502.08606. Cited by: §3.2. [11] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF international conference on computer vision (ICCV), p. 9630–9640. Cited by: §2. [12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In International conference on machine learning, p. 1597–1607. Cited by: §3.3. [13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §3.2. [14] T. W. Epps and L. B. Pulley (1983) A test for normality based on the empirical characteristic function. Biometrika 70 (3), p. 723–726. Cited by: §3.1.2. [15] R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. (2017) The “something something” video database for learning and evaluating visual common sense. In 2017 IEEE international conference on computer vision (ICCV), p. 5843–5851. Cited by: §3.3, §3.3. [16] J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. (2020) Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33, p. 21271–21284. Cited by: Appendix B, §1, §2, §3. [17] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi (2020) Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), Cited by: §2. [18] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. (2017) The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. Cited by: §3.3, §3.3. [19] L. Kuhn, G. Serra, R. Balestriero, and F. Buettner (2026) LeVLJEPA: end-to-end vision-language pretraining without negatives. External Links: 2607.00784, Link Cited by: §3. [20] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) Leworldmodel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §1, §6. [21] L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, N. Ballas, and A. Bardes (2026) V-jepa 2.1: unlocking dense features in video self-supervised learning. External Links: 2603.14482, Link Cited by: Figure 3, §5.5. [22] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §5.2. [23] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski (2024) DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research. Note: Featured Certification External Links: ISSN 2835-8856, Link Cited by: §2. [24] R. P. Rao and D. H. Ballard (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience 2 (1), p. 79–87. Cited by: §1, §1. [25] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. (2015) Imagenet large scale visual recognition challenge. International journal of computer vision 115 (3), p. 211–252. Cited by: §3.3. [26] E. S. Spelke, R. Kestenbaum, D. J. Simons, and D. Wein (1995) Spatiotemporal continuity, smoothness of motion and object identity in infancy. British journal of developmental psychology 13 (2), p. 113–142. Cited by: §1. [27] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §3.2. [28] Z. Tong, Y. Song, J. Wang, and L. Wang (2022) VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 10078–10093. External Links: Document, Link Cited by: §1, §2, §3.2, §4.1, §4.3. [29] S. Venkataramanan, M. N. Rizve, J. Carreira, Y. Asano, and Y. Avrithis (2024) Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In International Conference on Learning Representations, Vol. 2024, p. 31952–31973. Cited by: §3.3, §5.2, §5.3. [30] L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao (2023) Videomae v2: scaling video masked autoencoders with dual masking. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14549–14560. Cited by: §2. [31] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny (2021) Barlow twins: self-supervised learning via redundancy reduction. In International conference on machine learning, p. 12310–12320. Cited by: §2. [32] G. Zhou, H. Pan, Y. LeCun, and L. Pinto (2025) DINO-wm: world models on pre-trained visual features enable zero-shot planning. 2025 int. conf. Machine Learning. Cited by: §1, §6. [33] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong (2022) Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, External Links: Link Cited by: §2. Appendix A SIGReg Implementation Details The integral in Equation 3 is approximated by trapezoidal quadrature over 1717 knots t∈[0,3]t∈[0,3], and the statistic is computed separately for the embeddings of each view with M=1,024M=1,024 directions per step. Two properties of the formulation are of practical consequence. First, the empirical characteristic function and its gradient are bounded, so the loss is robust to outliers and requires none of the whitening or centering operations of variance–covariance-based regularizers [7]. Second, the formulation is compatible with distributed training at negligible cost: the empirical characteristic function is averaged across all workers with a single all-reduce operation before the statistic is computed, such that SIGReg is evaluated on the full global batch rather than on per-device shards. The communication overhead is a single 2×M×172× M× 17 tensor, independent of both the batch size and the embedding dimension. Appendix B Architectural Details Unless stated otherwise, the encoder is a ViT-B/16. The rotary embeddings partition the dimensions of each attention head into three groups, rotated according to the temporal, vertical, and horizontal coordinates of the token, respectively; no absolute positional embeddings are used. The projector is a two-layer multilayer perceptron composed of a linear layer (d→2,048d→ 2,048), batch normalization, a GELU nonlinearity, and a final linear layer (2,048→K2,048→ K) with K=256K=256; it is discarded after pretraining, and all downstream evaluations operate on the encoder’s representations. The Polyak average of the encoder weights uses decay 0.99990.9999, updated every 3232 optimizer steps; it receives no forward passes during training and does not appear in the objective, and is therefore distinct from the exponential-moving-average target encoders employed for collapse prevention in prior work [16, 6]. Appendix C Evaluation Details Attentive probing. The probe follows Bardes et al. [6] without modification. A single cross-attention layer attends from one learnable query token to all output tokens of the frozen encoder; the attended output is combined with the query through a residual connection, passed through a two-layer MLP with a GELU non-linearity and layer normalization, and classified by a linear layer. The probe and classifier are trained jointly on the downstream training set while all encoder parameters remain fixed; optimization hyperparameters are taken from Bardes et al. [6]. Linear probing on Kinetics-400. For Kinetics-400, the output tokens of the frozen encoder are averaged into a single vector and a linear classifier is trained on the pooled representation.