Paper deep dive
Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/21/2026, 3:47:12 AM
Summary
The paper introduces NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised learning framework for audio that uses a causal Transformer to predict the next patch embedding of a log-mel spectrogram. Unlike traditional methods relying on reconstruction or student-teacher setups, NAPE employs a minimalist approach using only causal masking and stop-gradient. It achieves state-of-the-art performance on several audio and speech benchmarks, demonstrating strong scalability and structured attention patterns.
Entities (10)
Relation Signals (9)
NAPE → uses → Causal Transformer
confidence 95% · NAPE is a self-supervised framework in which a causal Transformer predicts each next patch embedding
NAPE → predicts → Next Patch Embedding
confidence 94% · NAPE predicts each next patch embedding of a log-mel spectrogram from the previous ones
NAPE → employs → Stop-Gradient
confidence 93% · using causal masking and stop-gradient as its sole training signal
NAPE → employs → Causal Masking
confidence 93% · using causal masking and stop-gradient as its sole training signal
NAPE → trainson → AudioSet
confidence 92% · We pre-train NAPE on AudioSet (Gemmeke et al., 2017) without labels
NAPE → processes → Log-Mel Spectrogram
confidence 91% · predicts each next patch embedding of a log-mel spectrogram
NAPE → evaluatedon → IEMOCAP
confidence 90% · We further evaluate on IEMOCAP... NAPE achieves state-of-the-art fine-tuning performance
NAPE → evaluatedon → Speech Commands V1
confidence 90% · We further evaluate on Speech Commands V1... NAPE achieves state-of-the-art fine-tuning performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Tags
Links
- Source: https://arxiv.org/abs/2608.19863v1
- Canonical: https://arxiv.org/abs/2608.19863v1
Trouble viewing inline? Open PDF directly →
Full Text
84,008 characters extracted from source content.
Expand or collapse full text
Preprint LISTENING FORWARD: NEXT PATCH EMBEDDING PRE- DICTION ENABLES SCALABLE AUDIO LEARNERS Umberto Cappellazzo 1 , Xubo Liu 2 , Stavros Petridis 1 , Maja Pantic 1 1 Imperial College London 2 University of Surrey u.cappellazzo@imperial.ac.uk Project Page§ Code ABSTRACT Self-supervised learning (SSL) has driven substantial progress in audio represen- tation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre- training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio’s temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consis- tently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision. 1INTRODUCTION Self-supervised learning (SSL) has emerged as a foundational paradigm for representation learning across modalities, delivering strong transfer performance without the cost of human annotations (Rad- ford et al., 2018; Devlin et al., 2019; Brown et al., 2020; Chen & He, 2021; He et al., 2022; Oquab et al., 2023; Baevski et al., 2020; Kong et al., 2024; Chen et al., 2024). In audio, SSL methods largely adapt paradigms first developed in vision. Vision Transformer backbones (Dosovitskiy et al., 2020) are pre-trained on spectrograms either through masked-spectrogram modeling (Niizumi et al., 2022; Huang et al., 2022; Chong et al., 2023; Chen et al., 2022b; Dinkel et al., 2024; Niizumi et al., 2026) or through student-teacher distillation objectives (Ahmed et al., 2024; Chen et al., 2024; Alex et al., 2025; Yang et al., 2025b), with domain-specific augmentations tailored to the time-frequency structure of audio. Most of these methods pre-train on AudioSet (Gemmeke et al., 2017) and are evaluated primarily by fine-tuning performance on downstream classification tasks. These paradigms, however, rely on increasingly elaborate pre-training recipes. Existing methods typically depend on reconstruction decoders that map latent features back to raw mel content (Huang et al., 2022; Chen et al., 2024), separately-trained acoustic tokenizers that supply discrete semantic targets (Chen et al., 2022b), teacher-student setups with exponential moving average (EMA)-updated encoders (Chen et al., 2024; Alex et al., 2025; Ahmed et al., 2024), auxiliary regularization losses that stabilize training (Fei et al., 2023; Alex et al., 2025), and multi-codebook vector quantisation (Yang et al., 2025b). Recently, a distinct line of work has begun to reshape SSL by shifting away from reconstruction toward the direct prediction of latent embeddings. This shift extends a paradigm long 1 arXiv:2608.19863v1 [eess.AS] 20 Aug 2026 Preprint established in language modeling, where models are trained not as static feature extractors, but as predictive systems that model the data distribution through a single causal objective. Autoregressive prediction has thereby provided a unified pre-training interface across modalities, from discrete tokens in language to continuous embeddings in vision. Two variants of this philosophy have gained traction. Joint-embedding predictive approaches (Assran et al., 2023; Oquab et al., 2023; Fei et al., 2023; Bardes et al., 2024; Balestriero & LeCun, 2025; Yuksel et al., 2025; Huang et al., 2026; Wu et al., 2026) predict the latent embeddings of masked regions from a context view, produced by a target branch that is typically maintained via an EMA of the online encoder or stabilized by auxiliary regularization losses. While these methods are scalable and achieve strong performance, most of them rely on heavy heuristics (e.g., EMA, frozen layers, teacher-student architectures) to ensure training stability. Autoregressive next-embedding approaches (Teoh et al., 2025; Xu et al., 2025; Bredis et al., 2026; Yao et al., 2026; Maes et al., 2026b), in contrast, predict each latent embedding directly from the preceding ones, mirroring the causal next-token objective that drives modern large language models (Radford et al., 2018; Brown et al., 2020; Liu et al., 2024; Yang et al., 2025a). Both variants have quickly become a promising direction for representation learning in vision. Despite this progress, predictive next-embedding methods remain absent from audio SSL, and joint-embedding predictive approaches themselves have seen only limited exploration in the audio and speech settings (Fei et al., 2023; Yuksel et al., 2025; Tuncay et al., 2025). The absence of any next-embedding autoregressive method for audio is particularly striking because audio is arguably the modality most naturally suited to this paradigm. Unlike images, whose 2D structure is approximately isotropic and admits no canonical ordering, audio signals unfold along a well-defined temporal axis and sequential structure is intrinsic to the signal, not imposed on it. Predicting the next patch of a spectrogram from the past ones mirrors both how acoustic events emerge in time and how modern language models learn from sequential data. Motivated by this natural alignment, we seek a next-embedding prediction framework for audio that is deliberately minimalist while still delivering state-of-the-art downstream performance across audio and speech benchmarks. We therefore introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised frame- work that brings the causal next-embedding prediction paradigm to audio. Given a log-mel spectro- gram, NAPE first applies a patch embedding layer that splits the spectrogram into non-overlapping patches and projects each of them into ad-dimensional embedding, producing a 2D grid of patch embeddings. Since the causal Transformer that follows operates on a 1D sequence, this grid must be traversed under a scanning order. This choice is a design axis specific to audio: unlike static images, spectrograms have a strong temporal axis and a qualitatively different frequency axis, so the order in which the grid is linearized determines both the causal context available at each prediction step and the structural inductive bias of the model. We consider four scanning strategies, depicted in Figure 2. A causal Transformer encoder then processes the linearized sequence, and a lightweight predictor head produces an estimate of the next patch embedding, analogous to next-token prediction in language modeling, but operating in continuous embedding space rather than over a discrete vocabulary. Prediction quality is measured by negative cosine similarity to the target patch embedding under a stop-gradient. This objective requires no reconstruction decoder, no acoustic tokenizer, no student-teacher setup, and no auxiliary regularization losses: the entire learning signal comes from the model’s ability to anticipate the next embedding in the sequence. Extensive experiments on standard audio and speech benchmarks, including AudioSet (Gemmeke et al., 2017), ESC-50 (Piczak, 2015), Speech Commands V1 and V2 (Warden, 2018), and IEMO- CAP (Busso et al., 2008), show that NAPE achieves state-of-the-art performance on several tasks. NAPE exhibits favorable scaling properties across three encoder sizes: Small, Base, and Large, with 19/85/303 million parameters, respectively. Beyond fine-tuning, NAPE delivers strong linear-probing results despite being a purely predictive model whose objective is not aligned with linear separability, indicating that the learned representations remain discriminative even under strict feature-freeze evaluation. A qualitative analysis of NAPE’s attention patterns and embedding-space structure further confirms that NAPE learns meaningful, structured features from audio. Our main contributions are: •We introduce NAPE, the first self-supervised audio framework built around causal next-patch- embedding prediction, offering a substantially simpler alternative to reconstruction-based, masked-modeling, and joint-embedding predictive approaches for audio. • Through systematic ablations, we identify the design axes that make next-embedding prediction work in the audio setting—scanning order, predictor head, prediction target, 2 Preprint Embedding Layer ( ) Input Spectrogram ( ) Causal Transformer ( ) Predictor Head ( ) Similarity ( ) o o Embedding Layer Predictor/Task Head Layer Norm Input Spectrogram Transformer Layers LayerNorm QKV Causal Attention Layer Norm Layer Norm LayerScale LayerNorm FFN LayerScale + + ...... Input Spectrogram Embedding Layer PredictedTo predict at this stepUnknown Figure 1: Left: NAPE’s overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the next patch (red border) using only the embeddings of the preceding patches; patches at future positions are hidden by the causal attention mask. Middle: The NAPE pipeline: patch embeddingszare processed by the causal encoderhand predictorgto produce predictionsˆz, which are compared against the targetsz under stop-gradient using a similarity functionD(i.e., negative cosine similarity). Right: Encoder architecture (h): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization. patch embedding layer, and the three key components of causality, prediction shift, and stop-gradient—and converge on an optimal configuration for the framework. •Across six audio and speech benchmarks, NAPE achieves strong downstream results at three encoder scales, exhibits favorable scaling behavior, and delivers competitive linear-probing performance. •We provide qualitative evidence that NAPE learns structured representations, with attention patterns that reason jointly about the current spectral context and the frequency-consistent temporal history, and embedding-space behavior that groups acoustically-similar patches without any explicit labels. 2THE NAPE FRAMEWORK In this section, we describe the NAPE framework in detail. NAPE is a self-supervised pre-training method that trains a causal Transformer to predict the embedding of the next patch of a log-mel spectrogram from the preceding ones, relying only on causal masking and stop-gradient. We describe NAPE’s main components and downstream adaptation in the next subsections. Figure 1 depicts the overall framework. 2.1SPECTROGRAM PATCHIFICATION AND SCANNING ORDER Input Representation. Given a raw waveform, we compute a log-mel spectrogramx ∈ R 1×F×T frames withFfrequency bins andT frames time frames. We splitxinto non-overlapping square patches of sizeP × P, yielding a 2D grid ofN = T F · T T patches withT F = F/P andT T = T frames /P. Each patch is projected to ad-dimensional embedding by a patch embedding layerf, producing the sequencez 1 ,...,z N ∈R N×d . We use a standard Conv2d patch embedding layer by default, but we also consider a deeper convolutional stem with batch normalization (con- vstem) (Xiao et al., 2021) and a speech-oriented stem that treats the mel axis as feature channels and applies temporal-only convolutions (speechstem) (Team et al., 2026). We refer to Section 3.2 for details and results about the choice of f . Scanning Order. Transformer encoders operate on 1D sequences, so a 2D patch grid must be linearized before it can be processed. Under bidirectional attention, as used in prior audio SSL methods (Gong et al., 2021a; Huang et al., 2022; Chen et al., 2024), the choice of linearization is not a functional design choice: self-attention is permutation-equivariant given positional embeddings, so any consistent ordering yields the same representations. NAPE’s causal formulation, however, breaks this equivariance as the ordering determines which patches are “past” (visible to a given position) and 3 Preprint 18 19 20212223 121314151617 67891011 012345 Time Freq 3711151923 2610141822 159131721 048121620 Time Freq 232221201918 121314151617 11109876 012345 Time Freq 91317202223 5812161921 247111518 01361014 Time Freq Raster left right, bottom top Time-major bottom left, left right ZigzagDiagonal sweep by (freq + time)left right, right left Figure 2: Patch scanning orders. NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: raster (left-to-right, bottom-to-top), time-major (bottom-to-top within each time column, then advance in time), zigzag (raster with alternating row directions), and diagonal (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence. which are “future” (to be predicted), and therefore imposes a substantive inductive bias on the model. Since spectrograms have a strong temporal axis and a qualitatively different frequency axis, unlike natural images, whose 2D structure is approximately isotropic, the choice of scanning order for causal prediction is particularly consequential. Thus, we consider four orderings, illustrated in Figure 2. Raster (left-to-right, bottom-to-top) sweeps time before advancing in frequency and is the standard patch ordering used in vision and audio Transformer models, where each prediction is conditioned on the entire past time axis at every frequency traversed so far. Time-major (bottom-to-top within each time column, then advance in time) sweeps frequency before advancing in time, so each prediction is conditioned on the full frequency profile of every past time step. Zigzag is an alternating-direction variant of raster: rows alternate left-to-right and right-to-left, so consecutive patches in the sequence remain spatially adjacent even across row transitions. This avoids the spatial discontinuity raster incurs when jumping from the end of one row to the start of the next. Diagonal sweeps patches along anti-diagonals of the grid, mixing time and frequency progression at every step and thereby encoding 2D spatial priors more uniformly than the other three orders. 2.2NEXT-AUDIO-PATCH-EMBEDDING PREDICTION Prediction Task. Given the embedding sequencez =z 1 ,...,z N produced byfin the chosen scanning order, NAPE jointly trains an encoderhand a lightweight predictor headgso that, at each positiont, the model produces an estimate of the next patch embeddingz t+1 using only the previous patches: ˆz t+1 = g(h(z ≤t )),(1) wherez ≤t =z 1 ,...,z t denotes the patch embeddings at all positions up to and includingt. The restriction to the past patches is enforced by a causal attention mask that prevents each position from attending to patches at later positions in the sequence. This is directly analogous to next-token prediction in language modeling, but operates in a continuous embedding space rather than over a discrete vocabulary. Loss Function. Following SimSiam (Chen & He, 2021), we measure prediction quality by the negative cosine similarity between the target embedding z t+1 and the predicted embedding ˆz t+1 : D(z t+1 , ˆz t+1 ) = − z t+1 ∥z t+1 ∥ 2 · ˆz t+1 ∥ˆz t+1 ∥ 2 ,(2) where∥·∥ 2 is theℓ 2 -norm. Applying stop-gradient (stopgrad) on the target and averaging over all valid prediction positions yields the NAPE objective: L = 1 N − 1 N−1 X t=1 D(stopgrad(z t+1 ), ˆz t+1 ).(3) Thestopgradoperator treats the target embedding as a constant, so gradients flow only through the predicted side. Cosine similarity is magnitude-invariant, which prevents the trivial solution of shrinking both sides of the objective toward zero norm. We compare cosine similarity against alternative similarity functions in Section 3.2 and find it to yield the optimal results. 4 Preprint Causal Transformer Encoder (h). We use a Vision Transformer backbone (Dosovitskiy et al., 2020; Wang et al., 2026) with pre-norm design and a causal attention mask during pre-training (see Figure 1, right panel). For stability at depth, we adopt Rotary Position Embedding (RoPE) (Su et al., 2024) applied independently along the frequency and time axes, LayerScale (Touvron et al., 2021) on residual branches, and parameter-free query-key normalization (Henry et al., 2020) on the per-head query and key projections. We instantiate three configurations of increasing size to test NAPE’s scalability: Small (d = 384,12layers,6heads;∼19M parameters), Base (d = 768,12layers,12 heads;∼85M parameters), and Large (d = 1024, 24 layers, 16 heads;∼303M parameters). Predictor Head (g). The predictor headgdecouples the representation space ofhfrom the space in which predictions are made. Predicting directly with the encoder output forces the encoder to place its representations in the same space as the prediction targets, which can restrict feature richness. This asymmetric predictor design, together with the stop-gradient applied to the target branch, gives NAPE a structural resemblance to SimSiam (Chen & He, 2021): representation collapse is avoided without contrastive negatives or an EMA teacher, relying instead on the asymmetry between the two branches. NAPE differs from SimSiam in what the branches encode: SimSiam compares two augmented views of the same input under a symmetric encoder, whereas NAPE compares a prediction of the next patch to that patch’s actual embedding within a single autoregressive sequence, replacing view augmentation and siamese symmetry with temporal prediction as the source of the learning signal. We study multiple predictor styles in Section 3.2. Algorithm 1 NAPE pre-training algorithm. #f: Patch Embedding Layer #h: Causal Transformer Encoder #g: Predictor for x in loader: # x: [B,1,F,T_frames] z = f(x) # embeddings [B,T,D] z_hat = g(h(z)) # predictions [B,T,D] loss = D(z, z_hat) loss.backward(); # update(f,h,g) def D(z, z_hat): target = z[:,1:, :].detach() # stop-grad pred = z_hat[:,:-1, :] # AR shift pred = normalize(pred, dim=-1) target = normalize(target, dim=-1) return -(pred * target).sum(-1).mean() NAPE’s Key Components. Three complemen- tary mechanisms prevent NAPE from converging to trivial solutions and together define its training regime. (i) causality: enforced by the causal at- tention mask that restricts each position to attend only to prior patches, it prevents the encoder from attending to the target patch when producing its prediction, blocking the trivial identity mapping. (i) The prediction shift between input and target positions ensures that the model at positiontpre- dicts the embedding at positiont+1rather than the embedding at its own position, so causality alone cannot be side-stepped by copying the current input through. (i) stop-gradient on the target embedding prevents gradients from flowing through both sides of the loss simultaneously, which would otherwise allow the encoder to collapse all embeddings toward a shared constant. Together, these three ingredients constitute the core of NAPE’s self- supervised recipe, and we analyze the individual contribution of each in Section 3.2. The pseudocode of NAPE’s pre-training is given in Algorithm 1. 3EXPERIMENTS In this section, we carry out extensive experiments to evaluate NAPE at different scales and con- figurations. (1) We first perform several ablations on key design choices to converge on the best NAPE configuration (Section 3.2); (2) we then compare that configuration against state-of-the-art methods at multiple scales (Section 3.3 and 3.4); (3) we assess linear separability of the learned features (Section 3.5), and (4) we finally analyze how NAPE organizes acoustic information through its attention patterns and the structure of its learned embeddings (Section 3.6). 3.1EXPERIMENTAL SETUP Pre-training Data. We pre-train NAPE on AudioSet (Gemmeke et al., 2017) without labels, combining the unbalanced and balanced training splits. We obtained and processed1,964,222clips from the unbalanced split,20,961clips from the balanced split, and18,900clips from the evaluation split, consistent with prior work (Chen et al., 2022b; 2024; Alex et al., 2025). All input waveforms are resampled to mono at16kHz and converted into log-mel spectrograms with128mel bands using a25ms Hanning window and10ms hop size. A10-second clip yields a spectrogram of size 5 Preprint 1× 128× 1008(channel, frequency, time), corresponding to8× 63 = 504non-overlapping16× 16 patches per clip (as in (Gong et al., 2021a; Chen et al., 2022b; 2024)). Downstream Benchmarks.For downstream evaluation, we fine-tune on AudioSet-2M (AS-2M, un- balanced) and AudioSet-20K (AS-20K, balanced), applying the weighted sampling strategy of Huang et al. (2022) on AS-2M to mitigate class imbalance. We further evaluate on ESC-50 (Piczak, 2015), a50-class environmental sound classification benchmark of2000clips at5s each, using5-fold cross-validation; Speech Commands V1 and V2 (KS1, KS2) (Warden, 2018), keyword spotting tasks with12and35classes respectively; and IEMOCAP (ER) (Busso et al., 2008), a4-class speech emotion recognition benchmark with5-fold cross-validation. Together these tasks span both audio and speech domains. We report mean average precision (mAP) on the multi-label AudioSet tasks and top-1 classification accuracy on the single-label tasks; for ESC-50 and IEMOCAP we report the mean across cross-validation folds. Pre-training Details.We pre-train with the AdamW optimizer (Loshchilov & Hutter, 2017) using a base learning rate of5× 10 −3 with cosine decay, a weight decay of0.05,β 1 = 0.9,β 2 = 0.95, and a batch size of256for the Small and Base configurations, and128for NAPE Large. We pre-train NAPE Small and NAPE Large for25epochs and NAPE Base for30epochs, with a warmup ratio of 10%. All models are trained using HuggingFace Trainer with distributed data parallelism (DDP) on NVIDIA L40s GPUs (46GB). Fine-tuning Details. For each downstream task, we initialize from the pre-trained encoder and attach a linear classifier on top of mean-pooled patch tokens. We disable the causal attention mask during fine-tuning so that attention is bidirectional over the full patch sequence. We provide an ablation study on the optimal pooling method and whether to use causal/bidirectional attention in the Appendix D. We use AdamW withβ 2 = 0.999, cosine learning-rate decay, and layer-wise learning-rate decay (Clark et al., 2020; Bao et al., 2021). We use binary cross-entropy for AudioSet and soft-target cross-entropy for the single-label tasks. Following prior work (Gong et al., 2021a; Chen et al., 2024; 2022b), we apply a standard augmentation stack (SpecAugment (Park et al., 2019), Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), DropPath (Huang et al., 2016), temporal roll, additive noise, and label smoothing (Szegedy et al., 2016)). For AS-2M and AS-20K we additionally maintain an exponential moving average (EMA) (Polyak & Juditsky, 1992) of the fine-tuning weights (decay0.99995and0.999respectively); no EMA is used for the other tasks. Full task-specific hyperparameters are reported in the Appendix C. Linear Probing Details. For linear probing, we freeze the pre-trained encoder and train only the classifier plus a preceding LayerNorm (Ba et al., 2016). All augmentations are disabled. Baselines. We primarily compare against recent in-domain self-supervised methods, includ- ing Audio-MAE (Huang et al., 2022), BEATs (Chen et al., 2022b), A-JEPA (Fei et al., 2023), ASiT (Ahmed et al., 2024), EAT (Chen et al., 2024), SSLAM (Alex et al., 2025), and SPEAR (Yang et al., 2025b). We also include results from out-of-domain and in-domain supervised pre-training. For all baseline methods, fine-tuning results are taken from the original papers. 3.2NAPE’S OPTIMAL CONFIGURATION In this section, we ablate the main design axes of NAPE to identify its optimal configuration. Unless otherwise specified, all ablations use the NAPE Base model and the raster scanning order. Each ablation modifies a single design axis at a time, holding the rest of the configuration fixed at NAPE’s defaults. PredictionShift/Stop-gradient/Causality. Table 1 disentangles the three key mechanisms of NEPA: the prediction shift, the stop-gradient on the target, and the causal attention mask. Removing either the prediction shift or the stop-gradient causes pre-training to diverge, matching the analysis in Section 2.2: without the shift, the model at positiontcan trivially satisfy the objective by copying its own input embedding through; without the stop-gradient, gradients flow through both sides of the loss and the encoder collapses all embeddings toward a shared constant. 6 Preprint Table 1:AblationonmainNAPE’sdesignelements: prediction shift, stop-gradient, and causal objective. Pred stop causalAudio TasksSpeech Tasks shift grad mask AS-2M AS-20K ESC-50 KS1 KS2 ER ✘✓Diverge ✓✘✓Diverge ✓✘41.824.868.9 96.1 97.3 57.0 ✓ 49.6 39.1 94.2 97.9 98.8 64.9 Removing the causal mask, in contrast, does not diverge: the prediction shift alone con- tinues to define a nominal prediction tar- get, but degrades downstream performance sharply, with the largest drops on the audio benchmarks (−7.8mAP on AS-2M,−14.3 on AS-20K, and−25.4points on ESC-50; smaller but consistent drops on the speech tasks). Without causality, the encoder can attend to the target patch while producing its prediction, so the objective is trivially satisfied by a near-identity mapping that routes each target back to its predicted position (the loss saturates near−1within a few thousand steps of pre-training, see Appendix F): the loss decreases during pre-training, but the encoder is not forced to learn useful structure. All three mechanisms are therefore jointly necessary, none can be dropped without either destabilizing training or degrading the learned representations to a degree that fine-tuning cannot recover. Table 2:Ablationonthepatchembeddinglayer. Patch Emb.Audio TasksSpeech Tasks LayerAS-2M AS-20K ESC-50 KS1 KS2 ER Convstem46.734.389.1 97.4 98.3 63.6 Speechstem47.633.188.4 98.1 98.9 63.0 Conv2d49.6 39.1 94.2 97.9 98.8 64.9 PatchEmbeddingLayerf.Table 2 compares three patchifiers, all configured to produce the same sequence length (504 tokens per10s clip). The default Conv2d is a single strided convolu- tion with kernel and stride16× 16, which treats time and frequency as symmetric 2D axes and matches the standard ViT design (Dosovitskiy et al., 2020). We also consider two alternatives motivated by observations from computer vision and speech literature. The Convstem, following Xiao et al. (2021), replaces the single Conv2d with four3× 3stride-2 convolutions interleaved with batch normalization with the channel count growing progressively towardd; deeper convolutional stems have been shown to stabilize optimization and improve downstream performance in ViT-based image models. The Speechstem, inspired by the audio front-end of speech-oriented models such as Gemma 3n and Gemma 4 (Team et al., 2026), flattens the mel axis into feature channels and applies temporal-only3× 3convolutions with a temporal downsampling factor of2, producing a 1D time-only sequence of504tokens that reflects the frame-based processing typical of speech recognition systems. Both alternatives underperform Conv2d across all benchmarks, with the largest gaps on AudioSet: on AS-20K, Convstem loses4.8mAP points and Speechstem loses6.0mAP spoints relative to Conv2d. Convstem’s added non-linearity and speechstem’s temporal-first inductive bias therefore do not translate into gains for our causal spectrogram prediction objective, treating frequency and time as symmetric 2D axes at the patchification stage is important for the pretraining signal that NAPE exploits. Conv2d remains NAPE’s default. Table 3: Ablationonthepredictor-stylevariants. Predictor #Par. Audio TasksSpeech Tasks StyleAS-2M AS-20K ESC-50 KS1 KS2 ER None-48.737.893.3 98.1 98.8 64.2 2-MLP1.2M49.438.593.6 98.0 98.8 64.2 Transformer 14.2M 49.238.493.0 98.2 98.7 65.0 SimSiam1.8M 49.6 39.1 94.2 97.9 98.8 64.9 Predictorg. Table 3 compares four pre- dictor variants: no predictor (encoder out- put used directly as the prediction), a two-layer MLP (2-MLP;Linear(d,d) → GELU → Linear(d,d)), a SimSiam- style (Chen & He, 2021) three-layer predictor with intermediate LayerNorms (Linear(d,d) → LN → GELU → Linear(d,d) → LN → GELU → Linear(d,d)), and a Transformer predictor in the style of JEPA-family methods: a 2-layer causal Transformer (16heads) operating in the encoder’s hidden dimension (we apply a causal mask to prevent future leaking as for the encoder). Adding any predictor improves over using the encoder output directly, confirming that decoupling the representation and prediction spaces benefits the learned features. Among the three predictor styles, the SimSiam variant performs best on the audio benchmarks and on IEMOCAP, while remaining on par on the keyword-spotting tasks. Notably, the lightweight SimSiam predictor outperforms the Transformer predictor despite being nearly8× smaller in parameters, indicating that additional predictor capacity is not the bottleneck for NAPE, 7 Preprint 48.0 48.2 48.4 48.6 48.8 49.0 49.2 49.4 49.6 mAP AS-2M 49.39 49.27 49.13 48.55 35.5 36.0 36.5 37.0 37.5 38.0 38.5 39.0 mAP AS-20K 38.50 38.72 38.43 36.37 90.0 91.0 92.0 93.0 94.0 Accuracy (%) ESC-50 93.60 94.06 93.75 91.04 97.0 97.2 97.4 97.6 97.8 98.0 98.2 Accuracy (%) GSC v1 98.00 97.7997.79 97.37 98.3 98.4 98.5 98.6 98.7 98.8 98.9 Accuracy (%) GSC v2 98.80 98.64 98.5598.55 62.0 63.0 64.0 65.0 66.0 Accuracy (%) IEMOCAP 64.20 65.84 63.48 62.63 RasterDiagonalZigzagTime-major Figure 3:NAPE’sperformanceacrossfourscanordersonsixbenchmarks. consistent with the observation in the siamese self-supervised setting (Chen & He, 2021) that a compact MLP predictor suffices when combined with a well-designed encoder and stop-gradient target. The SimSiam-style predictor is NAPE’s default. PatchScannningVariants.Figure 3 compares the four scanning orders introduced in Section 2.1 (we use the 2-MLP predictor style). Diagonal, raster, and zigzag all perform comparably well across the six benchmarks, with diagonal and raster slightly ahead of zigzag on most tasks. In contrast, time-major consistently underperforms the other three orders across all tasks. We interpret this pattern as reflecting the temporal structure of audio: diagonal, raster, and zigzag all advance in time as the causal sequence progresses, allowing the model to accumulate temporal context in a way that matches how acoustic events unfold. Time-major, instead, exhausts each frequency column before advancing in time, so predictions early in the sequence are conditioned on rich instantaneous spectra but only limited temporal context, a mismatch with the temporal nature of audio classification. Given their superior performance, we retain both raster and diagonal in the main comparison against state-of-the-art methods in Section 3.4. Table 4:AblationonNAPE’stargettopredict. PredictedAudio TasksSpeech Tasks TargetAS-2M AS-20K ESC-50 KS1 KS2 ER 1st enc. layerDiverge Raw Mel49.738.0 94.8 97.7 98.6 64.2 Patch embed 49.6 39.194.2 97.9 98.8 64.9 PredictionTarget. Table 4 compares three choices for the target of the auto-regressive pre- diction: (i) the patch embeddingz t+1 produced by the shared embedding layerf(the default); (i) the raw mel content of the next patch i.e., the flattened mel-spectrogram values inside the patch, following the target formulation used by masked reconstruction methods such as Audio MAE (Huang et al., 2022); (i) the output of the first encoder layer, in the style of JEPA methods that use deeper encoder features as prediction targets. For the raw mel variant, we add a linear projection on top of the predictor g to map its output from d dimensions to the raw patch dimensionality. Patch embedding and raw mel yield comparable results across all benchmarks, indicating that both are valid target choices for NAPE. Using the first encoder layer as target, however, causes pre-training to diverge: the target itself depends on the encoder being trained, and stop-gradient alone is insufficient to prevent the encoder from collapsing both sides of the loss to a shared constant. JEPA-family methods circumvent this instability with additional regularization such as EMA teachers (Assran et al., 2023; Fei et al., 2023), variance-covariance regularizers such as VISReg (Wu et al., 2026) and sketched isotropic gaussian regularizers such as SIGReg (Balestriero & LeCun, 2025), which we do not employ here. Since the patch embedding approach on average performs better than raw mel and it is adopted in (Xu et al., 2025) as well, we retain the patch embedding as NAPE’s default. Table 5: Ablationonthesimilarityfunction. SimilarityAudio TasksSpeech Tasks FunctionAS-2M AS-20K ESC-50 KS1 KS2 ER L1Diverge L2Diverge Cross-entropy 48.837.493.6 98.0 98.7 64.5 Cosine49.6 39.1 94.2 97.9 98.8 64.9 SimilarityFunction(D).In Table 5 we com- pare four choices for the similarity functionD in Eq. 2: the negative cosine similarity (NAPE’s default), a soft cross-entropy formulation that treats prediction and target as distributions over thedchannels after applying a softmax (as in Chen & He (2021)), and theℓ 1 andℓ 2 dis- tances between the twod-dimensional vectors. Bothℓ 1 andℓ 2 cause pre-training to diverge: un- 8 Preprint SmallBaseLarge 48.0 48.8 49.6 50.4 mAP AS-2M SmallBaseLarge 36.0 37.5 39.0 40.5 mAP AS-20K SmallBaseLarge 93.0 94.0 95.0 96.0 Accuracy (%) ESC-50 SmallBaseLarge 97.2 97.5 97.8 98.1 98.4 Accuracy (%) KS1 SmallBaseLarge 98.0 98.2 98.5 98.8 99.0 Accuracy (%) KS2 SmallBaseLarge 64.5 66.0 67.5 69.0 Accuracy (%) IEMOCAP (ER) RasterDiagonal Figure 5: NAPE’s results at different scales under raster and diagonal scan orders. like cosine similarity, which is magnitude-invariant, these distance-based objectives can be trivially minimized by shrinking the norm of both predicted and target embeddings toward zero, a form of representation collapse in which the encoder outputs converge to a shared low-magnitude constant. The cross-entropy variant trains stably (softmax normalization implicitly bounds the target magnitude) but underperforms cosine on all benchmarks except the keyword-spotting tasks, where the two are on par. Cosine similarity, combining magnitude-invariance with a directionally informative loss signal, yields the best overall results and is retained as NAPE’s default. NAPE’s Optimal Configuration (i) Conv2d patch embedding layer, (i) raster/diagonal scanning order, (i) SimSiam-style predictor with three-layer MLP and intermediate LayerNorms, (iv) patch embedding as the prediction target, and (v) negative cosine similarity loss with stop-gradient on the target branch. Additional Ablation Results. We refer to Appendix D for additional ablations studies on: 1) the use of normalization layers (LayerNorm vs RMSNorm), 2) freezing/unfreezing the patch embedding layer, 3) the optimal positional encoding (absolute encoding vs RoPE), 4) the optimal pooling method during fine-tuning, 5) the optimal attention type during fine-tuning (causal vs bidirectional), (6) the additional use of random masking during pre-training, and 7) NAPE’s performance in terms of different compute budgets. 3.3SCALING NAPE Small ~20M Base ~85M Large ~300M 45.0 46.5 48.0 49.5 mAP AS-2M 45.0 47.3 47.4 47.6 49.6 50.2 Small ~20M Base ~85M Large ~300M 32.0 34.0 36.0 38.0 40.0 mAP AS-20K 32.1 37.1 37.6 36.2 39.1 40.5 Audio-MAENAPE Figure 4: Scaling comparison between Audio- MAE and NAPE, raster. Figure 5 shows downstream performance for the raster and diagonal variants of NAPE at three en- coder scales: Small (∼19M parameters, NAPE-S), Base (∼85M, NAPE-B), and Large (∼303M, NAPE- L). NAPE scales positively across the board: on every one of the six benchmarks, NAPE-L improves over NAPE-B, which in turn improves over the small ver- sion. While this improvement tends to diminish as we scale from the base to the large model, we observe consistent scaling gains for most of the tasks, while the keyword-spotting tasks show smaller absolute gains, reflecting their already-saturated accuracy levels. Raster and diagonal track each other closely at the Small scale, with essentially identical performance across all six tasks. At the Base and Large scales, diagonal slightly outperforms raster on most benchmarks, while raster achieves the strongest result on AS-2M with NAPE-L reaching50.18 mAP. In Figure 4, we compares NAPE (raster) and Audio-MAE (Huang et al., 2022) at three encoder scales on AS-2M and AS-20K. NAPE outperforms Audio-MAE at every scale, with particularly large margins at the Small size (+2.6mAP on AS-2M,+4.1mAP on AS-20K). NAPE also exhibits more favorable scaling behavior from Base to Large. 3.4COMPARISON WITH STATE-OF-THE-ART We compare NAPE against prior audio pre-training methods in Table 6. We use the best configuration identified in Section 3.2, reporting both raster and diagonal variants at the Base and Large scales. NAPE-B with diagonal scan delivers strong performance across all six benchmarks, matching or 9 Preprint Table 6: Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We gray-out the models and results with additional supervised training on external datasets. Best results are in bold, second-best are underlined. Model#Par. Pre-trainAudio TasksSpeech Tasks DataAS-2M AS-20KESC-50 KS1KS2ER Out-of-domain Supervised Pre-training PSLA (Gong et al., 2021b)14MIN44.431.9--96.3- AST (Gong et al., 2021a)86MIN45.934.788.795.5 98.1 56.0 HTS-AT (Chen et al., 2022a)31MIN47.1---98.0- Audio-CLIP (Guzhov et al., 2022)93MTI+AS25.9-96.7--- In-domain Supervised Pre-training AST (Gong et al., 2021a)86MIN+AS45.9-95.6-97.9- HTS-AT (Chen et al., 2022a)31MIN+AS47.1-97.0--- Audio-MAE (Huang et al., 2022)86MAS--97.4--- Self-Supervised Pre-training S-AST (Gong et al., 2022a)89MAS+LS-31.088.896.0 98.0 59.6 MAE-AST (Baade et al., 2022)86MAS+LS-30.690.095.8 97.9 59.8 CAV-MAE (Gong et al., 2022b)86MIN+AS44.934.2---- Audio-MAE (Huang et al., 2022)86MAS47.337.194.196.9 98.3- Audio-MAE L (Huang et al., 2022)304MAS47.437.7---- data2vec (Baevski et al., 2022)94MAS-34.5---- MaskSpec (Chong et al., 2023)86MAS47.132.389.6-97.7- BEATs iter3 (Chen et al., 2022b)90MAS48.038.395.697.7 98.3 64.5 A-JEPA (Fei et al., 2023)86MAS48.638.496.397.7 98.5- ASiT (Ahmed et al., 2024)86MAS48.038.695.398.2 98.9- EAT (Chen et al., 2024)88MAS48.640.295.9-98.3- SSLAM (Alex et al., 2025)88MAS50.240.996.2 98.8 98.1- SPEAR a Large (Yang et al., 2025b) 327MAS49.739.3---- NAPE-B raster85MAS49.639.194.297.9 98.864.9 NAPE-B diagonal85MAS49.739.294.897.9 98.6 67.1 NAPE-L raster303MAS50.240.596.097.9 98.868.0 NAPE-L diagonal303MAS50.040.496.298.298.9 68.8 exceeding every self-supervised competitor of comparable size. When we scale it further, NAPE-L raster attains results on par with the strongest baseline, SSLAM (Alex et al., 2025), tying it on AS-2M (50.2mAP) and approaching it closely on AS-20K (40.5vs40.9mAP) and ESC-50 (96.0 vs96.2%). What is notable about these results is the simplicity of underlying NAPE’s recipe. Unlike SSLAM, which relies on audio-mixture supervision, a student-teacher architecture, and a reconstruction decoder, NAPE requires none of these ingredients: its pre-training objective consists of a stop-gradient and a negative cosine similarity between the predicted next patch embedding and its target. NAPE also outperforms A-JEPA (Fei et al., 2023) on5benchmarks despite A-JEPA relying on an auxiliary target encoder updated via EMA, predicting multiple masked patches in parallel rather than causally, and adopting a regularized masking strategy during fine-tuning. Finally, NAPE transfers particularly well to speech tasks: on IEMOCAP, NAPE-L reaches68.0% accuracy, a+3.5-point improvement over the strongest baseline result reported at any scale (BEATs iter3 at 64.5%), suggesting that NAPE learns representations that generalize well beyond acoustic-event classification. 3.5LINEAR PROBING RESULTS WhichLayertoProbe? Linear probing measures the linear separability of the fea- tures produced by the pretrained encoder, treating the encoder as a fixed feature extrac- tor and training only a linear classification head.Following recent observations that the most classification-relevant features in deep Transformer models often lie in interme- diate rather than final layers (Skean et al., 2025; Bolya et al., 2026), we begin our lin- ear probing study by measuring downstream performance as a function of encoder depth. 10 Preprint 0%20%40%60%80%100% Relative encoder depth 10.0 12.5 15.0 17.5 20.0 mAP Small (6 layers) Base (12 layers) Large (24 layers) Figure 6:Layer-wiselinearprobinganalysis. Figure 6 reports AS-20K mAP obtained by linearly probing each layer of NAPE-S, NAPE-B, and NAPE- L, using the best raster configuration identified in Section 3.2. Across all three scales, the best probing layer lies at roughly the middle of the encoder: layer 2for NAPE-S, layer6for NAPE-B, and layer11 for NAPE-L. Beyond this mid-network optimum, performance declines steadily toward the final layer, dropping by roughly3-5mAP points. This pattern is consistent with the interpretation that the top layers of a NAPE-pretrained encoder specialize for the next-patch-embedding prediction objective, while the mid-layers retain more general and classification-relevant information. Table 7:Linearprobingresults. Model Layer AS-2M AS-20K ESC-50 Small2nd23.218.979.8 Base6th25.019.781.7 Large 11th 27.1 20.4 83.5 LinearProbingAcrossScalesandTasks.We then use the best probing layer identified per model to compare NAPE-S, NAPE-B, and NAPE-L on AS-2M, AS-20K, and ESC-50. Table 7 reports the results. NAPE scales positively under linear probing: larger models yield stronger probes on every task, from AS-2M (+3.9mAP from small to large) to ESC-50 (+3.7in accuracy). This trend reinforces the scaling behavior observed under fine-tuning (Section 3.3). At the same time, the absolute linear-probing numbers are noticeably lower than their fine-tuning counterparts. This gap is expected for prediction-based self-supervised methods: the pretraining objective encourages the encoder to learn features that support the next-patch prediction task, which do not necessarily align with the linear separability needed for classification. 3.6QUALITATIVE RESULTS To gain insight into what NAPE learns, we complement the quantitative benchmarks with two qualitative analyses of a pretrained NAPE-L (raster) on the AudioSet evaluation set. Both use the model after pretraining, with no fine-tuning. More qualitative results can be found in the Appendix G. PredictionFidelity. We measure the cosine similarity between the predicted embeddingsˆz t+1 and the true patch embeddingsz t+1 . Figure 7 (top left) shows the similarity averaged across500 held-out AudioSet clips, and the top middle and right panels report the same measurement for two individual clips: NAPE predicts the next patch embedding accurately almost everywhere on the grid, with similarity close to the ceiling of1.0both on average and per clip. The remaining low-similarity regions have clear structural explanations. The very first patch has the lowest similarity, since no previous context is available for the prediction to condition on. The first mel row is harder on average, since it combines limited past context with low-frequency patches. Finally, the rightmost patches show slightly lower similarity because the last time columns correspond to zero-padded frames appended to reach the target clip length. Away from these boundary regions, NAPE satisfies its pre-training objective on unseen audio. AttentionandEmbeddingAnalysis.To understand how NAPE arrives at its predictions, we select a query patch (marked in red in Figure 7, bottom left) and analyze the attention and embedding structure it induces. The attention map (bottom middle), which conveys the aggregated attention from the query position to every other patch averaged over all layers and heads, reveals a highly structured pattern with two distinct components: NAPE attends strongly to the current time column, integrating the full spectral profile of the current moment, and to same-mel-frequency patches earlier in the clip, tracking the temporal evolution of the frequency it is about to predict. The embedding-similarity map (bottom right), which compares the predicted embeddingˆz t+1 against every actual patch embedding in the clip, shows thatˆz t+1 is most similar to patches that share acoustic structure and energy with the query, regions of the spectrogram carrying comparable spectral content, with similarity gradually decaying at the temporal and spectral extremes. This grouping into coherent acoustic components emerges without any explicit labels or region annotations, suggesting that despite being trained only on a local next-patch objective, NAPE develops representations that capture the broader acoustic structure of the clip. 11 Preprint Figure 7: Top:PredictionQualityAnalysis. We report the cosine similarity between the predicted and true patch embeddings, averaged over500audio clips (left) and for two individual clips (middle, right). Bottom:Attention/EmbeddingAnalyses. Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embeddingˆz t+1 and every actual patch embedding in the spectrogram. 4CONCLUSION We presented NAPE, a self-supervised framework for audio representation learning based on causal next patch embedding prediction. Departing from the reconstruction- and masking-based approaches that dominate audio SSL, NAPE relies on a deliberately minimalist recipe: a single causal Transformer encoder, a lightweight predictor head, and a negative cosine similarity loss with stop-gradient on the target branch. Across six audio and speech benchmarks, NAPE achieves strong downstream performance while relying on a substantially simpler pre-training recipe. NAPE also exhibits favorable scaling behavior across three encoder sizes and delivers competitive linear-probing results. A qualitative analysis further shows that the model develops structured attention patterns and organizes its learned embeddings into acoustically coherent regions without any explicit supervision. Together, these results establish autoregressive next-embedding prediction as a simple, scalable, and effective self-supervised objective for audio, and open a direct path to bringing the causal predictive paradigm into audio representation learning. ACKNOWLEDGMENTS We thank Andrew Rouditchenko (Nvidia) for his insightful and valuable discussions. REFERENCES Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Sara Atito Ali Ahmed, Muhammad Awais, Wenwu Wang, Mark D Plumbley, and Josef Kittler. Asit: Local-global audio spectrogram vision transformer for event classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3684–3693, 2024. Tony Alex, Sara Atito, Armin Mustafa, Muhammad Awais, and Philip Jackson. Sslam: Enhancing self- supervised models with audio mixtures for polyphonic soundscapes. In International Conference on Learning Representations, volume 2025, p. 22608–22626, 2025. Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding pre- dictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15619–15629. IEEE, 2023. 12 Preprint Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Alan Baade, Puyuan Peng, and David Harwath. Mae-ast: Masked autoencoding audio spectrogram transformer. arXiv preprint arXiv:2203.16691, 2022. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, p. 1298–1312. PMLR, 2022. Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021. Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024. Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Bangalath, et al. Perception encoder: The best visual embeddings are not at the output of the network. Advances in Neural Information Processing Systems, 38:60884–60937, 2026. George Bredis, Nikita Balagansky, Daniil Gavrilov, and Ruslan Rakhimov. Next embedding prediction makes world models stronger. arXiv preprint arXiv:2603.02765, 2026. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359, 2008. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 646–650. IEEE, 2022a. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, p. 1691– 1703. PMLR, 2020. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058, 2022b. Wenxi Chen, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, and Xie Chen. Eat: Self-supervised pre-training with efficient audio transformer. arXiv preprint arXiv:2401.03497, 2024. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 15745–15753. IEEE, 2021. Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng. Masked spectrogram prediction for self-supervised audio pre-training. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. IEEE, 2023. 13 Preprint Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Ieee, 2009. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186, 2019. Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, and Bin Wang. Scaling up masked audio encoder learning for general audio classification. arXiv preprint arXiv:2406.06992, 2024. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. IEEE, 2023. Zhengcong Fei, Mingyuan Fan, and Junshi Huang. A-jepa: Joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311.15830, 2023. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 776–780. IEEE, 2017. Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. In INTERSPEECH, 2021a. Yuan Gong, Yu-An Chung, and James Glass. Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292–3306, 2021b. Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, p. 10699–10709, 2022a. Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022b. Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 976–980. IEEE, 2022. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 15979–15988. IEEE, 2022. Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 4246–4253, 2020. Chen Huang, Xianhang Li, Vimal Thilak, Etai Littwin, and Josh Susskind. Text-conditional jepa for learning semantically rich visual representations. arXiv preprint arXiv:2605.03245, 2026. 14 Preprint Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, p. 646–661. Springer, 2016. Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in neural information processing systems, 35:28708–28720, 2022. Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Ilya Loshchilov and Frank Hutter.Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. Lucas Maes, Quentin Le Lidec, Luiz Facury, Nassim Massaudi, Ayush Chaurasia, Francesco Capuano, Richard Gao, Taj Gillin, Dan Haramati, Damien Scieur, et al. stable-worldmodel: A platform for reproducible world modeling research and evaluation. arXiv preprint arXiv:2605.21800, 2026a. Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworld- model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026b. Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representa- tion. In HEAR: Holistic Evaluation of Audio Representations, p. 1–24. PMLR, 2022. Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Noboru Harada, and Nobutaka Ono. Rethinking masking strategies for masked prediction-based audio self-supervised learning. arXiv preprint arXiv:2603.23810, 2026. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779, 2019. Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, p. 1015–1018, 2015. Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PmLR, 2021. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 15 Preprint Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2818–2826, 2016. Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor C ̆ arbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, and John Langford. Next-latent prediction transformers learn compact world models. arXiv preprint arXiv:2511.05963, 2025. Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In 2021 IEEE/CVF international conference on computer vision (ICCV), p. 32–42. IEEE, 2021. Ludovic Tuncay, Etienne Labbé, Emmanouil Benetos, and Thomas Pellegrini.Audio-jepa: Joint-embedding predictive architecture for audio representation learning.arXiv preprint arXiv:2507.02915, 2025. Feng Wang, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad, Cihang Xie, and Alan Yuille. Vit-5: Vision transformers for the mid-2020s. arXiv preprint arXiv:2602.08071, 2026. Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018. Haiyu Wu, Randall Balestriero, and Morgan Levine. Visreg: Variance-invariance-sketching regular- ization for jepa training. arXiv preprint arXiv:2606.02572, 2026. Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 4563–4567. IEEE, 2022. Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in neural information processing systems, 34: 30392–30400, 2021. Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, Weiyang Jin, Joyce Chai, Saining Xie, and Stella X Yu. Next-embedding prediction makes strong vision learners. arXiv preprint arXiv:2512.16922, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, and Phil Woodland. Spear: A unified ssl framework for learning speech and audio representations. arXiv preprint arXiv:2510.25955, 2025b. Yumeng Yao, Jingzhi Dong, Haowen Gu, Tao Chen, Zonghan Wu, Xiaoshui Huang, and Yazhou Yao. Rethinking point clouds as sequences: A causal next-token predictive learning framework. arXiv preprint arXiv:2605.17566, 2026. Goksenin Yuksel, Pierre Guetschel, Michael Tangermann, Marcel van Gerven, and Kiki van der Heijden. Wavjepa: Semantic learning unlocks robust audio foundation models for raw waveforms. arXiv preprint arXiv:2509.23238, 2025. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, p. 6023–6032, 2019. Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural informa- tion processing systems, 32, 2019. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017. 16 Preprint AAPPENDIX BRELATED WORK Supervised Audio Pre-training.Supervised pre-training for audio has been mainly explored in two regimes. Out-of-domain approaches adapt models originally trained on labeled image datasets such as ImageNet (Deng et al., 2009) to spectrogram inputs, typically by modifying the input layer from three RGB channels to a single-channel spectrogram. Early work of this kind used CNN backbones such as EfficientNet (Gong et al., 2021b); more recent work adopts Transformer-based architectures, notably AST (Gong et al., 2021a) and HTS-AT (Chen et al., 2022a), which have driven substantial gains on audio classification benchmarks. In-domain approaches instead pre-train directly on the target modality. CLAP (Elizalde et al., 2023) adapts the CLIP (Radford et al., 2021) recipe to audio via contrastive language-audio alignment on supervised text-audio pairs, while Audio-CLIP (Guzhov et al., 2022) and Wav2clip (Wu et al., 2022) extends the CLIP backbone with an additional audio encoder trained on AudioSet (Gemmeke et al., 2017). Despite delivering strong results, these methods rely on large quantities of labeled data, which are expensive and time-consuming to obtain in practice. Self-Supervised Audio Pre-training. Self-supervised learning has driven substantial progress in audio representation learning, largely by transferring ideas developed in the vision domain. Most recent methods extract log-mel spectrograms as input and follow the masked image modeling paradigm (He et al., 2022): Audio-MAE (Huang et al., 2022), MaskSpec (Chong et al., 2023) and MSM-MAE (Niizumi et al., 2022) directly applie the MAE reconstruction objective to spectrogram patches; and BEATs (Chen et al., 2022b) instead trains an iterative acoustic tokenizer to provide discrete semantic prediction targets. Waveform-based approaches such as wav2vec 2.0 (Baevski et al., 2020) and data2vec (Baevski et al., 2022) bypass spectrogram preprocessing altogether, and predict latent representations of the raw audio signal. More recent efforts push these directions further by combining ideas from multiple objectives. EAT (Chen et al., 2024) combines an MAE- style reconstruction with the data2vec latent-target formulation. A-JEPA (Fei et al., 2023) adapts the joint-embedding predictive paradigm to audio, using a target encoder updated via EMA and predicting masked patch embeddings. SSLAM (Alex et al., 2025) extends EAT with an additional source retention loss trained on artificially mixed audio to improve robustness to polyphonic content. Recently, SPEAR (Yang et al., 2025b) unifies speech and general audio representation learning through the distillation of complementary knowledge from specialized teacher models. Despite their diversity, these methods all rely on bidirectional prediction of masked or corrupted audio, and typically require a reconstruction decoder, a student-teacher setup, or auxiliary regularization losses to achieve competitive performance. Next-token/embedding Prediction. Predictive learning has long been a central principle in rep- resentation learning across modalities. In language, GPT-style models (Radford et al., 2018; 2021; Achiam et al., 2023) established autoregressive next-token prediction as a scalable pre-training objec- tive, with subsequent work confirming that the same paradigm transfers to vision (Image-GPT (Chen et al., 2020)) and beyond. More recent efforts move away from predicting raw signal tokens toward predicting embeddings directly. A first line of work adopts an autoregressive formulation: NEPA (Xu et al., 2025) introduces next-embedding predictive autoregression for visual representation learning; PointNTP (Yao et al., 2026) adapts causal next-token predictive learning to 3D point clouds; and LeWorldModel (Maes et al., 2026b) together with stable-worldmodel (Maes et al., 2026a) extend the paradigm to next-frame prediction for control tasks. A second line of work adopts a joint-embedding formulation: I-JEPA (Assran et al., 2023) and A-JEPA (Fei et al., 2023) predict latent representations of masked regions from context using dual encoders and an EMA teacher, while LeJEPA (Balestriero & LeCun, 2025) dispenses with masking and instead enforces invariance across augmented views with a variance-covariance regularizer. Together, these works suggest that predicting future tokens or embeddings can serve as a unified and scalable pre-training principle across modalities. In audio, how- ever, this paradigm has remained largely unexplored: existing SSL methods rely almost exclusively on bidirectional masked modeling, and it is unclear a priori whether causal, per-position prediction is compatible with the non-stationary, temporally structured nature of audio signals. NAPE is, to the best of our knowledge, the first method to demonstrate that causal next-embedding prediction is a competitive and scalable pre-training objective for audio spectrograms, matching or surpassing more elaborate masked-modeling and joint-embedding methods with a substantially simpler recipe. 17 Preprint Table 8: Hyperparameter list. When an hyperparameterhvaries between the base (h b ) and large (h l ) model, we include both values like (h b /h l ). ∗ Following (Chen et al., 2022b), we balance each class to 50% of the size of the unknown class for each training epoch. Hyperparameters Pre-trainingFine-tuning AS-2MAS-2MAS-20KESC-50KS1KS2ER OptimizerAdamW Opt. Momentum (β 1 ,β 2 ) (0.9,0.95)(0.9,0.999) Weight Decay0.05 Learning Rate SchedulerCosine Decay Layer-Wise LR Decay1.00.7/0.90.8/0.90.7/0.90.7/0.8 0.7/0.8 0.7/0.9 Base Learning Rate5e-31.25e-3 Epochs30/2520/1530/20100505050 Warm-up Epochs34/36/510555 Batch Size256/12864 GPUs8441114 Weighted sampling✗✓✗✓∗✗ MultilabelN/A✓✗ EMA Decay Rate0.99990.999950.999✗ Label SmoothingN/A0.0.0.10.10.0.1 Roll Augmentation✗✓✗✓ Drop Path0.0.1 SpecAug (time/freq)N/A(96,16) (24,16)/(96,16) (96,24)/(24,16) (24,16) (24,16) (48,24) Mixup (alpha/prob.)N/A(0.8,1.0)(0.8,0.8)(0.8,0.5)(0.8,0.8) (0.8,0.8) (0.8,0.5) Cutmix (alpha/prob.)N/A(1.0,1.0)(1.0,0.8)(1.0,0.5)(1.0,0.8) (1.0,0.8) (1.0,0.5) Noise Augmentation✗✓ Loss FunctionNeg Cos Sim BCEBCECEBCEBCECE Dataset Mean for Norm.-6.84-6.84-6.84-6.84-9.11-9.16-13.74 Dataset Std for Norm.5.385.385.385.384.534.613.88 CFULL HYPERPARAMETER LIST Fine-tuning Setting.We report the full hyperparameter list for fine-tuning NEPA Base and Large in Table 8. Those values refer to the Raster scan variant, with the Diagonal variant having almost the same hyperparameters (the only difference is in the number of epochs needed to converge for AS-20K, which can vary of only a few epochs). Regarding NEPA Small, we used the same hyperparameters as NEPA Base, with the only difference being the value of the layer-wise learning rate decay (LLRD) for some of the downstream tasks as this hyperparameter depends on the number of layers of the model. Specifically, we setLLRD = 0.6for AS-2M andLLRD = 0.6for KS1/KS2. No other changes have been made. Linear Probing Setting. For the linear probing experiments, we made the following changes (all other settings are the same as the fine-tuning hyperparameters): we increase the learning rate to1e−2 and we disable all augmentation techniques. DADDITIONAL ABLATION STUDIES In this section, we report additional ablations studies on NAPE. For all experiments we use the raster scan variant. We use the SimSiam predictor for all ablations except for the ablation on the pre-training budget where we use the 2-MLP predictor. FreezingthePatchEmbeddingLayer.Table 10 examines whether freezing the patch embedding layer during fine-tuning affects downstream performance. In Xu et al. (2025), freezing the embedding layer yields significant improvements. However, in our setting, the two configurations are essentially indistinguishable on all three tasks tested. We therefore leave the patch embedding layer trainable during fine-tuning, matching the standard practice of prior audio Transformer methods. 18 Preprint Table 10:Ablationonfreezingtheemb.layer. Freeze emb AS-2M AS-20K KS2 ✓49.6139.14 98.73 ✘49.5839.08 98.80 Table 11:Ablationonpositionalencoding. Positional Enc. AS-2M AS-20K Absolute47.836.7 RoPE49.6 39.1 PositionalEncoding.Table 11 compares learned absolute positional encodings with Rotary Position Embedding (RoPE) (Su et al., 2024), applied independently along the frequency and time axes. RoPE substantially outperforms absolute encodings on both AudioSet benchmarks, with gains of+1.8 mAP on AS-2M and+2.4mAP on AS-20K. Beyond the accuracy improvement, RoPE also offers a practical benefit: because it encodes relative positions rather than absolute ones, it transfers naturally to clips of different lengths at inference time, whereas absolute encodings require interpolation. We therefore adopt RoPE as NAPE’s default positional encoding. Table 9:Ablationonthepoolingmethod andattentiontypeatfine-tuning. Attention PoolingTask Type Mode AS-2M AS-20K KS2 BidirecCLS Tok 49.738.998.7 BidirecLast Tok 49.638.7 98.8 Bidirec Avg Pool 49.6 39.1 98.8 CausalLast Tok 49.438.9 98.8 AttentionTypeandPooling.Table 9 jointly ablates the attention mask used during fine-tuning and the pooling strategy for producing the clip-level representation. Under causal attention, the natural pooling choice is the last token: only the final position has access to the full preceding context, and the CLS token functions more as a BOS- like anchor at the start of the sequence than as a readout summary of the clip (this is in line with (Xu et al., 2025)). Under bidirectional attention, however, every position sees the entire sequence, so CLS, last-token, and mean pooling all in principle have access to the same information. This is reflected in the numbers: the three bidirectional variants are essentially on par on the three tested tasks. Keeping causal attention with last-token pooling remains competitive but slightly worse than its bidirectional counterpart, with a0.2mAP drop on both AS-2M and AS-20K. We therefore use bidirectional attention with mean pooling as NAPE’s default fine-tuning configuration. Table 12:Ablationonnormalizationlayer style. NormAudio TasksSpeech Tasks Type AS-2M AS-20K ESC-50 KS1 KS2 ER LN49.639.194.2 97.9 98.8 64.9 RMS49.538.894.4 97.8 98.8 65.2 NormalizationLayer. Table 12 compares Layer- Norm (Ba et al., 2016) (LN) and RMSNorm (Zhang & Sennrich, 2019) in the encoder. RMSNorm drops the mean-centering step of LayerNorm and normalizes each feature vector by its root-mean-square, which has become common in recent language and vision Transformers. In our setting, the two variants are ef- fectively on par, with small variations across the six downstream tasks. For this reason, we retain Layer- Norm as NAPE’s default for consistency with prior audio Transformer works. 6121830 Pre-training epochs 48.0 48.2 48.5 48.8 49.0 49.2 49.5 49.8 50.0 mAP AS-2M 48.49 48.66 48.87 49.39 49.45 49.63 49.85 49.89 6121830 Pre-training epochs 36.0 37.0 38.0 39.0 40.0 mAP AS-20K 36.51 36.83 37.75 38.50 37.15 37.69 38.60 39.94 BaseLarge Figure 8:Pre-trainingbudgetablation. AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on. Pre-trainingBudget. Figure 8 reports down- stream mAP on AS-2M and AS-20K when fine-tuning from NAPE-Base and NAPE-Large checkpoints obtained after6,12,18, and30 epochs of pre-training. Performance improves monotonically with the pre-training budget for both scales and both benchmarks: on AS-2M, NAPE-Large improves from49.45mAP after 6epochs to49.89after30; on AS-20K, from 37.15to39.94. Notably, NAPE already de- livers strong results after only a few epochs of pre-training. After just6epochs, NAPE- Large reaches49.45mAP on AS-2M — already matching or exceeding most published baselines (cf. Table 6) — making NAPE an appealing choice under tight compute budgets. Longer pre-training 19 Preprint continues to yield returns at30epoch, and NAPE-Large consistently benefits more than NAPE-Base from a larger budget, consistent with the scaling behavior discussed in Section 3.3. Table 13:Ablationonrandommaskingap- pliedtotheinputembeddings. MaskAudio TasksSpeech Tasks Ratio AS-2M AS-20K ESC-50 KS1 KS2 ER 0 47.6 36.2 92.9 97.4 98.3 64.6 2047.3 36.292.2 97.6 98.5 63.9 5047.235.291.7 97.5 98.3 63.9 RandomMasking. Table 13 examines whether adding random masking to the input embeddings during pre-training, in the style of masked modeling methods, helps NAPE. We consider three masking ratios: no masking (0%, the default),20%, and50%. We use NAPE small with raster scan order. Across all six benchmarks, masking either has no effect or slightly de- grades downstream performance, with the largest drop on AS-20K (−1.0mAP at50%) and ESC-50 (−1.2 points at50%). This differs from methods such as Audio-MAE (Huang et al., 2022) or BEATs (Chen et al., 2022b), where high masking ratios are central to the pre-training signal: the model is asked to reconstruct or predict the masked content from the visible context. In NAPE, the causal attention mask already prevents each position from accessing its target, so additional input masking removes useful context without changing the difficulty of the prediction task. We therefore leave the input unmasked in NAPE’s default configuration. ENAPE VS JEPA VS LEWORLDMODEL FORMULATION E.1NAPE VS JEPA-STYLE METHODS NAPE and JEPA methods Assran et al. (2023); Fei et al. (2023) learn representations by predicting one set of patch embeddings from another, therefore it is worth clarifying how the two paradigms differ structurally. Figure 9 (left and right) illustrates the two frameworks. JEPA-style methods use two encoders: a context encoderhthat processes the visible portion of the inputx, and a target encoder that produces the prediction targets from the complementary regiony. The target encoder is not trained by gradient descent; instead, its weights are updated as an EMA of the context encoder’s weights, so that its output remains a moving target that the predictor tries to match. The predictor itself is a Transformer that receives both the context representation and an additional variablec, which encodes the target positions to be predicted, and produces the predicted target embeddingsˆz y in parallel. The context encoder operates with bidirectional attention over the visible patches. To prevent representation collapse under this regime, JEPA methods typically require additional machinery beyond stop-gradient: an EMA teacher (I-JEPA (Assran et al., 2023), A-JEPA (Fei et al., 2023)), auxiliary variance-covariance regularizers such as VICReg (Bardes et al., 2021), or masking strategies carefully tuned so that the two views do not degenerate into trivial solutions. NAPE, in contrast, uses a single encoder (h) and takes its prediction targets directly from the shared patch embedding layerf. Becausefis a shallow, non-recurrent module, its outputs provide a stable target signal that does not need to be maintained by a separate EMA-tracked encoder. NAPE also features a lightweight MLP head (g) that operates on a single position at a time, rather than a Transformer that jointly reasons about context and target positions: a structurally simpler design that reflects NAPE’s per-position causal prediction task. There is no masking: the model processes the full input as a causal sequence, and predictions are made one patch at a time under a causal attention mask. Collapse prevention relies solely on the three key components discussed in Section 2.2: causality, prediction shift, and stop-gradient, with no auxiliary regularizers or student-teacher setups. E.2NAPE VS LEWORLDMODEL METHODS A closely related recent method is LeWorldModel (Maes et al., 2026b), a JEPA designed for world modeling that also autoregressively predicts next-frame embeddings from raw pixels (Figure 9, middle). Like NAPE, LeWM aims to simplify the JEPA recipe and dispenses with EMA teachers and pretrained encoders, training its encoder and predictor jointly end-to-end. Unlike NAPE, LeWM relies on the SIGReg regularizer (Balestriero & LeCun, 2025), an auxiliary loss that projects the learned embeddings onto random directions and enforces Gaussian-distributed marginals, to prevent collapse, whereas NAPE requires only stop-gradient on the target branch. LeWM uses MSE as 20 Preprint JEPA LeWorldModel NAPE Context Encoder Target Encoder (EMA) Predictor EncoderEncoder Predictor SIGRegSIGReg Embedding Layer Predictor Figure 9: Comparison between JEPA, LeWorldModel, and NAPE architectures. its prediction loss (kept stable by SIGReg), whereas NAPE uses cosine similarity combined with stop-gradient, which together avoid the shrink-to-zero collapse mode observed withℓ 1 andℓ 2 under a magnitude-sensitive loss (Section 3.2). Finally, the two methods target different problems: LeWM predicts across time steps of an action-conditioned trajectory for planning in latent space, while NAPE predicts across patches within a single input for representation learning applicable to classification tasks. NAPE therefore constitutes a distinct point in the JEPA design space: one that shares LeWM’s end-to-end simplicity but arrives at collapse prevention through a fundamentally different mechanism. FTRAINING LOSS VISUALIZATIONS To better understand NAPE’s training dynamics, in Figure 10 we report the pre-training loss curves of six NAPE runs across multiple scales and configurations. The top row reports runs of the default NAPE configuration (raster scanning, SimSiam predictor, cosine similarity, all three key components enabled) at the three model scales — Small (top left), Base (top middle), and Large (top right). All three curves follow the same qualitative trajectory: a rapid descent from around−0.3to below−0.9 within the first∼ 20,000steps, followed by a smooth and steady decrease that asymptotes just below −0.98, close to the theoretical minimum of−1. The Large model runs for roughly twice as many steps as Small and Base, its batch size is halved to fit into memory, but reaches a comparable final loss along a trajectory of the same shape, indicating that the NAPE objective is well-conditioned across encoder capacities. The bottom row reports three ablation runs at the Base scale. Without the causal mask (bottom left), the loss collapses almost immediately, saturating near−1.0within∼ 2,000steps and remaining there. The model has found the trivial identity mapping enabled by bidirectional attention: the objective is satisfied without learning any structure that transfers downstream, as reflected in the sharp mAP drop reported in Table 1. The configuration with L1 loss (bottom middle) diverges: the loss drops rapidly to∼ 0.05within a few hundred steps, plateaus briefly, and then climbs steadily. The initial descent reflects the model finding low-norm solutions in which both predicted and target embeddings shrink toward zero, since L1 is magnitude-sensitive and trivially reduced by scaling the outputs; once the encoder approaches this degenerate regime, the objective becomes ill-conditioned and the loss reverses. The cross-entropy loss (bottom right), in contrast, converges to a reasonable minimum around step∼ 70,000before gradually overfitting. Unlike L1, CE does not collapse: this suggests that collapse prevention is not exclusive to cosine similarity. In our setting, however, CE still yields worse downstream performance than cosine (Table 5), and its tendency to overfit after the initial descent makes it a less robust choice for NAPE pre-training. GADDITIONAL QUALITATIVE RESULTS We include additional qualitative results for NAPE-L with raster order in Figure 11) and NAPE-L with diagonal order in Figure 12). 21 Preprint 050000100000150000200000 Training step 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Training loss NAPE Small 050000100000150000200000 Training step 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Training loss NAPE Base 050000100000150000200000250000300000350000400000 Training step 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 Training loss NAPE Large 010000200003000040000 Training step 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Training loss NAPE w/o causal mask 02000400060008000 Training step 0.05 0.10 0.15 0.20 0.25 Training loss NAPE w/ L1 loss 020000400006000080000100000120000140000 Training step 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 Training loss NAPE w/ CE loss Figure 10: Pre-training loss curves for multiple NEPA models across scales and configurations. Figure 11: Additional qualitative results for NAPE-L with raster order. 22 Preprint Figure 12: Additional qualitative results for NAPE-L with diagonal order. 23