Paper deep dive
GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets
Gaspard Botté, Séverin Baroudi, Samir Sadok, Francesco Paissan, Thomas Hueber, Xavier Alameda-Pineda, Ricard Marxer, Mirco Ravanelli
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.
Tags
Links
- Source: https://arxiv.org/abs/2609.37798v1
- Canonical: https://arxiv.org/abs/2609.37798v1
Trouble viewing inline? Open PDF directly →
Full Text
25,692 characters extracted from source content.
Expand or collapse full text
GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets Gaspard Botté, Séverin Baroudi, Samir Sadok, Francesco Paissan, Thomas Hueber, Xavier Alameda-Pineda, Ricard Marxer, Mirco Ravanelli †thanks: ⋆Correspondence: gaspard003.botte@gmail.com, ricard.marxer@lis-lab.fr, mirco.ravanelli@concordia.ca Abstract Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder’s continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe. Index Terms: self-supervised learning, speech representations, JEPA, Gaussian regularization †address: 1Concordia University 2Mila – Québec AI Institute 4CNRS, ILLS, Univ Toulon 3Inria, Univ. Grenoble Alpes, CNRS, LJK 6Univ Toulon, Aix Marseille Univ, CNRS, LIS 5Université Laval 7Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France 1 Introduction Speech self-supervised learning (SSL) has emerged as a powerful paradigm, achieving state-of-the-art performance across diverse downstream tasks. By allowing a pretrained encoder to transfer effectively across recognition, speaker analysis, and spoken-language understanding, SSL demonstrates remarkable robustness, particularly in low-resource scenarios where labeled data is scarce [4, 27, 8]. This success motivates a closer look at the core ingredients required for compact, efficient encoders. A critical design dimension in masked-prediction speech SSL is target construction, which defines what the encoder learns to infer from context and directly shapes its learning signal. For instance, Wav2vec 2.0 relies on quantized representations [4], HuBERT and WavLM utilize clustered hidden units [15, 8], data2vec 2.0 leverages continuous targets from an exponential-moving-average (EMA) encoder [2], and S-JEPA predicts soft Gaussian-mixture assignments [16]. These mechanisms do more than resist collapse: they fundamentally shape the information content and stability of the prediction signal. Simplified recipes such as S-JEPA still construct dedicated targets [16]. This raises a fundamental question: Are specially engineered prediction targets strictly necessary to achieve competitive performance in speech SSL? 00202040406060808010010055101015152020TERAvq-wav2vecwav2vecDeCoAR 2.0S-JEPAGLaS-JEPABase encoders(95M)Parameters (M)WER (%) Figure 1: Frozen SUPERB ASR versus parameter count. All plotted models are pretrained on LibriSpeech 960 h, except S-JEPA, pretrained on 83k h [27, 16]. Dashed line: data2vec 2.0 WER (4.81%) [28]. Speech Acoustic frontend Mask m(⋅)m(·) Encoder fθf_θ Encoder fθf_θ TargetszzPredictionsz z SIGReg MSE hFull viewpαp_αpαp_αstop-gradientsharedweights Figure 2: GLaS-JEPA: shared encoder and token-wise projector pαp_α. MSE uses masked predictions and stop-gradient targets; SIGReg backpropagates through the full view. The projector is discarded downstream. To answer this, we introduce GLaS-JEPA (Gaussian Latent Speech-JEPA), a framework that directly predicts the current encoder’s continuous representations at masked positions, bypassing discrete assignments or an EMA target encoder. Following the joint-embedding predictive architecture (JEPA) principle [18, 1], prediction occurs entirely in representation space, meaning targets evolve jointly with the model. However, optimizing these unconstrained targets risks representation collapse, where the model minimizes prediction error by outputting constant vectors and discarding speech information. Rather than relying on a target generator to prevent this, GLaS-JEPA uses Sketched Isotropic Gaussian Regularization (SIGReg) [5] to regularize the space directly. This shifts collapse prevention from complex target construction to a simple regularization term encouraging a non-collapsed Gaussian distribution. We evaluate our approach on the SUPERB benchmark [27]. Our core contributions are: (1) introducing GLaS-JEPA, a speech SSL framework that learns by predicting the encoder’s continuous representations at masked positions; (2) showing that SIGReg is sufficient to prevent collapse in this setting, dispensing with the need for target-generation mechanisms commonly used in speech SSL; and (3) establishing strong performance across content, speaker, and semantic tasks, showing that competitive speech representations can emerge from a substantially simpler training pipeline. 2 Related Work Constructing prediction targets. Quantization supplies discrete latent targets in wav2vec 2.0 and vq-wav2vec [4, 3], whereas HuBERT and WavLM predict clustered pseudo-labels [15, 8]. BEST-RQ simplifies this process with a fixed random-projection quantizer [10], retaining a dedicated target mapping. Continuous latent prediction is already established by data2vec 2.0 [2]. It constructs targets by averaging normalized upper-layer representations from a slowly moving EMA teacher, and regresses them with a separate temporal CNN predictor. The distinction in GLaS-JEPA is not continuity, but using current-encoder representations with explicit distributional regularization instead of processed EMA targets. Its token-wise linear projector maps latents independently into a 128-dimensional loss space; unlike the data2vec 2.0 predictor, it does not mix neighboring positions. S-JEPA [16] instead predicts soft GMM posteriors, using MFCC features initially and EMA-encoder features. These alternatives motivate testing whether discrete target mappings and EMA-based target generation can be replaced by direct regularization of current-encoder representations. Regularizing representations directly. VICReg and Barlow Twins explicitly constrain representation statistics, providing an alternative to predictor and stop-gradient mechanisms such as those used in SimSiam[6, 30, 9]. LeJEPA introduces SIGReg for distributional regularization [5], and LeWorldModel applies it to visual trajectories [21]. Building on SIGReg, we investigate whether direct representation regularization can enable current-encoder speech latent prediction without discrete assignments or an EMA teacher. Speech produces many latent frames per utterance, yielding large token populations within each batch; we study how to group them in Sec. 3.3. Scope of comparison. Related non-distilled speech encoders include modified CPC, APC, PASE+, TERA, wav2vec, and DeCoAR 2.0 [24, 11, 23, 20, 25, 19], covering contrastive prediction, acoustic reconstruction, and multi-task objectives. A-JEPA, Audio-JEPA, and WavJEPA explore related predictive representations for general audio [12, 26, 29]; we discuss them but exclude unmatched numerical comparisons because their general-audio training and evaluation differ from our speech-only setting. 3 GLaS-JEPA Table 1: Frozen-encoder SUPERB results (%). Baselines: SUPERB/S-JEPA [27, 16]; SD: WavLM [8]; data2vec 2.0: ASR/ER/SF [28], SD [7]. Bold: best non-distilled sub-90M score. GLaS-JEPA uses Full Marginal SIGReg. Parameters rounded to millions. Content Speaker Paralinguistic Semantic Model Params (M) Pretrain (h) ASR WER ↓ SD DER ↓ ER Acc. ↑ SF F1 ↑ SF CER ↓ Base encoders (approximately 95M parameters) wav2vec 2.0 Base 95 960 6.43% 6.08 63.43 88.30 24.77 HuBERT Base 95 960 6.42% 5.88 64.92 88.53 25.20 WavLM Base 95 960 6.21% 4.55 65.94 89.38 22.86 data2vec 2.0 Base 94 960 4.81% 6.5 66.66 89.67 22.09 Non-distilled sub-90M baselines TERA 21 960 18.17% 9.96 56.27 67.50 54.17 wav2vec 33 960 15.86% 9.90 59.79 76.37 43.71 vq-wav2vec 34 960 17.71% 9.93 58.24 77.68 41.54 DeCoAR 2.0 90 960 13.02% 6.59 62.47 83.28 34.73 S-JEPA 52 83k 12.10% – 64.83 83.05 33.17 GLaS-JEPA 57 960 6.89% 6.46 57.91 87.72 25.87 3.1 Architecture and forward pass We introduce GLaS-JEPA (Gaussian Latent Speech Joint-Embedding Predictive Architecture), a dual-path framework for direct prediction of continuous speech representations. As illustrated in Fig. 2, an acoustic frontend first processes the input speech into a sequence of continuous feature vectors h. This sequence is then branched into two parallel processing pathways: The full view provides the complete, uncorrupted feature sequence h to the encoder fθf_θ, and a subsequent projector pαp_α maps each token independently into a dedicated loss space to target representations: =pα(fθ()).z=p_α (f_θ(h) ). The masked view applies a masking function m(⋅)m(·) that replaces contiguous spans of the input features with a learnable mask token. The same encoder fθf_θ processes these features and learns to infer missing content from temporal context. The shared projector pαp_α then maps its representations into the loss space: ^=pα(fθ(m())). z=p_α (f_θ (m(h) ) ). The encoder learns prediction during training; there is no separate temporal predictor or EMA teacher. The token-wise linear projector provides a 128-dimensional bottleneck that separates the optimization loss space from the encoder representations used downstream. It can assist optimization but does not mix neighboring positions, and is discarded after pretraining. 3.2 Loss function and regularization Prediction. The model learns by minimizing the mean squared error (MSE) between the masked-view predictions z and the full-view targets z. We simplify the notation by letting ℳ⊆1,…,B×1,…,T,M \1,…,B\×\1,…,T\, represent the set of all masked time indices across a batch. The prediction loss is computed exclusively at these masked positions: ℒpred=1|ℳ|∑(b,t)∈ℳ‖^b,t−sg(b,t)‖22.L_pred= 1|M| _(b,t) \| z_b,t-sg(z_b,t) \|_2^2. (1) where sg(⋅)sg(·) denotes the stop-gradient operation. Because the targets z are generated dynamically using the current, actively updating network weights, the stop-gradient ensures the prediction loss routes gradients only through the masked forward path. Regularization. SIGReg [5] regularizes unmasked targets z to prevent collapse. It averages weighted squared discrepancies between empirical characteristic functions of random unit projections and the standard Gaussian function e−u2/2e^-u^2/2 at frequency u, encouraging isotropic Gaussian representations. We optimize: ℒ=(1−λ)ℒpred+λℒreg,with λ=0.01.L=(1-λ)L_pred+ _reg, λ=0.01. (2) Klindt et al. [17] prove that LeJEPA guarantees linear identifiability under independent Gaussian latents and isotropic transitions, a property unique to Gaussian distributions. This raises a key empirical question for speech: is Gaussian regularization sufficient to extract useful, non-collapsed representations from complex, correlated audio? GLaS-JEPA investigates this by shifting collapse prevention entirely to ℒregL_reg. Here, gradients from ℒregL_reg flow exclusively through the unmasked full-view representations, preventing representation collapse without strictly enforcing exact Gaussianity on finite-length speech latents. 3.3 Applying Distributional Regularization Speech representations are temporally correlated. For a latent batch Z∈ℝB×T×dpZ ^B× T× d_p, we must choose which population SIGReg should regularize. We consider two alternatives (Fig. 3). (a) Time-conditional(b) Shuffled Marginalp(z∣t)p(z t): fixed indexp(z)p(z): mixed indicesA1A_1B1B_1C1C_1A2A_2B2B_2C2C_2A3A_3B3B_3C3C_3A4A_4B4B_4C4C_4A1A_1C3C_3A4A_4B2B_2A3A_3C1C_1C4C_4B1B_1B4B_4A2A_2C2C_2B3B_3Population sample → SIGReg → (0,I)N(0,I) reference SIGReg ASR WER ↓ ER Acc. ↑ SD DER ↓ Time-conditional 13.99 57.92 7.65 Shuffled Marginal 12.24 58.60 6.65 Figure 3: Time-conditional (left) and Shuffled Marginal (right), with SUPERB results for the 30M model. Colors/letters: utterances; subscripts: time indices. Outlines mark groups of B tokens. Time-conditional SIGReg. This regularizes p(z∣t)p(z t). At each fixed latent index t, the population contains B representations from different utterances. We average the T penalties, as in LeWorldModel [21]. Using distinct utterances limits within-utterance dependence, although shared-speaker dependence may remain. Marginal SIGReg. This regularizes the batch-induced marginal p(z)p(z), ignoring the time index. Tokens across times and utterances belong to the same population, allowing both sources of variation to maintain a non-collapsed signal. We use two estimators of this marginal objective. Shuffled Marginal, used only in the controlled four-layer ablation, shuffles the BTBT tokens into T groups of B and averages their penalties. This matches Time-conditional SIGReg in group size, number of penalties, and computation, isolating population construction rather than sample size. Full Marginal, used in our main 57M model, applies one penalty to all BTBT tokens. Both target the same marginal population p(z)p(z) and differ only in finite-sample estimation. Under IID sampling, both converge to the same population-level objective as sample size grows; speech tokens need not satisfy this assumption. With all other settings fixed, Shuffled Marginal improves ASR WER from 13.99% to 12.24%, ER accuracy from 57.92% to 58.60%, and SD DER from 7.65% to 6.65% (Fig. 3). These gains motivate Marginal SIGReg for the main model. 4 Experimental Setup Encoder and training. We pretrain on LibriSpeech 960 h [22]. The eight-layer Conformer [14] has width 576, eight heads, FFN width 2,048, and convolution kernel 31. The frontend uses 80-bin log-Mel features (25 ms window, 5 ms hop) with temporal stride four, yielding 20 ms latent spacing. A token-wise linear projector maps encoder representations to a 128-dimensional loss space. The downstream encoder contains approximately 57M parameters, excluding the projector. We mask 50% of positions in contiguous spans of ten latents (200 ms). Training uses 220,000 AdamW updates, 40,000-step linear warmup followed by cosine decay, peak learning rate 10−410^-4, weight decay 10−310^-3, and λ=0.01λ=0.01. Crops span 2–15.6 s, with approximately 4,000 s of audio per optimization step. We use Full Marginal SIGReg on all tokens in the batch. Evaluation. For the four SUPERB tasks [27], we freeze the encoder and train task heads over a learned weighted sum of hidden layers. We follow WavLM Base learning rates and batch sizes [8] for ASR, speaker diarization (SD), emotion recognition (ER), and slot filling (SF); ASR uses no external language model. Published baselines are contextual comparisons, not matched-budget reimplementations. Linear probes assess 41-phone classification using CPC’s LibriSpeech protocol and speaker identification from time-pooled features; RankMe [13] measures effective rank. 5 Results and Discussion 5.1 Downstream transfer Table 1 tests whether current-representation prediction supports useful transfer. GLaS-JEPA achieves 6.89% WER and 25.87% SF concept error rate (CER), improving over the best listed non-distilled sub-90M baselines by 43.1% and 22.0% relatively. Against S-JEPA, it uses 57.36M rather than 51.8M parameters and 960 rather than approximately 83,000 pretraining hours. Baseline comparisons use different architectures and training budgets, so they cannot isolate the SSL objective’s effect. Accordingly, these comparisons test whether such a target-free model can be competitive, rather than whether its objective outperforms alternatives under controlled conditions. The gains are task-dependent: SF F1 reaches 87.72%, but ER accuracy (57.91%) trails S-JEPA (64.83%). GLaS-JEPA reaches 6.46% SD DER, slightly below DeCoAR 2.0 (6.59%) and data2vec 2.0 (6.5%), with approximately 39% fewer parameters. S-JEPA does not report SD here. Base encoders remain stronger in ASR, including WavLM (6.21% WER) and data2vec 2.0 (4.81%). 5.2 Collapse, representations, and limitations Without SIGReg, effective rank [13] falls to 1 within a few thousand updates; phone accuracy is ∼ 16% with imbalanced classes. Fig. 4 tracks phone and speaker linear-probe test accuracy across encoder layers, indicating where each type of information is accessible. Depth is normalized by the number of layers to compare the 8-layer GLaS-JEPA and 12-layer WavLM Base. Their phone peaks are 83.9/85.4% at layers 6/8 and 11/12; speaker peaks are 96.5/92.7% at 3/8 and 4/12. Strong intermediate speaker encoding, then greater upper-layer loss than WavLM with retained phonetics, suggests invariance to slowly varying/global acoustic factors relevant to ER. From ∼ 140k to 220k steps in the same run, ASR WER improved from 7.52% to 6.89%, but ER fell from 59.20% to 57.91%. This suggests optimization increasingly favors content over paralinguistic information, though it does not establish causality. Stable current-encoder training at ∼ 95M remains ongoing work; 57M competitiveness does not establish equivalent scaling to Base WavLM, HuBERT, or data2vec 2.0. Figure 4: Phone and speaker linear-probe test accuracy versus normalized encoder depth for GLaS-JEPA and WavLM Base. 6 Conclusion At the demonstrated scale, GLaS-JEPA replaces engineered targets with representation regularization, reaching 6.89% ASR WER at 57M parameters with competitive content, speaker, and semantic transfer. Competitive speech representations can thus emerge from current-encoder prediction without dedicated target generation. Acknowledgments. We thank Dr. Thomas Hueber (GIPSA-lab, CNRS) for making this internship project possible. We thank Lovanya Jain for help with experiments. M. Ravanelli acknowledges support from NSERC, the Digital Research Alliance of Canada (alliancecan.ca), and Translated for funding through the Immediate Research Grant. This work benefited from support from the French National Research Agency through the ANR-20-CE23-0012-01 (MIM) grant and from access to the HPC resources of IDRIS under allocation A0191014044. References [1] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas (2023) Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, p. 15619–15629. External Links: Link Cited by: §1. [2] A. Baevski, A. Babu, W. Hsu, and M. Auli (2023) Efficient self-supervised learning with contextualized target representations for vision, speech and language. In ICML, PMLR, Vol. 202, p. 1416–1429. External Links: Link Cited by: §1, §2. [3] A. Baevski, S. Schneider, and M. Auli (2020) vq-wav2vec: self-supervised learning of discrete speech representations. In ICLR, External Links: Link Cited by: §2. [4] A. Baevski, H. Zhou, A. Mohamed, and M. Auli (2020) Wav2vec 2.0: a framework for self-supervised learning of speech representations. NeurIPS 33, p. 12449–12460. External Links: Link Cited by: §1, §2. [5] R. Balestriero and Y. LeCun (2025) LeJEPA: provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544. External Links: 2511.08544, Link Cited by: §1, §2, §3.2. [6] A. Bardes, J. Ponce, and Y. LeCun (2022) VICReg: variance-invariance-covariance regularization for self-supervised learning. In ICLR, External Links: Link Cited by: §2. [7] H. Chang, S. Bhati, J. Glass, and A. H. Liu (2025) USAD: universal speech and audio representation via distillation. In IEEE ASRU, External Links: Link Cited by: Table 1. [8] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei (2022) WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Topics Signal Process. 16 (6), p. 1505–1518. External Links: Link Cited by: §1, §2, Table 1, §4. [9] X. Chen and K. He (2021) Exploring simple siamese representation learning. In CVPR, p. 15750–15758. External Links: Link Cited by: §2. [10] C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu (2022) Self-supervised learning with random-projection quantizer for speech recognition. In ICML, PMLR, Vol. 162, p. 3915–3924. External Links: Link Cited by: §2. [11] Y. Chung, W. Hsu, H. Tang, and J. Glass (2019) An unsupervised autoregressive model for speech representation learning. In Interspeech, p. 146–150. External Links: Document, Link Cited by: §2. [12] Z. Fei, M. Fan, and J. Huang (2023) A-JEPA: joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311.15830. External Links: 2311.15830, Link Cited by: §2. [13] Q. Garrido, R. Balestriero, L. Najman, and Y. LeCun (2023) RankMe: assessing the downstream performance of pretrained self-supervised representations by their rank. In ICML, PMLR, Vol. 202, p. 10929–10974. External Links: Link Cited by: §4, §5.2. [14] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang (2020) Conformer: convolution-augmented transformer for speech recognition. In Interspeech, p. 5036–5040. External Links: Link Cited by: §4. [15] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed (2021) HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio Speech Lang. Process. 29, p. 3451–3460. External Links: Link Cited by: §1, §2. [16] G. Ioannides, A. Kieback, J. Goldfeder, L. Pang, A. Chadha, A. Elkins, Y. LeCun, and R. Shwartz-Ziv (2026) S-JEPA: soft clustering anchors for self-supervised speech representation learning. arXiv preprint arXiv:2606.19398. External Links: 2606.19398, Link Cited by: Figure 1, §1, §2, Table 1. [17] D. Klindt, Y. LeCun, and R. Balestriero (2026) When does LeJEPA learn a world model?. arXiv preprint arXiv:2605.26379. External Links: Link Cited by: §3.2. [18] Y. LeCun (2022) A path towards autonomous machine intelligence. Note: Position paper, OpenReviewVersion 0.9.2 External Links: Link Cited by: §1. [19] S. Ling and Y. Liu (2020) DeCoAR 2.0: deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659. External Links: Link Cited by: §2. [20] A. T. Liu, S. Li, and H. Lee (2021) TERA: self-supervised learning of transformer encoder representation for speech. IEEE/ACM Trans. Audio Speech Lang. Process. 29. External Links: Document, Link Cited by: §2. [21] L. Maes, Q. Le Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. External Links: 2603.19312, Link Cited by: §2, §3.3. [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) LibriSpeech: an ASR corpus based on public domain audio books. In ICASSP, p. 5206–5210. External Links: Link Cited by: §4. [23] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio (2020) Multi-task self-supervised learning for robust speech recognition. In ICASSP, External Links: Link Cited by: §2. [24] M. Rivière, A. Joulin, P. Mazaré, and E. Dupoux (2020) Unsupervised pretraining transfers well across languages. In ICASSP, External Links: Link Cited by: §2. [25] S. Schneider, A. Baevski, R. Collobert, and M. Auli (2019) wav2vec: unsupervised pre-training for speech recognition. In Interspeech, p. 3465–3469. External Links: Document, Link Cited by: §2. [26] L. Tuncay, E. Labbé, E. Benetos, and T. Pellegrini (2025) Audio-JEPA: joint-embedding predictive architecture for audio representation learning. In ICME, External Links: 2507.02915, Link Cited by: §2. [27] S. Yang, P. Chi, Y. Chuang, C. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G. Lin, T. Huang, W. Tseng, K. Lee, D. Liu, Z. Huang, S. Dong, S. Li, S. Watanabe, A. Mohamed, and H. Lee (2021) SUPERB: speech processing universal performance benchmark. In Interspeech, p. 1194–1198. External Links: Document, 2105.01051, Link Cited by: Figure 1, §1, §1, Table 1, §4. [28] J. W. Yoon, S. M. Kim, and N. S. Kim (2023) MCR-Data2vec 2.0: improving self-supervised speech pre-training via model-level consistency regularization. In Interspeech, p. 2833–2837. External Links: Document, Link Cited by: Figure 1, Table 1. [29] G. Yuksel, P. Guetschel, M. Tangermann, M. van Gerven, and K. van der Heijden (2025) WavJEPA: semantic learning unlocks robust audio foundation models for raw waveforms. arXiv preprint arXiv:2509.23238. External Links: Link Cited by: §2. [30] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny (2021) Barlow twins: self-supervised learning via redundancy reduction. In ICML, PMLR, Vol. 139, p. 12310–12320. External Links: Link Cited by: §2.