Paper deep dive
CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations
Gabriel Meseguer-Brocal, Yuexuan Kong, Romain Hennequin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Joint-Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space. However, it typically relies on teacher--student architecture with an EMA to stabilise training, and can tend to yield uninformative representations. Contrastive learning is stable to train and produces strong global representations, but remains limited on local tasks by the global nature of its objective. In this work, we combine both into CoJEPA: a single shared backbone jointly trained with a JEPA objective on masked sequence tokens and a contrastive objective on the class token. The contrastive gradient provides stability, removing the need for an EMA teacher entirely, while JEPA enriches the sequence tokens via local predictions that contrastive learning alone cannot provide. Crucially, no extra parameters are added to the backbone: the same model is guided towards richer representations purely through the design of its training signal. CoJEPA takes the best of both worlds, outperforming or matching both individual methods across global and local MIR tasks, with a particularly strong advantage on tonal and harmonic understanding, and without any task-specific architectural changes. CoJEPA shows that combining objectives with complementary inductive biases can substitute for scale, encouraging future work to invest in smarter training objectives over ever-larger models.
Tags
Links
- Source: https://arxiv.org/abs/2608.30974v1
- Canonical: https://arxiv.org/abs/2608.30974v1
Trouble viewing inline? Open PDF directly →
Full Text
44,274 characters extracted from source content.
Expand or collapse full text
CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations Abstract Joint-Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space. However, it typically relies on teacher–student architecture with an EMA to stabilise training, and can tend to yield uninformative representations. Contrastive learning is stable to train and produces strong global representations, but remains limited on local tasks by the global nature of its objective. In this work, we combine both into CoJEPA: a single shared backbone jointly trained with a JEPA objective on masked sequence tokens and a contrastive objective on the class token. The contrastive gradient provides stability, removing the need for an EMA teacher entirely, while JEPA enriches the sequence tokens via local predictions that contrastive learning alone cannot provide. Crucially, no extra parameters are added to the backbone: the same model is guided towards richer representations purely through the design of its training signal. CoJEPA takes the best of both worlds, outperforming or matching both individual methods across global and local MIR tasks, with a particularly strong advantage on tonal and harmonic understanding, and without any task-specific architectural changes. CoJEPA shows that combining objectives with complementary inductive biases can substitute for scale, encouraging future work to invest in smarter training objectives over ever-larger models. 1 Introduction Self-supervised learning (SSL) has emerged as an impactful paradigm in music information retrieval (MIR) in recent years. It enables the learning of rich audio representations without requiring heavy manual annotations, thereby circumventing common issues seen in supervised learning, such as label ambiguity, annotation cost, and lack of generalization. Moreover, SSL aligns with how human learns internal musical representations, through exposure, interaction and prediction [1, 2, 3], as musical training is not necessary for requiring musical knowledge [4]. In this paradigm, a backbone is trained via a pretext task where the supervisory signal is derived directly from the data itself, rather than from human annotations. The backbone aims to produce useful representations that can be adapted to a wide range of downstream tasks with minimal effort, usually via probing, i.e., training a lightweight classifier on top of the frozen backbone representations. What makes SSL particularly attractive is not only the little requirement of annotations and the reusability of the learned representations, but also the fact that they have managed to surpass supervised approaches on a broad set of downstream tasks [5, 6]. Yet a central challenge remains: designing better training objectives that guide models toward more rich and diverse features. As one of the latest family in SSL, Joint-Embedding Predictive Architecture (JEPA) was recently proposed [7, 8]. The core idea of of JEPA is to predict masked regions directly in a learned latent space. In practice, JEPA relies on a teacher–student framework in which a teacher encoder produces latent targets while a student network learns to predict them. The teacher is updated through an exponential moving average (EMA) mechanism to stabilize training and avoid collapse [8]. However, while powerful in learning representations, JEPA is still hard and unstable to train, with extensive hyperparameter search, and prone to collapse [8, 9]. At the same time, as an easy-to-train and stable SSL paradigm, contrastive learning performs worse on local tasks since the losses are typically applied to capture global variance and invariance [10, 11, 12]. Moreover, recent analyses of transformer-based contrastive learning models have revealed emergent musical properties in sequence tokens [13]. Surprisingly, even when contrastive objectives are applied only to a global class token, sequence tokens still encode local musical information useful for frame-level downstream tasks. Building on these ideas, we propose CoJEPA an efficient and novel method for combining the strengths of JEPA and contrastive learning. To a traditional JEPA, we apply a contrastive loss on the class token of a transformer, as a regularisation to constrain the latent space, removing the need for EMA and thereby stabilizing training. As a result, CoJEPA improves the quality of global and local representations compared to standard JEPA and contrastive pre-training and it outperforms either method alone on downstream tasks, and remains competitive against a much larger model, MusicFM [5]. Figure 1: Overview of the proposed method. Two full views ax_a and bx_b are encoded by a shared backbone; their CLS tokens are contrasted via ℒcontrastiveL_contrastive. The same backbone acts as a teacher (full ax_a, stop-gradient) and as a student (masked ax_a), a predictor gϕg_φ reconstructs the missing tokens from context representations and learnable positional placeholders, with gradients from ℒJEPAL_JEPA flowing back through the predictor and student path. The backbone is thus trained simultaneously as a teacher via contrastive and as a student via JEPA, removing the need for an EMA teacher. 2 Literature review We review prior work on contrastive learning and JEPA: self-supervised methods that propose different ways to derive the training signal from the data itself. Joint embedding architectures (JEA) learn representations by mapping different views of the same input to nearby locations in a shared embedding space. These views are typically obtained through data augmentations or different contextual transformations of the same audio excerpt. Different approaches use different losses to control the invariance and variance needed in the latent space [14, 15, 16]. In music representation learning, joint embedding methods-and more specifically, contrastive learning—have proven particularly effective for learning global semantic representations that transfer well across downstream tasks [10, 17, 18, 19, 13]. However, since the losses are applied to a global representation, these approaches often underperform on tasks requiring fine-grained temporal information, such as beat tracking or chord estimation [13]. Masking modeling methods learn by predicting masked portions of the input [20, 21]. By forcing the model to predict masked regions from contextual information, these methods encourage representations to capture rich information. In audio and music, masking-based approaches have shown strong performance, with models such as MusicFM [5] and MERT [6] establishing state-of-the-art results across a wide range of MIR tasks. However, reconstructing signals directly in the input domain may encourage the model to focus on low-level details that are not always aligned with perceptually meaningful musical semantic features. To address the limitations of masked modelling, Joint-Embedding Predictive Architectures (JEPA) was recently proposed [7, 8]. Rather than reconstructing missing content in the raw input space, JEPA predicts masked regions directly in a learned latent space. This shifts the objective away from low-level reconstruction and toward semantic prediction. It has shown strong performance in audio and music domain [22, 23]. Although effective, JEPA requires EMA, therefore remains difficult to optimize in practice due to training instability and sensitivity to hyperparameters [24, 25]. Equivariant SSL learns representations that transform equivariantly under known input transformations [26, 27, 28, 29, 30, 31]. Applying a transformation to the input is expected to induce an equivariant transformation in the latent representation. Specific to this family, the pretext task is explicitly aligned with the downstream task of interest. While effective for specialized tasks, equivariant approaches are tightly coupled to the modelled transformation and therefore less transferable across diverse downstream tasks. While these works have traditionally been studied in isolation, recent work has begun exploring hybrid approaches that leverage multiple SSL principles simultaneously. For instance, in MIR, [12] combines joint embedding learning with equivariance constraints, demonstrating that incorporating geometric structure can enhance representation quality. Similarly, other works have explored combinations such as contrastive learning with masked prediction [32, 33], suggesting that different SSL objectives capture complementary aspects of the data structure. Furthermore, to address the limitations of JEPA, recent work combines predictive latent-space learning with regularized joint embedding objectives [34], by combine I-JEPA [8] with VICReg [15], introducing explicit variance and covariance regularization terms to stabilize the latent space and prevent collapse. However, EMA is still necessary to stabilize the training. 3 Methodology In this section, we describe the two SSL paradigms at the core of our framework, contrastive learning and JEPA, and present how we combine them into a unified training framework. We work with an updated ViT-1D transformer [13] (Section 4) as the backbone encoder, which processes audio inputs as sequences of tokens, with a class token prepended at the beginning of the token sequence. This choice is central to our approach, as the self-attention mechanism and the class token architecture of transformers play a key role in enabling the combination of both paradigms. 3.1 JEPA JEPA [7, 8] extends masked modeling by operating in the representation space: tokens are masked after encoding, and a predictor is trained to reconstruct the missing latent representations from the visible ones. Given a sequence of T input audio tokens =x1,…,xTx=\x_1,…,x_T\ where xi∈ℝdex_i ^d_e (see Section 4.1): Teacher. A teacher encoder fθtf_θ_t processes the full sequence x, producing a sequence of token representations t=[z1t,…,zTt]=fθt()z^t=[z_1^t,…,z_T^t]=f_θ_t(x), where each zit∈ℝdez_i^t ^d_e captures the contextualised semantics of the whole sequence. The tokens at the masked positions serve as target tokens. Student. A student encoder fθsf_ _s processes only the visible (context) tokens c=xi:mi=1x^c=\x_i:m_i=1\, defined by a binary mask ∈0,1Tm∈\0,1\^T, producing ∑imi _im_i context tokens c=zic:mi=1z^c=\z_i^c:m_i=1\ with zi∈ℝdez_i ^d_e . Predictor. A lightweight predictor gϕg_φ takes z and prior information about the missing tokens as input; commonly, the positional embeddings pip_i of the masked tokens. It then estimates the representations: ^=gϕ(c,pi:mi=0) z=g_φ(z^c,\p_i:m_i=0\) gϕg_φ is itself a transformer where masked tokens attend to context positions via cross-attention, encouraging z to encode useful information for predicting the missing content. Loss. The training objective combines L2 distance and cosine similarity between predicted and target tokens, computed only at the masked positions: ℒJEPA=1∑imi∑i:mi=0(∥z^i−zit∥22+1−z^it⋅zit‖z^it‖‖zit‖)L_JEPA= 1 _im_i _i:\,m_i=0 (\| z_i-z^t_i\|_2^2+1- z^t_i· z^t_i\| z^t_i\|\|z^t_i\| ) The L2 term penalises magnitude differences and prevents scale ambiguity. Cosine similarity enforces directional alignment between predicted and target. Their combination is motivated by the decomposition of SSL objectives into alignment and uniformity components [11]: L2 alone is prone to representation collapse through magnitude shrinkage, whereas cosine similarity induces higher-order gradient dynamics that provide stronger collapse resistance in non-contrastive settings [35]. By predicting in latent space rather than in the raw signal domain, JEPA encourages the model toward semantically meaningful representations, abstracting away low-level reconstruction details. A key challenge in JEPA is that the target tz^t is not fixed but evolve throughout training, since the teacher encoder fθtf_ _t is updated through an EMA of the student weights: θt←τθt+(1−τ)θsθ_t←τ _t+(1-τ) _s where τ∈[0,1)τ∈[0,1) is a momentum coefficient progressively increased toward 11 [36, 8]. This moving target creates a co-adaptation dynamic: the student is pushed toward semantically rich representations in order to be useful to predict the targets, while the targets are pressured to become easier to predict, leading to the potential representational collapse [34]. Although EMA slows the process by ensuring targets evolve smoothly, it does not fully prevent collapse and does not necessarily ensure useful features [9, 34, 15], while also introducing more hyperparameters related to τ. 3.2 Contrastive learning Contrastive learning learns representations by mapping different views of the same input to nearby locations in a shared embedding space. We obtain the two views by sampling from the same track two random temporally proximate segments [19, 10, 17, 18, 13]. Intuitively, two excerpts from the same song should share sufficient musical semantics to yield similar representations, while segments from different tracks should be distinguishable. Given a batch of B audio samples, each sample yields two views x and ′x . A dedicated class token is prepended at the beginning of the token sequence, i.e., position 0, initialized with trainable parameters and the average of the sequence tokens, used to aggregate information from the full sequence via self-attention. Two views are passed through the same encoder and class tokens are obtained as: z0=fθ()[CLS],z0′=fθ(′)[CLS]z_0=f_θ(x)_ [CLS],z _0=f_θ(x )_ [CLS]. Loss. We use the NT-Xent loss [14]. For a positive pair (z0,q,z0,q′)(z_0,q,z _0,q) obtained from the same audio sample q (we add q for a clearer demonstration of contrastive loss), the per-pair loss is defined as: ℓ(q)=−logexp(sim(z0,q,z0,q′)/t)∑k≠qexp(sim(z0,q,z0,k)/t), (q)=- \! (sim(z_0,q,z_0,q )/t ) Σ _k≠ q \! (sim(z_0,q,z_0,k)/t ), the denominator includes representations from all other samples in the batch, which act as negative examples. The total loss ℒNT-XentL_NT-Xent is computed for all positive pairs. 3.3 Combining JEPA and contrastive learning Recent work has shown that transformers trained with a contrastive objective on the class token already develop semantically meaningful representations in their sequence tokens, even without any direct supervision on them [13]. This suggests that contrastive learning naturally sets up a representation space where JEPA can operate effectively and benefit from. We leverage this observation and combine the two paradigms into CoJEPA: contrastive learning provides a well-structured, stable global embedding space, while JEPA explicitly pushes individual sequence tokens to encode rich local information. Figure 1 provides an overview of the proposed method and the flow of gradients through each objective. Shared backbone. Our key design contribution is to use a single shared backbone fθf_θ (i.e. with θ=θt=θsθ= _t= _s) for the teacher and student roles, without any EMA mechanism. Depending on the input it receives, fθf_θ operates in two modes (annotations slightly differ from Section 3.1): • Teacher mode: fθf_θ processes the full sequence x and produces a complete representation =fθ()z=f_θ(x), including a class token z0z_0 and sequence tokens 1:Nz_1:N. • Student mode: fθf_θ processes only the visible context tokens cx^c, producing context representations cz^c. Forward pass. Each training step processes two views (,′)(x,x ) sampled from the same track. Both views are passed through fθf_θ in teacher mode, producing: =fθ(),′=fθ(′)z=f_θ(x), =f_θ(x ) The class tokens z0z_0 and z0′z _0 are used to compute the NT-Xent contrastive loss ℒNT-XentL_NT-Xent. In parallel, the sequence tokens at the masked positions zi:mi=0\z_i:m_i=0\ serve as JEPA prediction targets. These are detached from the computational graph (sg(zi:mi=0)sg(\z_i:m_i=0\)), so that the JEPA loss does not propagate through teacher mode. The masked views cx^c are then processed in student mode, and the predictor recovers the target representations: ^=gϕ(fθ(c),pi:mi=0) z=g_φ\! (f_θ(x^c),\,\p_i:m_i=0\ ) Gradient flow. The backbone fθf_θ receives gradients from two sources: • ℒNT-XentL_NT-Xent: flows through the teacher forward pass, shaping the global structure of the embedding space. • ℒJEPAL_JEPA: flows through the predictor and the student forward pass, pushing sequence tokens to encode locally predictive information. The total training objective is: ℒ=ℒNT-Xent+ℒJEPAL=L_NT-Xent+L_JEPA Eliminating EMA. Unlike standard JEPA, CoJEPA does not require EMA mechanism. The contrastive loss directly regularizes the backbone, preventing representational collapse by enforcing a well-structured global embedding space. The teacher is thus not a slowly drifting copy of the student but the same model, updated jointly and stably through ℒNT-XentL_NT-Xent. 4 Model 4.1 Inputs Our model takes as input the raw waveform sampled at 1616 kHz and computes a normalized mel-frequency spectrogram with 128128 frequency bins and a frame rate of 31.531.5 Hz. This representation is widely used in MIR for contrastive learning and audio tagging task [18, 10]. The spectrogram is patchified such that each time frame is projected into the transformer embedding dimension, yielding a sequence of tokens =[1,…,T] x=[ x_1,…, x_T] with i∈ℝde x_i ^d_e, de=192d_e=192, and T=126T=126 for a 44-second segment. 4.2 Backbone The backbone is updated based on a Vision Transformer with 1-D patches (ViT-1D) [13], using an embedding dimension of 192192, a depth of 1212 blocks, and 33 attention heads per block, Tiny configuration from the original ViT paper [37]. We incorporate three architectural upgrades from modern large language models [38]: (a) rotary positional embeddings (RoPE) in place of standard learned positional embeddings, (b) RMSNorm instead of LayerNorm, and (c) feed-forward layers with gated activations. These upgrades result in a backbone with 7,1M trainable parameters. Class token. The class token (CLS token) is a special token prepended to the input sequence, responsible for aggregating global sequence information via self-attention. It is defined as a learnable vector of dimension de=192d_e=192, to which we add the mean of the projected input tokens, spe x_spe. And it is where the joint-embedding loss is applied. Masking technique. We apply contiguous temporal masking: a single block of consecutive time frames is masked, with all frequency bins zeroed out. Since our patchification maps one time frame to one token, this is equivalent to masking a contiguous block of tokens in the input sequence. The mask length per sequence is sampled uniformly from 50%50\% to 75%75\% of the sequence length T. The starting position is drawn uniformly at random from the valid range. Each sample in the batch receives an independently sampled mask length, introducing variability in the amount of context available to the student encoder. 4.3 Predictor The predictor has the same architecture as the backbone but smaller: an embedding dimension of 192192, a depth of 66 blocks, resulting in 3.53.5M parameters. Ablation of these design choices is left for future work; which is relevant given that intermediate layers of the backbone are the ones capturing the richest representations, suggesting the predictor’s depth plays a role in shaping them Section 5. Masked tokens. At each masked position, a learnable token of dimension 192192 is injected as a placeholder, to which the corresponding rotary embedding is added to encode the position of the missing token within the sequence. This allows the predictor to attend to masked positions while distinguishing them from the visible context tokens. Task Dataset Size nb. classes Global tasks Tagging MTAT 25k 50 Key FMAKv2 5.5k 24 Pitch TinySOL 3k 84 Instruments TinySOL 3k 14 BPM GiantStepstempo 664 72 Local tasks Chords RWCpop + SWD 124 25 Beat Ballroom + GTZANrhythm 1.7k – Table 1: Statistics of downstream datasets. We omit the nb. classes for beat tracking (More details in 5.2). Layer 4 - early Layer 8 - mid Layer 12 - last Baseline JEPA Contrastive CoJEPA JEPA Contrastive CoJEPA JEPA Contrastive CoJEPA MusicFM Task Metric seq CLS seq CLS seq seq CLS seq CLS seq seq CLS seq CLS seq seq Tagging MAPmacro .404 .265 .266 .410 .429 .393 .266 .266 .436 .436 .376 .393 .395 .395 .421 .454 ROC-AUCmacro .888 .804 .803 .891 .896 .884 .804 .802 .903 .902 .875 .882 .881 .884 .897 .906 Key w-acc .245 .232 .243 .633 .609 .167 .234 .236 .640 .518 .147 .539 .550 .503 .396 .558 Pitch acc .959 .962 .966 .983 .979 .877 .962 .962 .973 .973 .699 .925 .962 .873 .921 .969 Instruments acc .938 .692 .688 .935 .945 .911 .695 .688 .949 .952 .873 .942 .952 .942 .925 .897 BPM acc@1 .375 .444 .403 .389 .604 .806 .389 .396 .757 .826 .833 .854 .847 .840 .826 .771 acc@2 .389 .444 .403 .389 .632 .847 .389 .396 .799 .861 .875 .917 .895 .910 .875 .819 Table 2: Global tasks at three transformer depths. JEPA is evaluated only on mean-pooled sequence tokens (seq). Contrastive and CoJEPA are evaluated on both the CLS token and mean-pooled sequence tokens. MusicFM is reported as a single baseline without layer ablation. Underlined: best overall. Bold: best among our three models. Red: best JEPA. Yellow: best Contrastive. Blue: best CoJEPA. 5 Experiments 5.1 Pretraining We train three models: JEPA, contrastive, and CoJEPA. For the JEPA baseline we follow the standard EMA-based training procedure [8]. Dataset. We use an in-house dataset consisting of about 22M full tracks to pre-train our models. The dataset is curated to be as musically balanced and diverse as possible, in order to improve generalization to unseen data. Hyperparameters. All models are trained for 999999 epochs with 512512 steps per epoch and a batch size of 128128. We note that the batch size of 128128 is small relative to common contrastive learning practice, which typically relies on large batches to provide sufficient negative pairs [10]. We use the AdamW optimizer with β=(0.9,0.98)β=(0.9,0.98) and a cosine learning rate schedule with 1010 warm-up epochs and a minimum learning rate of 1.5×10−61.5× 10^-6. Weight decay follows a cosine schedule from 0.040.04 to 0.010.01. For the JEPA EMA baseline, the teacher momentum is scheduled from 0.9960.996 to 0.99990.9999 over the full training. The masking ratio is fixed at 75%75\%. All models are trained on a single GPU. 5.2 Downstream tasks We group tasks into global (sequence-level prediction: tagging, key, pitch, BPM) and local (frame-level prediction: beat tracking, chord estimation). Table 1 summarizes all evaluation datasets. Global tasks require a sequence-level prediction capturing the overall musical content, while local tasks require frame-level predictions at a resolution adapted to the downstream task. Global tasks. For music tagging, we train and evaluate on MagnaTagATune using the split of [39], and report macro ROC-AUC and mAP. For instrument recognition and pitch estimation, we use TinySOL [40] with a random 8:1:1 split, evaluating top-1 accuracy over 14 instrument classes and 82 pitch classes respectively. For key estimation, we train on FMAKv2 [28] and test on GiantSteps [41], using the MIREX weighted accuracy metric [42]. For BPM estimation, we use Giantsteps-tempo-dataset[41] with a split of 8:1:1 for training, validation and evaluation. We report ACC1ACC_1, accuracy within a 4% tolerance, and ACC2ACC_2 accuracy with the same tolerance additionally allowing octave errors of 2, 3, 1/2, and 1/3. Local tasks. For beat tracking, we train on Ballroom Dataset [43] and evaluate on GTZAN Rhythm [44]. We apply temporal smoothing, reporting the F1F_1-score with a ± 70 ms70\,ms tolerance window. For chord estimation, we use a combined corpus of RWC-POP [45] and Schubert Winterreise Dataset [46] with an 8:1:1 split, and evaluate frame-level accuracy over 24 major/minor chord classes plus a “no chord” class. Chords Beat Layer Model overlap F-measure P-score 4 - early JEPA .211 .778 .745 Cont. .258 .549 .551 CoJEPA .399 .778 .747 8 - mid JEPA .097 .803 .774 Cont. .256 .551 .551 CoJEPA .275 .793 .764 12 - last JEPA .070 .806 .780 Cont. .241 .737 .707 CoJEPA .160 .789 .757 Baseline MusicFM .193 .866 .851 Table 3: Local tasks. The probe is applied independently to each sequence token (no pooling). 5.3 Evaluation protocols We follow the standard SSL linear probing protocol: backbone weights are frozen and a single linear layer (192→192→ nb_classes + softmax) is trained per task for 5050 epochs, 128128 steps per epoch, batch size 128128, learning rate 10−310^-3 and fixed random seed. We probe at three transformer depths: layer 44 (early), layer 88 (mid), and layer 1212 (last), since different layers capture qualitatively different information [47, 13], allowing us to better attribute performance differences to each training objective. Global tasks. We probe a single fixed-length sequence representation using one of two pooling strategies: • CLS token: for Contrastive and CoJEPA, explicitly optimized to aggregate global information. • Mean-pooled sequence tokens: The only option for JEPA (no CLS token). We computed for all three models as it provides a complementary view of how well the local tokens capture global musical content. Local tasks. The linear probe is applied independently to each token in the sequence. This evaluates the quality of token-level representations and is relevant for assessing the benefit of the JEPA objective on local features. Baseline. We compare against MusicFM [5], a recent SSL foundation model based on a Conformer encoder (330330M parameters, de=1024d_e=1024). We extract features from its last transformer layer only and train our linear probes on 1010-second segments (consistent with pretraining) or on the longest available segment when the audio is shorter. We acknowledge that intermediate layers may yield stronger results [47]. MusicFM thus serves as an indicative reference point against a recent large-scale model. 6 Results Comparison of JEPA, Contrastive, and CoJEPA. Tables 2 and 3 report linear probing results across all tasks at three layers. CoJEPA achieves the best or tied-best result on the majority of tasks, confirming that combining both objectives takes the best of both worlds: the local representational richness of JEPA and the globally structured embedding space of contrastive learning. The only exceptions are rhythmic tasks: BPM, where Contrastive (CLS) marginally leads, and beat tracking, where JEPA leads; yet CoJEPA remains competitive in both cases. Tasks requiring different musical understanding: tagging, key, and chords; show the largest benefit, with substantial margins over either individual method. JEPA representations peak early. For most tasks, JEPA representations are strongest at layer 44 and degrade with depth: pitch drops from .959.959 to .699.699, key from .245.245 to .147.147, and chords from .211.211 to .070.070. This suggests that the JEPA pretext task, as a local temporal prediction objective, produces its most transferable features early in the network but does not provide sufficient guidance for deeper layers to develop music rich semantic representations. This is consistent with findings showing that JEPA alone requires large-capacity models to avoid representation collapse in later layers [8]. The notable exception is BPM and beat tracking, where JEPA improves with depth (BPM acc@1: .375→.833.375→.833, beat F-measure: .778→.806.778→.806). This suggests that rhythmic regularity, which is temporally localised and periodically structured, is naturally captured by the JEPA when representations mature, unlike tonal and semantic features that require explicit auxiliary objectives to steer training beyond JEPA’s inherent temporal bias. Contrastive representations crystallise at the last layer. Contrastive models produce weak early-layer representations for most tasks (e.g., key w-acc at layer 44: .243.243, instruments at layer 44: .692.692) but improve dramatically at the final layer (key: .550.550, instruments: .952.952). This indicates that contrastive learning encodes discriminative information primarily in the last-layer global embedding, leaving intermediate representations relatively uninformative. On local tasks, this weakness is most apparent: beat F-measure reaches only .737.737 even at the last layer, well below JEPA (.806.806) and CoJEPA (.793.793), reflecting the lack of direct supervision on individual sequence tokens. CoJEPA peaks at intermediate depth. CoJEPA achieves its best performance at layer 88 for most tasks: tagging MAP (.436.436), key (.640.640), instruments (.952.952), beat (.793.793); a qualitatively distinct behaviour from JEPA (peaks early) and Contrastive (peaks late), suggesting the two objectives interact to produce well-structured representations at intermediate depth. BPM follows the rhythmic trend of JEPA, improving steadily with depth (CLS at L12: .840.840). Particularly striking is the early emergence of tonal representations. Pitch, and chords peak at layer 44 (.983.983 and .399.399) and key at layer-8 (.64.64). This is unexpected given that harmonic information has been shown to be difficult to capture in general music representation learning models [6, 48] without a specifically designed training objective [28]. These features are not preserved at depth, likely discarded as contrastive increasingly dominates. CLS vs. Sequence Tokens. For Contrastive models, the gap between CLS and mean-pooled sequence tokens is consistently small, suggesting the contrastive signal passively enriches sequence tokens through self-attention. For CoJEPA, the gap is larger and task-dependent: the CLS token dominates on key (.640.640 vs. .518.518 at layer 88), While sequence tokens are better on BPM and tagging, particularly at early and intermediate layers where the JEPA objective is most influential. This layer-dependent divergence suggests that the two token types carry complementary information in CoJEPA: the CLS encodes a globally summary, enforced by the contrastive objective; the sequence tokens retain local structure, encouraged by the JEPA objective. Comparison with MusicFM. Despite its much larger scale, MusicFM leads only on tagging and beat tracking, and matching results for on pitch. As a purely masked prediction model, MusicFM also mirrors the behaviour of JEPA on rhythmic versus tonal tasks (strong on beat, weaker on chords) suggesting that the masked prediction objective, without a global discriminative component, faces the same structural limitations regardless of scale. Given the large gap in model capacity (7.17.1M, de=192d_e=192 vs 330330M, de=1024d_e=1024), these results further support the effectiveness of principled objective design over sheer scaling. 7 Conclusion We presented CoJEPA, a self-supervised framework that unifies JEPA and contrastive learning. The two objectives are complementary: JEPA drives sequence tokens to encode rich local structure; while contrastive learning shapes a globally discriminative embedding space, its gradient preventing collapse without the need for an EMA teacher. Crucially, both objectives operate on the same backbone: rather than adding parameters, CoJEPA guides the model towards richer representations purely through the design of its training signal. Despite its small scale (7.17.1M), CoJEPA outperforms or matches both individual methods across global and local tasks, with a particularly strong advantage on harmonic understanding. Interestingly, the richest representations emerge at intermediate depth rather than the final layer, a behaviour that neither JEPA nor contrastive learning exhibits alone. While CoJEPA is expected to benefit from scaling model and batch sizes (as contrastive learning improves with more negative pairs [10]) its strong performance at small scale shows the benefits of complementary training objectives. We encourage future work to invest more in smarter designs, notably the combination of different self-supervised learning objectives. References [1] J. S. Bamberger (1991) The mind behind the musical ear: how children develop musical intelligence. Harvard University Press. Cited by: §1. [2] D. J. Levitin (2012) What does it mean to be musical?. Neuron 73 (4), p. 633–637. Cited by: §1. [3] H. Honing (2011) The illiterate listener: on music cognition, musicality and methodology. Amsterdam University Press. Cited by: §1. [4] C. L. Krumhansl (2001) Cognitive Foundations of Musical Pitch. Oxford University Press. External Links: ISBN 978-0-19-514836-7, Link, Document Cited by: §1. [5] M. Won, Y. Hung, and D. Le (2024) A foundation model for music informatics. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Cited by: §1, §1, §2, §5.3. [6] L. Yizhi, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, et al. (2024) MERT: acoustic music understanding model with large-scale self-supervised training. In International Conference on Learning Representations, Cited by: §1, §2, §6. [7] Y. LeCun (2022) A path towards autonomous machine intelligence. Open Review 62 (1), p. 1–62. Cited by: §1, §2, §3.1. [8] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas (2023) Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 15619–15629. Cited by: §1, §2, §2, §3.1, §3.1, §5.1, §6. [9] R. Balestriero and Y. LeCun (2025) LeJEPA: provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544. Cited by: §1, §3.1. [10] M. C. McCallum, F. Korzeniowski, S. Oramas, F. Gouyon, and A. F. Ehmann (2022) Supervised and unsupervised learning of audio representations for music understanding. In Proc. International Society for Music Information Retrieval Conference (ISMIR), Cited by: §1, §2, §3.2, §4.1, §5.1, §7. [11] T. Wang and P. Isola (2020) Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning (ICML), Cited by: §1, §3.1. [12] Y. Kong, V. Lostanlen, R. Hennequin, M. Lagrange, and G. Meseguer-Brocal (2025) Multi-class-token transformer for multitask self-supervised music information retrieval. In 2025 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), p. 1–5. Cited by: §1, §2. [13] Y. Kong, G. Meseguer-Brocal, V. Lostanlen, M. Lagrange, and R. Hennequin (2025) Emergent musical properties of a transformer under contrastive self-supervised learning. In Proceedings of the International Society on Music Information Retrieval (ISMIR) Conference, Cited by: §1, §2, §3.2, §3.3, §3, §4.2, §5.3. [14] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), p. 1597–1607. Cited by: §2, §3.2. [15] A. Bardes, J. Ponce, and Y. LeCun (2022) VICReg: variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, Cited by: §2, §2, §3.1. [16] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny (2021) Barlow Twins: Self-Supervised Learning via Redundancy Reduction. arXiv (en). Note: arXiv:2103.03230 [cs] External Links: Link, Document Cited by: §2. [17] J. Spijkervet and J. A. Burgoyne (2021) Contrastive learning of musical representations. arXiv preprint arXiv:2103.09410. Cited by: §2, §3.2. [18] G. Meseguer-Brocal, D. Desblancs, and R. Hennequin (2024) An experimental comparison of multi-view self-supervised methods for music tagging. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1141–1145. Cited by: §2, §3.2, §4.1. [19] A. Ferraro, X. Favory, K. Drossos, Y. Kim, and D. Bogdanov (2021) Enriched music representations with multiple cross-modal contrastive learning. IEEE Signal Processing Letters 28, p. 733–737. Cited by: §2, §3.2. [20] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick (2022) Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16000–16009. Cited by: §2. [21] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed (2021) HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, p. 3451–3460. Cited by: §2. [22] A. Riou, S. Lattner, G. Hadjeres, M. Anslow, and G. Peeters (2024) Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation. arXiv (en). Note: arXiv:2408.02514 [cs] External Links: Link, Document Cited by: §2. [23] A. Quelennec, P. Chouteau, G. Peeters, and S. Essid (2025) MATPAC++: enhanced masked latent prediction for self-supervised audio representation learning. arXiv preprint arXiv:2508.12709. Cited by: §2. [24] X. Chen and K. He (2020) Exploring simple siamese representation learning. CoRR abs/2011.10566. External Links: Link, 2011.10566 Cited by: §2. [25] Y. Tian, X. Chen, and S. Ganguli (2021) Understanding self-supervised learning dynamics without contrastive pairs. CoRR abs/2102.06810. External Links: Link, 2102.06810 Cited by: §2. [26] E. Quinton (2022) Equivariant self-supervision for musical tempo estimation. In Proceedings of the International Society on Music Information Retrieval (ISMIR) Conference, Cited by: §2. [27] A. Riou, S. Lattner, G. Hadjeres, and G. Peeters (2023) PESTO: pitch estimation with self-supervised transposition-equivariant objective. In Ismir 2023 Hybrid Conference, Cited by: §2. [28] Y. Kong, V. Lostanlen, G. Meseguer-Brocal, S. Wong, M. Lagrange, and R. Hennequin (2024) STONE: self-supervised tonality estimator. In International Society for Music Information Retrieval Conference (ISMIR), Cited by: §2, §5.2, §6. [29] Y. Kong, G. Meseguer-Brocal, V. Lostanlen, M. Lagrange, and R. Hennequin (2025) S-key: self-supervised learning of major and minor keys from audio. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §2. [30] B. Gfeller, C. Frank, D. Roblek, M. Sharifi, M. Tagliasacchi, and M. Velimirović (2020) SPICE: self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, p. 1118–1128. Cited by: §2. [31] F. Cwitkowitz and Z. Duan (2024) Toward fully self-supervised multi-pitch estimation. arXiv preprint arXiv:2402.15569. Cited by: §2. [32] Y. Gong, A. Rouditchenko, A. H. Liu, D. Harwath, L. Karlinsky, H. Kuehne, and J. Glass (2023) Contrastive audio-visual masked autoencoder. In International Conference on Learning Representations, Cited by: §2. [33] Y. Chung, Y. Zhang, W. Han, C. Chiu, J. Qin, R. Pang, and Y. Wu (2021) W2v-bert: combining contrastive learning and masked language modeling for self-supervised speech pre-training. arXiv preprint arXiv:2108.06209. Cited by: §2. [34] S. Mo and S. Tong (2024) Connecting joint-embedding predictive architecture with contrastive self-supervised learning. In Advances in Neural Information Processing Systems, Cited by: §2, §3.1. [35] H. Bao, Y. Nagano, and K. Nozawa (2023) Feature normalization prevents collapse of non-contrastive learning dynamics. arXiv preprint arXiv:2309.16109. Cited by: §3.1. [36] J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. (2020) Bootstrap your own latent: a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, Vol. 33, p. 21271–21284. Cited by: §3.1. [37] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §4.2. [38] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.2. [39] J. Lee, J. Park, K. L. Kim, and J. Nam (2017) Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms. In Proc. International Conference on Sound and Music Computing (SMC), Cited by: §5.2. [40] C. E. Cella, D. Ghisi, V. Lostanlen, F. Lévy, J. Fineberg, and Y. Maresz (2020) OrchideaSOL: a dataset of extended instrumental techniques for computer-aided orchestration. In Proc. International Computer Music Conference (ICMC), Cited by: §5.2. [41] P. Knees, Á. Faraldo, P. Herrera, R. Vogl, S. Böck, F. Hörschläger, and M. L. Goff (2015) Two datasets for tempo estimation and key detection in electronic dance music annotated from user corrections. In Proc. International Society for Music Information Retrieval Conference (ISMIR), Cited by: §5.2. [42] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel (2014) Mir_eval: a transparent implementation of common mir metrics.. In Proc. International Society for Music Information Retrieval Conference (ISMIR), Vol. 10, p. 2014. Cited by: §5.2. [43] F. Gouyon and S. Dixon (2004) A review of rhythm description systems. In Proc. International Society for Music Information Retrieval Conference (ISMIR), Cited by: §5.2. [44] U. Marchand, Q. Fresnel, and G. Peeters (2015) Gtzan-rhythm: extending the gtzan test-set with beat, downbeat and swing annotations. Proc. International Society for Music Information Retrieval Late-breaking/Demo (ISMIR-LBD). Cited by: §5.2. [45] M. Goto and H. Hashiguchi (2002) RWC music database: popular, classical, and jazz music databases. Proc. International Society for Music Information Retrieval Conference (ISMIR). Cited by: §5.2. [46] C. Weiß, F. Zalkow, V. Arifi-Müller, M. Müller, H. V. Koops, A. Volk, and H. G. Grohganz (2021) Schubert winterreise dataset: a multimodal scenario for music analysis. Journal on Computing and Cultural Heritage (JOCCH) 14 (2), p. 1–18. Cited by: §5.2. [47] A. Pasad, J. Chou, and K. Livescu (2021) Layer-wise analysis of a self-supervised speech representation model. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 3179–3183. Cited by: §5.3, §5.3. [48] Y. Ding and A. Lerch (2024) Parameter-efficient transfer learning for music foundation models. arXiv preprint arXiv:2411.19371. Cited by: §6.