Paper deep dive
Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization
Habibeh Naderi, Behrouz Haji Soleimani, Stan Matwin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/27/2026, 7:37:49 PM
Summary
HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training) is a cross-attentive multimodal framework designed for learning document-level audio-text representations from long, segmented sequences in low-resource settings. The model uses frozen pre-trained speech and language encoders to extract segment-level features, which are then aggregated via cross-modal attention and self-attentive pooling. To address dimensional imbalance between audio and text, HILBERT employs a reciprocal dual contrastive objective that aligns both modalities to a joint representation. It incorporates two auxiliary regularizers: Centered Kernel Alignment (CKA) loss for structural consistency and Mutual Information (MI) loss for information balancing. For downstream tasks, it utilizes a Mixture-of-Experts (MoE) classifier to handle heterogeneous label regimes.
Entities (12)
Relation Signals (10)
HILBERT → employs → Mixture-of-Experts (MoE)
confidence 100% · For downstream prediction, HILBERT employs a Mixture-of-Experts (MoE) classifier
HILBERT → incorporates → Centered Kernel Alignment (CKA)
confidence 100% · Two auxiliary regularizers further stabilize long-sequence fusion: a Centered Kernel Alignment (CKA) loss
HILBERT → incorporates → Mutual Information (MI) Loss
confidence 100% · and a mutual information balancing loss that prevents dominance of a single modality
T5 → isa → Text Encoder
confidence 100% · Similarly, for text encoding, we employ pre-trained language models (e.g. T5, RoBERTa)
Whisper → isa → Audio Encoder
confidence 100% · For audio encoding, we utilize pre-trained models (e.g. Whisper, HuBERT)
HILBERT → leverages → T5
confidence 100% · pre-trained language models (e.g., T5, RoBERTa)
HILBERT → leverages → Whisper
confidence 100% · For audio encoding, we utilize pre-trained models (e.g., Whisper, HuBERT)
HILBERT → uses → Mutual Information (MI) Loss
confidence 100% · and a mutual information balancing loss that prevents dominance of a single modality
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained speech and language encoders to extract segment-level features, which are aggregated via cross-modal attention and self-attentive pooling to form modality-specific document representations and a joint cross-attentive embedding. To align modalities while preserving modality-specific structure under severe audio-text dimensional imbalance, we introduce a reciprocal dual contrastive objective that simultaneously aligns audio-to-joint and text-to-joint representations, rather than directly contrasting audio and text alone. Two auxiliary regularizers further stabilize long-sequence fusion: a Centered Kernel Alignment (CKA) loss that preserves structural consistency between each modality and the joint embedding, and a mutual information balancing loss that prevents dominance of a single modality by equalizing information flow from audio and text into the joint space. For downstream prediction, HILBERT employs a Mixture-of-Experts (MoE) classifier over concatenated audio, text, and joint representations to accommodate heterogeneous label regimes. Extensive evaluation across multiple audio-text backbone combinations demonstrates that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings.
Tags
Links
- Source: https://arxiv.org/abs/2604.16247v1
- Canonical: https://arxiv.org/abs/2604.16247v1
Trouble viewing inline? Open PDF directly →
Full Text
48,170 characters extracted from source content.
Expand or collapse full text
Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization Habibeh Naderi 1 (), Behrouz Haji Soleimani 1 , and Stan Matwin 1 Dalhousie University, Halifax NS, Canada habibeh.naderi@dal.ca, behrouz.hajisoleimani@dal.ca, stan@cs.dal.ca Abstract. We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive mul- timodal framework for learning document-level audio-text representa- tions from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained speech and language encoders to extract segment-level features, which are aggregated via cross-modal attention and self-attentive pooling to form modality-specific document represen- tations and a joint cross-attentive embedding. To align modalities while preserving modality-specific structure under severe audio-text dimen- sional imbalance, we introduce a reciprocal dual contrastive objective that simultaneously aligns audio-to-joint and text-to-joint representa- tions, rather than directly contrasting audio and text alone. Two auxil- iary regularizers further stabilize long-sequence fusion: a Centered Ker- nel Alignment (CKA) loss that preserves structural consistency between each modality and the joint embedding, and a mutual information bal- ancing loss that prevents dominance of a single modality by equalizing information flow from audio and text into the joint space. For down- stream prediction, HILBERT employs a Mixture-of-Experts (MoE) clas- sifier over concatenated audio, text, and joint representations to accom- modate heterogeneous label regimes. Extensive evaluation across mul- tiple audio-text backbone combinations demonstrates that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings. Keywords: Multimodal Representation Learning· Contrastive Learn- ing· Mixture of Experts· Mental Disorders Prediction. 1 Introduction Multimodal representation learning has emerged as a crucial research area [8], leveraging the co-occurrence of observations from interdependent sources, such as paired audio and text, which act as a form of weak supervision. By integrat- ing information across modalities, multimodal learning outperforms unimodal approaches in feature learning [4]. However, effectively aligning representations from different modalities while preserving their distinctive characteristics re- mains a significant challenge [13]. Specifically, in audio-text multimodal learning, arXiv:2604.16247v1 [cs.LG] 17 Apr 2026 2H. Naderi et al. a key challenge arises from the high dimensionality of audio representations com- pared to text, potentially causing imbalanced contributions from each modality. One of the key approaches for multimodal representation learning is con- trastive learning. This method minimizes the distance between semantically re- lated pairs while maximizing the distance between unrelated pairs in the embed- ded space. Through this objective, contrastive learning produces high-quality multimodal representations that exhibit robustness to distribution shifts and zero-shot transferability. CLIP exemplifies the success of contrastive learning in multimodal domains, playing a key role in advancing text-to-image generation techniques [7]. Simultaneously, sparsely activated Mixture-of-Experts (MoE) models [14] have proven effective for expanding model capacity while maintaining manage- able computational costs. By dynamically selecting a subset of parameters for each input, MoE models are particularly well-suited for multimodal learning since expert layers can learn an appropriate partitioning of modalities. These models enhance representation learning, improve multitask performance, and mitigate catastrophic forgetting in continual learning. The sparse MoE architec- ture provides several advantages: 1) increased model capacity without a propor- tional increase in computation, as only a subset of experts is typically active for any given input, 2) specialization of experts, where different experts can focus on different aspects of the data or different tasks, and 3) improved handling of heterogeneous data, which is common in multimodal scenarios. In this work, we introduce HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a novel framework for multi- modal audio-text representation learning that addresses the challenges of effec- tive cross-modal alignment while preserving modality-specific information, par- ticularly designed for long sequence representation. HILBERT balances modality contributions while preserving both shared and modality-specific features. Our approach leverages frozen pre-trained foundation models for feature extraction, employs a sophisticated dual contrastive learning strategy for cross-modal align- ment, and utilizes an MoE architecture for downstream task learning. We design specialized loss functions including Centered Kernel Alignment (CKA) and Mu- tual Information (MI) losses to ensure balanced and informative joint represen- tations. Our framework effectively handles the inherent imbalance between audio and text modalities, enabling more effective utilization of multimodal informa- tion for downstream tasks. HILBERT efficiently learns high-quality multimodal representations ensuring that the learned embeddings preserve semantic richness, effectively capturing both shared and modality-specific features. The task of multimodal audio-text representation learning aims to map paired audio and text data into a joint representation space where semantically re- lated pairs are placed close together, while unrelated pairs are pushed apart. A key challenge is balancing the contributions from different modalities, especially given the disparity in dimensionality and information density between audio and text data. Ensuring that the joint representation captures complementary infor- mation from both modalities, rather than being dominated by one, is crucial Joint-Centric Dual Contrastive Alignment with Structure-Preserving3 for effective multimodal learning. HILBERT incorporates a dual multimodal contrastive loss, designed to preserve structural alignment between modality- specific and joint representations. Unlike conventional contrastive methods that primarily focus on inter-modality alignment, our approach enforces both inter- and intra-modality consistency, preventing semantically similar representations from being pushed apart in the embedding space. Inspired by prior contrastive learning frameworks such as SimCLR [1] and CrossCLR [13], HILBERT ex- tends contrastive learning principles to multimodal data, improving the quality of learned joint embeddings [6]. HILBERT targets a learning regime that is fundamentally distinct from CLAP-style [2] and related cross-modal pretraining frameworks. CLAP variants are designed for large-scale audio-caption corpora and optimized for short au- dio clips through global embedding alignment, making them ill-suited for long, document-length audio-text inputs. In contrast, HILBERT is explicitly devel- oped for long-sequence, document-level representation learning, leveraging cross- modal self-attention to model segment-level interactions, auxiliary CKA and mutual information losses to balance modality contributions, and an MoE ar- chitecture for multi-task prediction. Unlike CLAP’s contrastive objective, which operates on pooled audio and text embeddings and does not capture long-range temporal structure, HILBERT directly models cross-modal dependencies across extended sequences. We therefore view CLAP and similar large-scale pretraining methods as complementary, while HILBERT provides a lightweight, backbone- agnostic framework for segment-aware audio-text integration under small-data and constrained training conditions. By integrating frozen pre-trained models, contrastive objectives, and MoE- based adaptation, HILBERT advances the state-of-the-art in multimodal repre- sentation learning for long sequences. Our experiments demonstrate that HILBERT achieves competitive performance in capturing semantically meaningful repre- sentations, making it well-suited for a wide range of downstream tasks. The main contributions of this work are as follows: – We introduce a dual contrastive learning strategy that simultaneously en- forces cross-modal and modality-specific consistency. Unlike conventional contrastive methods that primarily focus on inter-modality alignment, our approach explicitly preserves intra-modality structure, preventing semanti- cally similar instances from being separated in the embedding space. – We incorporate specialized loss functions including CKA and MI losses to enhance cross-modal alignment while preserving information-theoretic rele- vance. CKA loss ensures inter-modal and intra-modal alignment while MI loss ensures equitable contribution from both modalities to the joint rep- resentation. These specialized loss functions improve the quality of learned representations by ensuring that the joint embedding space remains seman- tically meaningful and well-structured. – We develop a multimodal joint encoder architecture that effectively combines frozen pre-trained foundation models with cross-modal self-attention mech- anisms to capture complex dependencies between audio and text modalities. 4H. Naderi et al. This multi-head self-attention mechanism generates high quality document- level representations from segment-level embeddings, capturing complex de- pendencies among segments. – We integrate an MoE approach for downstream task learning that adaptively leverages information from different experts based on input features, enabling efficient specialization across heterogeneous data distributions. 2 Proposed Method In this section, we present our novel framework for multimodal audio-text rep- resentation learning, which we call HILBERT (HIerarchical Long-sequence Bal- anced Embedding with Reciprocal contrastive Training). Our approach leverages pre-trained models for effective representation learning and addresses the chal- lenge of alignment in multimodal learning. The proposed framework extends the principles of multimodal contrastive representation learning and builds on prior state-of-the-art [6, 9] while introducing several innovations to enhance represen- tation quality and cross-modal alignment. The task of multimodal audio-text representation learning aims to learn effec- tive joint representations from paired audio and text data. LetD =(x a i ,x t i ,y i ) N i=1 denote a dataset containing N audio-text pairs, where x a i represents the audio input, x t i denotes the corresponding text input, and y i is the associated label (e.g., emotion, speaker identity, or semantic category). Our objective is to learn a joint representation space where semantically related audio and text pairs are mapped close to each other, while unrelated pairs are pushed apart. More formally, we aim to train a set of encoders that can map audio inputs x a and text inputs x t into a shared latent space, ensuring that the joint repre- sentation effectively captures complementary information from both modalities. This embedding space preserves high-quality modality-specific features while also modeling cross-modal interactions, making it well-suited for downstream tasks. HILBERT Model Architecture: Our HILBERT framework comprises four main components as illustrated in Figure 1. This architecture is specifically de- signed to process long sequences, such as large documents and extended audio files. It includes a segmentation step, which can be performed manually or auto- matically, to break the input into smaller segments before passing them to the model. 1. Frozen Pre-trained Models and Segment Embedding: We utilize pre- trained models for audio and text to extract rich feature representations from raw inputs. These pre-trained models have been trained on large-scale datasets and can provide high-quality embeddings that capture semantic and structural information. 2. Document Embeddings and Multimodal Joint Learning: The segment- level embeddings from the pre-trained models are processed through a multi- head self-attention mechanism to generate document-level representations. A multimodal joint encoder then combines information from both modalities. Joint-Centric Dual Contrastive Alignment with Structure-Preserving5 Frozen Pre-Trained Models Segment Embedding Downstream Learning Dual Contrastive Learning Document Embeddingsand Multimodal Joint Learning Pre-trained AudioEncoder (e.g.HuBERT) Pre-trained TextEncoder (e.g.T5) Segment embeddings Segment embeddings She's very kind and giving. She loves to be doing something all the time. Contrastive Loss A1 A2 A3 A4 A5 J5J4J3J2J1 Classification Head MLP Concatenation Multi-head self-attention Multi-head self-attention Multimodal Joint Encoder Multi-head self-attention with attention pooling Shared Projector (MLP) Expert MLP 1 Expert MLP 2 Gating Network Sparse Mixture of Experts (MoE) Encoder Network Expert MLP N ... Contrastive Loss T1 T2 T3 T4 T5 J5J4J3J2J1 Attention Pooling Attention Pooling 푍 ,푍 ௧ ,푍 ௧ 퐷 ,퐷 ௧ ,퐷 ௧ 푆 ,푆 ௧ 푋 ,푋 ௧ Fig. 1: Our proposed HILBERT model architecture. 3. Dual Contrastive Learning: This step consists of a shared encoder that maps inputs into a joint representation. We employ a contrastive learning approach that involves contrasting positive pairs (semantically related audio- text pairs) against negative pairs (unrelated pairs). Our designed contrastive loss incorporates multimodal knowledge into the each modality embeddings and aligns representations from different modalities through a Centered Ker- nel Alignment (CKA) loss. To further enhance relevance, we introduce a Mu- tual Information (MI) loss, ensuring the joint embedding retains meaningful information from each modality. 4. Downstream Learning with Mixture of Experts (MoE): For down- stream tasks, we utilize an MoE architecture to adaptively leverage informa- tion from different experts based on the input. Our pipeline is hierarchical: long audio and text are segmented, encoded with frozen models, and aggregated via document-level multi-head self-attention with modality-specific pooling and cross-modal fusion. This reduces complexity 6H. Naderi et al. from frame-level O(T 2 ) to segment-level O(L 2 ), with L in the tens. Only the segment-to-document and fusion layers are trained, keeping compute bounded while preserving long-range dependencies. In the following subsections, we elabo- rate on each component and discuss how they contribute to effective multimodal representation learning. 2.1 Frozen Pre-trained Models and Feature Extraction For audio encoding, we utilize pre-trained models (e.g., Whisper, HuBERT) that has been trained on large-scale audio data. The pre-trained audio encoder f a E (·) extracts rich acoustic and semantic features from the raw audio signal. Given an audio input X a , the pre-trained encoder processes it to generate segment-level embeddings, S a = f a E (X a )∈ R L×d a , where L is the number of segments, and d a is the dimension of the audio embedding space. Similarly, for text encoding, we employ pre-trained language models (e.g., T5, RoBERTa) that has been trained on large-scale text corpora. The pre-trained text encoder f t E (·) processes the text input X t to generate segment-level embed- dings, S t = f t E (X t ) ∈ R L×d t , where L again is the number of segments, and d t is the dimension of the text embedding space. These segment-level embeddings capture semantic information from the text. It is important to note that the parameters of these pre-trained models are kept frozen during training to avoid the computational overhead associated with fine-tuning large models and to prevent overfitting on limited data. This ap- proach aligns with the philosophy of leveraging the rich knowledge encoded in pre-trained models while keeping the training process efficient. 2.2 Document Embeddings and Multimodal Joint Learning Modality-Specific Document Embedding: We first project segment-level embeddings of audio and text (i.e. S a d a ,S t d t ) to the same dimension to ob- tain S a d ,S t d . Then to obtain document-level representations for audio and text (i.e. D a , D t ) from their corresponding segment-level embeddings, we employ multi-head self-attention mechanisms with attention pooling. The multi-head self-attention mechanism allows the model to attend to different parts of the input sequence and capture complex dependencies and relationships among seg- ments. Multimodal Joint Encoder: The multimodal joint learning module contains a joint encoder f joint E (·) designed to efficiently fuse the audio-text information. It consists of a cross-modal fusion layer and follows attention-based embedding layers. To achieve this, we take the audio-to-text embedding S a→t to show the cross-modal attention: S a→t = CM a→t (S a d ,S t d ) = softmax W Q a S a d W T K t S t d T √ d ! W V t (1) Joint-Centric Dual Contrastive Alignment with Structure-Preserving7 Similarly, we can get text-to-audio embedding S t→a . The obtained S a→t and S t→a will be concatenated together and projected to the latent space as the joint document embedding via a self-attention layer followed by attention pooling: D joint = f AttPool (SelfAttention(S a→t ⊕ S t→a ))(2) The multimodal joint encoder combines information from both modalities and captures cross-modal interactions. The design of the joint encoder is crucial for effective multimodal representation learning. Unlike simple concatenation or averaging, our joint encoder employs a more sophisticated fusion mechanism that can adaptively weigh the contributions from different modalities based on their informativeness and reliability. 2.3 Contrastive Learning and Cross-Modal Alignment To enhance the alignment between different modalities and promote effective joint representation learning, we employ a contrastive learning approach. Con- trastive learning aims to bring the representations of semantically related sam- ples closer while pushing apart the representations of unrelated samples in the embedding space. This module consists of a shared projector to map all inputs to the same latent space. Given the presence of a joint encoder in our architec- ture, we employ a dual contrastive learning strategy, which enforces alignment between (i) audio and joint representations and (i) text and joint representa- tions. Our specialized dual contrastive loss function is composed of three key components: 1) contrastive loss, 2) Centered Kernel Alignment (CKA) loss, and 3) Mutual Information (MI) loss. Shared Projector: The document-level representations D a and D t from both modalities as well as the joint document-level embedding D joint are then fed into a shared projector implemented as a multilayer perceptron (MLP) to map them into a common latent space, Z a = f shared (D a ) ∈ R d latent , Z t = f shared (D t ) ∈ R d latent , Z joint = f shared (D joint )∈ R d latent where d latent is the dimension of the common latent space and f shared (·) is the shared projection function. Multimodal Contrastive Loss: We utilize a multimodal contrastive loss to align the joint representation with modality-specific representations. Given a batch of embeddings B = Z a i ,Z t i ,Z joint i |B| i=1 , the positive pairs are defined as the joint embedding with its corresponding modality-specific embeddings, i.e., (Z a i ,Z joint i ) and (Z t i ,Z joint i ). All other pairing combinations are treated as negative pairs. Let c(x i ,x j ) represent the cosine similarity between two embeddings x i and x j , and τ be the temperature hyperparameter. The scaled similarity is defined as sim(x i ,x j ) = exp c(x i ,x j ) τ and the negative pairs for contrastive learning are defined as: Ω m i = X j̸=i sim(Z m i ,Z m j ) + sim(Z m i ,Z joint j ) + sim(Z joint i ,Z joint j ) (3) 8H. Naderi et al. where m ∈ a,t indicates the modality type. The contrastive loss for all data embeddings is then formulated as: L con (B) =− 1 |B| |B| X i=1 log sim(Z a i ,Z joint i ) Ω a i + sim(Z t i ,Z joint i ) Ω t i ! (4) This loss encourages the joint representation to be semantically aligned with both the audio and text representations of the same sample while being distinct from the representations of other samples. Unlike pairwise audio-text, modality- modality, joint-joint schemes, we deliberately avoid audio-text contrast in the objective and instead anchor both modalities to the fused joint space, then en- force covariance alignment and information balance. This reduces modality col- lapse without over-penalizing intra-modality neighbors, and is tailored to high- dimensional audio versus compact text and to our long-sequence hierarchical fusion. Centered Kernel Alignment (CKA) Loss: We propose to use Centered Ker- nel Alignment (CKA) as a measure of similarity between representation spaces. CKA is a powerful tool for measuring the similarity between neural network rep- resentations and has been shown to be invariant to orthogonal transformations and isotropic scaling [5]. Given two sets of representations X and Y , the CKA between them is defined as: CKA(K, L) = HSIC(K, L) p HSIC(K, K)· HSIC(L, L) (5) where K = X T and L = Y Y T are the kernel matrices, and HSIC (Hilbert- Schmidt Independence Criterion) [3] is defined as HSIC(K,L) = 1 (n−1) 2 trace(KHLH) where n is the number of samples, H = I n − 1 n 1 n 1 T n is the centering matrix, I n is the identity matrix, and 1 n is a vector of ones of length n. Given the centered data matrices ̃ X = HX and ̃ Y = HY , we can express the HSIC in terms of covariance: HSIC(K,L) = 1 (n− 1) 2 trace( ̃ X ̃ X T ̃ Y ̃ Y T ) =∥Cov( ̃ X T , ̃ Y T )∥ 2 F (6) where Cov(X,Y ) is the covariance matrix and ∥·∥ 2 F is the Frobenius norm of the matrix. With these definitions, the CKA can be formulated as: CKA(X, Y ) = ∥Cov(X, Y )∥ 2 F ∥Cov(X, X)∥ F ·∥Cov(Y, Y )∥ F (7) CKA ranges between 0 and 1, therefore for our multimodal scenario, the CKA loss components become: L m CKA = 1− ∥Cov(Z joint ,Z m )∥ 2 F ∥Cov(Z joint ,Z joint )∥ F ·∥Cov(Z m ,Z m )∥ F (8) Joint-Centric Dual Contrastive Alignment with Structure-Preserving9 where m∈a,t is the modality. The total CKA loss remains: L CKA = L a CKA + L t CKA (9) This covariance formulation highlights that CKA measures the normalized alignment between the covariance structures of the representations, providing deeper insight into how information is preserved across modalities in the joint representation. Mutual Information (MI) Loss: To further enhance the informativeness of the joint representation, we introduce an MI loss that maximizes the mutual information between the joint representation and each modality-specific repre- sentation. Mutual information measures the amount of information obtained about one random variable through observing another random variable. Since direct computation of mutual information is challenging, we adopt the InfoNCE estimator, which provides a lower bound on the mutual information: I(Z joint ;Z m )≥ E 1 |B| |B| X i=1 log f(Z joint i ,Z m i ) 1 |B| P |B| j=1 f(Z joint i ,Z m j ) = L m MI (10) where f(x,y) is a scoring function that measures the compatibility between x and y. We use the dot product as the scoring function, f(x,y) = exp(x T y/γ) where γ is a temperature parameter. The total MI term balances contributions of modalities in the joint embedding and prevents over-emphasis on a single modality: L MI =−(L a MI + L t MI ) + (L a MI − L t MI ) 2 (11) By minimizing the difference between the mutual information of the joint rep- resentation and each modality-specific representation, we ensure that the joint representation captures the essential information from both modalities. HILBERT uses CKA and MI as mechanisms for modality balancing, not for forcing audio and text to share identical covariance structure. The joint en- coder, via cross-attention, naturally integrates both modalities, CKA encourages structural consistency between each modality and the joint embedding, while MI prevents dominance of a single modality. In HILBERT, the modality gap is ad- dressed in three ways: 1) A shared projector maps both modality-specific and joint document embeddings into a common latent space, encouraging a compara- ble representation scale. 2) CKA between each modality and the joint embedding (not audio↔text directly), ensuring structural information from both inputs is retained without forcing identical covariance. This reduces representational mis- match while preserving modality-specific structure. 3) The MI term explicitly balances the mutual information between the joint embedding and each modal- ity, penalizing dominance of one modality and promoting equitable information flow and stabilizing long-sequence fusion. This is related in spirit to Cacophony’s [12] modality-balancing goals but integrated into a long-sequence cross-attentive encoder and MoE classifier. 10H. Naderi et al. 2.4 Downstream Learning with Mixture of Experts (MoE) For downstream tasks such as classification, we propose an MoE architecture that can adaptively leverage information from different experts based on the input. The MoE architecture consists of multiple expert networks and a gating network that determines the contribution of each expert to the final output. Each expert network is implemented as a multilayer perceptron (MLP) that takes the joint representation as input and produces a task-specific output E i (Z) = f i MLP (Z) where Z = [Z a ;Z joint ;Z t ] represents the concatenation of audio, text, and joint features. The gating network determines the contribution of each expert to the final output. It takes the concatenated features from all modalities as in- put and outputs a probability distribution over the experts g = f gate (Z)∈ R N E where N E is the number of experts, and f gate (·) is implemented as a neural network with softmax activation to ensure that the gating values sum to 1: g i = exp(f i gate (Z)) P N E j=1 exp(f j gate (Z)) (12) The gating network enables the model to dynamically select the most relevant experts based on the input features. This is particularly useful in multimodal scenarios where different experts may specialize in different aspects of the data. The outputs from all experts are combined using the gating values to produce the final representation, Z MoE = P N E i=1 g i · E i (Z). This weighted combination allows the model to adaptively leverage the strengths of different experts based on the input. For classification tasks, the MoE output is passed through a final classification head implemented as an MLP, ˆy = f sup (Z MoE )∈ R C where C is the number of classes, and f sup (·) is a multilayer perceptron with softmax activation. For the supervised classification loss, we use the cross-entropy loss: L sup =− 1 |B| |B| X i=1 C X c=1 y i,c log(ˆy i,c )(13) where y i,c is a binary indicator if class label c is the correct classification for sample i, and ˆy i,c is the predicted probability that sample i belongs to class c. Total Loss Function: The overall loss function for our multimodal representa- tion learning framework combines the supervised classification loss, contrastive loss, CKA loss, and MI loss: L total = L sup + λ con L con + λ CKA L CKA + λ MI L MI (14) where L sup is the supervised classification loss (e.g., cross-entropy loss for classi- fication tasks), and λ con , λ CKA , and λ MI are hyperparameters that control the contribution of each loss term. Joint-Centric Dual Contrastive Alignment with Structure-Preserving11 Table 1: Dataset information for the parent and offspring data. LevelTaskNum SamplesNum ClassesImbalanced (%)Label 0Label 1Label 2Label 3Label 4 Parent Document affect484 3 93.18 23 124 337 - - Parent Document warmth485 3 39.69 117 194 174 - - Parent Document overprotection 487 3 85.77 267 182 38 - - Parent Document cohesion492 5 92.63 14 14 132 190 142 Parent Document criticism492 4 89.22 232 166 69 25 - Parent Document worry381 4 93.29 164 147 59 11 - Parent Cognitive spectrum363 4 87.25 129 149 66 19 - Parent Cognitive introvert369 2 49.39 245 124 - - - Parent Cognitive extrovert369 2 63.84 271 98 - - - Parent Cognitive ADHD369 2 67.74 279 90 - - - Parent Cognitive anxiety369 2 51.81 249 120 - - - Parent Cognitive depression369 2 89.19 333 36 - - - Offspring Document affect621 4 97.73 16 11 109 485 - Offspring Document coherence616 5 95.83 11 33 109 264 199 Offspring Document richness614 3 96.70 82 515 17 - - Offspring Cognitive spectrum85 4 86.11 36 27 17 5 - Offspring Cognitive introvert85 2 39.62 53 32 - - - Offspring Cognitive extrovert85 2 53.45 58 27 - - - Offspring Cognitive ADHD85 2 53.45 58 27 - - - Offspring Cognitive anxiety85 2 39.62 53 32 - - - Offspring Cognitive depression85 2 83.56 73 12 - - - 3 Dataset The data used in this work consists of audio speech samples from 369 sub- jects participating in the Families Overcoming Risks and Building Opportunities for Well Being (FORBOW) research project [10]. Participants are parents, 266 mothers and 103 fathers, in the age range of 28-51 years. In these clinical inter- views, parents were asked to talk about their children for five minutes without interruption. Out of these subjects, 149 were diagnosed with Major Depressive Disorder (MDD), 66 were diagnosed with Bipolarity Disorder (BD), 19 were di- agnosed with Schizophrenia, and 129 were the control group with no major mood disorders. In addition to the parents’ interview files, FORBOW research project started collecting interviews with the children themselves. The audio interviews of children consists of 3 parts: 1) a three minute interview where children talk about themselves, 2) a two minute interview talking about a positive experience they had, and 3) a two minute interview where they talk about a negative expe- rience they had. All three interviews are uninterrupted with a total of 7 minutes of speech from each child. We transcribed and broke down each sample into multiple segments based on changes in emotion, sentiment, objectivity/subjectivity, etc. Average word count in a segment is 17 and average audio length for a segment is 6.47 sec- onds.Table 1 summarizes the dataset statistics, including the set of prediction tasks at different levels of granularity (segment-level, document-level, and psy- chological and cognitive tasks), as well as the label distributions and the degree of class imbalance. 12H. Naderi et al. 4 Experimental Results In this section, we provide the experimental results of our proposed contrastive learning framework. We chose six different language models, nliRoBERTa, nliDis- tilRoBERTa, nliMPNet, paraTinyBERT, sentenceT5XL, and allMiniLM12. We also chose five pre-trained audio models, whisperMedium, wav2vec2Large-FineTune, hubertLargeFineTune, conformerLargeFineTune, and spectrogram. This results in a total of 30 combinations, trained separately on each individual task for both parent and offspring data. For the dimensionality of the shared projector in contrastive embedding we tested 64, 128, and 256 and the quality of predictions was not sensitive to the dimensionality of the contrastive latent space. Therefore, in the following we only report the results achieved with dimensionality of 128. The expert network consists of 8 experts each of which is a 2 layer MLP with 32 and 32 units. The classification head consists of 3 layer dense network, a 32-unit layer followed by a 16-unit layer followed by a softmax layer with the size corresponding to the classification task. All results in this section are based on a 25-fold cross- validation. Table 2 reports AUC (%) from 25-fold cross-validation over 6 backbone com- binations and 4 architecture configurations on 6 document-level and 7 psycho- logical spectrum tasks using parent data, with methods sorted by average perfor- mance across all tasks. Overall, the results demonstrate a clear and consistent advantage of the full HILBERT (Dual Contrastive + MoE) framework across nearly all backbone combinations. Adding MoE to transfer learning yields mod- est but systematic improvements, while introducing contrastive learning leads to further gains, particularly on more challenging psychological traits. The com- plete HILBERT configuration achieves the strongest performance overall, with pronounced improvements on document-level affective tasks and gains of up to 5-10 AUC points over CLAP-based pre-training, highlighting the benefit of task- aware multimodal contrastive learning over generic audio-text alignment. Across backbones, nMPNet and nRoBERTa combined with hubERT or w2v2 achieve the highest absolute AUCs (e.g., ∼ 80 for affect and ∼ 70 for cohe- sion/mood), indicating that stronger text encoders better leverage dual-contrastive supervision. Performance gains are consistent across both task families, suggest- ing improved generalization rather than task-specific overfitting. Notably, Tiny- BERT+ConformerLarge (HILBERT) attains the best result on the spectrum task (66.75% AUC), which is the most important task with 4 classes of mental disorders including depression, bipolar, Schizophrenia, and control group. It is also the most complicated task due to our low resource and highly imbalanced data. Table 3 presents AUC (%) results from 25-fold cross-validation on offspring data across 3 document-level and 7 psychological spectrum tasks. Similar to the parent setting, the full HILBERT (Dual Contrastive + MoE) framework con- sistently outperforms transfer-only and partial ablation variants across nearly all backbone combinations. Incorporating MoE yields steady improvements over plain transfer learning, while contrastive learning further enhances performance, Joint-Centric Dual Contrastive Alignment with Structure-Preserving13 Table 2: AUC (%) on a 25 fold cross-validation for document-level and psycho- logical spectrum predictions using parent data. Document-level tasksCognitive and Psychological tasks affectwarmthoverprotectcohesioncriticismworrySpectrumIntrovertExtrovertADHDAnxietyDepressionMood CLAP-LAION [11]71.84 56.11 51.73 60.65 56.96 54.8653.09 55.11 56.18 54.07 57.03 59.86 57.51 TinyBERT+hubLgFT (Baseline: Transfer)72.88 59.67 57.21 63.93 57.02 59.3358.53 61.50 56.77 57.11 57.61 59.20 63.60 TinyBERT+hubLgFT (Transfer + MoE)73.12 59.21 57.49 63.70 58.16 60.8858.44 61.76 57.59 55.85 58.82 60.24 64.13 TinyBERT+hubLgFT (Contrastive + MoE) 74.68 61.98 58.89 64.91 59.41 61.7259.47 63.09 59.35 58.88 60.18 61.77 65.61 TinyBERT+hubLgFT (HILBERT: DualC + MoE)76.30 62.70 57.51 67.67 61.98 63.0261.20 65.38 59.05 59.87 62.82 61.85 67.63 TinyBERT+confLgFT (Baseline: Transfer)71.17 60.73 50.65 60.82 55.31 53.7562.78 55.23 59.67 58.99 58.50 58.17 64.99 TinyBERT+confLgFT (Transfer + MoE)72.18 61.75 52.44 62.10 56.11 55.0064.43 55.31 59.94 59.52 59.77 58.76 65.15 TinyBERT+confLgFT (Contrastive + MoE) 74.90 64.31 53.30 63.83 57.26 56.6866.11 56.26 62.05 61.14 61.84 60.81 67.10 TinyBERT+confLgFT (HILBERT: DualC + MoE) 74.85 66.54 54.91 63.46 60.37 57.4166.75 58.16 65.14 61.52 63.57 63.00 69.62 nRoBERTa+hubLgFT (Baseline: Transfer)70.31 61.50 54.31 59.33 55.93 59.8853.80 56.02 58.97 54.92 60.44 59.88 66.05 nRoBERTa+hubLgFT (Transfer + MoE)71.91 61.32 56.29 59.34 56.76 58.5555.47 57.53 59.64 55.21 60.98 62.66 67.68 nRoBERTa+hubLgFT (Contrastive + MoE)73.43 63.40 56.99 61.76 58.61 60.6056.01 59.07 60.51 58.10 62.47 63.65 69.31 nRoBERTa+hubLgFT (HILBERT: DualC + MoE)75.30 64.59 57.61 63.52 58.92 58.6058.21 61.85 59.21 58.49 64.89 67.87 70.99 TinyBERT+w2v2LgFT (Baseline: Transfer)71.80 60.26 53.54 63.11 55.61 53.3058.40 58.12 61.34 56.44 56.46 59.85 63.88 TinyBERT+w2v2LgFT (Transfer + MoE)72.99 59.76 55.03 65.54 56.06 54.6260.64 60.25 60.57 56.33 58.14 59.01 63.96 TinyBERT+w2v2LgFT (Contrastive + MoE)73.84 61.87 55.60 65.40 58.05 56.6861.18 60.69 62.69 58.37 60.07 61.79 66.39 TinyBERT+w2v2LgFT (HILBERT: DualC + MoE)74.58 63.29 58.97 66.24 58.70 57.7361.64 62.49 66.37 60.42 61.40 63.08 66.09 nMPNet+hubLgFT (Baseline: Transfer)74.47 63.98 56.39 63.20 54.30 55.1255.75 57.19 59.94 57.13 56.29 58.01 60.38 nMPNet+hubLgFT (Transfer + MoE)74.71 63.38 56.04 63.14 56.71 55.0756.77 56.74 60.78 58.32 56.78 59.78 60.36 nMPNet+hubLgFT (Contrastive + MoE)77.21 65.69 57.77 64.65 57.17 57.5757.68 58.61 62.52 59.48 59.38 61.17 61.93 nMPNet+hubLgFT (HILBERT: DualC + MoE)79.96 67.24 59.19 70.29 59.19 58.2858.80 61.70 64.31 59.33 62.18 61.98 65.92 nRoBERTa+w2v2LgFT (Baseline: Transfer)73.98 59.61 48.47 62.92 58.31 54.1951.81 57.82 64.94 57.08 55.71 59.89 63.68 nRoBERTa+w2v2LgFT (Transfer + MoE)73.97 62.21 49.31 61.92 58.87 56.5452.47 58.41 64.94 57.59 57.16 61.59 63.96 nRoBERTa+w2v2LgFT (Contrastive + MoE)76.11 63.40 50.85 64.83 59.33 56.8054.66 60.42 66.41 59.02 58.33 61.68 65.80 nRoBERTa+w2v2LgFT (HILBERT: DualC + MoE)80.34 63.72 52.53 68.25 60.88 59.0254.98 62.61 67.83 61.90 61.46 66.57 68.17 particularly on psychological traits. The strongest gains are observed in document- level tasks such as affect, coherence, and richness, where HILBERT achieves substantial margins over CLAP-based pre-training, indicating that task-aware multimodal contrastive learning remains effective despite the increased variabil- ity in offspring speech. Across architectures, TinyBERT+WhisperM, aMiniLM12, and larger text encoders (e.g., sT5-XL, nDRoBERTa) benefit substantially from HILBERT, achieving the highest AUCs on several tasks. Notably, aMiniLM12+spec (HILBERT) attains the best performance on multiple psychological outcomes, including ADHD, Depression, and Mood, while TinyBERT+WhisperM (HILBERT) achieves the strongest results on affect and coherence. On the ‘spectrum’ task, HILBERT achieves 67.33% AUC with TinyBERT and Whisper backbone on offspring data. These trends suggest that audio cues play a more prominent role in offspring data, where prosody, intonation, and pauses provide complementary informa- tion beyond lexical content. Performance improvements are broad rather than task-specific, suggesting improved generalization under dual-contrastive super- vision. Overall, these results confirm that the advantages of HILBERT extend 14H. Naderi et al. Table 3: AUC (%) on a 25 fold cross-validation for document-level and psycho- logical spectrum predictions using offspring data. Document-level tasksCognitive and Psychological tasks affectcoherencerichnessSpectrumIntrovertExtrovertADHDAnxietyDepressionMood CLAP-LAION [11]64.23 58.48 60.8152.67 59.14 55.26 54.35 54.17 61.74 60.56 TinyBERT+whisperM (Baseline: Transfer)79.99 76.09 73.2363.84 64.00 56.09 54.15 57.60 63.31 71.90 TinyBERT+whisperM (Transfer + MoE)80.82 75.98 74.0364.29 65.18 58.59 52.63 56.39 62.73 73.36 TinyBERT+whisperM (Contrastive + MoE)81.94 78.61 75.0564.87 66.61 58.70 55.13 59.31 66.13 73.81 TinyBERT+whisperM (HILBERT: DualC + MoE)83.85 79.80 76.9667.33 68.42 60.50 55.97 60.20 66.06 77.83 aMiniLM12+spec (Baseline: Transfer)70.22 59.47 61.6652.71 57.85 63.47 60.95 69.93 70.70 77.10 aMiniLM12+spec (Transfer + MoE)71.83 60.44 62.2954.02 62.20 64.70 62.56 70.90 72.55 79.17 aMiniLM12+spec (Contrastive + MoE)73.04 62.29 63.8456.83 61.80 66.38 64.26 72.05 73.92 80.77 aMiniLM12+spec (HILBERT: DualC + MoE)73.17 63.14 64.6659.74 63.17 69.75 65.13 74.16 74.58 85.31 sT5XL+whisperM (Baseline: Transfer)79.52 73.46 75.0650.12 57.42 67.17 56.38 56.24 63.71 64.32 sT5XL+whisperM (Transfer + MoE)78.09 74.68 77.2953.21 59.94 67.21 57.13 56.39 64.90 66.16 sT5XL+whisperM (Contrastive + MoE)80.60 76.06 77.4053.80 60.31 69.47 59.82 58.30 66.65 66.98 sT5XL+whisperM (HILBERT: DualC + MoE)80.81 78.21 79.9756.09 61.82 71.05 61.63 59.86 69.65 71.60 nDRoBERTa+w2v2LgFT (Baseline: Transfer)72.42 73.18 74.1148.98 62.58 67.85 51.99 53.95 67.57 62.06 nDRoBERTa+w2v2LgFT (Transfer + MoE)72.04 73.49 74.5850.15 61.53 68.06 53.14 56.38 68.21 61.31 nDRoBERTa+w2v2LgFT (Contrastive + MoE)75.36 73.91 76.5252.33 64.24 71.35 54.20 58.11 71.17 64.28 nDRoBERTa+w2v2LgFT (HILBERT: DualC + MoE)76.39 73.88 78.1657.43 64.95 75.23 58.13 59.14 73.66 68.15 nRoBERTa+w2v2LgFT (Baseline: Transfer)60.50 70.63 66.8453.24 56.51 59.35 56.13 72.36 63.32 70.63 nRoBERTa+w2v2LgFT (Transfer + MoE)60.14 72.58 67.3152.76 58.22 60.35 54.71 71.96 63.40 73.45 nRoBERTa+w2v2LgFT (Contrastive + MoE)62.24 72.88 69.3455.91 60.24 62.40 58.94 73.81 65.41 74.62 nRoBERTa+w2v2LgFT (HILBERT: DualC + MoE)63.50 74.37 70.3356.34 61.15 62.56 60.73 74.56 67.46 74.35 aMiniLM12+whisperM (Baseline: Transfer)77.68 73.93 73.8650.51 61.07 55.11 53.81 61.83 60.99 61.41 aMiniLM12+whisperM (Transfer + MoE)79.65 75.89 74.8349.56 63.06 54.92 55.18 62.69 61.47 61.00 aMiniLM12+whisperM (Contrastive + MoE)79.79 76.68 76.8752.36 62.54 56.67 56.97 64.51 63.09 62.93 aMiniLM12+whisperM (HILBERT: DualC + MoE) 81.97 79.51 76.0354.60 65.89 56.85 60.89 66.80 63.92 64.61 robustly to offspring data, even under noisier acoustic conditions and greater inter-speaker variability. Ablation Studies of the Auxiliary Loss Functions: Our final objective incorporates three auxiliary losses: contrastive, CKA, and mutual information (MI). We conduct a full brute force ablation over all non-empty combinations of these losses ( P N r=1 N r = 2 N − 1 where N = 3) to assess the contribution of each component. The study is performed using the paraTinyBERT and hubert- LargeFineTune configuration, which yields the strongest average performance on parent data. Table 4 reports, for each auxiliary loss, the best validation AUC achieved across combinations that include the loss versus those that exclude it. All three losses provide consistent gains. The contrastive loss yields the largest improvement in classification performance. The CKA loss substantially enhances multimodal representation alignment. The MI loss further stabilizes training by balancing modality contributions. Based on these observations, we apply distinct weighting coefficients (λ) to appropriately emphasize each auxiliary loss in the final objective. Joint-Centric Dual Contrastive Alignment with Structure-Preserving15 Table 4: Across 7 combinations, each row shows the best AUC (%) of all com- binations that included the auxiliary loss✓vs. those that did not✗. Validation AUC (%) is the average of 5 folds classification on parent data. Validation Auxiliary Loss✗✓ Contrastive 65.8 66.1 CKA65.7 65.9 MI65.7 65.8 5 Conclusion In this work, we introduced HILBERT, a novel framework that effectively ad- dresses the challenges of multimodal audio-text representation learning. Our approach successfully tackles the fundamental problem of aligning cross-modal information while preserving modality-specific characteristics, particularly fo- cusing on the dimensional imbalance between audio and text representations. The experimental results demonstrate the effectiveness of HILBERT across multiple document-level and psychological spectrum tasks. We achieved an AUC score of 66.75% and 67.33% on the ‘spectrum’ task using parent and offspring data, respectively. The ‘spectrum’ task which is the mental disorder prediction is a highly imbalanced multi-class problem and is the most challenging task overall. Interestingly, our findings reveal that the audio component has a more significant impact on offspring data compared to parent data, likely because audio features such as emotions, intonations, and pauses provide additional information beyond spoken words in capturing coherence and mental state in offspring participants. Several key observations emerge from our experiments. First, the choice of audio and text models significantly influences performance across different tasks, with hubert and whisper showing particular strength in parent and offspring data respectively. Second, smaller language models often outperformed larger models, suggesting that more compact models may be better suited for limited sample sizes in mental health applications. HILBERT’s dual contrastive learning approach, coupled with specialized loss functions and an MoE architecture, offers several advantages over tradi- tional methods. By simultaneously enforcing both inter-modal alignment and intra-modal consistency, our framework creates more balanced and semantically meaningful joint representations. The incorporation of CKA and MI losses en- sures that both modalities contribute equitably to the final representation while preserving information-theoretic relevance. Additionally, our multimodal joint encoder effectively captures complex dependencies between modalities through cross-modal self-attention mechanisms, resulting in high-quality document-level representations. In conclusion, HILBERT advances the state-of-the-art in multimodal rep- resentation learning for long sequences, particularly in the context of mental health prediction tasks. Our framework’s ability to effectively balance modality 16H. Naderi et al. contributions while preserving both shared and modality-specific features makes it well-suited for a wide range of downstream applications requiring nuanced understanding of multimodal data. References 1. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: International conference on machine learning. p. 1597–1607. PMLR (2020) 2. Elizalde, B., Deshmukh, S., Al Ismail, M., Wang, H.: Clap learning audio concepts from natural language supervision. In: ICASSP. p. 1–5. IEEE (2023) 3. Gretton, A., Bousquet, O., Smola, A., Schölkopf, B.: Measuring statistical de- pendence with hilbert-schmidt norms. In: International conference on algorithmic learning theory. p. 63–77. Springer (2005) 4. Huang, W., Han, A., Chen, Y., Cao, Y., Xu, Z., Suzuki, T.: On the comparison between multi-modal and single-modal contrastive learning. Advances in Neural Information Processing Systems 37, 81549–81605 (2024) 5. Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network rep- resentations revisited. In: International conference on machine learning. p. 3519– 3529. PMLR (2019) 6. Poklukar, P., Vasco, M., Yin, H., Melo, F.S., Paiva, A., Kragic, D.: Geometric multimodal contrastive representation learning. In: International Conference on Machine Learning. p. 17782–17800. PMLR (2022) 7. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PMLR (2021) 8. Shi, Y., Paige, B., Torr, P., et al.: Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in neural information processing systems 32 (2019) 9. Shi, Y., Paige, B., Torr, P.H., Siddharth, N.: Relating by contrasting: A data-efficient framework for multimodal generative models. arXiv preprint arXiv:2007.01179 (2020) 10. Uher, R., Cumby, J., MacKenzie, L.E., Morash-Conway, J., Glover, J.M., Aylott, A., Propper, L., Abidi, S., Bagnell, A., Pavlova, B., et al.: A familial risk enriched cohort as a platform for testing early interventions to prevent severe mental illness. BMC psychiatry 14(1), 344 (2014) 11. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., Dubnov, S.: Large- scale contrastive language-audio pretraining with feature fusion and keyword-to- caption augmentation. In: ICASSP. p. 1–5. IEEE (2023) 12. Zhu, G., Darefsky, J., Duan, Z.: Cacophony: An improved contrastive audio-text model. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024) 13. Zolfaghari, M., Zhu, Y., Gehler, P., Brox, T.: Crossclr: Cross-modal contrastive learning for multi-modal video representations. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 1450–1459 (2021) 14. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., Fedus, W.: St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906 (2022)