Paper deep dive
Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning
Donghuo Zeng, Hao Niu, Masato Taya
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 2:34:59 AM
Summary
HSC-MAE is a dual-path teacher-student framework for unsupervised audio-visual representation learning. It enforces hierarchical semantic consistency across three levels: global canonical-geometry correlation (via DCCA), local neighborhood-semantics correlation (via teacher-mined soft top-k affinities), and sample-level conditional-sufficiency correlation (via masked autoencoding). The framework uses an EMA teacher to provide stable targets and learnable multi-task weights to reconcile objectives, achieving significant mAP improvements on AVE and VEGAS benchmarks.
Entities (6)
Relation Signals (4)
HSC-MAE → evaluatedon → AVE
confidence 100% · Experiments on AVE and VEGAS demonstrate substantial mAP improvements
HSC-MAE → evaluatedon → VEGAS
confidence 100% · Experiments on AVE and VEGAS demonstrate substantial mAP improvements
HSC-MAE → uses → DCCA
confidence 95% · global-level canonical-geometry correlation via DCCA
HSC-MAE → uses → InfoNCE
confidence 95% · a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hierarchical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation - from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embeddings within a shared modality-invariant subspace; (ii) local-level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational structure among semantically similar instances; and (iii) sample-level conditional-sufficiency correlation via masked autoencoding, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canonical geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio-visual representations.
Tags
Links
- Source: https://arxiv.org/abs/2604.04229v1
- Canonical: https://arxiv.org/abs/2604.04229v1
Trouble viewing inline? Open PDF directly →
Full Text
31,469 characters extracted from source content.
Expand or collapse full text
Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio–Visual Representation Learning Donghuo Zeng KDDI Research, Inc., Saitama, Japan do-zeng@kddi-research.jp Hao Niu KDDI Research, Inc.,Saitama, Japan ha-niu@kddi.com Masato Taya KDDI Research, Inc., Saitama, Japan ma-taya@kddi.com Abstract—Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hier- archical Semantic Correlation-aware Masked AutoEncoder), a dual-path teacher–student framework that enforces semantic consistency across three complementary levels of representa- tion — from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embed- dings within a shared modality-invariant subspace; (i) local- level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational struc- ture among semantically similar instances; and (i) sample- level conditional-sufficiency correlation via masked autoencod- ing, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canoni- cal geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio–visual representations. IndexTerms—MaskedAutoencoder,Unsupervised audio–Visual Learning, DCCA, Contrastive Learning I. INTRODUCTION Learning aligned representations across heterogeneous modalities is a fundamental challenge in multimedia and em- bodied perception. Audio–visual representation learning seeks to discover shared semantic structure between sound and sight so that systems can ground concepts, reason about events, and act robustly in real-world scenes. Humans naturally fuse auditory and visual cues (e.g., a flash followed by thunder, the cadence of footsteps implying an approaching person) to form context-rich, resilient concepts; reproducing this ability in machines, without relying on costly manual labels, would enable perception and interaction in the wild. In this work, we focus on unsupervised approaches that learn cross-modal grounding from paired but unlabeled data, explicitly address- ing the practical issues of noise, weak alignment, and limited access to raw sensor modalities. Unsupervised audio–visual representation learning from paired but unlabeled clips is appealing but practically difficult. Real-world corpora are noisy and weakly aligned: clips often contain multiple concurrent events, temporal misalignment is common, and spurious co-occurrences violate the hard- positive assumption in standard contrastive learning. Two practical constraints further complicate the problem. First, many modern pipelines expose only compact, pre-extracted feature vectors (rather than raw pixels or spectrogram patches); methods must therefore build strong intra-modal semantics from these condensed descriptors. Second, cross-modal posi- tives are inherently ambiguous, since many clips admit mul- tiple valid positive matches across modalities. Consequently, the widely adopted one-positive-per-anchor assumption in contrastive learning misrepresents the local geometry of the embedding space and risks introducing confirmation bias [1]. To address these issues, we introduce HSC-MAE (Hierar- chical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher–student framework that explicitly enforces hierarchical semantic correlations (HSC) across three com- plementary levels of representation — from coarse to fine: (1) Global-level canonical-geometry correlation — audio and visual embeddings should share a common low-dimensional subspace that captures modality-invariant semantic factors. DCCA imposes this coarse distributional alignment, estab- lishing a well-structured embedding manifold on which finer- grained constraints can build. (2) Local-level neighborhood- semantics correlation — local neighborhoods should preserve multi-positive relational structure among semantically simi- lar instances. Building on the global manifold, a soft top- k InfoNCE with teacher-mined affinities shapes these neigh- borhoods, relaxing the brittle single-positive assumption that misrepresents semantic relatedness in multi-event clips. (3) Sample-level conditional-sufficiency correlation — individual embeddings must retain discriminative semantic content suf- ficient to predict missing feature dimensions. Sample-level masked autoencoding enforces this instance-level robustness, which is critical when working with pre-extracted descriptors where feature dimensions may be noisy or missing. Together, these hierarchical constraints reduce a principled upper bound on retrieval risk. HSC-MAE realizes them with two coordinated training modes on a shared encoder: a student MAE path (value-masked inputs) that enforces sample-level conditional sufficiency via reconstruction and local neighbor- arXiv:2604.04229v1 [cs.M] 5 Apr 2026 hood consistency via soft top-k InfoNCE, and a CCA path (clean inputs with gradient masking) that serves as Exponential Moving Average (EMA) [2] teacher enforcing global canonical geometry via DCCA. The teacher produces stable canonical embeddings and affinity weights that mine soft positives and provide geometric targets for optional distillation; the stu- dent learns corruption-robust local semantics while inheriting global structure from the teacher. An EMA teacher stabilizes neighborhood mining and prevents confirmation bias, while learnable multi-task weights reconcile competing objectives across the hierarchical levels during optimization. We evaluate HSC-MAE on AVE and VEGAS benchmarks using mean Average Precision (mAP). Results show sub- stantial and consistent gains over strong unsupervised base- lines. Ablations confirm that global correlation, local neigh- borhood discrimination, and sample-level reconstruction each contribute complementary and non-redundant improvements to retrieval geometry. Contributions can be summarized as: (i) We introduce HSC-MAE, a dual-path masked autoencoder that enforces hierarchical semantic correlations — from global canonical geometry, through local neighborhood semantics, to sample-level conditional sufficiency — for unsupervised au- dio–visual representation learning from compact pre-extracted features. (i) We introduce a teacher-guided soft top-k neigh- borhood mining strategy, where an EMA teacher produces stable affinity weights to define multi-positive relationships for contrastive learning. This formulation mitigates brittle one- positive assumptions and reduces confirmation bias in weakly aligned, multi-event settings. (i) We use a principled multi- task weighting scheme and optional distillation to reconcile reconstruction, correlation, and contrastive objectives, which stabilizes optimization across the hierarchical levels. (vi) We demonstrate significant mAP improvements on AVE and VEGAS and provide thorough ablations validating how the hierarchical constraints jointly and complementarily improve audio–visual cross-modal retrieval. I. RELATED WORK A. Audio–Visual Learning Classical methods align modalities with linear or shallow projections such as Canonical Correlation Analysis (CCA) [3], while nonlinear extensions (DCCA) learn powerful shared subspaces via deep encoders [4]. More recent unsupervised methods leverage contrastive objectives adapted to paired audio–visual data to encourage cross-modal alignment [5]–[8], and metric-learning variants such as triplet losses have also been applied to multimedia retrieval [9]. These approaches, however, often assume strict positive pairings and can struggle on weakly aligned with multi-event clips. HSC-MAE builds on DCCA’s global alignment strengths while mitigating brittle pair assumptions via soft multi-positive contrast. B. Masked Autoencoders Masked autoencoders (MAE) have shown strong self- supervision by reconstructing masked patches in images or spectrograms [10], [11]. When only pre-extracted feature vectors are available, patch masking is not applicable; instead, feature-level denoising and cross-modal prediction have been proposed (e.g., denoising autoencoders and feature recon- struction) [12]. HSC-MAE adopts sample-level masking and feature-dimension reconstruction to learn robust intra-modal representations from compact features, and couples these reconstructions with cross-modal decoders so reconstruction benefits retrieval. C. Metric Learning Metric learning losses (contrastive, triplet) provide many ways to shape embedding geometry [9], [13]. In unsupervised settings, mining reliable positives/negatives is critical: com- mon strategies include top-K neighbors, mutual nearest neigh- bors, clustering, and graph diffusion over kNN graphs [1]. Recent work advocates soft positives or affinity-weighted losses to reflect uncertainty in mined labels [14]. Our soft top- k InfoNCE constructs weighted multi-positive targets from a teacher’s affinity estimates, reducing noise from ambiguous matches common in multi-event clips. D. Knowledge distillation The exponential moving average (EMA) teacher models and bootstrapping methods (BYOL, DINO) have proven ef- fective to avoid collapse and to provide stable targets in self-supervision [2], [15]. In cross-modal learning, teacher- student schemes also stabilize mining and reduce confirmation bias by supplying smoother labels [8]. HSC-MAE leverages an EMA teacher both to produce stable correlation targets for soft neighborhood mining and to distill consistency into the student; combined with learnable multi-task weighting, this reduces optimization conflicts between reconstruction and alignment. I. METHOD A. Problem formulation and notation Let D = (x a,i ,x v,i ) N i=1 be an unlabeled collection of paired audio and visual feature vectors, x a,i ∈ R d a and x v,i ∈ R d v . Our goal is to learn modality encoders f a (·;θ) and f v (·;θ) that map both modalities into a shared embedding space R d . Retrieval is performed by a similarity s(·,·) (e.g., cosine) on L2-normalized embeddings. HSC-MAE imposes semantic consistency across multiple levels of representation. B. Dual-path forward pass HSC-MAE executes two coordinated forward modes on the same encoder parameters: (1) CCA-path (teacher/global geometry): Inputs are passed in a clean mode (values pre- served); gradients from selected input dimensions may be masked to protect canonical geometry. Clean-view embeddings Z cca a ,Z cca v are optimized by a DCCA objective that encourages a shared low-dimensional subspace across modalities. An EMA (momentum) copy of the encoder is maintained as a teacher and evaluated on this CCA path; the teacher’s clean embeddings provide stable affinity estimates and geometric targets for mining and distillation. Gradients do not flow into the teacher. (2) MAE-path (student/robust semantics): Inputs Fig. 1: Overview of the HSC-MAE architecture. Pre- extracted audio and visual features are processed by shared encoders and a cross-attention fusion block under two coordi- nated training modes. The student MAE path applies sample- level value masking and is optimized by reconstruction, a teacher-guided soft top-k InfoNCE, producing robust embed- dings (Z a ,Z v ). In parallel, the EMA teacher CCA path pre- serves input values and enforces global cross-modal alignment via DCCA, yielding clean embeddings (Z ′ a ,Z ′ v ). The teacher provides stable semantic affinities and geometric targets for neighborhood mining and distillation, while gradients are blocked from flowing into the teacher. are value-masked at the sample level (a fraction of feature dimensions zeroed) and passed through the same encoder and a decoder to reconstruct missing components. Masked-view embeddings Z mae a ,Z mae v are trained with reconstruction, a teacher-guided soft multi-positive InfoNCE, and an optional consistency loss that aligns student embeddings to teacher geometry. This path encourages embeddings that are locally coherent and robust to partial observations while inheriting global structure from the teacher. C. HSC-MAE architecture The architecture (Fig. 1) enforces hierarchical semantic correlations across three complementary levels — global subspace, local neighborhood, and individual sample — via four objectives detailed below. Consistent with the hierarchy introduced in Sec. I, we present them from coarse to fine. 1) Global shared-subspace alignment (canonical-geometry correlation): Denote the clean-view audio-visual embeddings by Z cca a ,Z cca v . DCCA finds canonical directions whose corre- lations ρ m r m=1 are maximized. We minimize the negative sum of top-r canonical correlations: L cca = − r X m=1 ρ m Z cca a ,Z cca v . Maximizing these correlations encourages audio and visual representations to occupy a common low-dimensional sub- space of modality-invariant factors. It establishes the global ge- ometric structure on which local and sample-level constraints are built. 2) Localneighborhoodpreservation(neighborhood- semantics correlation): Let Z mae i be the masked-view embedding of sample i and let W ij ≥ 0 be nonnegative affinity weights produced by the teacher over the mined top-k neighborhood, normalized so that P j W ij = 1. Using cosine similarity s(·,·) and temperature τ , we define: L infoNCE =−E i h X j W ij log exp s(Z mae i ,Z mae j )/τ P ℓ exp s(Z mae i ,Z mae ℓ )/τ i . This affinity-weighted multi-positive objective shapes local neighborhood structure within the globally aligned manifold, relaxing the brittle single-positive assumption by tolerating multiple plausible positives per anchor — a necessary property for weakly aligned, multi-event clips. 3) Sample-level MAE reconstruction (conditional suffi- ciency correlation): Given an input x ∼ D, a masked view ̃x ∼ q( ̃x | x) is generated via a stochastic masking process, and reconstructed as ˆx=D(E( ̃x)). The reconstruction objective minimizes the mean squared error: L rec = E x∼D, ̃x∼q( ̃x|x) ∥x− ˆx∥ 2 2 . Minimizing L rec requires the masked-view embedding Z mae =E( ̃x) to retain sufficient statistics of x given the cor- rupted observation ̃x, thereby enforcing conditional sufficiency at the sample level and ensuring robustness to noisy or incom- plete descriptors common in pre-extracted feature pipelines. a) Optional consistency (distillation): The EMA teacher evaluated on the clean path produces stable teacher embed- dings Z t . We optionally align student embeddings to these targets: L dis = E i ∥Z mae i − Z t i ∥ 2 2 . This distillation transfers teacher geometry into the masked student and reduces confirmation bias during mining. b) Overall objective and optimization: The individual terms are combined with learnable log-variance weights σ = (σ rec ,σ infoNCE ,σ cca ,σ dis ) following an uncertainty- based scheme [16]: L total (θ,σ) = X m exp(−σ m )L m (θ) + σ m , where the sum runs over the active losses m ∈ rec, infoNCE, cca, dis. Both encoder parameters θ and the log-variance parametersσ are updated by gradient descent. The EMA teacher parameters θ t are updated with a momentum rule θ t ← ρθ t + (1 − ρ)θ after each optimization step; gradients are not propagated into θ t . Motivated by DCCA [4], we append a linear CCA projection layer at the end of the encoder network, applied during both training and inference. This layer continuously reduces embedding dimensionality and emphasizes maximally correlated canonical directions, yielding a more discriminative and geometrically structured retrieval space throughout optimization. IV. EXPERIMENTS A. Datasets and Evaluation We evaluate HSC-MAE on two standard audio–visual benchmarks [7]: AVE [7], consisting of 1,955 clips across 15 categories (1,564 for training and 391 for testing), and VEGAS [7], containing 28,103 YouTube clips (2–10s) from 10 categories (22,482 for training and 5,621 for testing). Audio features are extracted as 128-D embeddings using VGGish [17], while visual features are 1,024-D descriptors derived by average-pooling InceptionV3 [18] frame-level fea- tures. Unsupervised cross-modal retrieval (UCMR) is evalu- ated in both audio-to-visual (A2V) and visual-to-audio (V2A) directions using cosine similarity between embeddings. The final score is the mean of the two Mean Average Precision (MAP) values, following prior works [7], [19]. Evaluation is class-based, using dataset categories only for testing, not for training. B. Implementation Details HSC-MAE employs modality-specific three-layer MLP en- coders. The audio encoder ([128,1024,1024,1024]) and vi- sual encoder ([1024,1024,1024,1024]) use BatchNorm in the first layer, LayerNorm thereafter, Tanh as activations, and Dropout (0.2). Cross-modal fusion is achieved via multi- head attention (64 heads). Linear projectors map audio and visual embeddings to a 32-D retrieval space. Training uses a dual-path masking scheme: (i) GradMask in the CCA path to enable selective gradient flow, and (i) value masking in the MAE path for reconstruction. Mask ratios are 0.3 for VEGAS and 0.2 for AVE. Soft top-k mining (k=5) selects pseudo-positive pairs via model predictions, selections are weighted via a temperature-scaled softmax (τ =0.05) and used in symmetric InfoNCE losses. Loss terms are balanced with uncertainty-based multi-task weighting, preceded by a 5-epoch warmup with fixed weights ([L rec ,L cca ,L dis ,L infoNCE ] = [1, epoch×0.1, 0.1, 0.05]) to ensure stable initialization be- fore learnable balancing is introduced. This warmup prevents cold-start sensitivity and yields consistent weight convergence across runs without manual tuning. We optimize with AdamW (LR=3×10 −4 , weight decay = 1×10 −4 ), gradient clipping = 1.0, a cosine-annealing schedule with T max = 50. A teacher network is maintained via EMA of student weights, with momentum ρ annealed from 0.95 to 0.999. After training, a linear CCA (output dim=10) is fitted on the training embeddings for final retrieval alignment. Batch sizes are 512 (VEGAS) and 400 (AVE); models are trained for 100 epochs. All models were trained on an NVIDIA TABLE I: mAP comparison across state-of-the-art meth- ods. The best results are shown in bold, and the second-best results are underlined. A2V2AAvg.A2V2AAvg. Random0.1270.1240.1260.1100.1090.109 CCA0.1900.1890.1900.3320.3270.330 KCCA0.30620.18180.24400.2880.2730.281 DCCA0.33820.34380.34100.75410.75740.7558 InfoNCE0.46460.46430.46440.73590.70970.7228 Contrastive0.39430.39120.39270.50890.51010.5095 Triplet0.45190.45240.45210.43060.43100.4308 DUMCH0.46110.47190.46650.56000.54390.5519 UCCH0.45350.44980.45170.59860.60040.5995 DECH0.33680.33460.33570.62800.62570.6269 CAV-MAE0.61230.62070.61650.74530.76170.7535 Ours0.77470.77280.77370.80130.80390.8026 Methods AVE datasetVEGAS dataset RTX 3080 GPU (10 GB). Code available at https://github.com/ ZenzenDatabase/UnsupervisedMAE C. Baselines We evaluate HSC-MAE against a diverse set of classical and recent unsupervised audio–visual retrieval baselines. All meth- ods use the same pre-extracted features and, where applicable, are reimplemented with comparable encoder capacity and training budgets to ensure fair comparison. Specifically, we include a Random baseline as a lower-bound reference. Clas- sical correlation-based methods include linear CCA [3], kernel CCA (KCCA) [20], and deep CCA (DCCA) [4] implemented with shallow MLP encoders. We further compare against contrastive learning objectives, including InfoNCE with single positives [21] using same-sample audio–visual pairs, a CLIP- style symmetric cross-modal contrastive loss [22], and triplet- based metric learning [9]. Finally, we benchmark against recent unsupervised cross-modal hashing and masked autoen- coding approaches, including DECH [23], DUMCH [24], UCCH [25], and CAV-MAE [26]. Whenever feasible, baselines are retrained using the identical training settings. We report results of CCA and KCCA from the original publications. D. Main results Table I reports retrieval performance (mAP) of HSC-MAE and all baselines on AVE and VEGAS. HSC-MAE consistently outperforms classical correlation-based methods, contrastive objectives, and recent MAE-based approaches across both retrieval directions (A2V and V2A) as well as averaged mAP. Quantitatively, HSC-MAE substantially improves upon the strongest prior method, CAV-MAE, on AVE. Specifically, A2V mAP increases from 0.6123 to 0.7747 (+26.24%), V2A from 0.6207 to 0.7728 (+15.21%), and the averaged mAP from 0.6165 to 0.7737 (+15.72%). On VEGAS, the improvements are smaller but consistent: A2V improves from 0.7453 to 0.8013 (+15.60%), V2A from 0.7617 to 0.8039 (+4.22%), and the averaged mAP from 0.7535 to 0.8026 (+4.91%). Com- pared with standard contrastive learning baselines, including Contrastive, InfoNCE, and Triplet losses, HSC-MAE achieves TABLE I: Component-wise impact on the final objective loss for different methods on the AVE and VEGAS datasets. A2V2AAvg.A2V2AAvg. ✓0.77470.77280.77370.80840.82280.8156 −✓0.62290.62390.62340.77500.78820.7816 ✓−0.74860.74270.74570.80600.81020.8081 ✓−✓0.67350.67460.67410.78050.77490.7777 ✓−0.57280.57010.57150.79030.78940.7899 ✓−✓0.61430.61450.61440.79400.78830.7911 ✓−0.56880.56980.56930.78910.78560.7873 CCA Rec. Soft In- foNCE Self- distillation AVE datasetVEGAS dataset markedly higher performance on both datasets, demonstrating that hierarchical semantic modeling and the proposed dual- path masking strategy provide more effective cross-modal alignment than vanilla contrastive objectives. E. Ablation studies We conduct ablation studies to quantify the impact of individual loss components and mask ratio in HSC-MAE, validating the design choices described in Sec. I. 1) Impact of loss components: We evaluate the contribu- tion of each major component in HSC-MAE by selectively removing or modifying individual components. The ablated variants include removing self-distillation (w/o EMA), dis- abling the CCA objective for global alignment, replacing soft top-k mining with strict single-positive InfoNCE, and etc. The quantitative impact of these ablations on AVE and VEGAS is summarized in Table I. Removing any component leads to a consistent performance drop, with the largest degrada- tion observed when masked reconstruction or soft InfoNCE is removed, confirming their central role in learning robust cross-modal representations. Disabling CCA further degrades performance, highlighting the importance of preserving global cross-modal correlation. Figure 2 visualizes the training loss decomposition and cor- responding test mAP under different ablations. Removing self- distillation or CCA leads to slower convergence and higher final losses, accompanied by degraded retrieval accuracy, whereas omitting InfoNCE yields fast, stable convergence with low loss but inferior mAP, indicating weak representation geometry and limited generative capacity due to missing local geometric constraints. In contrast, the full model exhibits smoother optimization and consistently higher mAP, validating the effectiveness of HSC-MAE. 2) Impact of mask ratio:We analyze the sensitiv- ity of HSC-MAE to the masking ratio, evaluated over 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7 on both AVE and VE- GAS. Figure 3 illustrates the effect of masking on retrieval performance (mAP). For both datasets, performance improves rapidly as the mask ratio increases from 0.0, peaks at moderate values (0.1 and 0.3), and degrades when masking becomes too aggressive. This behavior highlights the trade-off between learning robust representations through sample-level masked Fig. 2: (Left) Decomposition of training losses; (Right) test mAP over epochs (1-100) on AVE under component-wise ablations. Fig. 3: Effect of mask ratio on UCMR task on both AVE and VEGAS datasets. Solid curves show the average mAP, while shaded regions indicate the absolute gap between the two retrieval directions. reconstruction and preserving sufficient semantic information for cross-modal alignment. We additionally observe dataset-dependent effects of the mask ratio in the shaded regions of Figure 3: on the larger VEGAS, moderate masking (0.2, 0.3) yields the highest mAP but a larger A2V–V2A gap, indicating residual directional bias, whereas on the smaller AVE dataset the gap is consis- tently smaller, reflecting more symmetric embeddings; in both cases, low masking under-regularizes the model and excessive masking disrupts cross-modal semantics. Fig. 4: Qualitative audio–visual cross-modal retrieval results on AVE. For each query (audio or visual), the top-10 retrieved results are shown. F. Qualitative Results Figure 4 shows A2V and V2A retrieval examples on AVE to illustrate semantic alignment. Using a truck audio as the A2V query, the model returns 7 correct truck visuals in the top-10 (non-matches: bus at ranks 2, 8, 9). Using the paired truck visual for V2A, retrieval again yields 7 correct audio clips with bus confusions at ranks 6, 8, 10. These results indicate HSC-MAE reliably captures cross-modal semantics; remaining errors are primarily between acoustically and vi- sually similar classes (e.g., bus vs. truck), reflecting inherent category ambiguity rather than misalignment. V. CONCLUSION We introduced HSC-MAE that combines sample-level masked reconstruction, DCCA-driven global alignment, and teacher-guided soft top-k contrast within a dual-path teacher–student design. By decoupling reconstruction and correlation objectives—and coupling them through cross- attention, an EMA teacher, and learnable multi-task weights, HSC-MAE enforces semantic consistency at multiple struc- tural levels: canonical geometry (shared modality subspace), neighborhood semantics (multi-positive local structure), and conditional sufficiency (robust intra-modal features). Exper- iment shows substantial mAP improvements on AVE and VEGAS compared to unsupervised baselines. Ablations con- firm that reconstruction, soft-neighborhood discrimination, and global correlation each provide complementary benefits, and their combination yields the best retrieval geometry. Beyond retrieval, HSC-MAE offers a practical approach for label-scarce multimodal grounding in embodied agents and related systems. Future work will explore additional modal- ities, finer temporal modeling, and scalability improvements for large-scale or online settings. REFERENCES [1] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bo- janowski, and Armand Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” in NeurIPS, 2020. [2] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ́ e J ́ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, “Emerging properties in self-supervised vision transformers,” in ICCV, 2021. [3] Hardoon David, R., Szedm ́ ak S ́ andor, and Shawe-Taylor John, “Canon- ical correlation analysis: An overview with application to learning methods,” Neural Computation., vol. 16, no. 12, p. 2639–2664, 2004. [4] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu, “Deep canonical correlation analysis,” in ICML, 2013, p. 1247–1255. [5] Donghuo Zeng, Yi Yu, and Keizo Oyama, “Deep triplet neural networks with cluster-cca for audio-visual cross-modal retrieval,” ACM TOMM, vol. 16, no. 3, p. 1–23, 2020. [6] Donghuo Zeng, Yi Yu, and Keizo Oyama, “Audio-visual embedding for cross-modal music video retrieval through supervised deep cca,” in IEEE ISM, 2018, p. 143–150. [7] Donghuo Zeng, Jianming Wu, Gen Hattori, Rong Xu, and Yi Yu, “Learning explicit and implicit dual common subspaces for audio-visual cross-modal retrieval,” ACM TOMM, vol. 19, no. 2s, p. 1–23, 2023. [8] Donghuo Zeng and Kazushi Ikeda, “Metric learning with progressive self-distillation for audio-visual embedding learning,” in ICASSP, 2025, p. 1–5. [9] Florian Schroff, Dmitry Kalenichenko, and James Philbin, “Facenet: A unified embedding for face recognition and clustering,” in CVPR, 2015, p. 815–823. [10] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll ́ ar, and Ross Girshick, “Masked autoencoders are scalable vision learners,” in CVPR, 2022. [11] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer, “Masked autoencoders that listen,” NeurIPS, vol. 35, p. 28708–28720, 2022. [12] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol, “Extracting and composing robust features with denoising autoencoders,” ICML, 2008. [13] Raia Hadsell, Sumit Chopra, and Yann LeCun, “Dimensionality reduc- tion by learning an invariant mapping,” in CVPR. IEEE, 2006, vol. 2, p. 1735–1742. [14] David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi, “Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives,” in AAAI, 2022, vol. 36, p. 897–905. [15] Jean-Bastien Grill, Florian Strub, Florent Altch ́ e, Corentin Tallec, et al., “Bootstrap your own latent: A new approach to self-supervised learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. [16] Alex Kendall, Yarin Gal, and Roberto Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in CVPR, 2018, p. 7482–7491. [17] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, et al., “CNN architectures for large-scale audio classification,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, p. 131–135. [18] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, et al., “Youtube-8m: A large-scale video classification benchmark,” arXiv:1609.08675, 2016. [19] Donghuo Zeng, Yanan Wang, Jianming Wu, and Kazushi Ikeda, “Com- plete cross-triplet loss in label space for audio-visual cross-modal retrieval,” in ISM, 2022, p. 1–9. [20] Pei Ling Lai and Colin Fyfe, “Kernel and nonlinear canonical correlation analysis,” International Journal of Neural Systems, vol. 10, no. 5, p. 365–377, 2000. [21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals, “Representation learning with contrastive predictive coding,” in arXiv:1807.03748, 2018. [22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al., “Learning transferable visual models from natural language supervi- sion,” in ICML, 2021. [23] Xin Yang, Li Zhang, and Peng Li,“Deep evidential cross-modal hashing,” IEEE TMM, 2023. [24] Jingjing Wang, Zhen Zhang, and Xuelong Li,“Deep unsupervised momentum contrastive hashing for cross-modal retrieval,” in ACM M, 2022. [25] Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, and Xi Peng, “Unsupervised contrastive cross-modal hashing,” IEEE TPAMI, vol. 45, no. 3, p. 3877–3889, 2022. [26] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass, “Contrastive audio- visual masked autoencoder,” arXiv:2210.07839, 2022.