Paper deep dive
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
Francisco Mena, Dino Ienco, Roberto Interdonato, Cassio F. Dantas, Simon Besnard
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-modal classification leverages complementary information across diverse data sources to enhance predictive performance. However, real-world scenarios subject to operational constraints, such as sensor failures or privacy restrictions, lead to inconsistent modality availability between training and inference times. To handle missing modalities, prior studies have mainly covered bimodal data setups and focused on designing robust fusion processes. Instead, we adopt a multi-modal co-learning framework that prioritizes inter-modal collaboration rather than multi-modal fusion. Specifically, we consider that any subset of modalities may be absent, without assuming predefined missing-modality patterns, an inference scenario we refer to as missing arbitrary modalities. To address this challenge, we introduce two alternative approaches that leverage information at both feature- and decision-level. Experiments on two multi-modal classification benchmarks demonstrate significant robustness gains in various missing modality conditions. The first method shows more robust behavior under minimal missing conditions, where a single modality is absent, whereas the second performs better under extreme missing conditions, where all-but-one modalities are missing. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.24683v1
- Canonical: https://arxiv.org/abs/2607.24683v1
Trouble viewing inline? Open PDF directly →
Full Text
41,433 characters extracted from source content.
Expand or collapse full text
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification Francisco Mena 1[0000−0002−5004−6571] (), Dino Ienco 2,3[0000−0002−8736−3132] , Roberto Interdonato 4,3[0000−0002−0536−6277] , Cassio F. Dantas 2,3[0000−0002−1934−0625] , and Simon Besnard 1[0000−0002−1137−103X] 1 GFZ Helmholtz Center for Geosciences, Potsdam, Germany mena@gfz.de 2 INRAE, UMR TETIS, University of Montpellier, Montpellier, France 3 INRIA, EVERGREEN, University of Montpellier, Montpellier, France 4 CIRAD, UMR TETIS, University of Montpellier, Montpellier, France Abstract. Multi-modal classification leverages complementary informa- tion across diverse data sources to enhance predictive performance. How- ever, real-world scenarios subject to operational constraints, such as sen- sor failures or privacy restrictions, lead to inconsistent modality availabil- ity between training and inference times. To handle missing modalities, prior studies have mainly covered bimodal data setups and focused on designing robust fusion processes. Instead, we adopt a multi-modal co- learning framework that prioritizes inter-modal collaboration rather than multi-modal fusion. Specifically, we consider that any subset of modali- ties may be absent, without assuming predefined missing-modality pat- terns, an inference scenario we refer to as missing arbitrary modalities. To address this challenge, we introduce two alternative approaches that leverage information at both feature- and decision-level. Experiments on two multi-modal classification benchmarks demonstrate significant ro- bustness gains in various missing modality conditions. The first method shows more robust behavior under minimal missing conditions, where a single modality is absent, whereas the second performs better under ex- treme missing conditions, where all-but-one modalities are missing. Our code is available at https://github.com/fmenat/Co4Miss. Keywords: Multi-modal classification· Co-learning· Missing modali- ties· Robustness. 1 Introduction Multi-modal classification aims to fuse complementary data sources to enhance predictive performance and support reliable decision-making [22]. Although re- lated research primarily focuses on standard multi-modal setups such as im- age–text–audio [22], real-world applications often involve heterogeneous modal- ities beyond this setup. Nowadays, modern sensing technologies collect a vast amount of data modalities with varying characteristics, e.g. acquisition processes, resolutions, and physical meaning. In the Earth Observation (EO) domain, satel- lite imagery, radar, LiDAR, and in-situ measurements provide structurally dis- tinct yet complementary views of the Earth’s surface. In human-centered studies, arXiv:2607.24683v1 [cs.CV] 27 Jul 2026 2Mena et al. wearable and biometric sensors produce diverse physiological and motion signals of a subject. Whereas such diversity enriches representation, it also poses signif- icant challenges if training modalities are absent at inference time. In real-world applications, data modalities cannot be assumed to be consis- tently accessible across the same phenomenon and time span. This is because data acquisition can be constrained by environmental, operational, and cost fac- tors. As a result, modalities used at training time may be partially observed or entirely missing at inference [21]. This challenge is evident in EO, where optical imagery may be affected by cloud coverage, satellite missions can be discontin- ued, or may operate only over restricted geographic areas, leading to abrupt gaps in multimodal data sources [18]. Similar issues arise in human-centered systems relying on wearable sensors, where motion signals may be corrupted, interrupted by battery depletion, or unavailable due to user non-compliance. These situa- tions underscore the need for multi-modal models capable of remaining robust when one or more modalities are missing at deployment. The challenge of missing modalities has been addressed with simple data pro- cessing techniques (e.g. imputation) to more advanced DL approaches [21]. Most DL methods enforce the fusion process to be robust, e.g. by randomly dropping modalities during training. This Modality Dropout (ModDrop) technique [16] can be implemented in various ways in the literature. For instance, data modal- ities are replaced with zero at input- [16], or feature-levels [8], filled up with a learnable parameter [21] masked out from the attention [10], or completely ig- nored during fusion [13]. To recover the full set of training modalities before the fusion time, models like SMIL [11], and ActionMAE [20] perform cross-modal re- construction. Beyond designing robust fusion processes, Ma et al. [10] introduce a sharing weights mechanism across modality-dedicated components. Moreover, McKinzie et al. [12] harness the knowledge distillation framework with a full- modal teacher who guides a student with partial modalities available. Recently, the multi-modal co-learning framework has emerged for addressing the challenge of missing modalities. In the co-learning paradigm, multiple models are trained to cooperate (share knowledge), aiming to enhance their individual performance [17]. Beyond its common usages for domain adaptation, noisy la- bels, and knowledge distillation, few works have harnessed it in multi-modal data setups. This involves the collaboration between modality-dedicated mod- els or components, either in a model-, feature-, or decision-based approach. For instance, MLA [25] shares the last layers of modality-specific models (model- based) and adjusts gradient directions to avoid overwriting modality-specific knowledge. MDiCo [15] enforces the learning of shared and specific features in modality-dedicated models (feature-based) for unimodal boosting in a bimodal setup. MV-HFMD [1] uses mutual distillation, previously introduced in [26], to guide individual per-modality predictions toward a full-modality consensus. Despite efforts to handle missing modalities, important challenges remain. Most existing methods focus on bimodal setups [21], where models are trained with two modalities, and one is missing at inference time. This limits the applica- bility of methods such as SMIL [11], DisCoM [6], MDiCo [15], and hallucination- Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification3 Training time Full-modal availability Audio ()Elevation () Ground-sensor ()Remote radar ()Weather ()Text () RGB optical () Remote optical () Inference time Minimal missing Audio ()Elevation () Ground-sensor ()Remote radar ()Weather () RGB optical () Moderate missing Audio ()Elevation () Ground-sensor () Text () RGB optical () Extreme missing Elevation () Remote optical () Remote optical () Remote optical () Text ()Remote radar () Text ()Remote radar () Audio () RGB optical () Inference time Inference time Weather () Weather ()Ground-sensor () Fig. 1: Illustration of three missing modality conditions at inference time: mini- mal (a missing case when a single modality is missing), moderate (various modal- ities are missing), and extreme (all-but-one modalities are missing). based approaches [5] to broader multi-modal setups. In addition, some works specifically address the missing of all-but-one modalities, like EmbrNet [2], dis- tillation [21], and cross-reconstruction-based methods [4]. In contrast, we address the more general scenario of missing arbitrary modalities, where any subset of the training modalities may be available at inference time. As illustrated in Fig. 1, this encompasses a broad spectrum of missing modality conditions, from minimal (single modality) to moderate and extreme (all-but-one modalities). To address missing arbitrary modalities, we formulate a framework grounded in the principles of multi-modal co-learning and knowledge distillation. Rather than focusing the model design on robust fusion processes as previous works have done [16, 2, 10, 20, 4, 13], we prioritize inter-modal collaboration to encour- age each modality to contribute to and benefit from the others. Concretely, we introduce two decision-level fusion methods that leverage information at both feature- and decision-level to enhance the classification robustness: Co-learning for Missing (Co-Miss) and Full Co-learning (FullCo). Each method is co-guided to learn modality-shared and modality-specific features with volume-based con- trastive loss and a modality discriminant classifier. The Co-Miss method also adopts ModDrop, enforcing missing modality predictions to imitate a consensus derived from full-modality data, a strategy we refer to as missing distillation. In contrast, the FullCo method employs the mutual distillation strategy to guide individual per-modality predictions toward the full-modality consensus. We validate our methods via two multi-modal classification benchmarks. The Multi-CropHarvest dataset for crop-type recognition with four sensor modali- ties, and HL-Opportunity for human activity recognition with nineteen sensor 4Mena et al. modalities. The results demonstrate the classification robustness of our methods against three baselines and seven recent state-of-the-art approaches. Further- more, the results reveal complementary strengths between our approaches: the Co-Miss method proves particularly effective under minimal missing conditions, while the FullCo method stands out in moderate to extreme conditions. Taken together, these findings highlight the potential of our co-learning strategies for handling missing arbitrary modalities in real-world multi-modal classification scenarios. 2 Related Work Multi-modal Co-learning The co-learning paradigm has been applied to multi-modal data to handle noisy modalities and weak supervision [17]. This consists of unimodal models employing a feature-, decision-, or model-based col- laboration. The feature-based approach usually involves the design of a shared space across modalities. For instance, contrastive learning frameworks maximize the cosine-similarity among modalities [23], while Ienco et al. [6] align repre- sentations with a modality classifier trained adversarially. In decision-based co- learning, the objective is to exchange the unimodal predictive knowledge. In image classification, Black et al. [1] use the mutual distillation strategy to guide per-modality predictions into a full-modality consensus. In contrast, the model- based approach exchanges components across unimodal networks. For example, Zhang et al. [25] propose a modality-shared prediction head with orthogonal gra- dients among modalities. Moreover, Zadeh et al. [24] provide theoretical evidence that multi-modal training can improve performance even if a single modality is available at inference time, suggesting the benefit from auxiliary modalities. Missing Modalities The literature has addressed the challenge of missing modalities with diverse strategies [21]. A common technique corresponds to im- puting missing data with a zero placeholder. However, this carries strong bias with a drop in predictive performance [10]. An alternative consists of designing a robust fusion process in multi-modal learning. Recent models like ShaSpec [19] and MissModal [8] use the ModDrop technique [16] to make the multi-modal fu- sion invariant to missing modalities. Additionally, missing-modality predictions can be regularized to remain less confident than their full-modality counterparts [9], or optimized such that their loss is explicitly higher than that of the full- modality case [7]. To encourage robustness, Choi et al. [2] propose to select a random modality (for each feature) in the fusion process. Data-driven recovery has also been explored by reconstructing the modalities that are missing from the available ones. Models like SMIL [11] and SimMMDG [4] reconstruct the data modalities for the underlying full-modality fusion process. Beyond the fo- cus on robust fusion design, knowledge- and self-distillation frameworks have been employed to increase robustness to missing modalities [12, 8]. A standard distillation setup consists of a full-modal teacher who guides student models trained with a subset of all training modalities. Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification5 Co-learning for Missing Modalities The collaboration of modality-dedicated components has been leveraged to improve robustness when modalities are miss- ing. For instance, Hoffman et al. [5] use a hallucination approach that enforces an unimodal model mimicking the behavior of a potential missing modality. Mena et al. [15], use a feature-based co-learning strategy to enforce unimodal models learning modality-shared and -specific features in a bimodal setup. Zhang et al. [25] use a model-based co-learning strategy that exchanges parameters between unimodal models. As decision-based co-learning, Mena et al. [14] introduced mu- tual distillation in unimodal models tailored for EO data classification. These works have shown the potential of co-learning for missing modalities. Among these works, only a limited number address the challenge of missing arbitrary modalities and extend beyond bimodal setups. For instance, EmbrNet [2], FCoM [13], and MissModal [8] rely on random modality masking to improve the fusion robustness, yet their performance with missing modalities remains limited. To this end, we formulate two methods grounded in the principles of multi-modal co-learning and knowledge distillation to enhance the classification robustness in inference scenarios with missing arbitrary modalities. 3 Methodology Multi-Modal Setup Let us considerM as the set of M =|M| training modalities, andX = X m m∈M the multi-modal input data with label y ∈ 1,...,K. At inference time, any arbitrary subset of these modalities ̃ M ⊆M (with ̃ M = | ̃ M|) may be accessible, expressed by ̃ X = X m m∈ ̃ M . We define this scenario, encompassing any missing modality case, as missing arbitrary modalities. To address this challenge, we introduce two multi-modal methods employ- ing decision-level fusion, also known as late fusion. In concrete, we employ a simple fusion process where per-modality probability predictions ˆ y m ∈ [0, 1] K are averaged to yield the full-modality estimation (called consensus), given by ˆ y full = M −1 · P m∈M ˆ y m . Thus, if modalities are missing at inference time, they are just disregarded from the aggregation, expressed by ̃ y = ̃ M −1 · P m∈ ̃ M ˆ y m . For optimization, we use a standard cross-entropy loss to guide the main prediction with full-modality data, defined as L main =L CE (y, ˆ y full ) ,(1) where L CE (p, q) = − P k 1(p = k)· logq k is the loss function between the true label p and probabilities q, and 1(·) is the indicator function. We introduce additional loss terms at the feature-level (via co-learning) and at the decision- level (via knowledge distillation) as follows. 3.1 Feature-level Learning Criteria We assume that training modalities have shared (invariant among modalities) and specific (unique to each modality) information among them relevant for the 6Mena et al. classification. The shared space corresponds to class-relevant features that can be extracted from either modality, while the specific space contains class-relevant features that can only be extracted from a specific modality. For instance, a high- resolution and a low-resolution optical satellite image have common data (the optical part) and specific data (related to the differences in spatial resolutions). In concrete, we use modality-dedicated encoders that learn to extract both the shared z sha m ∈R d and specific z spe m ∈R d features explicitly, given by z sha m , z spe m =E m (X m ) .(2) For learning the specific features per modality, we use a modality discriminant loss function (based on the cross-entropy), expressed by L mod = 1 M X m∈M L CE (m,P spe (z spe m )) ,(3) whereP spe (·) is an auxiliary linear classifier that is fed with the specific features from either modality and has to predict the correct one. In this way, the specific features per sample have to be linearly distinguished among modalities. For learning the shared features among modalities, we use a volume-based contrastive loss function [3], defined by L cont = 1 2 L GRAM z sha a ,Z sha a ;γ +L GRAM Z sha a , z sha a );γ ,(4) where L GRAM (·,·;γ) is the GRAM multi-modal contrastive loss defined in [3] and parametrized by a scalar γ, a ∈M is the anchor modality, andZ sha a = z sha m : m ∈M/a. We use γ = 0.07, following [15]. The minimization of the Gramian volume among the shared features is applied over all modalities for each sample, enforcing them to be closer in space. Individual Prediction To obtain per-modality predictions, our model concate- nates the modality-shared and modality-specific features (from Eq. (2)) followed by a modality-dedicated linear headP m (·). This is given by ˆ y m =P m (z sha m ||z spe m ). 3.2 Decision-level Learning Criteria We consider two knowledge distillation variants used at the decision-level to improve the predictive robustness to missing arbitrary modalities. Missing Distillation Expecting any missing modality condition (see Fig. 1), we incorporate the ModDrop technique at the decision-level to expose the model to missing modalities during training. This missing modality prediction is ex- pressed by ˆ y miss = M −1 drop · P m∈M (1−d m )· ˆ y m , where d m ∼ Bern(α) the randomly drawn decision if modality m is masked out, and M drop = P m∈M d m the number of modalities available. However, instead of drawing a single random case per Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification7 A Logits Logits Logits Loss function Main prediction Specific features Shared features Missing distillation loss Modality discrim. loss Contrastive loss Main predictive loss Ground truth C2 C3 C1 Fused logits C2 C3 C1 C2 C3 C1 Fused logits Volume-based (GRAM) Modality-dedicated head ( ) Modality-dedicated head ( ) Modality-dedicated head ( ) C2 C3 C1 C2 C3 C1 C2 C3 C1 Modality 1 Modality 2 Modality-dedicated encoder ( ) Modality-dedicated encoder ( ) Modality-dedicated encoder ( ) C Modality 2 Modality 1 Modality label Modality Modality 2 Modality 1 Modality label Modality Modality 2 Modality 1 Modality label Modality C C C Concat / Average A / A Modality Case 1Case 2Case 3Case 4Case 5Case 6Case 7 ModDrop Fig. 2: Illustration of the Co-Miss method with four loss functions. sample, we exhaustively simulate all missing modality cases, i.e. 2 M − 1 combi- nations. Then, we enforce this missing modality prediction ˆ y miss to imitate the consensus ˆ y full , as well as the ground truth y, by the following loss function L miss =L CE (y, ˆ y miss ) + λ·L KD ( ˆ y full , ˆ y miss ;τ) ,(5) where λ is a weighting factor (we use λ = τ 2 following [14]), and L KD (·,·;τ) is the knowledge distillation function parametrized by the temperature τ (we use τ = 0.5 following [14]). We refer to this strategy as missing distillation. This Co-learning for Missing (Co-Miss) method is optimized via an unweighted sum of all afore-mentioned loss terms, expressed by L total =L main +L mod +L cont +L miss .(6) This method, illustrated in Fig. 2, follows a feature-based co-learning (in L mod and L cont ) combined with a distillation process (in L miss ). Mutual Distillation To address extreme missing conditions (See Fig. 1), we follow a decision-based co-learning strategy called mutual distillation, where each per-modality prediction ˆ y m has to imitate both the consensus ˆ y full and ground truth y. The loss function of this mutual distillation strategy considers the av- erage of the individual per-modality distillations, given by L mut = 1 M X m∈M L (m) mut (7) L (m) mut =L CE (y, ˆ y m ) + λ·L KD ( ˆ y full , ˆ y m ;τ) ,(8) where λ = τ 2 and τ = 0.5, following the same criteria as for the Co-Miss method. This Full Co-learning (FullCo) method is learned by optimizing an 8Mena et al. Volume-based (GRAM) Modality-dedicated head ( ) Modality-dedicated head ( ) Modality-dedicated head ( ) C2 C3 C1 C2 C3 C1 Logits C2 C3 C1 Logits Modality Modality 1 Modality 2 Modality-dedicated encoder ( ) Modality-dedicated encoder ( ) Modality-dedicated encoder ( ) C Modality 2 Modality 1 Modality label Modality Modality 2 Modality 1 Modality label Modality Modality 2 Modality 1 Modality label Modality C C Fused logits C2 C3 C1 Ground truth C2 C3 C1 Loss function Main prediction Specific features Shared features C Concat / Average Main predictive loss Contrastive loss Mutual distillation loss Modality discrim. loss A A / Logits Fig. 3: Illustration of the FullCo method with four loss functions. unweighted sum of the following loss terms: L total =L main +L mod +L cont +L mut .(9) This method, illustrated in Fig. 3, follows a full co-learning strategy, with a feature-based (in L mod and L cont ) and decision-based (in L mut ) collaboration. Our methods differ in how the decision-level learning is carried out: through the missing distillation (L miss ) in Co-Miss or mutual distillation (L mut ) in FullCo. 4 Experiments 4.1 Experimental Setup Datasets We use the following multi-modal classification benchmarks. Multi-CropHarvest 5 We consider a crop-type recognition problem by using the CropHarvest dataset. This benchmark contains 29 642 samples around the globe between 2016 and 2021. The label covers ten different crop-type groups (other, beverage spice, cereals, leguminous, fruits nuts, root tuber, oilseeds, vegetables melons, sugar, non crop). The input data consist of three multi-temporal sensor modalities, at 10[m] spatial resolution: multi-spectral optical data from Sentinel- 2, radar data from Sentinel-1, and weather variables. In addition, topographic features (mono-temporal modality) are available for each sample. Following pre- vious research [14, 15], we perform a 10-fold cross-validation. 5 https://github.com/nasaharvest/cropharvest (Accessed 27.07.2026). Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification9 HL-Opportunity 6 We consider a high-level activity recognition problem by us- ing the Opportunity dataset. This benchmark consists of data from six subjects performing different activities (five high-level: relaxing, coffee time, early morn- ing, cleanup, sandwich time). There are 19 modalities, corresponding to various sensors positioned on the body of the subjects (e.g. food, back, arms, hip). Fol- lowing the evaluation setup by Choi et al. [2], a total of 264 279, 6 528, and 12 688 samples are available for training, validation, and testing, respectively. Comparison We compare our methods against the following approaches. Baselines We consider unimodal models trained exclusively for each modality (i.e. no multi-modal interaction during training). As multi-modal baselines, we include two models employing feature- (Feat) and decision-level (Dec) fusion, respectively, where missing modalities are imputed with zeros at inference time. Competitors We include seven methods from the recent literature, general enough to accommodate multi-modal beyond bimodal setups. Five methods use feature- level fusion: EmbrNet [2], a method that randomly selects a single modality per feature, and the following ModDrop-based approaches: MissModal [8], ShaSpec [19], FCoM [13], and SimMLM [7]. In addition, two decision-level fusion methods that leverage the mutual distillation strategy: DML [26], and DSensD + [14]. Evaluation details For the classification assessment, we calculate the weighted F1 (F1) score with full-modality data and under every missing modality case. For each missing modality condition (i.e. minimal, moderate, and extreme), we average the F1 values of the corresponding missing modality cases. We also consider the macro missing performance as the average between the following settings: full-modality data, minimal missing, moderate missing, and extreme missing conditions. In addition, the method rank (according to performance, i..e rank 1= best) is calculated and averaged across all missing modality cases. Implementation We apply a z-score normalization on the input data. We use standard encoder architectures for the selected datasets [14, 2]. This corresponds to a 1D CNN for multi-temporal modalities, and an MLP for mono-temporal ones. For each encoder, we use two layers with 128 units and 20% of dropout on all encoders in the Multi-CropHarvest dataset, and four layers with 64 units for the HL-Opportunity. We apply a layer normalization to the per-modality predictions to scale the logits magnitudes. For optimization, we use AdamW with a learning rate of 10 −3 , batch-size of 128, and early stopping. The stopping considers the F1 score (macro missing) calculated on the validation set. We use weights in the main predictive loss function L main (inverse to the number of samples per class) to cope with class imbalance. For all competitors, we retain the default hyperparameter settings as reported in their original works. 6 w.opportunity-project.eu/challengeDataset.html (Accessed 27.07.2026). 10Mena et al. 4.2 Experimental Results We report the crop-type recognition results in Table 1. Here, the lowest drop in performance occurs under minimal missing conditions, varying whether the optical, radar, weather, or topographic modality is missing. In the Dec baseline, this ranges from 0.6 points when the topographic is missing to 17.1 points when the optical is missing. In contrast, the Co-Miss method reduces these drops to 0 and 12.1 points, respectively. Overall, the drop in performance is more evident in the extreme missing conditions. In the baselines, there is around 40 points of performance drop in the extreme (average) condition. In this case, our Co-Miss method has a drop of 25.2, while FullCo has 23.9 points. In addition, our methods outperform all competitors with significant differences in each missing condition, due to different advantages. The Co-Miss method outperforms all approaches in full-modality data, as well as in minimal missing conditions, except when missing the weather modality. The FullCo method achieves the best results under mod- erate and extreme missing conditions. This is expected, as the FullCo method is explicitly trained to handle unimodal data (via the distillation of per-modality predictions). Besides, FullCo achieves the second-best results in full-modality data and minimal missing conditions. These complementary advantages make our methods tie as the best average ranking with a 1.5 rank. Table 1: F1 score in the crop-type recognition task (Multi-CropHarvest). The sensor modalities are optical (O), radar (R), weather (W), and topographic (T). Bold indicates the best mean performance. Underlined indicates values that are not statistically different from the best method (Welch’s t-test, p > 0.05). ∗ Averaged over available cases. O R W T Uni- modal Feat Dec Embr- Net Miss- Modal Sha- Spec FCoM Sim- MLM DML DSen- sD + Co- Miss Full- Co ✓ 71.1 73.3 73.6 72.1 70.6 67.1 75.1 73.9 66.5 74.3 79.9 79.7 ×✓ 57.5 56.5 56.6 63.7 56.4 64.4 62.3 58.7 65.1 67.8 67.7 ✓ ×✓ 66.1 69.4 69.6 68.1 64.9 73.2 71.7 64.2 73.1 79.0 78.9 ✓ ×✓ 65.0 59.7 65.6 69.5 65.9 74.8 73.2 69.0 74.5 79.379.4 ✓ × 70.3 73.0 72.5 70.5 67.0 75.2 73.8 67.2 74.7 79.9 79.7 minimal 64.7 64.7 66.1 67.9 63.6 71.9 70.3 64.8 71.9 76.5 76.4 moderate 51.8 51.4 56.2 62.0 55.6 65.1 62.5 59.5 65.7 69.369.6 ✓ × × × 72.5 57.8 56.4 64.4 67.4 63.4 73.3 70.8 77.8 73.6 78.578.8 ×✓ × × 55.6 37.1 31.8 38.8 54.2 43.5 54.2 51.0 56.6 56.9 57.9 59.2 × ×✓ × 46.5 30.9 34.8 39.9 47.2 39.7 46.1 42.7 47.4 48.2 51.352.0 × × ×✓ 21.8 10.0 3.1 8.3 22.9 12.9 21.5 18.3 23.9 28.4 31.2 33.5 extreme 49.1 34.0 31.5 37.8 47.9 39.9 48.8 45.7 49.8 51.854.755.8 Macro 60.1 ∗ 55.8 55.4 58.0 62.2 56.3 65.4 63.2 61.2 65.8 70.170.4 Ranking 9.7 9.6 8.0 6.5 9.2 3.9 5.8 6.8 3.5 1.5 1.5 Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification11 Table 2: F1 score in the high-level activity recognition task (HL-Opportunity). Bold indicates the best mean performance. Underlinedindicates values that are not statistically different from the best method (Welch’s t-test, p > 0.05). ∗ Averaged over available cases Inference Uni- modal Feat Dec Embr- Net Miss- Modal Sha- Spec FCoM Sim- MLM DML DSen- sD + Co- Miss Full- Co full-modal 77.4 71.9 69.6 71.0 78.777.0 78.878.276.0 79.5 78.1 79.0 minimal 70.6 68.7 70.7 78.5 76.6 78.5 77.9 76.0 79.4 77.7 78.7 moderate 57.5 56.4 62.2 72.2 67.6 69.9 69.7 73.5 74.9 70.0 74.0 extreme 59.1 28.0 27.1 35.1 49.6 32.3 41.1 42.6 60.5 57.0 43.3 56.4 Macro 68.2 ∗ 57.0 55.5 59.7 69.8 63.4 67.0 67.1 71.572.7 67.0 72.0 Ranking 10.1 10.6 8.9 3.7 7.5 5.4 5.6 4.3 1.4 6.1 2.4 Finally, the macro missing results in the Multi-CropHarvest dataset indicate that, regardless of the modalities missing at inference time, the expected F1 performance is 70.1 and 70.4 for Co-Miss and FullCo methods, respectively. This value improves around 4 points compared to the closest competitor (DSensD + ), and up to 15 points compared to the Feat and Dec baselines. Moreover, the improvements achieved by our methods are statistically significant in all cases. Besides, no significant differences are observed among our methods across the inference scenarios, except when radar or topographic modalities are available. The results in the activity recognition task are displayed in Table 2. We only report the average of the three missing conditions, as with 19 sensor modalities, there are 2 19 − 1 missing modality cases possible. Similarly to previous results, the performance degrades the least under minimal missing conditions and more noticeably under extreme missing ones. The potential information redundancy among the 19 sensor modalities explains the limited performance drop when a single modality is missing (i.e., 18 modalities are still available). In our methods, the performance decreases by just 0.3 points with FullCo and 0.4 with Co-Miss. Overall, the best results are achieved by the DSensD + method in all missing conditions except for the extreme missing one, where DML obtains the best re- sults. Among our methods, FullCo competes with DSensD + and DML methods, obtaining the second-best results across all missing conditions (except extreme), and being statistically similar to the best results in both full-modal and moder- ate conditions. Moreover, the average ranking shows FullCo as the second-best method overall with a 2.4 rank. At last, the macro missing results indicate that, regardless of the modalities missing at inference time, the expected F1 perfor- mance of the FullCo method remains 72.0. This represents a marginal difference of only 0.7 points compared to DSensD+, which is not statistically significant. Increasing missing modalities Figure 4 shows that the classification performance decreases non-linearly as the number of missing modalities increases. This drop is more pronounced for the baseline methods, which lack any dedicated mech- 12Mena et al. Fig. 4: F1 score by increasing the number of modalities missing at inference time. The 95% confidence interval is included as shaded colors for each method. anism to handle missing modalities. The class-relevant redundancy among the 19 modalities is noted in the HL-Opportunity, where the performance remains mostly unchanged until five modalities are missing. Moreover, the performance difference and variability between the methods increase considerably as more modalities are missing. This trend highlights the growing robustness challenge as we move from minimal to moderate and extreme missing conditions. Similar to previous results, the performance relates to the considered dataset. In Multi- CropHarvest, FullCo and Co-Miss methods achieve the best robustness curves along missing modalities, while in HL-Opportunity, the FullCo has comparable behavior to the best competing approaches (i.e. DSensD + and DML). 4.3 Ablation Results In this subsection, we vary individual components to isolate the key factors characterizing the behavior of our methods. These results are reported in Ta- ble 3 for the Multi-CropHarvest dataset, while similar results are observed in HL-Opportunity. We observe that the most relevant loss term is the contrastive one (L cont ), followed by the decision-level term, either mutual (L mut ) or missing (L miss ) distillation. On the other hand, the loss term that least affects perfor- mance is the modality discriminant one. This indicates that for our co-learning method, the modality-shared information is more crucial than the modality- specific counterpart. Furthermore, we compare the volume-based contrastive loss against its standard pair-wise version (computed over all pairs [23]). These results suggest that the volume-based formulation is more effective for missing condi- tions, yielding a more structured modality-shared space within our approach. Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification13 Table 3: F1 score in the Multi-CropHarvest dataset under different configurations in our methods. The best and second-best values are highlighted. Ablation caseFull-modal Minimal Moderate ExtremeMacro Co-Miss 79.9 76.5 69.3 54.770.1 w/o L mod 79.9 76.5 69.3 54.4 70.0 w/o L cont 79.2 75.7 68.554.0 69.4 w/o L mod and L cont 79.2 75.6 68.5 54.1 69.3 pair-wise contrastive 77.5 75.4 68.5 53.4 68.7 FullCo 79.7 76.4 69.6 55.8 70.4 w/o L mod 79.576.369.555.570.2 w/o L cont 78.8 75.5 68.7 55.0 69.5 w/o L mod and L cont 78.9 75.7 68.9 55.2 69.7 pair-wise contrastive 77.4 74.1 67.5 54.0 68.2 w/o L mut or L miss 79.8 76.1 68.7 53.4 69.5 Fig. 5: F1 gap between the default version of our methods and different variants. We display the performance gap (difference between the F1 score of a variant and the default version of our approaches) in Fig. 5. In this case, removing the feature-level loss terms (L mod and L cont ) causes the largest performance drop when few modalities are missing. Conversely, removing the decision-level loss term (either L mut or L miss ) degrades performance most when more modalities are missing. This reflects the complementarity of our feature- and decision-level learning criteria: feature-level collaboration is key to robustness when facing moderate to minimal missing conditions, while decision-level distillation is cru- cial when facing moderate to extreme missing conditions at inference time. 4.4 Model Comparison & Limitations The Co-Miss method has a scalability limitation that future work should address, as its training complexity scales asO(2 M ). For example, with M = 10 modalities, it must simulate 1023 missing-modality combinations. Given this limitation and its weaker performance under severe missing conditions, we recommend FullCo, particularly for moderate to extreme missing inference scenarios. When a few 14Mena et al. modalities are available for training (i.e., M < 10), Co-Miss remains a suitable choice, especially for full-modality and minimal missing inference scenarios. Both of our methods share similarities to MDiCo [15]. Despite the learning of a modality-shared space by contrastive learning (Sec. 3.1), we use a volume- based version instead of the pairwise loss as in [15]. However, MDiCo is a purely multi-modal co-learning approach, i.e. no fusion is performed among modalities. Thus, MDiCo does not employ any decision-level learning criteria as us (Sec. 3.2). We validate our approaches in two applications, while conventional multi- modal benchmarks (e.g., image-text-audio) can demonstrate their generalization beyond sensor-based domains. Future work should also consider adaptive mod- eling for varying amounts of modality-shared and modality-specific information. 5 Conclusion Missing arbitrary modalities at inference time is expected in real-world appli- cations, degrading the classification performance of multi-modal models. In this work, we present two multi-modal co-learning methods to improve the clas- sification robustness under inference scenarios with missing arbitrary modali- ties. Our methods learn modality-shared and -specific feature spaces through feature-based co-learning, while promoting predictive robustness via decision- level knowledge distillation. Evaluated on two benchmarks, our methods con- sistently outperform (with significant differences) all competing approaches on one dataset, and achieve competitive (significantly similar) results in the other. The evidence highlights the potential and synergies of co-learning and knowledge distillation for enhancing the robustness of multi-modal classification models. Acknowledgments. Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 414984028 – SFB 1404 FONDA. References 1. Black, S., Souvenir, R.: Multi-view classification using hybrid fusion and mutual distillation. In: Proceedings of the IEEE/CVF WACV. p. 270–280 (2024) 2. Choi, J.H., Lee, J.S.: EmbraceNet: A robust deep learning architecture for multi- modal classification. Information Fusion 51, 259–270 (2019) 3. Cicchetti, G., Grassucci, E., Sigillo, L., Comminiello, D.: Gramian multimodal representation learning and alignment. In: ICLR (2025) 4. Dong, H., Nejjar, I., Sun, H., Chatzi, E., Fink, O.: SimMMDG: A simple and effective framework for multi-modal domain generalization. NeurIPS 36, 78674– 78695 (2023) 5. Hoffman, J., Gupta, S., Darrell, T.: Learning with side information through modal- ity hallucination. In: Proceedings of the IEEE CVPR. p. 826–834 (2016) 6. Ienco, D., Dantas, C.F.: DisCoM-KD: Cross-modal knowledge distillation via dis- entanglement representation and adversarial learning. In: The 35th BMVC (2024) 7. Li, S., Chen, C., Han, J.: SimMLM: A simple framework for multi-modal learning with missing modality. In: Proceedings of the IEEE ICCV. p. 24068–24077 (2025) Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification15 8. Lin, R., Hu, H.: MissModal: Increasing robustness to missing modality in mul- timodal sentiment analysis. Transactions of the Association for Computational Linguistics 11, 1686–1702 (2023) 9. Ma, H., Zhang, Q., Zhang, C., Wu, B., Fu, H., Zhou, J.T., Hu, Q.: Calibrating multimodal learning. In: ICML. p. 23429–23450 (2023) 10. Ma, M., Ren, J., Zhao, L., Testuggine, D., Peng, X.: Are multimodal transformers robust to missing modality? In: Proceedings of the IEEE/CVF CVPR. p. 18177– 18186 (2022) 11. Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: SMIL: Multimodal learning with severely missing modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, p. 2302–2310 (2021) 12. McKinzie, B., Shankar, V., Cheng, J.Y., Yang, Y., Shlens, J., Toshev, A.T.: Ro- bustness in multimodal learning under train-test modality mismatch. In: ICML. p. 24291–24303 (2023) 13. Mena, F., Arenas, D., Dengel, A.: Missing data as augmentation in the Earth observation domain: A multi-view learning approach. Neurocomputing 638 (2025) 14. Mena, F., Ienco, D., Cassio, D.F., Interdonato, R., Dengel, A.: Multi-sensor model for Earth observation robust to missing data via sensor dropout and mutual dis- tillation. IEEE Access 13, 83930 – 83943 (2025) 15. Mena, F., Ienco, D., Dantas, C.F., Interdonato, R., Dengel, A.: Multi-modal co- learning for Earth observation: Enhancing single-modality models via modality collaboration. Machine Learning 114(12), 279 (2025) 16. Neverova, N., Wolf, C., Taylor, G., Nebout, F.: ModDrop: Adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelli- gence 38(8), 1692–1706 (2015) 17. Rahate, A., Walambe, R., Ramanna, S., Kotecha, K.: Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions. In- formation Fusion 81, 203–239 (2022) 18. Shen, H., Li, X., Cheng, Q., Zeng, C., Yang, G., Li, H., Zhang, L.: Missing informa- tion reconstruction of remote sensing data: A technical review. IEEE Geoscience and Remote Sensing Magazine 3(3), 61–85 (2015) 19. Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., Carneiro, G.: Multi-modal learning with missing modality via shared-specific feature modelling. In: Proceedings of the IEEE/CVF CVPR. p. 15878–15887 (2023) 20. Woo, S., Lee, S., Park, Y., Nugroho, M.A., Kim, C.: Towards good practices for missing modality robust action recognition. In: Proceedings of the AAAI Confer- ence on Artificial Intelligence. vol. 37, p. 2776–2784 (2023) 21. Wu, R., Wang, H., Chen, H.T., Carneiro, G.: Deep multimodal learning with miss- ing modality: A survey. Transactions on Machine Learning Research (2026) 22. Yan, X., Hu, S., Mao, Y., Ye, Y., Yu, H.: Deep multi-view learning methods: A review. Neurocomputing 448, 106–129 (2021) 23. Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., Faieta, B.: Multimodal contrastive training for visual representation learning. In: Proceedings of the IEEE/CVF CVPR. p. 6995–7004 (2021) 24. Zadeh, A., Liang, P.P., Morency, L.P.: Foundations of multimodal co-learning. Information Fusion 64, 188–193 (2020) 25. Zhang, X., Yoon, J., Bansal, M., Yao, H.: Multimodal representation learning by alternating unimodal adaptation. In: Proceedings of the IEEE/CVF CVPR. p. 27456–27466 (2024) 26. Zhang, Y., Xiang, T., Hospedales, T.M., Lu, H.: Deep mutual learning. In: Pro- ceedings of the IEEE CVPR. p. 4320–4328 (2018)