Paper deep dive
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen, Manuel Lage Cañellas, Miguel Bordallo López
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
Tags
Links
- Source: https://arxiv.org/abs/2610.15891v1
- Canonical: https://arxiv.org/abs/2610.15891v1
Trouble viewing inline? Open PDF directly →
Full Text
30,056 characters extracted from source content.
Expand or collapse full text
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose RecognitionThanks: ∗These authors contributed equally to this work. Yunxiang Luo†∗, Constantino Álvarez Casado†‡∗, Xiaoting Wu†, Nhi Nguyen†, Le Nguyen†, Manuel Lage Cañellas†‡, Miguel Bordallo López† Abstract Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of 75.23%±0.6075.23\%± 0.60 averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control. Index Terms: sensor fusion, missing modalities, contactless sensing, V-JEPA2, activity recognition †address: †Center for Machine Vision and Signal Analysis, University of Oulu, Finland ‡Candour Ltd, Oulu, Finland 1 Introduction Figure 1: (a) Shared inference architecture of the final query model and the earlier Subset-query (B3). Grey marks frozen feature extraction and blue trained pooling and fusion. The example declares depth absence, and insets show representative observations. (b) Final all-view task supervision evaluates one batch under all seven non-empty subsets, averages the pose/activity losses, and makes one update, without a latent prediction loss. Multimodal contactless human sensing combines cameras and radar to extract physiological and behavioural information for health monitoring and human state assessment [10]. RGB, depth, and radar observe the appearance, geometry, and motion of the same activity and can provide complementary information about breathing and body pose, but the available sensor configuration may change between data collection and deployment. Power, synchronization, and communication requirements can restrict which sensors are deployed [5]. Here, a missing modality denotes an unavailable sensor stream, rather than degradation within a stream that remains available. Modality dropout and adaptation of pretrained networks address this condition [5, 15], yet multimodal transformers remain sensitive to missing inputs [13]. Performance with complete inputs therefore does not establish performance with missing sensors. We use V-JEPA2, a video model trained with a joint-embedding predictive architecture (JEPA) to predict target representations from available context [2]. The central question is how downstream pooling and supervision affect the use of these frozen representations for breathing activity and pose classification when sensor availability changes. The contributions are as follows: 1. Availability-aware multimodal fusion. One classifier with query pooling, fixed sensor slots, and explicit availability covers all seven non-empty RGB, depth, and radar subsets, reaching 75.23±0.6075.23± 0.60% activity macro-F1 averaged across them. 2. Controlled attribution of training gains. With matched downstream parameters, supervising all subsets in each batch improves the query model by 1.923 percentage points (p), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. 3. Evidence on representation use and compression. Radar probes, token reductions, and an exact zero-weight control test transferred information, compression sensitivity, and a complete-to-subset prediction loss. Predictive dynamics and planning remain outside the evaluated scope. 2 Related Work Multimodal learning joins representation, alignment, and fusion across sensor streams [3], and models trained on complete observations can deteriorate when an input stream is unavailable [15, 13]. Proposed responses include modality dropout [5, 14], feature reconstruction and transformer fusion [18, 9], shared and sensor-specific representations [16], logit distillation [17], availability-conditioned paths [12], and parameter-efficient adaptation [15]. They do not address frozen JEPA representations of heterogeneous RGB, depth, and radar inputs evaluated across all seven subsets. Joint-embedding predictive architectures predict representations rather than input samples, from I-JEPA for images [1] to V-JEPA and V-JEPA2 for video [4, 2]. Predictive alignment has also been studied across modalities and time series [11, 7, 6], while V-JEPA2-AC adds action-conditioned latent dynamics for robotic planning [2]. We instead use V-JEPA2 as a frozen feature extractor and test a complete-to-subset prediction objective against directly supervised fusion. 3 Proposed Methodology All models use frozen V-JEPA2 features with the same fusion transformer and task heads, as summarised in Table 1, and the final query/all-view model uses the B3 inference architecture, as depicted in Figure 1(a). 3.1 Sensor input construction Let ℳ=RGB,D,RM=\RGB,D,R\ denote the sensors and ∅≠S⊆ℳ ≠ S the available subset. RGB and depth inputs contain 64 frames at 256×256256× 256 pixels. Depth is clipped at the 1st and 99th within-sample percentiles and scaled to the encoder range, and its frozen input convolution averages the pretrained colour kernels. Radar contributes 200 samples of phase displacement, filtered breathing, and filtered cardiac traces, which a complex Morlet wavelet transform maps to a stacked 256×256256× 256 scalogram replicated over three channels and two identical frames. Every modality uses the frozen vjepa2-vitl-fpc64-256 encoder without its predictor. Encoded sequences are adaptively averaged to at most 128 tokens and cached. Radar time lies along a scalogram axis, so duplicated frames do not establish temporal alignment with video tokens. 3.2 Availability-aware fusion Query pooling uses a learned, modality-specific query to attend to the cached tokens and produce a 128-dimensional vector, whereas mean pooling averages them before projection. The sensor slot is given by: m=poolm(m)+m+1,m∈S,m+0,m∉S,s_m= casespool_m(T_m)+e_m+v_1,&m∈ S,\\ e_m+v_0,&m∉ S, cases (1) where mT_m holds the tokens, me_m identifies the modality, and 0,1v_0,v_1 encode absence and presence. A summary token and the three slots enter a pre-norm transformer with two layers, four heads, width 128, and feed-forward width 256. Its summary output Sh_S feeds heads for three poses and four activities, whose decisions define 12 joint classes and whose mean cross-entropy is ℒtaskL_task. Absent slots carry no reconstructed measurement. 3.3 Capacity and supervision controls The factorial comparison crosses query and matched mean pooling with balanced and all-view supervision. Matched mean pooling adds a normalized residual 128→256→128128→ 256→ 128 perceptron per sensor, giving both downstream models 882,695 parameters against 683,399 for Subset-mean (B2), without equalizing computation or inductive bias. Balanced supervision draws one subset per batch, with selection counts across the seven subsets differing by at most one per epoch. All-view supervision evaluates every subset on the same batch and makes one update with: ℒall=17∑∅≠S⊆ℳℒactivity(S)+ℒpose(S)2.L_all= 17 _ ≠ S L_activity(S)+L_pose(S)2. -2.84526pt (2) The final model uses this objective alone, which gives seven supervised views per example and so increases exposure. 3.4 Complete-to-subset prediction Consensus-JEPA adds a training-only complete-input branch. For S⊊ℳS , S=qϕ(gθ(S))z_S=q_φ(g_θ(h_S)) predicts ℳ=sg[gθ(ℳ)]z_M=sg[g_θ(h_M)] from the same online network, where sgsg stops gradients through the complete-input target. The projector and predictor are two-layer perceptrons with hidden width 256 and output width 128. Training minimizes ℒtask+λℒcjL_task+ _cj with ℒcj=1−cos(S,ℳ)L_cj=1- (z_S,z_M). The complete pass runs in evaluation mode without a moving-average teacher. Zero-loss (A0) and Consensus (C1) execute the same graph with λ=0λ=0 and 0.0030.003, and both auxiliary modules are removed at inference. 4 Experimental Evaluation Table 1: Earlier controlled reference models and their out-of-fold (OOF) recording-level macro-F1 (%). All use frozen V-JEPA2 tokens, a fusion transformer, and supervised pose/activity heads, and universal models sample one subset per batch. Activity Avg-7 is mean ± sample SD, pose and joint are Avg-7 means, and Worst-7 averages the within-seed minimum. Model (ID) Nets Training subsets Pooling Auxiliary target (weight) Act. Avg-7 Pose Joint Worst-7 Full-only (B0) 1 Full input only Mean None 57.92±2.6757.92± 2.67 78.99 48.82 22.35 Subset-mean (B2) 1 Balanced seven Mean None 71.88±0.7471.88± 0.74 89.80 64.15 54.04 Subset-query (B3) 1 Balanced seven Query None 74.05±0.3674.05± 0.36 90.09 66.64 55.22 Specialists (B1) 7 One fixed subset Query None 74.74±0.5574.74± 0.55 90.29 67.63 60.00 Zero-loss (A0) 1 Balanced seven Query Full latent branch (0) 73.24±1.5673.24± 1.56 90.07 65.92 54.26 Consensus (C1) 1 Balanced seven Query Full latent, predictor (0.003) 73.23±1.1673.23± 1.16 90.06 65.91 54.73 Direct (A1) 1 Balanced seven Query Full latent, no predictor (0.003) 73.69±0.8273.69± 0.82 89.82 66.08 54.93 Logit-KD (KD) 1 Balanced seven Query Full logits (0.5, temperature 2) 73.81±1.3273.81± 1.32 89.91 66.55 54.49 Dataset and acquisition protocol. The complete aligned OMuSense-23 subset contains 39 subjects [10], each recorded standing, sitting, and lying in four nominally 30-second conditions separated by 20-second gaps: normal breathing, reading aloud, guided breathing, and breath holding. The 117 sessions give 468 activity recordings. Using TIMER-defined activity intervals, we extract 10-second windows at a nominal 1-second stride, appending a complete end-aligned window when the regular grid does not reach the interval end. This yields 10,530 aligned samples. Lying-pose RGB and depth are rotated by 180∘180 using the pose field, so deployment would need label-independent orientation metadata, and the camera position and background of this pose may affect pose and joint scores. Activity is therefore the primary endpoint, although acquisition effects cannot be excluded. Subject split, aggregation, and metrics. Five outer subject-group folds each reserve five training subjects for validation, with subjects disjoint across training, validation, and testing. Recording-level activity macro-F1 selects checkpoints, on the training subset for Full-only and specialists and on the seven-subset mean for universal models. Window logits are averaged per recording before y^r=argmaxcℓ¯r,c y_r= _c _r,c, and the five test folds are pooled per seed, giving 468 predictions per subset. For task t with CtC_t classes, F1,t(S) F_1,t(S) =1Ct∑c=1Ct2TPc2TPc+FPc+FNc, = 1C_t _c=1^C_t 2\,TP_c2\,TP_c+FP_c+FN_c, (3) Avg7t 7_t =17∑∅≠S⊆ℳF1,t(S). = 17 _ ≠ S F_1,t(S). Activity Avg-7 is primary, with pose and joint Avg-7 and activity Worst-7, the within-seed minimum subset score. Paired 95% intervals use 5,000 subject-cluster bootstrap draws shared between compared methods. They are conditional on the fitted models and folds, excluding retraining and new-cohort uncertainty, and exploratory without multiplicity correction. Comparators and optimization. In Table 1, C1 must be compared with A0 to isolate the auxiliary weight, since B3 shares their inference architecture but not their training graph. Logit-KD targets complete-input logits of the same online network. These 14 configurations give 210 training tasks over five folds and three seeds, with AdamW at learning rate 3×10−43× 10^-4, weight decay 10−410^-4, batch size 128, gradient clipping at 1, at most 120 epochs, and patience 15. The factorial experiment keeps these settings, seeds 20260816–20260818, and validation Avg-7 selection, covering 60 training cells, 50 of them new. Representation probes and token reduction. Linear probes fit separate logistic-regression heads (L-BFGS, C=1C=1, at most 2,000 iterations) to seven fixed representations, giving 70 heads over five folds without tuning or repeated seeds, with standardization fitted on training windows only. Radar uses a 16×1616× 16 pooled scalogram, 135 spectral/statistical features (40 Hann-periodogram log-power bins over 0.1–4 Hz and five statistics per trace), mean V-JEPA2 features of 1,024 dimensions, and their 256-dimensional whitened PCA fitted on training data (seed 20260908). Mean RGB, mean depth, and full concatenation complete the set. Token reduction evaluates two 64-token variants of all 15 final checkpoints without retraining. The RGB/depth cache holds 32 temporal groups and four spatial stripes, so spatial aggregation gives 32×232× 2 groups and temporal aggregation 16×416× 4, while radar uses the same reduction in both. 5 Results and Discussion Subset-aware training addresses the largest deficit. Full-only reaches 80.51% activity macro-F1 with all sensors but only 57.92% Avg-7, whereas Subset-mean reaches 71.88%, as shown in Table 1. This 13.96-point difference includes subset-aware checkpoint selection. It gains 31.69, 27.47, and 29.87 points for depth, radar, and depth+radar, but changes by -1.03 with complete input, so it mainly corrects the mismatch between training and available sensors. Frozen predictive features support the task, but the downstream decision must still be trained for the observed subsets. Matched capacity qualifies the pooling claim. The earlier 2.17 p gain of Subset-query over Subset-mean also changes parameter count. With matched capacity, as shown in Table 2 and depicted in Figure 2(b), mean minus query is +0.206 p [-1.083, 1.457] under balanced supervision, while query exceeds mean by 1.433 p [0.024, 2.879] under all-view supervision, close to the interval boundary. The interaction (Qall−Qbal)−(Mall−Mbal)(Q_all-Q_bal)-(M_all-M_bal) of 1.639 p [-0.061, 3.350] does not establish a special synergy. Table 2: Completed capacity-matched comparison. All four models have 882,695 downstream parameters. Scores are recording-level macro-F1 (%), with three-seed sample SD for activity Avg-7 and means for the remaining columns. Pooling / supervision Act. Avg-7 Pose Joint Worst-7 Query / balanced 73.31±0.2173.31± 0.21 89.41 65.49 54.43 Mean / balanced 73.51±0.9973.51± 0.99 89.61 65.58 56.09 Mean / all-view 73.80±1.0073.80± 1.00 90.47 66.72 56.70 Query / all-view 75.23±0.6075.23± 0.60 91.25 68.85 59.59 Figure 2: (a) Activity macro-F1 per sensor subset (three-seed means) for the local query/balanced control, the final query/all-view model, and the earlier specialists, with D depth and R radar. (b) Activity Avg-7 of the four capacity-matched models with training-seed SD, not subject intervals. Lower panels give paired contrasts in p with 95% subject-bootstrap intervals. All-view supervision improves the matched query model. Query/all-view reaches 75.23%±0.6075.23\%± 0.60 activity Avg-7 against 73.31±0.2173.31± 0.21 for its local query/balanced control, a paired gain of 1.923 p [0.647, 3.209] that is positive for each seed. Pose and joint improve by 1.845 p [1.069, 2.642] and 3.368 p [1.919, 4.918]. The earlier B3 score of 74.05 records a different execution, not this same-environment reference. As depicted in Figure 2(a), the gain concentrates in incomplete inputs, with depth rising from 54.43 to 59.59% and radar from 67.39 to 71.45%. Depth remains the weakest subset, and the final model does not dominate every sensor condition. One network compared with seven specialists. The final model is 0.490 p [-0.607, 1.626] above the earlier specialist bank, which establishes neither superiority nor equivalence, and the bank was not rerun in the new environment. The specialists remain higher for depth and radar alone. Its 882,695 downstream parameters cover every availability state, against 4,381,745 for a bank pruned to its required branches, both excluding the shared frozen encoder of about 304 million parameters. This concerns storage and maintenance, not end-to-end speed or energy. Radar probes test representation transfer. As shown in Table 3, with the same classifier type, mean V-JEPA2 features exceed pooled CWT pixels by 6.319 p [2.751, 9.996] in activity F1, and the PCA control retains 3.951 p [0.287, 7.729] at the same input dimension. This supports transferred information beyond the pooled-image descriptor, although PCA fitting and pixel pooling prevent complete isolation of the encoder. The 2.143 p [-1.217, 5.500] difference over spectral/statistical features leaves activity superiority uncertain. The radar probe also exceeds the radar-only shared model (76.31 against 71.45%), so feature information is not equally used by every downstream configuration, although the gap does not identify interference. Table 3: Fixed radar linear probes, with classifier input dimension and radar-only recording-level macro-F1 (%), not seven-subset averages. Each row pools five OOF folds without repeated seeds. Radar representation Dim. Activity Pose Joint Pooled CWT pixels 256 69.99 71.30 45.06 Spectral/statistical 135 74.17 76.00 52.91 V-JEPA2 mean + PCA 256 73.95 82.22 60.49 V-JEPA2 mean 1,024 76.31 84.58 64.17 Post-training compression approximately preserves mean scores. At 64 tokens the final checkpoints give activity Avg-7 of 75.26%±0.3475.26\%± 0.34 with spatial and 75.10%±0.2075.10\%± 0.20 with temporal aggregation. Spatial 64 minus the 128-token reference is +0.025 p [-0.431, 0.484], and temporal minus spatial -0.152 p [-0.589, 0.272], proving neither equivalence nor a preferred geometry, so 128 tokens remain the reference. The FP32 payload halves from 1.50 to 0.75 MiB per window, although the 128-token cache is still read. An earlier B3 reduction to 32 tokens lowered Avg-7 from 74.05 to 59.24%. As reduction follows global encoder attention, neither experiment isolates local spatial or temporal information. The matched predictive loss does not improve Avg-7. Consensus minus Zero-loss is -0.005 p [-1.350, 1.278], a contrast that changes only the auxiliary weight. Direct alignment and Logit-KD do not exceed Subset-query, but change parameters or targets and do not show that such objectives generally fail. Over 15 Consensus checkpoints and six incomplete subsets, the median weighted auxiliary/task gradient-norm ratio is 0.065%, and gradient cosine is negative in ten of 90 conditions. This indicates a weak auxiliary contribution at this weight, without establishing its cause or representation collapse, so the null result is specific to the tested target and protocol. What this contributes to JEPA-based world models. The study separates transfer from a JEPA backbone, training an observation classifier for missing sensors, and learning predictive dynamics. This encoder could precede a separate transition model, but classification does not establish a physically sufficient or sensor-independent state. The transferable lesson is to test observation encoders across deployment subsets with controlled capacity, exposure, and auxiliary-loss baselines. Incremental evaluation and published sensing baselines. Table 4 places our stages beside published models that process engineered camera/radar or derived cardiorespiratory signals [10, 9, 8]. The initial distillation used an RGB–radar teacher and rolling leave-one-subject-out evaluation with three validation subjects and window metrics, whereas the controlled stage adds subject-group folds, recording aggregation, and corrected orientation and radar windowing. Because protocols differ, 75.23% Avg-7 cannot be interpreted as a controlled improvement over 70.43%. Likewise, the full-input 82.23% macro-F1 and published 87.2% hybrid accuracy cannot establish a performance ranking because both the metrics and evaluation protocols differ. Table 4: Published OMuSense-23 baselines and stages of this work, as activity accuracy/macro-F1 (%). Published protocols are subject fivefold, except the 30/10/10 split of physiological SSL. Stages are not matched reruns, earlier F1 aggregation differs from pooled OOF scoring, and a dash marks an untabulated result. Study/stage (representation) Full input Seven subsets Published, N=50N=50, window-level protocols OMuSense-23 [10] (477 handcrafted) 81.0/81.0 – Physiological SSL [9] (four 1-D) 74.0/– – Hybrid fusion [8] (handcrafted+deep) 87.2/– – Initial stage, N=39N=39, rolling LOSO/window V-JEPA2 (frozen RGB, depth, radar) 77.99/76.27 – Distillation (experts + teacher) 81.33/79.80 72.39/70.43 Controlled stage, N=39N=39, fivefold OOF/recording Full-only (B0), corrected cache –/80.51 –/57.92 Subset-query (B3), same cache –/81.18 –/74.05 Query/all-view, final –/82.23 –/75.23 Scope and remaining evidence. We test declared absence, not failure detection or corrupted inputs, and all seven subsets appear in training. Unseen sensors, incomplete records, free-living activity, new sites, and other backbones are not evaluated. Training compute and specialist exposure are unmatched, pose may use acquisition context, and repeated development on this cohort limits the intervals. An exposure control with one update from seven independent batches differs from all-view by +0.140 and -0.488 p in two validation cells, so a unique benefit from reusing examples is not established. Saved predictions and paired intervals are available. Full reproduction also requires model checkpoints, cached features, and subject splits. Other small fusion and auxiliary-loss screens remain exploratory. 6 Conclusion We present a shared multimodal fusion model that recognizes breathing activity and pose from frozen V-JEPA2 representations of RGB, depth, and radar under all seven non-empty sensor subsets, without reconstructing missing measurements. Supervising all subsets in each batch yields an activity Avg-7 of 75.23%±0.6075.23\%± 0.60 and improves the matched control by 1.923 p, primarily for incomplete inputs. Under matched capacity, query pooling shows no clear advantage over matched mean pooling under balanced supervision, and the Consensus-JEPA loss yields no measurable gain. The evidence therefore supports direct supervision over the known availability states with explicit checks of representation, capacity, uncertainty, and compression, whereas predictive state modelling for JEPA-based world models remains a separate question. ACKNOWLEDGMENT The research was supported by the Research Council of Finland Smart Video Sensorization for Secure Healthcare Monitoring (SViSenS) (Grant Number: 370277370277), the Interreg Aurora ResilientEdge project (Grant Number: 20373282) and Profi7 Hybrid intelligence program (352788). The authors wish to acknowledge the use of AI-assisted tools for language editing, with a focus on grammar checking and improving readability, and coding support. In addition, the Elicit platform was used during the literature search as an AI-assisted discovery aid for candidate records. All inclusion decisions, exclusions, data extraction, synthesis, interpretation, and final writing decisions were performed by the authors. COMPLIANCE WITH ETHICAL STANDARDS This work analyses existing recordings, including RGB video, from the OMuSense-23 acquisition [10]. That study was conducted in accordance with the Declaration of Helsinki and institutional regulations, and all participants provided written informed consent. No new data were collected and no participant intervention was performed. The face shown in Figure 1 is pixelated to prevent identification. References [1] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas (2023) Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15619–15629. Cited by: §2. [2] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, et al. (2025) V-JEPA 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. External Links: Document Cited by: §1, §2. [3] T. Baltrušaitis, C. Ahuja, and L. Morency (2019) Multimodal machine learning: a survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2), p. 423–443. External Links: Document Cited by: §2. [4] A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas (2024) Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471. External Links: Document Cited by: §2. [5] S. de Blois, M. Garon, C. Gagné, and J. Lalonde (2020) Input dropout for spatially aligned modalities. In 2020 IEEE International Conference on Image Processing (ICIP), p. 733–737. Cited by: §1, §2. [6] U. Dutta, G. Pastrana, S. Khoshfetrat Pakazad, and H. Ohlsson (2026) Giving sensors a voice: multimodal JEPA for semantic time-series embeddings. arXiv preprint arXiv:2605.31580. External Links: Document Cited by: §2. [7] A. M. Girgis, A. Valcarce, and M. Bennis (2024) Time-series JEPA for predictive remote control under capacity-limited networks. arXiv preprint arXiv:2406.04853. External Links: Document Cited by: §2. [8] M. Lage Cañellas, C. Álvarez Casado, M. Malin, N. Prencipe, S. Sharifipour, and M. Bordallo López (2025) Fusion of handcrafted and deep-learned cardiorespiratory features for breathing pattern classification. In Proceedings of the 33rd European Signal Processing Conference (EUSIPCO), Palermo, Italy, p. 1627–1631. External Links: ISBN 978-9-46-459362-4, Link Cited by: Table 4, §5. [9] M. Lage Cañellas, C. Álvarez Casado, L. Nguyen, and M. Bordallo López (2025) A self-supervised multimodal framework for 1D physiological data fusion in remote health monitoring. Information Fusion 124, p. 103397. External Links: Document Cited by: §2, Table 4, §5. [10] M. Lage Cañellas, L. Nguyen, A. Mukherjee, C. Álvarez Casado, X. Wu, N. Nguyen, P. Susarla, S. Sharifipour, D. B. Jayagopi, and M. Bordallo López (2026) OMuSense-23: a multimodal dataset for contactless breathing pattern recognition and biometric analysis. IEEE Internet of Things Journal, p. 1–14. External Links: Document Cited by: §1, §4, Table 4, §5, COMPLIANCE WITH ETHICAL STANDARDS. [11] H. Lei, X. Cheng, Q. Qin, D. Wang, H. Huang, Q. Gu, Y. Wu, and L. Ji (2025) M3-JEPA: multimodal alignment via multi-gate MoE based on the joint-embedding predictive architecture. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 33902–33917. Cited by: §2. [12] S. Li, C. Chen, and J. Han (2025) SimMLM: a simple framework for multi-modal learning with missing modality. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 24068–24077. External Links: Document Cited by: §2. [13] M. Ma, J. Ren, L. Zhao, D. Testuggine, and X. Peng (2022) Are multimodal transformers robust to missing modality?. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 18156–18165. Cited by: §1, §2. [14] N. Neverova, C. Wolf, G. W. Taylor, and F. Nebout (2016) ModDrop: adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (8), p. 1692–1706. External Links: Document Cited by: §2. [15] M. K. Reza, A. Prater-Bennette, and M. S. Asif (2025) Robust multimodal learning with missing modalities via parameter-efficient adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (2), p. 742–754. External Links: Document Cited by: §1, §2. [16] H. Wang, Y. Chen, C. Ma, J. Avery, L. Hull, and G. Carneiro (2023) Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15878–15887. Cited by: §2. [17] Q. Wang, L. Zhan, P. Thompson, and J. Zhou (2020) Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, External Links: Document Cited by: §2. [18] S. Woo, S. Lee, Y. Park, M. A. Nugroho, and C. Kim (2023) Towards good practices for missing modality robust action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 2776–2784. External Links: Document Cited by: §2.