Paper deep dive
Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations
Ioannis Ziogas, Ensieh Khazaei, Bilal Taha, Aamna Al Shehhi, Ahsan H. Khandoker, Leontios J. Hadjileontiadis, Dimitrios Hatzinakos
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning disentangled representations is a key requirement for developing versatile, general-purpose, and sustainable models in multi-modal wearable computing. However, existing approaches do not operate as full-stack wearable processors, i.e., they do not simultaneously address task-specific classification performance, disentangled and interpretable representation learning, fusion, and generative modeling of highly heterogeneous multi-modal time series. To address this gap, we introduce Omni-modal Variational Decomposition Autoencoders (OmniDecVAEs), a framework that efficiently learns multi-purpose representations in a unified and scalable manner from arbitrarily many modalities. OmniDecVAEs extend DecVAEs by learning modality-conditioned time-frequency latent subspaces through a multi-view self-supervised decomposition loss and a shared asymmetric autoencoder (AE) architecture. Results on a challenging omni-modal human activity recognition (HAR) setting with up to thirty modalities, demonstrate the ability of OmniDecVAEs to learn full-stack wearable representations. When compared to transformer-based and VAE-based methods, OmniDecVAEs full-stack disentangled representation properties lead to accuracy improvements of 1.01% and 6.75% in activity and identity recognition, respectively. Furthermore, OmniDecVAEs synthesize realistic omni-modal time-frequency data that manifest with enhanced reconstructions (mean absolute error improves by 76.84%) and distributional similarity between real and synthetic data (maximum mean discrepancy improves by 13.85%). Our results highlight OmniDecVAEs potential as a lightweight model suitable for intelligent edge wearables and clinical healthcare, unifying processing requirements and abilities in a single model, through its enhanced representational capacity, modality-invariant spatial complexity (4.1M parameters), and real-time latency.
Tags
Links
- Source: https://arxiv.org/abs/2608.07385v1
- Canonical: https://arxiv.org/abs/2608.07385v1
Trouble viewing inline? Open PDF directly →
Full Text
87,207 characters extracted from source content.
Expand or collapse full text
Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations Ioannis N. Ziogas Member, IEEE Ensieh Khazaei Bilal Taha Aamna Al Shehhi Ahsan H. Khandoker Member, IEEE Leontios J. Hadjileontiadis Member, IEEE Dimitrios Hatzinakos , IEEE Manuscript received . This research was funded by the Healthcare Engineering Innovation Group (HEIG), Khalifa University of Science and Technology (KU-HEIG). Ioannis N. Ziogas is with the Department of Biomedical Engineering and Biotechnology, Khalifa University, 127788 Abu Dhabi, UAE. Part of this work was conducted during his time as a Visiting Graduate Researcher at the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada (e-mail: ioannis.ziogas@ku.ac.ae).Ensieh Khazaei is with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada.Bilal Taha is with the MIT Media Lab and Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto.Aamna Al Shehhi is with the Department of Electrical Engineering, Khalifa University, Abu Dhabi, UAE.Ahsan H. Khandoker is with the Department of Biomedical Engineering and Biotechnology, Khalifa University, 127788 Abu Dhabi, UAE.Leontios J. Hadjileontiadis is with the Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, GR 54124 Thessaloniki, Greece, and with the Department of Biomedical Engineering and Biotechnology, Khalifa University, 127788 Abu Dhabi, UAE (e-mail: leontios@auth.gr, .hadjileontiadis@ku.ac.ae).Dimitrios Hatzinakos is with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada (e-mail: dimitris@comm.utoronto.ca) Abstract Learning disentangled representations is a key requirement for developing versatile, general-purpose, and sustainable models in multi-modal wearable computing. However, existing approaches do not operate as full-stack wearable processors, i.e., they do not simultaneously address task-specific classification performance, disentangled and interpretable representation learning, fusion, and generative modeling of highly heterogeneous multi-modal time series. To address this gap, we introduce Omni-modal Variational Decomposition Autoencoders (OmniDecVAEs), a framework that efficiently learns multi-purpose representations in a unified and scalable manner from arbitrarily many modalities. OmniDecVAEs extend DecVAEs by learning modality-conditioned time-frequency latent subspaces through a multi-view self-supervised decomposition loss and a shared asymmetric autoencoder (AE) architecture. Results on a challenging omni-modal human activity recognition (HAR) setting with up to thirty modalities, demonstrate the ability of OmniDecVAEs to learn full-stack wearable representations. When compared to transformer-based and VAE-based methods, OmniDecVAEs full-stack disentangled representation properties lead to accuracy improvements of 1.01%1.01\% and 6.75%6.75\% in activity and identity recognition, respectively. Furthermore, OmniDecVAEs synthesize realistic omni-modal time-frequency data that manifest with enhanced reconstructions (mean absolute error improves by 76.84%76.84\%) and distributional similarity between real and synthetic data (maximum mean discrepancy improves by 13.85%13.85\%). Our results highlight OmniDecVAEs potential as a lightweight model suitable for intelligent edge wearables and clinical healthcare, unifying processing requirements and abilities in a single model, through its enhanced representational capacity, modality-invariant spatial complexity (4.1M4.1M parameters), and real-time latency. Our code will be made available at https://github.com/GiannisZgs/OmniDecVAEs. IEEEkeywords Omni-modal, multi-modal, wearable computing, disentanglement, variational autoencoders (VAEs), variational decomposition autoencoders (DecVAEs), omni-modal DecVAEs (OmniDecVAEs), human activity recognition (HAR) 1 Introduction Wearable sensing systems are increasingly generating large-scale, heterogeneous multi-modal data, requiring robust and expressive representation learning for applications in healthcare, human-computer interaction, and biometric security [23]. Recent advances in sensing hardware enable the integration of diverse physiological and behavioral modalities within unified wearable platforms, resulting in increasingly complex multi-modal data representations. This evolution motivates a transition from conventional multi-modality to omni-modality, defined as the unification of arbitrarily many heterogeneous data streams into universal and expressive representations through a single model [39]. In supervised multi-modal approaches, large amounts of labeled data are required, often leveraging transformer [30] or state-space models [2] to capture long-range dependencies across time and modality dimensions. However, fusing heterogeneous streams such as inertial, physiological, and behavioral data [10] requires significant per-modality design considerations. Even for closely related modalities fusion often necessitates explicit allocation of latent capacity to model intra-modal and cross-modal interactions [13]. This challenge is amplified when combining disparate modalities, such as video and inertial data [17], or biosignals and audio [37]. While supervised approaches can achieve strong performance on specific tasks, their reliance on labeled data often limits generalization across unseen conditions, tasks, and modality configurations [36]. Self-supervised learning (SSL) has emerged as a powerful alternative, enabling the learning of informative and transferable representations with minimal or no annotations [5]. In unimodal wearable sensing, SSL methods typically rely on contrastive objectives to distinguish between activities [38], or reconstruction-based objectives that improve robustness through input perturbations [29]. However, extending SSL to multi-modal wearable settings requires explicit modeling of cross-modal interactions, often through architectural modifications [41], graph-based formulations [42], or carefully designed multi-modal learning objectives [17]. Despite these advances, existing approaches primarily focus on task-specific performance and do not jointly address disentanglement and generative modeling. Autoencoder (AE)-based methods [16, 34, 17] and generative adversarial networks (GANs) [8] promote disentangled representations, but typically rely on modality-specific encoders or decoders [34, 40]. Disentangled representations are particularly important in wearable sensing, as they enable applications such as biometric privacy and identity-aware modeling [34]. To the best of our knowledge, no existing approach jointly addresses scalable multi-modal fusion, disentanglement, and generation within a single unified framework. Motivated by these challenges, we introduce OmniDecVAE, a structure-aware learning framework for omni-modal wearable computing that learns full-stack representations. These representations are multi-modal, unsupervised, and characterized by strong generalization, task informativeness, disentanglement, and generative capabilities. OmniDecVAE is a self-supervised, AE-based model that treats individual modality streams as components of a unified system through an adversarial contrastive objective inspired by signal decomposition dynamics [43]. The model processes omni-modal inputs—up to thirty modalities in our experiments—including inertial, physiological, and audio signals. All inputs are transformed into the time-frequency (TF) domain using the short-time Fourier transform (STFT) and Mel-scale representations for audio. A shared convolutional encoder learns modality-specific latent subspaces, while a shared fully-connected (FC) decoder establishes sample-level correspondence and reconstructs multi-modal signals. The variational formulation enforces a Gaussian latent structure, enabling stochastic generation of unseen samples. Following pre-training, the learned representations are evaluated using conventional supervised classifiers. We assess OmniDecVAE on HARWE [10], a large-scale multi-modal dataset for human activity and identity recognition. Experimental results demonstrate that OmniDecVAE achieves strong performance across recognition, generation, and computational efficiency benchmarks, outperforming supervised transformer-based fusion methods [10], VAE-based models[25],[20], and classical approaches such as Independent Component Analysis (ICA)[21] and Principal Component Analysis (PCA)[32]. These capabilities potentiate OmniDecVAEs as a foundational paradigm for next generation models in intelligent wearable and clinical healthcare. First, modality-invariance promotes resilience in fast changing medical environments where equipment is upgraded constantly, requiring intelligent AI to adapt accordingly. In turn, disentangled representations provide an interpretable and transparent information carrier for medical applications, as isolation of identity information is critical for patient-centric healthcare. In addition, disentanglement allows for explicit control of the identity information, thus enabling biometric security applications and anonymization by operating directly on the unified disentangled representation. Finally, the generative capability can improve sustainability in healthcare and in-the-wild deployments by enabling sensor reduction, reconstruction of missing data, and synthesis of multi-modal signals. In this work, we make the following contributions: • We propose a novel multi-modal SSL objective that enables scalable fusion of arbitrarily many modalities within a structured latent space, using a modality-agnostic architecture. • We introduce a unified architecture that scales to a large number of modalities through a shared encoder, without relying on transformer-based designs, and enables multi-modal TF data generation through a shared decoder. • We learn a disentangled latent representation where modality, activity, and identity factors are well structured, leading to markedly improved performance in downstream recognition tasks. The remainder of this paper is organized as follows. Section 2 reviews related work in multi-modal fusion, SSL, and generative modeling. Section 3 presents the proposed OmniDecVAE framework. Section 4 describes the experimental setup. Section 5 reports and discusses the results. Finally, Section 6 concludes the paper. 2 Related Work 2.1 Multi-modal Fusion in Wearable Computing Effective fusion methods must capture cross-modal interactions while preserving modality-specific characteristics in heterogeneous data streams. A common approach is concatenation-based fusion, which aggregates modalities either at the input (early fusion) or within latent representations (late fusion), typically relying on separate feature extractors per modality. For example, Kumar et al. [33] employ late fusion TF representations and convolutional branches. Transformer-based architectures provide a more flexible alternative by modeling intra- and inter-modal dependencies through attention mechanisms. However, these approaches often still depend on modality-specific feature extraction pipelines. For instance, Dissanayake et al. [9] combine Convolutional Neural Networks (CNN)-based feature extractors with late fusion, while Wu et al. [37] employ a transformer module for emotion recognition. To enhance cross-modal alignment, contrastive learning has been introduced in multimodal settings. Nguyen et al. [31] reduce the latent distance between modalities of the same sample to enforce coherence. Nevertheless, such approaches still rely on separate encoders per modality. Overall, existing fusion methods do not naturally scale to an arbitrary number of modalities without architectural modifications, limiting their applicability in omni-modal wearable settings. 2.2 Self-supervised Learning in Multi-modal Wearable Computing Wearable sensing scenarios are often constrained by limited annotations, motivating the use of SSL to learn representations from unlabeled data. SSL leverages surrogate objectives, such as reconstruction or contrastive learning, to extract meaningful structure from data [18].In unimodal wearable sensing, SSL approaches typically rely on contrastive objectives to discriminate between activities [38], or reconstruction-based objectives to improve robustness through perturbations [29]. In multi-modal settings, SSL has also been used to facilitate fusion. For example, Dissanayake et al. [9] and Wu et al. [37] employ augmentation prediction tasks as auxiliary objectives. More advanced approaches, such as TCLHAR [7], construct contrastive pairs across temporal neighborhoods and perform fusion using CNN-based architectures. Similarly, He et al. [17] align modalities through reconstruction objectives across separate branches. However, in most existing methods, SSL objectives primarily enhance intra-modal representations, while cross-modal fusion is handled through architectural design rather than the learning objective itself. 2.3 Multi-modal Generation through Disentangled Representations in Wearable Computing Disentangled representation learning aims to separate underlying factors of variation in high-dimensional data [6], which is particularly relevant in wearable sensing for decoupling identity from activity [34]. AE-based approaches have been used to achieve disentanglement in multi-modal settings. Han et al. [15] proposed a multi-branch architecture for reconstruction, adversarial subject invariance, and classification. However, their evaluation does not assess generative quality. Other methods are architecture-based disentanglement, such as liquid neural networks for single-modal physiological signals [40], or separate decoder branches for identity disentanglement in electromyography-based systems [34]. GANs have also been employed for modality transfer between related signals, such as ECG and PCG [8, 24]. While these methods achieve high-quality signal synthesis, they often do not evaluate the utility of the learned latent representations for downstream tasks. Figure 1: Overview of OmniDecVAEs and MMVDA, the proposed omni-modal full-stack wearable disentangled representation learning framework. (a) Three-step generative process θ: a latent event associated with activity and identity variables is generated, decomposed into M latent modality variables, and expressed through wearable sensor measurements across physiological, inertial, and acoustic domains. (b) MMVDA instantiated through OmniDecVAE aims to learn, via the inference model ϕφ, a reverse mapping from inputs X to a modality-separated latent space Z, ultimately recovering the underlying activity- and identity-conditioned latent event. The interplay between the encoder ϕφ, the SSLDec structure-aware latent objective, and the decoder θ enables effective disentanglement, supports downstream classification tasks, and facilitates data synthesis. 3 Proposed Method 3.1 Problem Statement Let X=x1,x2,…,xN∈ℝT×MX=\x_1,x_2,...,x_N\ ^T× M denote a multivariate time-series dataset with N i.i.d. observations, where T represents the temporal dimension and M denotes the number of modalities. Each observation xn∈ℝT×Mx_n ^T× M is assumed to be generated by a three-step generative process θ, involving a global latent variable z and a set of modality-specific latent variables z1,z2,…,zM\z_1,z_2,...,z_M\ that act as components of z. In the multi-sensor wearable setting, different sensors capture the same underlying phenomenon from distinct perspectives (e.g., physiological, inertial, and acoustic), as illustrated in Fig. 1a. We interpret the latent variable z as a sensing event, while each zmz_m represents a modality-specific view of that event. We further assume that each latent component zmz_m encodes unique information that is orthogonal to the other components. Under this assumption, the relationships between latent variables are defined as: ⟨zi,zj⟩=∑zizj∗=0,∀i≠j z_i,z_j =Σ z_iz_j^*=0, ∀ i≠ j (1) z=∑i=1Mziz= _i=1^Mz_i (2) where ⟨⋅,⋅⟩ ·,· denotes the dot product in latent space Z and (⋅)∗(·)^* denotes the complex conjugate. These equations describe an idealized setting in which modality-specific latent variables are mutually independent (zero correlation), and their aggregation fully reconstructs the underlying latent event z. This formulation naturally motivates an omni-modal setting, where increasing the number of sensing modalities can lead to progressively richer representations of the latent event. In the traditional variational decomposition autoencoding (VDA) framework [43], only a univariate signal X∈ℝTX ^T is observed. VDA employs a decomposition model to estimate intermediate components C embedded within X, and subsequently learns latent subspaces z1,z2,…,zC\z_1,z_2,...,z_C\. The final latent representation is then obtained by aggregating these subspaces. In contrast, in the multi-modal setting considered here, the dataset is already decomposed into its constituent modalities, i.e., X=X1,X2,…,XMX=\X_1,X_2,...,X_M\. Therefore, instead of performing decomposition, the objective becomes to learn a composition model that reconstructs how the latent event z is expressed in the input space. Method Overview. The proposed omni-modal framework outlined across the following sections can be summarized as follows: (1) Input: given a multi-modal observation x=x1,…,xMx=\x_1,…,x_M\, the anchor x0=∑i=1Mxix_0= _i=1^Mx_i is constructed. (2) Encoding: multi-modal observations xix_i and the anchor x0x_0 are fed to a shared encoder fϕ(⋅)f_φ(·) to obtain latent representations hih_i and h0h_0. (3) Latent projection: intermediate representations hih_i and h0h_0 are mapped to modality-specific latent variables zi∼qϕ(zi∣x,z ),i∈0,Mz_i q_φ(z_i x,z_ i),i∈\0,M\. (4) Self-supervised decomposition: the objective in Eq.10 enforces (i) alignment of hih_i with the anchor h0h_0 via Llat_reconL_lat\_recon (Eq.8) and (i) orthogonality via LorthoL_ortho (Eq.9), using omni-modal weighting (Eq.12f). (5) Latent aggregation: modality-specific subspaces are combined into the full latent representation z=[z0,z1,z2,…,zM]z=[z_0,z_1,z_2,...,z_M]. (6) Decoding: latent variables zi,i∈0,Mz_i,i∈\0,M\ are sampled via reparameterization (Eq.16) and reconstruct inputs using the conditional decoder pθ(x∣z,c)p_θ(x z,c). (7) Optimization: the model is trained using the (conditional) DELBO objective (Eqs.14,17). 3.2 Multi-modal Variational Decomposition Autoencoding We extend variational decomposition autoencoding (VDA) to the multi-modal setting, referred to as multi-modal VDA (MMVDA), by instantiating a three-step generative process θ that bypasses the need for an explicit decomposition model. As illustrated in Fig. 1, for each observation xnx_n, the generative process θ produces modality-specific components xn1,xn2,…,xnM\x_n1,x_n2,...,x_nM\ from latent variables z1,z2,…,zM\z_1,z_2,...,z_M\, which are sampled from prior distributions p(z1),p(z2),…,p(zM)\p(z_1),p(z_2),...,p(z_M)\. The aggregated observation in the input space, denoted as x0x_0, is then generated through a conditional distribution: pθ(x0∣z1,z2,…,zM)≡pθ(x∣z).p_θ(x_0 z_1,z_2,...,z_M)≡ p_θ(x z). (3) The parameters of the generative process θ and the latent variables z,z1,…,zM\z,z_1,...,z_M\ are unknown and must be inferred. The joint distribution of the generative process is defined as: pθ(X,Z1,…,ZM) p_θ(X,Z_1,...,Z_M) =∏n=1Npθ(xn∣z1n,…,zMn)pθ(z1n)⋯pθ(zMn) = _n=1^Np_θ(x^n z_1^n,...,z_M^n)\,p_θ(z_1^n)·s p_θ(z_M^n) (4a) pθ(zn) p_θ(z^n) =pθ(z1n)⋯pθ(zMn) =p_θ(z_1^n)·s p_θ(z_M^n) (4b) pθ(xn∣zn) p_θ(x^n z^n) =pθ(xn∣z1n,…,zMn), =p_θ(x^n z_1^n,...,z_M^n), (4c) where Eq. (4b) expresses the prior over the latent event as a product of modality-specific priors. Each term in Eq. (4a) is modeled as: pθ(x∣z1,…,zM) p_θ(x z_1,.,z_M) =(x∣0,diag(z1,…,zM)) =N(x 0,diag(z_1,.,z_M)) (5) pθ(zi) p_θ(z_i) =(zi∣0,σzi2I),∀i∈1,…,M =N(z_i 0, _z_i^2I), ∀ i∈\1,.,M\ where θ parameterizes the generative model and each prior pθ(zi)p_θ(z_i) is an isotropic multivariate Gaussian with zero mean [43]. The conditional distribution pθ(x∣z1,…,zM)p_θ(x z_1,...,z_M) assumes a diagonal covariance structure, reflecting the orthogonality of the latent subspaces ziz_i. Since the true posterior pθ(z∣x)p_θ(z x) is intractable [43, 25], we introduce a recognition model qϕ(z∣x)q_φ(z x) to approximate it. The approximate posterior over the dataset is expressed as: qϕ q_φ (Z∣X)=∏n=1Nqϕ(z1n∣xn,z2n,…,zMn)⋯ (Z X)= _n=1^Nq_φ(z_1^n x_n,z_2^n,.,z_M^n)\,·s (6) ⋯qϕ(z2n∣xn,z1n,z3n,…,zMn)qϕ(zMn∣xn,z1n,…,zM−1n) ·s\,q_φ(z_2^n x_n,z_1^n,z_3^n,.,z_M^n)\,q_φ(z_M^n x_n,z_1^n,.,z_M-1^n) Each conditional factor in Eq. (6) is modeled as: qϕ(zi∣x,z∖i)=(zi∣fμzi(x,z∖i),fσzi2(x,z∖i))q_φ(z_i x,z_ i)=N (z_i f_ _z_i(x,z_ i),f_σ^2_z_i(x,z_ i) ) (7) where z∖iz_ i denotes all latent variables except ziz_i, and each posterior distribution is modeled as an isotropic multivariate Gaussian [43]. As in VDA, the generative model defined in Eqs. (4)–(5) is not explicitly factorized to enforce the reconstruction constraints described in Eqs. (1) and (2). To avoid the need for separate inference models per modality, MMVDA employs shared generative and recognition networks (θ,ϕ)(θ,φ), while enforcing the desired structure in the latent space through a self-supervised decomposition loss. 3.3 Multi-modal Multiple Views Self-supervised Latent Decomposition Contrastive Learning To enable omni-modal latent representation learning, we employ the self-supervised latent decomposition (SSLDec) loss introduced in DecVAEs [43]. SSLDec operationalizes the assumptions introduced in Section 3.1, namely that modality-specific components form a composite system and exhibit orthogonality in the latent generative space (Eqs. 1, 2). Therefore, SSLDec is used to enforce these structural constraints in the input and latent spaces through a self-supervised objective defined as: Llat_recon=∑i=1MwposiDKL(DJS(hi,h0)∥p), L_lat\_recon= _i=1^Mw^i_pos\,D_KL (D_JS(h_i,h_0)\,\|\,p ), (8) where p(y)=ϵ,∀y∈1,…,d,H∈ℝd×N p(y)=ε,\ ∀ y∈\1,.,d\,\ H ^d× N Lortho=∑i=1M∑j=i+1Mwneg(i,j)DKL(DJS(hi,hj)∥n), L_ortho= _i=1^M _j=i+1^Mw^(i,j)_neg\,D_KL (D_JS(h_i,h_j)\,\|\,n ), (9) where n(y)=1,∀y∈1,…,d n(y)=1,\ ∀ y∈\1,.,d\ ℒSSLDec=Llat_recon+LorthoL_SSLDec=L_lat\_recon+L_ortho (10) where wposiw^i_pos and wneg(i,j)w^(i,j)_neg are weighting coefficients that control the contribution of positive and negative interactions, respectively. DKLD_KL denotes the KLD and DJSD_JS denotes the Jensen–Shannon divergence. The target distributions p and n are uniform, with ϵε representing a small positive constant (i.e. 1e−61e-6). The SSLDec objective as defined by Eqs.(8,9,10) is a multi-view contrastive learning formulation with an adversarial structure that promotes disentanglement. The latent reconstruction term Llat_reconL_lat\_recon minimizes the divergence between each modality-specific representation hih_i and an anchor representation h0h_0, encouraging alignment across modalities. In contrast, the orthogonality term LorthoL_ortho maximizes the divergence between all pairs of modality representations, promoting separation and reducing redundancy. The use of KLD formulates this adversarial objective into stable cross-entropy terms and mitigates representation collapse [22]. Notably, SSLDec is computed in an intermediate latent space h, which is derived from the primary latent space z through a latent generative mapping rψr_ψ, where ψ⊂ϕψ⊂φ [43]. Although SSLDec inherently supports multi-modal interactions, its effectiveness depends on the quality of the anchor representation h0h_0 used in the Llat_reconL_lat\_recon term. In classical VDA, h0h_0 is derived from a unimodal input, while hih_i correspond to decomposed components of that input. In the multi-modal setting considered here, each hih_i corresponds to a distinct modality, and thus an explicit multi-modal anchor must be constructed. We define the anchor in the input space as: x0=∑i=1Mxix_0= _i=1^Mx_i (11) and obtain the corresponding latent representation as h0=f(x0)h_0=f(x_0) using the recognition model ϕφ. 3.4 Increasing the Number of Contrastive Pairs In real-world wearable sensing systems, each modality often consists of multiple channels. For example, inertial sensors are tri-axial, and different sensing modalities (e.g., accelerometer (ACC) and gyroscope (GYR), or physiological signals such as electrodermal activity (EDA) and blood volume pulse (BVP)) may exhibit varying degrees of correlation. Consequently, interactions between signals are not uniform: channels originating from the same sensor or related sensing modalities tend to share stronger relationships than those from unrelated sources. The standard SSLDec formulation assumes uniform interactions between latent components. However, this assumption is not well suited for omni-modal wearable settings, where relationships between channels and modalities are inherently heterogeneous. To address this, we extend SSLDec into an asymmetric contrastive learning formulation, where the strength of interactions between pairs is explicitly controlled. To systematically model these interactions, we define the positive and negative weighting terms wposiw^i_pos and wneg(i,j)w^(i,j)_neg using an omni-modal vector VOMposV_OMpos and matrix WOMnegW_OMneg. These weights are designed to reflect the relative importance of different interactions across channels and modalities. We introduce imbalance factors IFposIF_pos and IFnegIF_neg to regulate the strength of weak and strong interactions between wearable channels and the anchor. The resulting weighting scheme is defined as follows: wpos−base=1∑iMCiw_pos-base= 1 _i^MC_i (12a) wpos−weak=(1−IFpos)∗wpos−basew_pos-weak=(1-IF_pos)*w_pos-base (12b) wpos−strong=(1+IFpos∗Npos−weakNpos−strong)∗wpos−basew_pos-strong=(1+IF_pos* N_pos-weakN_pos-strong)*w_pos-base (12c) wneg−base=1∑j∑iMCijw_neg-base= 1 _j _i^MC_ij (12d) wneg−weak=(1−IFneg)∗wneg−basew_neg-weak=(1-IF_neg)*w_neg-base (12e) wneg−strong=(1+IFneg∗Nneg−weakNneg−strong)∗wneg−basew_neg-strong=(1+IF_neg* N_neg-weakN_neg-strong)*w_neg-base (12f) where CiC_i denotes the number of channels in the i-th modality. The parameters IFposIF_pos and IFnegIF_neg control the relative scaling of weak and strong interactions and are experimentally set to 0.250.25 and 0.40.4, respectively. The terms Npos−weak,Npos−strongN_pos-weak,N_pos-strong denote the number of weak and strong positive interactions contributing to Llat_reconL_lat\_recon, while Nneg−weak,Nneg−strongN_neg-weak,N_neg-strong denote the corresponding quantities for negative interactions in LorthoL_ortho. The weighting scheme is guided by domain-specific relationships between modalities. Since the anchor representation is constructed as the sum of all input signals (Eq. 11), modalities with higher signal complexity or dimensionality, such as audio, tend to exert a stronger influence on the anchor. Therefore, stronger weights are assigned to anchor–modality interactions involving such modalities. For negative interactions, smaller weights are assigned to pairs of channels that are expected to be similar, such as physiological signals (e.g., BVP, EDA, temperature), corresponding axes of inertial sensors (e.g., X-axis of ACC and GYR), and channels originating from the same sensor (e.g., X, Y, Z axes of ACC). In contrast, dissimilar modalities are assigned stronger repulsive weights to encourage separation in the latent space. The omni-modal vector VOMposV_OMpos and upper-triangular matrix WOMnegW_OMneg are then given by: VOMpos=[wpos−weak,wpos−strong]∑iMCiV_OMpos=[w_pos-weak,w_pos-strong] _i^MC_i (13a) WOMneg= W_OMneg= (13b) [wneg−weakwneg−weakwneg−strong⋯wneg−strong⋯wneg−strong0wneg−weakwneg−strong⋯wneg−strong⋯wneg−strong00wneg−strong⋯wneg−strong⋯wneg−strong000⋯wneg−weak⋯wneg−strong⋮⋯⋱⋯⋮00000⋯wneg−strong] [ array[]@ 2ptc@ 2ptc@ 2ptc@ 2ptc@ 2ptc@ 2ptc@ 2ptc@ 2pt 2.0pt @intercol w_neg-weak 2.0&w_neg-weak 2.0&w_neg-strong 2.0&·s 2.0&w_neg-strong 2.0&·s 2.0&w_neg-strong 2.0\\ 2.0pt @intercol 0 2.0&w_neg-weak 2.0&w_neg-strong 2.0&·s 2.0&w_neg-strong 2.0&·s 2.0&w_neg-strong 2.0\\ 2.0pt @intercol 0 2.0&0 2.0&w_neg-strong 2.0&·s 2.0&w_neg-strong 2.0&·s 2.0&w_neg-strong 2.0\\ 2.0pt @intercol 0 2.0&0 2.0&0 2.0&·s 2.0&w_neg-weak 2.0&·s 2.0&w_neg-strong 2.0\\ 2.0pt @intercol 2.0&·s 2.0& 2.0& 2.0& 2.0&·s 2.0& 2.0\\ 2.0pt @intercol 0 2.0&0 2.0&0 2.0&0 2.0&0 2.0&·s 2.0&w_neg-strong 2.0\\ array ] AOM= _OM= (13c) [BVP−EDA⋯BVP−AccX⋯BVP−GyrX⋯BVP−AudioFn0EDA−TempEDA−AccX⋯EDA−GyrX⋯EDA−AudioFn00Temp−AccX⋯Temp−GyrX⋯Temp−AudioFn000⋯AccX−GyrX⋯AccX−AudioFn⋮⋯⋱⋯⋮0000⋯0AudioFn−1−AudioFn] [ array[]@ 1ptc@ 1ptc@ 1ptc@ 1ptc@ 1ptc@ 1ptc@ 1ptc@ 1.0pt @intercol \!-\!EDA 1.0&·s 1.0&BVP\!-\!Acc_X 1.0&·s 1.0&BVP\!-\!Gyr_X 1.0&·s 1.0&BVP\!-\!Audio_Fn\\ 1.0pt @intercol 0 1.0&EDA\!-\!Temp 1.0&EDA\!-\!Acc_X 1.0&·s 1.0&EDA\!-\!Gyr_X 1.0&·s 1.0&EDA\!-\!Audio_Fn\\ 1.0pt @intercol 0 1.0&0 1.0&Temp\!-\!Acc_X 1.0&·s 1.0&Temp\!-\!Gyr_X 1.0&·s 1.0&Temp\!-\!Audio_Fn\\ 1.0pt @intercol 0 1.0&0 1.0&0 1.0&·s 1.0&Acc_X\!-\!Gyr_X 1.0&·s 1.0&Acc_X\!-\!Audio_Fn\\ 1.0pt @intercol 1.0&·s 1.0& 1.0& 1.0& 1.0&·s 1.0& \\ 1.0pt @intercol 0 1.0&0 1.0&0 1.0&0 1.0&·s 1.0&0 1.0&Audio_F_n-1\!-\!Audio_F_n\\ array ] where AOM[∑iMCi×∑iMCi]A_OM^[ _i^MC_i× _i^MC_i] is the omni-modal interaction matrix that encodes pairwise relationships between channels and modalities, and can be mapped directly to the weights in WOMnegW_OMneg. This formulation also allows the decomposition of complex modalities such as audio into finer-grained components (e.g., frequency channels or oscillatory components (OCs) AudioFi,i∈1,…,NAudio_F_i,i∈\1,...,N\), further increasing the number of contrastive pairs. Because the SSLDec objective is parameterized by VOMposV_OMpos, WOMnegW_OMneg, and Eq. 10, the proposed framework can scale to an arbitrarily large number of channels and modalities. This is achieved by extending the weighting structures to incorporate additional anchor and cross-modal relationships, without requiring modifications to the underlying model architecture. 3.5 Multi-modal Decomposition Evidence Lower Bound We instantiate MMVDA using a deep neural network architecture by parameterizing the generative and inference models pθp_θ and qϕq_φ with neural networks having parameters θ and ϕφ, respectively. The posterior latent Gaussian distributions fμzi(x,zj≠i)f_ _z_i(x,z_j≠ i) and fσzi2(x,zj≠i)f_σ^2_z_i(x,z_j≠ i), for i,j∈1,…,Mi,j∈\1,...,M\, are also modeled as neural networks. As in VAEs [25] and DecVAEs [43], the structural assumptions introduced in previous sections must be incorporated into an objective function that provides a tractable evidence lower bound (ELBO) for the inference model qϕq_φ with respect to the marginal likelihood of an observation xnx_n. To this end, we adapt the decomposition ELBO (DELBO) formulation of DecVAEs [43], extending it to accommodate omni-modal weighting through VOMposV_OMpos and WOMnegW_OMneg. When only the encoder qϕq_φ is used (i.e., without an explicit decoder), the DELBO objective ℒDELBOEncL_DELBO_Enc for an observation xnx_n with latent representations hn,znh_n,z_n is defined as: ℒDELBOEnc(ϕ;xn,zn,VOMpos,WOMneg,β) _DELBO_Enc(φ;x_n,z_n,V_OMpos,W_OMneg,β) (14) =∑iVOMpos(i)z∼qϕ(zn|xn,hn)[log(rψ⊂ϕ(hin|zn))] = _iV_OMpos^(i)E_z q_φ(z_n|x_n,h_n)[log(r_ψ⊂φ(h_i_n|z_n))] −∑i∑jWOMneg(i,j)DKL(hin,hjn) - _i _jW_OMneg^(i,j)D_KL(h_i_n,h_j_n) −β∑iDKL(qϕ(zn|xn,hin)||p(zn))+const. -β _iD_KL(q_φ(z_n|x_n,h_i_n)||p(z_n))+const. =Llat_recon−Lortho−βLprior+const. =L_lat\_recon-L_ortho-β L_prior+const. where rψr_ψ is the latent generative model defined in DecVAEs [43], and β controls the strength of the prior regularization term LpriorL_prior, following the β-VAE formulation [20]. When a decoder (generative model pθp_θ) is also included, the objective becomes: ℒDELBOEnc−Dec(θ,ϕ;xn,VOMpos,WOMneg,β) _DELBO_Enc-Dec(θ,φ;x_n,V_OMpos,W_OMneg,β) (15) =z∼qϕ(zn|xn)[log(pθ(xn|zn))] =E_z q_φ(z_n|x_n)[log(p_θ(x_n|z_n))] −∑iVOMpos(i)DKL(hi,h0) - _iV_OMpos^(i)D_KL(h_i,h_0) −∑i∑jWOMneg(i,j)DKL(hi,hj) - _i _jW_OMneg^(i,j)D_KL(h_i,h_j) −β∑iDKL(qϕ(zn|xn)||p(zn))+const. -β _iD_KL(q_φ(z_n|x_n)||p(z_n))+const. =z∼qϕ(zn|xn)[log(pθ(xn|zn))] =E_z q_φ(z_n|x_n)[log(p_θ(x_n|z_n))] −Llat_recon−Lortho−βLprior. -L_lat\_recon-L_ortho-β L_prior. In this formulation, the first term corresponds to the reconstruction likelihood of the generative model, replacing the latent generative component used in the encoder-only setting. The terms Llat_reconL_lat\_recon and LorthoL_ortho retain their role in enforcing alignment and orthogonality in the latent space, respectively. In our experiments, we evaluate both ℒDELBOEncL_DELBO_Enc and ℒDELBOEnc−DecL_DELBO_Enc-Dec. The objectives in Eqs. 14 and 15 encode the structural biases of MMVDA, namely the decomposition of representations into orthogonal modality-specific subspaces anchored to a shared multi-modal context, together with Gaussian regularization. Figure 2: Neural network architecture of the proposed OmniDecVAE scheme, that applies MMVDA. 3.6 OmniDecVAE: Omni-modal Variational Decomposition Autoencoder Fig. 2 illustrates the OmniDecVAE architecture, which instantiates the MMVDA framework. TF domain input streams Xf×(1+∑iMCi)X^f×(1+ _i^MC_i), corresponding to all modality channels and the anchor signal (Eq. 11), are processed by a shared single-branch encoder fW(⋅)f_W(·). The encoder is a seven-layer one-dimensional CNN based on Wav2Vec2 [4], equipped with LayerNorm normalization [3] and GELU activations [19]. It projects the inputs into an intermediate representation space Vv×(M+1)V^v×(M+1). A shared FC projection head fH(⋅)f_H(·) further maps these representations into a latent interaction space Hd×(M+1)H^d×(M+1), where the SSLDec objective (Eq. 10) is applied. Subsequently, modality-specific latent subspaces ZiZ_i are obtained via M+1M+1 FC projection layers that output the mean μ and variance logvarlogvar parameters of Gaussian distributions. The prior regularization term LpriorL_prior is independently computed for each modality subspace. An aggregation function g(⋅)g(·) then concatenates these subspaces into the final disentangled representation Zzdim×(M+1)Z^z_dim×(M+1). A key property of OmniDecVAE is that multi-modality is handled entirely through the SSLDec objective (Eq. 10), allowing the network architecture to remain simple and modality-agnostic. No modality-specific encoder branches are required, and modality-specific structure is only enforced at the level of the latent projections. A hyperparameter lSSLDec%l_SSLDec\% controls the percentage of latent frames used in the computation of Eqs. (14, 15) via random masking, acting as a form of regularization. In our experiments, the encoder-only variant of the model is denoted as OmniDecVAEncOmniDecVAE_Enc and is optimized using Eq. 14. 3.7 Asymmetric Modality-Conditioned Shared Decoder To enable multi-modal data generation, the OmniDecVAE architecture incorporates a decoder network (Fig. 2). The generative model θ is implemented as a FC network, resulting in an asymmetric autoencoder design. Asymmetric architectures, such as masked autoencoders (MAEs) [18], have demonstrated strong performance by leveraging self-supervised objectives in the latent space, reconstructing inputs from partially observed latent representations. This design allows the encoder to remain expressive while keeping the decoder lightweight. In our case, the asymmetric decoder maps both masked and unmasked latent representations from Z→XZ→ X. Notably, the anchor latent z0z_0 is not provided as input to the decoder, ensuring that reconstruction relies solely on modality-specific latent components. The decoder is instantiated as a shared four-layer FC network with LayerNorm normalization [3] and GELU activations [19], with no non-linearity at the output layer. To enable sampling from the posterior distributions qϕ(zi∣x)q_φ(z_i x), we employ the reparameterization trick [25]. Specifically, for each modality-channel latent subspace, we sample: zi~=μzi+τ∗σzi2∗ϵ,i∈1,…,M, z_i= _z_i+τ*σ^2_z_i*ε, i∈\1,.,M\, (16) wherezi∼(μ,σ2),ϵ∼(0,1) \ \ z_i (μ,σ^2),\ ε (0,1) where τ is a scaling factor that controls the influence of the stochastic term ϵε on the sampled representation zi~ z_i. Empirically, we observe that as the number of modalities increases, smaller values of τ in the range [0.1,1.0][0.1,1.0] lead to more stable training. To support expressive multi-modal generation within a single shared decoder, we introduce a conditional embedding c that encodes the specific channel–modality combination. This embedding acts as a control signal, providing the decoder with information about which modality-channel pair to reconstruct. The decoder input is formed as: LayerNorm([z~i,c])LayerNorm([ z_i,c]) where concatenation is followed by normalization. The decoder outputs reconstructed signals X~f×(∑iMCi) X^f×( _i^MC_i), where f denotes the dimensionality of the flattened TF-domain representation. Given the conditional nature of the decoder, the training objective becomes the conditional DELBO: ℒcDELBOEnc−Dec(θ,ϕ;xn,VOMpos,WOMneg,β,c) _cDELBO_Enc-Dec(θ,φ;x_n,V_OMpos,W_OMneg,β,c) (17) =z∼qϕ(zn|xn)[log(pθ(xn|zn,c))] =E_z q_φ(z_n|x_n)[log(p_θ(x_n|z_n,c))] −Llat_recon−Lortho−βLprior -L_lat\_recon-L_ortho-β L_prior In practice, for an input observation xnx_n and its reconstruction yny_n, the decoder is optimized using a smoothed mean absolute error (sMAE) loss: Lrecon=0.5(xn−yn)2,if|xn−yn|<1|xn−yn|−0.5,otherwiseL_recon= \ array[]l0.5(x_n-y_n)^2,if|x_n-y_n|<1\\ |x_n-y_n|-0.5,otherwise array . (18) Table 1: OmniDecVAE Hyperparameters Model Part Parameter Value Encoder number of channels in the convolution layers 512 kernel sizes for each convolution layer [8,3,3,2,2] kernel strides for each convolution layer [4,4,3,2,2] output of convolutional layers dimension 512 weight initialization strategy He Hidden and Latent Projections dimension of the projecting HiH_i and latent subspaces ZiZ_i 192 intermediate dimension in the hidden projection layers 192 dimensionality per ZiZ_i latent subspace 32 weight initialization strategy Xavier Decoder dimensions of fully-connected hidden projection layers [128,256,512,512] values for τ in Eq. 16 when number of channels C is [3,6,9,21,24,27,30] [1,0.8,0.5,0.1,0.1,0.25,0.15] conditional embedding c dimension 8 weight initialization strategy Xavier Optimization and training batch size for training 64 percentage of frames used in the SSLDec loss calculation Eq.(10) 50 peak learning rates for training with [Eq.(14),Eq.(17)] [8∗10−58*10^-5, 1∗10−51*10^-5] learning schedule for training with [Eq.(14),Eq.(17)] [constant with warmup, linear] learning rate warmup steps for training with [Eq.(14),Eq.(17)] [4320, 15120] b1 parameter of the Adam optimizer 0.5 b2 parameter of the Adam optimizer 0.999 L2 regularization penalty on large weights 10−410^-4 epsilon parameter of the Adam optimizer 10−610^-6 clip gradients if exceeding this maximum value 1 4 Experiments 4.1 Dataset To evaluate the OmniDecVAE framework, we utilize HARWE [10], a large-scale multi-modal HAR dataset. HARWE consists of video, audio, inertial and physiological recordings from thirty-five participants performing nine different daily activities in work environments: Walking around office, Talking to phone, Walking around office and talking to phone, Browsing monitor, Reading book, Writing on paper, Writing on paper and talking to phone, Typing on keyboard, Stretching on chair. In this work we do not use the video modality; instead we utilize the remaining time-series modalities and their channels as follows: • Audio: monaural audio sampled at 16kHz16kHz; we utilize the Filter Decomposition (FD) [43] to separate acoustic events into six frequency band-delimited OCs, thereby setting the number of audio channels to six. • Physiology: three physiological signals are captured through a smartwatch device, namely BVP (64Hz64Hz), EDA (4Hz4Hz), and Temperature (TEMP) (64Hz64Hz) modalities. • Inertial sensors: seven inertial measurements are sampled at 200Hz200Hz through a smartphone device, i.e., ACC, GYR, Gravity (GRA), Orientation (OR), Rotation (RO) Magnetometer (MG), Linear accelerometer (LACC). Each of these measurements is tri-axial, hence the number of channels for the inertial modalities is twenty-one. Each recording is accompanied by an activity annotation with a duration of approximately 10 min per activity. The HARWE dataset evaluation consists of two different partitioning schemes, Easy and Difficult [10]. The Easy scenario is a subject-dependent (SD) evaluation, where 80%80\% of each subject’s recordings are kept as the training set, leaving the remaining 20%20\% for testing. The Difficult scenario is a subject-independent (SI) scenario where 70%70\% of the subjects are kept as the training set, and the remaining 30%30\% as the testing set. In the Easy scenario, 14400 and 3776 samples are contained in the training and testing sets, respectively, whereas 12160 and 5952 samples are contained in Difficult scenario training and testing sets, respectively. 4.2 Comparisons and Benchmarks We compare the OmniDecVAE framework to a number of different approaches used on HARWE [10], standard benchmark approaches in the multi-modal fusion domain [11], [28], AE-based models [25], [20], [43] and eigenprojection models ICA [21], PCA [14], and kernel-PCA variants [32]. We also instantiate a multi-modal VAE model (MMVAE), by using the same architecture as in Fig.2, but with modality-dedicated branches instead of a shared branch. 4.3 Implementation and Optimization We follow the pre-processing of all modalities in HARWE according to Esmailzehi et al. [10]; then we calculate the STFT TF-domain representation of all physiological and inertial modalities in 3s3s frames. We use a 512-point STFT in the frequency domain with a 0.750.75 s Hamming window, with an overlap of 75%75\%, resulting in a spectrogram of size 64×364× 3 that is flattened to a final size of 192×1192× 1 and given as input to the 1D CNN extractor. For the audio signals, a Mel spectrogram is calculated with 64 Mel scales and 33 hops. For calculating the anchor signal (Eq. 11), we first superpose all modalities in the time domain before taking a TF-domain transform; in the case of mixing acoustic with inertial or physiological modalities, we take the Mel transform for the anchor, resulting in a cumulative “sonified” Mel representation of acoustic, physiological and inertial signals. Finally, all signals are normalized using batch statistics. Detailed hyperparameters and their values for our OmniDecVAE architectures are given in Table 1. We optimize our networks on a single NVIDIA RTX 6000 Ada Generation using PyTorch [1] and Huggingface [35] using the Adam optimizer [26] and train for approximately 120 epochs. Other VAE-based models are trained for 100 epochs as their optimization converges faster. Our learning scheme consists of a pre-training stage, where the model is trained to optimize one of Eqs. (14,15,17), and a post-training stage where a simple Support Vector Machine (SVM) classifier [12]) is utilized as a classification oracle on the learned representations. For combined CNN-Transformer-based fusion, we use the training schemes for supervised learning from Esmailzehi’s et al. [10]. 4.4 Evaluation Metrics 4.4.1 Representation Informativeness HARWE allows us to effectively perform the supervised classification tasks of HAR and identity recognition (IR). We evaluate these tasks using Accuracy, F1-score (F1) and Macro-averaged F1-score (MF1). For the SD scenario we evaluate for both HAR and IR, whereas in the SI scenario we evaluate only for HAR. 4.4.2 Generation Quality In the first generative experiment we sample from a latent noise distribution using the learned conditional subspaces and generate a distribution of multi-modal signals YgenY_gen. We then sample a subset of the testing set XrealX_real and evaluate distributional similarity between these two subsets using the following metrics: • Multi-Kernel-Maximum Mean Discrepancy (MK-MMD): uses multiple radial basis function (RBF) kernels K over multiple bandwidths σ, and averages individual MMDs between real and generated subsets, over those kernels: MK−MMDRBF2(Xreal,Ygen)=1K∑kK[1n2∑i,i′k(xi,xi′)+1m2∑j,j′k(yj,yj′)−2nm∑i,jk(xi,yj)]MK-MMD_RBF^2(X_real,Y_gen)= 1K _k^K[ 1n^2 _i,i k(x_i,x_i )+ 1m^2 _j,j k(y_j,y_j )- 2nm _i,jk(x_i,y_j)] • Diversity Score: this metric measures pairwise Euclidean distances dreal,dgend_real,d_gen within the sets Xreal,YgenX_real,Y_gen, and calculates DivScore=|dreal−dgen|DivScore=|d_real-d_gen|. In the second experiment, we evaluate the actual reconstruction ability of the decoder w.r.t. to input signals. Specifically, we compare inputs XnX_n with decoder reconstructions YnY_n using the following metrics: • Mean Squared Error (MSE): MSEn=1T∑t=1T(Xn[t]−Yn[t])2MSE_n= 1T _t=1^T(X_n[t]-Y_n[t])^2, over the T samples of the TF reconstructions Xn,YnX_n,Y_n. • Mean Absolute Error: MAEn=1T∑t=1T|Xn[t]−Yn[t]|MAE_n= 1T _t=1^T|X_n[t]-Y_n[t]| The reconstructed signals YnY_n are further fed into the OmniDecVAE encoder to obtain ZYnZ_Y_n and compared to the original latents ZnZ_n; the goal of the below metrics is to measure latent consistency: • Latent L2 distance: dL2=|Zn−ZYn|2d_L2=|Z_n-Z_Y_n|^2 • Latent cosine distance: dcos=ZnZYn‖Zn‖‖ZYn‖d_cos= Z_nZ_Y_n||Z_n||||Z_Y_n|| 4.4.3 Complexity Furthermore, to assess the suitability of our model for real-time processing and edge deployment potential, we measure floating-point operations (FLOPs), number of parameters (Param.), size on disk (SoD), and frames processed per second during inference or latency. 5 Results and Discussion Table 2: HARWE Omni-modal Disentangled Activity and Identity Recognition Performance - C=30C=30 Modality Channels Subject-Dependent (Easy) Subject-Independent (Difficult) Model HAR IR HAR Acc. ↑ F1 ↑ MF1 ↑ Acc. ↑ F1 ↑ MF1 ↑ Acc. ↑ F1 ↑ MF1 ↑ Random Classifier 11.18 ± 0.15 11.18 ± 0.15 11.17 ± 0.16 2.91 ± 0.10 2.91 ± 0.10 2.89 ± 0.09 11.07 ± 0.17 11.08 ± 0.16 11.06 ± 0.17 Majority Classifier 11.47 ± 0.00 2.36 ± 0.00 2.28 ± 0.00 3.36 ± 0.00 0.21 ± 0.00 0.19 ± 0.00 11.23 ± 0.00 2.27 ± 0.00 2.24 ± 0.00 Logistic Regression 38.07 ± 0.13 36.36 ± 0.20 36.22 ± 0.19 18.50 ± 0.35 17.72 ± 0.36 17.80 ± 0.36 38.33 ± 0.11 35.84 ± 0.17 35.85 ± 0.16 Random Forest 44.55 ± 0.16 43.39 ± 0.17 43.30 ± 0.17 25.18 ± 0.14 24.59 ± 0.13 24.67 ± 0.13 43.82 ± 0.13 41.69 ± 0.14 41.76 ± 0.13 SVM 42.64 ± 0.08 41.16 ± 0.09 41.05 ± 0.09 17.34 ± 0.10 16.12 ± 0.17 16.18 ± 0.09 43.09 ± 0.06 40.73 ± 0.06 40.77 ± 0.06 ICA [21] 77.00 ± 0.07 76.92 ± 0.08 77.22 ± 0.08 75.30 ± 0.10 75.44 ± 0.11 75.68 ± 0.10 62.06 ± 0.13 62.04 ± 0.15 62.73 ± 0.13 PCA [14] 77.00 ± 0.07 76.93 ± 0.08 77.23 ± 0.08 75.29 ± 0.10 75.42 ± 0.10 75.67 ± 0.10 62.06 ± 0.13 62.04 ± 0.13 62.73 ± 0.12 rbf-PCA [32] 11.72 ± 0.001 2.46 ± 0.001 2.33 ± 0.001 3.36 ± 0.001 0.21 ± 0.001 0.19 ± 0.001 11.63 ± 0.001 2.42 ± 0.001 2.31 ± 0.001 sigmoid-PCA [32] 17.76 ± 0.08 14.51 ± 0.08 14.47 ± 0.08 6.90 ± 0.07 5.25 ± 0.11 5.29 ± 0.10 15.32 ± 0.06 12.48 ± 0.08 12.63 ± 0.08 poly-PCA [32] 67.71 ± 0.07 67.34 ± 0.07 67.63 ± 0.07 51.42 ± 0.11 51.42 ± 0.10 51.54 ± 0.11 58.90 ± 0.09 58.33 ± 0.10 59.04 ± 0.10 YAMNet[11] & EEGNet[28] Late Fusion Single-Head 38.15 ± 5.40 30.26 ± 6.42 30.63 ± 6.37 50.22 ± 4.41 47.51 ± 4.59 47.31 ± 4.58 31.85 ± 3.77 24.19 ± 4.50 24.48 ± 4.49 YAMNet[11] & EEGNet[28] Late Fusion Dual-Head 49.88 ± 4.62 46.26 ± 5.26 46.71 ± 5.20 50.08 ± 4.32 47.06 ± 4.75 46.99 ± 4.79 NA NA NA YAMNet[11] & EEGNet[28] & Transformer Late Fusion Single-Head [10] 39.93 ± 6.03 35.70 ± 6.64 36.05 ± 6.70 67.36 ± 2.92 66.85 ± 3.06 66.60 ± 3.12 32.98 ± 2.32 28.30 ± 2.72 28.77 ± 2.69 YAMNet[11] & EEGNet[28] & Transformer Late Fusion Dual-Head [10] 53.92 ± 2.98 51.75 ± 3.23 52.17 ± 3.20 61.57 ± 3.55 60.13 ± 3.89 59.89 ± 3.87 NA NA NA MMVAE 83.55 ± 0.10 83.46 ± 0.11 83.67 ± 0.10 83.22 ± 0.15 83.26 ± 0.14 83.32 ± 0.14 63.06 ± 0.14 62.74 ± 0.12 63.46 ± 0.12 β-MMVAE (β=0.1β=0.1) 82.36 ± 0.13 82.31 ± 0.13 82.50 ± 0.13 79.42 ± 0.34 79.65 ± 0.35 79.77 ± 0.24 70.91 ± 0.10 70.37 ± 0.10 70.92 ± 0.09 OmniDecVAEEnc [43] 71.46 ± 1.13 71.26 ± 1.13 71.53 ± 1.14 62.12 ± 2.69 62.22 ± 2.73 62.43 ± 2.74 61.85 ± 0.41 61.48 ± 0.34 62.10 ± 0.34 β-OmniDecVAEEnc [43] (β=80β=80) 71.75 ± 0.93 71.55 ± 0.94 71.85 ± 0.94 63.08 ± 1.26 63.36 ± 1.23 63.52 ± 1.24 59.31 ± 1.01 58.82 ± 1.15 59.47 ± 0.89 OmniDecVAEEnc-Dec 82.18 ± 0.16 82.07 ± 0.15 82.31 ± 0.16 86.12 ± 0.11 86.12 ± 0.12 86.18 ± 0.12 60.82 ± 0.19 59.97 ± 0.20 60.71 ± 0.20 β-OmniDecVAEEnc-Dec (β=0.1β=0.1) 84.56 ± 0.21 84.48 ± 0.21 84.70 ± 0.20 88.97 ± 0.13 88.99 ± 0.13 89.00 ± 0.13 64.56 ± 0.12 64.09 ± 0.14 64.78 ± 0.13 (a) Acc.: Accuracy. MF1: Macro F1. Best score for each metric of each task is given in bold. 95%95\% confidence intervals are reported over n=25 random seeds. 5.1 Disentangled Representation Informativeness Table 2(a) presents the performance on HAR and IR informativeness from the learned representations when using all available modalities on HARWE. In the SD scenario, the β-OmniDecVAEEnc-Dec variant provides the best performance in terms of HAR and IR, closely followed by VAE-based methods [20, 25]. Eigenanalysis methods such as ICA [21] and PCA[32] provide relatively good performance as well, whereas supervised methods such as YAMNet and EEGNet feature extractors with Transformer fusion [11, 28, 10] have subpar performance. Despite the end-to-end training of the supervised methods to perform one of the two tasks (HAR and IR) their performance remains low, with the dual-head multi-task learning scheme performing evidently better. Notably, supervised methods also manifest with a much higher sensitivity to the random seed as the 95%95\% confidence intervals suggest. In the SI scenario, the β-MMVAE variant outperforms other alternatives, signifying that the multi-branch architecture is slightly more adaptable to unseen subjects compared to the proposed OmniDecVae. Fig. 3 illustrates the disentanglement properties of Transformer-based [10], VAE-based [25] and OmniDecVAE models. It can be seen that the CNN-Transformer model achieves partial linear separation of activities when trained end-to-end for that task (Fig. 3a), whereas β-MMVAE and β-OmniDecVAE ((Fig. 3b,c) learn unsupervised representations that are not linearly separable. However, β-MMVAE and β-OmniDecVAE achieve disentanglement in the sense of a structured modality space, as evident by Figs.3d,e. Notably, β-OmniDecVAE achieves very clear separation between different modality modes (Fig.3e) in contrast to β-MMVAE (Fig. 3d), attributed to the ℒDELBOEnc−DecL_DELBO_Enc-Dec objective (Eq. 15). Subsequently, this disentangled modality structured latent space, renders the representation more informative to the downstream tasks of HAR and IR; effectively, the clear representation of each modality’s information leads to a clearer multi-modal fusion, expressed through the relative location of the modality clusters. Each modality’s location is dictated by the SSLDec structural dynamics, by its relation to the anchor (latent reconstruction) and the other modalities (orthogonality). We see that the asymmetric weights assigned through VOMpos,WOMnegV_OM_pos,W_OM_neg, result in a representation where the anchor (‘center of mass’ of the representation) is shifted away from some modalities and closer to others. Figure 3: TSNE [27] visualizations of the learned latent representations. (a) Activity-colored latent from the CNN-Transformer [10], taken at the output of the Transformer layers. (b) Activity-colored latent from β-MMVAE. (c) Activity-colored latent from β-OmniDecVAEEnc-Dec. (d) Modality-colored latent from β-MMVAE. (e) Modality-colored latent from β-OmniDecVAEEnc-Dec. 5.2 Generative Performance Table 3: Omni-modal Subject-Independent Generative Performance - C=30C=30 Modality Channels Model Reconstruction Quality Latent Noise Generation MSE ↓ MAE ↓ dL2d_L2 ↓ dcosd_cos ↓ MK-MMD ↓ DivScore ↓ MMVAE 4.94 ± 0.041 1.048 ± 0.007 0.575 ± 0.002 0.150 ± 0.002 0.226 ± 0.044 18.555 ± 4.409 β-MMVAE 4.358 ± 0.33 1.019 ± 0.006 1.369 ± 0.003 0.182 ± 0.001 0.210 ± 0.040 15.468 ± 3.450 OmniDecVAE 0.225 ± 0.002 0.328 ± 0.001 0.327 ± 0.001 0.427 ± 0.002 0.186 ± 0.018 5.802 ± 0.528 β-OmniDecVAE 0.116 ± 0.001 0.236 ± 0.001 0.446 ± 0.001 0.242 ± 0.001 0.181 ± 0.019 4.094 ± 0.688 (a) Best metric score is given in bold. 95%95\% confidence intervals are reported over n=4627 omni-modal samples for a single random seed of the model. Importantly, imposing the above discussed disentanglement dynamics allows us to generate omni-modal synthetic samples of higher quality and realism compared to VAE-based models. Table 3(a) presents the generative performance of models trained in the SI scheme of HARWE with all available modalities. Notably, even though MMVAEs are trained with multiple branches allocated to each modality, their generation performance is subpar to that of OmniDecVAEs. Specifically, OmniDecVAEs are characterized by enhanced reconstruction quality as evident by the MSE and MAE metrics. Moreover, OmniDecVAE generated samples also provide more realistic latent embeddings; metrics dL2d_L2 and dcosd_cos showcase that OmniDecVAE-synthesized samples are more suitable to be used as synthetic data to train a neural network due to their higher similarity to real data embeddings, expressed through lower dL2d_L2 and dcosd_cos distances. Finally, when sampling latent noise from the learned representations of the models, OmniDecVAE models manifest with a higher distributional similarity to real data. Namely, a lower MK-MMD and DivScore suggest that the distribution learned by OmniDecVAE is more “realistic”, closer to real data. The above results are visually consolidated in Fig. 4. Notably, MMVAE models are not able to always capture the amplitude of the TF representations (Temperature, ACC, Audio), or their exact morphology, simultaneously when generating multiple modalities. On the contrary, OmniDecVAEs provide much more accurate reconstructions w.r.t. the amplitude and position of TF events, whereas they capture better the morphology of narrow-band frequency phenomena such as the decomposed audio events e.g. Audio OC1, OC4. This illustrates the robustness of OmniDecVAEs in generating simultaneously a diverse set of wearable modalities compared to MMVAEs. Figure 4: Flattened 3s TF representations reconstruction results from generative models trained on the SI HARWE scheme with C=30C=30 modality channels. 5.3 Ablation Studies 5.3.1 Contribution of Multi-modal Input Tables 4(a),5(a) presents downstream HAR and IR performance for different combinations of modalities. CNN-Transformer presents a stronger performance when only audio is used as the input, as it uses the YAMNet [11] as the feature extractor, a backbone designed for audio. For other modality combinations, β-OmniDecVAE provides stronger performance in some cases, whereas β-MMVAE also dominates some modality combinations, with comparable performance between these two methods. Notably, the full-modal scenario with C=30C=30 channels does not provide the best performance in the SD scenario, but rather the combinations with less modalities. Specifically, combinations that contain the inertial sensors demonstrate superior performance, hinting of their elevated importance for the task of HAR with prior subject knowledge. In the SI scenario though, the full-modal scenario provides the best results overall. Fig. 5 illustrates the generative performance metrics of MAE and MK-MMD. β-OmniDecVAE demonstrates a more robust behavior, less influenced by the number of input modality channels, compared to β-MMVAE. Table 4: Ablation Study on Number of Modalities - Activity Recognition Performance Modalities SD HAR ↑ SI HAR ↑ CNN-Transformer [10] ↑ β-MMVAE ↑ β-OmniDecVAEEnc-Dec ↑ CNN-Transformer [10] ↑ β-MMVAE ↑ β-OmniDecVAEEnc-Dec ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Smartwatch (BVP+EDA+TEMP, C=3C=3) 17.87 ± 1.20 16.23 ± 1.23 22.50 ± 0.15 21.99 ± 0.16 21.68 ± 0.20 21.51 ± 0.18 11.24± 1.03 8.69 ± 1.07 13.70 ± 0.19 13.58 ± 0.35 12.78 ± 0.17 12.63 ± 0.15 Audio (6 OCs, C=6C=6) 48.30 ± 9.26 45.87 ± 10.89 38.80 ± 0.19 37.93 ± 0.17 40.85 ± 0.28 39.75 ± 0.30 41.61 ± 8.31 38.48 ± 9.94 35.62 ± 0.20 33.80 ± 0.20 38.07 ± 0.16 36.46 ± 0.15 SmartWatch + Audio (C=9C=9) 43.55 ± 4.92 40.37 ± 5.43 47.33 ± 0.08 46.58 ± 0.11 49.62 ± 0.09 48.91 ± 0.11 38.75 ± 5.42 36.78 ± 5.54 37.81 ± 0.31 36.46 ± 0.32 38.33 ± 0.22 37.02 ± 0.23 Inertial (C=21C=21) 58.35 ± 4.26 55.92 ± 4.46 86.80 ± 0.15 86.80 ± 0.15 86.24 ± 0.11 86.23 ± 0.12 32.80 ± 2.77 26.75 ± 3.25 57.91 ± 0.06 56.90 ± 0.09 57.16 ± 0.11 56.93 ± 0.09 Inertial + Smartwatch (C=24C=24) 49.85 ± 3.56 46.63 ± 4.20 86.00 ± 0.15 86.23 ± 0.15 86.20 ± 0.17 86.19 ± 0.16 32.46± 2.20 27.09 ± 2.70 62.30 ± 0.12 62.11 ± 0.10 55.13 ± 0.21 54.81 ± 0.20 Inertial + Audio (C=27C=27) 40.63 ± 4.67 36.05 ± 4.07 87.21 ± 0.07 87.18 ± 0.08 81.13 ± 0.14 81.04 ± 0.14 36.02 ± 2.95 31.12 ± 3.65 69.42 ± 0.16 68.59 ± 0.17 62.63 ± 0.12 62.00 ± 0.10 Smartwatch + Inertial + Audio (C=30C=30) 39.93 ± 6.03 35.70 ± 6.65 82.36 ± 0.13 82.31 ± 0.13 84.56 ± 0.21 84.48 ± 0.21 32.98 ± 2.32 28.30 ± 2.72 70.91 ± 0.10 70.37 ± 0.10 64.56 ± 0.12 64.09 ± 0.14 (a) Acc.: Accuracy. Best Acc. and F1 score for each row of each task is given in bold. 95%95\% confidence intervals are reported over n=25 random seeds. Table 5: Ablation Study on Number of Modalities - Identity Recognition Performance Modalities SD IR ↑ CNN-Transformer [10] ↑ β-MMVAE ↑ β-OmniDecVAEEnc-Dec ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Acc. ↑ F1 ↑ Smartwatch (BVP+EDA+TEMP, C=3C=3) 38.57 ± 4.74 34.88 ± 4.92 50.50 ± 0.14 48.45 ± 0.13 45.50 ± 0.13 44.02 ± 0.09 Audio (6 OCs, C=6C=6) 3.35 ± 0.002 0.22 ± 0.001 17.56 ± 0.29 16.29 ± 0.29 16.75 ± 0.28 15.24 ± 0.28 SmartWatch + Audio (C=9C=9) 33.21 ± 4.35 29.26 ± 4.66 47.04 ± 0.29 46.68 ± 0.28 48.27 ± 0.15 47.59 ± 0.17 Inertial (C=21C=21) 74.04 ± 1.23 74.59 ± 1.21 88.24 ± 0.25 88.21 ± 0.26 88.01 ± 0.12 87.96 ± 0.12 Inertial + Smartwatch (C=24C=24) 62.77 ± 4.05 61.78 ± 4.38 92.77 ± 0.11 92.76 ± 0.10 92.66 ± 0.06 92.63 ± 0.06 Inertial + Audio (C=27C=27) 36.29 ± 4.17 35.41 ± 4.21 85.66 ± 0.12 85.68 ± 0.12 75.63 ± 0.16 75.63 ± 0.16 Smartwatch + Inertial + Audio (C=30C=30) 67.36 ± 2.92 66.85 ± 3.05 79.42 ± 0.34 79.65 ± 0.35 88.97 ± 0.13 88.99 ± 0.13 (a) Acc.: Accuracy. Best Acc. and F1 score for each row of each task is given in bold. 95%95\% confidence intervals are reported over n=25 random seeds. Figure 5: The effect of a varying number of input channels for β-MMVAEs (red) and β-OmniDecVAE (blue) models, on MAE (upper) and MK-MMD (lower) generative performance metrics. 5.3.2 Contribution of Optimization Terms Table 6 contains a detailed ablation of the four terms contained in Eq. ( 17); in terms of task-related performance, we see that utilizing the SSLDec terms alongside a supervised loss, gives the highest performance while maintaining disentanglement between activity and identity. It becomes evident that SSLDec and the decoder are the main drivers of disentanglement, as their absence results in higher imbalance in the performance of HAR or IR. Table 6: Ablation Study on OmniDecVAE Loss Components Loss Components SD HAR ↑ SD IR ↑ Acc. ↑ F1 ↑ Acc ↑ F1 ↑ SSLDecSSL_Dec 76.03 75.91 72.37 72.42 LreconL_recon 83.22 83.14 86.92 86.93 LpriorL_prior 65.30 64.97 47.69 48.09 Supervised 82.07 82.00 83.21 83.23 SSLDecSSL_Dec + LpriorL_prior 74.66 74.56 67.26 67.31 SSLDecSSL_Dec + LreconL_recon 82.09 82.01 86.58 86.63 SSL_Dec + Supervised 85.83 85.75 88.32 88.33 LreconL_recon + LpriorL_prior 81.31 81.23 85.70 85.69 LreconL_recon + Supervised 83.06 82.96 86.46 86.49 LpriorL_prior + Supervised 70.64 70.44 56.46 56.93 SSLDecSSL_Dec + LpriorL_prior + LreconL_recon 82.17 82.04 85.89 85.89 SSLDecSSL_Dec + LpriorL_prior + Supervised 72.48 72.43 62.28 62.45 SSLDecSSL_Dec + LreconL_recon + Supervised 83.80 83.71 87.71 87.73 LreconL_recon + LpriorL_prior + Supervised 82.11 82.02 85.35 85.35 SSLDecSSL_Dec + LpriorL_prior + LreconL_recon + Supervised 81.89 81.78 86.39 86.38 5.3.3 Contribution of Gaussian Prior Approximation Weight β The role of LpriorL_prior and the value of β is better understood through Fig.6. As discussed in DecVAEs [43], a strong β compression strength may not always result in a better representation; in Fig. 6a, HAR performance correlates directly with modality disentanglement at different values of β. Here higher β values result in a collapse of the disentangled structure. We also see that the absence of the LpriorL_prior through β=0β=0 is not detrimental to the downstream performance, mainly attributed to the disentanglement mechanisms of SSLDec. A different behavior is evident in the decoder-less variant of OmniDecVAE (see Fig. 6b); in the absence of a decoder, a higher β is needed for a disentangled representation. Figure 6: The effect of varying Gaussian prior approximation strength β on representation disentanglement for (a) HAR and (b) IR for β-OmniDecVAEEnc (orange) and β-OmniDecVAEEnc-Dec (orange). 5.4 Complexity In Table 7(a), we evaluate our proposed OmniDecVAE alongside a Transformer-based model and MMVAE on their complexity, by calculating per sample FLOPs and latency inside a batched input. The CNN-Transformer of [10] provides the faster choice with very low latency per sample. MMVAE performs the least number of FLOPs, whereas OmniDecVAE provides the lighter alternative in terms of SoD and number of parameters. Notably, OmniDecVAE adds a significant number of operations per sample due to the sophisticated optimization objective; at the same time, MMVAE presents increasing storage requirements due to the high number of modalities it accommodates as separate branches. Indeed, Fig. 7b showcases that the spatial complexity of MMVAEs increase with the number of modalities; on the other hand OmniDecVAEs are invariant to the number of modalities for that matter, while being more prone to fast inference as the number increases. Table 7: Performance on Complexity Metrics for C=30C=30 Modality Channels Model GFLOPs ↓ LatencyInf (ms) ↓ SoD (MB) ↓ Parameters (M) ↓ CNN-Transformer [10] 0.65 0.06 20.19 5.28 MMVAE 0.45 0.67 336.46 88.09 OmniDecVAEEnc [43] 29.61 3.51 13.20 3.43 OmniDecVAEEnc-Dec 85.85 4.00 15.82 4.11 (a) LatencyInf: Inference latency, SoD: size on disk Figure 7: The effect of a varying number of input channels on (a) inference latency and (b) number of parameters. 5.5 Discussion Wearable representation learning has significantly advanced in the latest years mainly due to breakthroughs in fusion architectures that has enabled integration of multiple modality streams across large-scale datasets, spearheaded by the prowess of transformer-based models. However, despite high HAR classification performance, the focus of these models does not extend to full-stack representations. Here, we presented omni-modal representation learning for wearable computing, by efficiently learning through a modality-invariant architecture; modality-specific data handling for OmniDecVAEs ends at the pre-processing stage with the selection of input representation. In contrast to transformer-based alternatives, we show that expressive fusion can be achieved through a learning objective without requiring architectural interventions. Significantly, we notice that even though our OmniDecVAE approach does not utilize the video modality, one of the most expressive modalities in HAR, it performs on par with a supervised transformer-based method in the HARWE Difficult SI scenario that utilizes video (Acc. of 68.5%68.5\% in SI HAR) [10]. In addition, our proposed full-stack structured representation learning framework promotes generalizable and widespread utility through an AE-based SSL task that guarantees disentangled representations. The importance of disentanglement can be understood when evaluating task informativeness; notably, our omni-modal AE networks performed better under SSL pre-training on both HAR and IR, compared to supervised transformer-based alternatives, that are trained separately to learn activity-specific and identity-specific representations, and VAE-based methods and other benchmarks (Table 2(a)). The proposed OmniDecVAEs consistently showed a greatly improved performance over supervised alternatives in the SI disentanglement tasks for HAR and IR. OmniDecVAEs also performed well in the SD scenario, which resembles a biometric identification scenario after a prior registration of biometrics has been performed for each subject. Moreover, the structure-informed learning objective that we adopted, is fully explainable and transparent. Visual inspection of the representations in Fig. 3 reveals well delimited modes that arise as a result of subspace learning. This is also translated to better multi-modal generation quality in Table 3(a) and Fig. 4. Multi-modal synthetic data generation is also of significant value for the expansion of modern AI systems, due to the limitations inherent in real data collection. Our results on unseen subjects in the SI scenario show an enhanced ability of OmniDecVAEs on generating realistic omni-modal data distributions, compared to VAE-based alternatives. Finally, our approach has a complexity-friendly design that is exemplified when the number of modalities aggressively scales; architectures with separate branches per modality quickly become unsustainable in real-life edge deployment scenarios due to the increasing number of parameters and storage required. In terms of real-time processing, OmniDecVAEs require much more FLOPs than compared methods, yet they still lie within the real-time processing window for edge applications, with a latency of 4ms per sample. Combined with a negligent storage requirement of as low as 15MB15MB, enhanced generation performance, and high downstream task performance, OmniDecVAEs are very attractive for cloud-based edge processing systems, clinical environments, wearable healthcare and in-the-wild monitoring. Despite the promising performance of our proposed framework, aspects for improvement in future work remain. Our current implementation of OmniDecVAE is based on TF representations; although TF representations through STFT and Mel scale are more informative and predominantly used in modern deep learning, the generation of synthetic data in the more raw format of time domain is preferable, allowing more freedom in downstream processing. Moreover, we did not experiment with cross-modal inference and missing modality scenarios; our results have showcased excellent omni-modal separation and conservation of hierarchical relations within the latent representations of OmniDecVAEs, hinting a promising direction of our work towards cross-modal generation. Other future extensions of our work will accommodate other types of modalities beyond time series, such as video or text. 6 Conclusion This work proposes OmniDecVAEs, a novel structured representation learning framework to learn full-stack wearable representations by fusing arbitrarily large multi-modal (or omni-modal) data streams. Our approach is powered by an SSL decomposition loss that facilitates disentanglement forces in a latent space, alongside VAE-based prior approximation and decoding. The proposed OmniDecVAEs operate in the TF-domain through a single-branched modality-invariant encoder-decoder scheme, where fusion, cross-modal alignment and intra-modal integrity are ensured through the learning objective. Our evaluations highlight full-stack representational capacity, with improvements across an array of tasks and over different model families, such as Transformers and VAEs. Namely, OmniDecVAEs simultaneously enhance HAR and IR accuracy by 6.75%6.75\%, and 1.01%1.01\%, multi-modal data synthesis MAE and MK-MMD by 13.85%13.85\% and 76.84%76.84\%, respectively, along with modality-invariant storage requirements of 15MB15MB or 4.1M4.1M parameters. These results underscore OmniDecVAEs as a foundational paradigm for next-generation full-stack models in wearable AI. Their robust modality-invariant footprint, transparent and interpretable representations, aid towards secure, patient-centric healthcare, enabling biometric security applications, whereas their generative capability allows for sustainable deployment through synthetic signals, and invariance to sensor reduction and missing data. References References [1] P. A. et al. ((2019)) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: §4.3. [2] K. Avramidis, D. Kunc, B. Perz, K. Adsul, T. Feng, P. Kazienko, S. Saganowski, and S. Narayanan (2024) Scaling Representation Learning from Ubiquitous ECG with State-Space Models. IEEE Journal of Biomedical and Health Informatics 28 (10), p. 5877–5889. External Links: Link, Document, ISSN 21682208 Cited by: §1. [3] J. L. Ba, J. R. Kiros, and G. E. Hinton (2016-07) Layer Normalization. External Links: Link Cited by: §3.6, §3.7. [4] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Advances in Neural Information Processing Systems 33, p. 12449–12460. External Links: Link Cited by: §3.6. [5] R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum (2023-04) A Cookbook of Self-Supervised Learning. arXiv preprint arXiv:2304.12210 (2023). External Links: Link Cited by: §1. [6] M. A. Carbonneau, J. Zaïdi, J. Boilard, and G. Gagnon (2024) Measuring Disentanglement: A Review of Metrics. IEEE Transactions on Neural Networks and Learning Systems 35 (7), p. 8747–8761. External Links: Document, ISSN 21622388 Cited by: §2.3. [7] X. Chen, X. Zhou, M. Sun, and H. Wang (2025) Temporal Contrastive Learning for Sensor-Based Human Activity Recognition: A Self-Supervised Approach. IEEE Sensors Journal 25 (1), p. 1839–1850. External Links: Link, Document, ISSN 15581748 Cited by: §2.2. [8] T. Dissanayake, T. Fernando, S. Denman, S. Sridharan, and C. Fookes (2023-02) Generalized Generative Deep Learning Models for Biosignal Synthesis and Modality Transfer. IEEE Journal of Biomedical and Health Informatics 27 (2), p. 968–979. External Links: Document, ISSN 21682208 Cited by: §1, §2.3. [9] V. Dissanayake, S. Seneviratne, R. Rana, E. Wen, T. Kaluarachchi, and S. Nanayakkara (2022) SigRep: Toward Robust Wearable Emotion Recognition with Contrastive Representation Learning. IEEE Access 10, p. 18105–18120. External Links: Document, ISSN 21693536 Cited by: §2.1, §2.2. [10] A. Esmaeilzehi, E. Khazaei, K. Wang, N. Kaur Kalsi, P. C. Ng, H. Liu, Y. Yu, D. Hatzinakos, and K. Plataniotis (2024-08) HARWE: A multi-modal large-scale dataset for context-aware human activity recognition in smart working environments. Pattern Recognition Letters 184, p. 126–132. External Links: Link, Document, ISSN 0167-8655 Cited by: §1, §1, §4.1, §4.1, §4.2, §4.3, §4.3, Figure 3, Figure 3, §5.1, §5.1, §5.4, §5.5, 2(a), 2(a), 4(a), 4(a), 5(a), 7(a). [11] D. E. et al. (2019) YAMNet: a pretrained audio event classifier. GitHub, https://github.com/tensorflow/models/tree/master/research/audioset/yamnet. External Links: Link Cited by: §4.2, §5.1, §5.3.1, 2(a), 2(a), 2(a), 2(a). [12] P. F. et al. ((2011)) Scikit-learn: machine learning in python. The Journal of Machine Learning Research 12, p. 2825–2830. Cited by: §4.3. [13] T. Fan, F. Tian, S. Wang, H. Zhang, G. Luo, L. Zhu, J. Liu, L. Jiang, R. Cai, Q. Dong, and B. Hu (2026) CMDˆ3: Cross-Modal Decoupled Deformable Distillation for EEG-fNIRS Fusion. IEEE Transactions on Affective Computing. External Links: Link, Document, ISSN 19493045 Cited by: §1. [14] M. Greenacre, P. J.F. Groenen, T. Hastie, A. I. D’Enza, A. Markos, and E. Tuzhilina (2022-12) Principal component analysis. Nature Reviews Methods Primers 2 (1), p. 1–21. External Links: Link, Document, ISSN 26628449 Cited by: §4.2, 2(a). [15] M. Han, O. Ozdenizci, T. Koike-Akino, Y. Wang, and D. Erdogmus (2021-08) Universal Physiological Representation Learning with Soft-Disentangled Rateless Autoencoders. IEEE Journal of Biomedical and Health Informatics 25 (8), p. 2928–2937. External Links: Link, Document, ISSN 21682208 Cited by: §2.3. [16] M. Han, O. Özdenizci, Y. Wang, T. Koike-Akino, and D. Erdoǧmuş (2020) Disentangled adversarial autoencoder for subject-invariant physiological feature extraction. IEEE Signal Processing Letters 27, p. 1565–1569. External Links: Document, ISSN 15582361 Cited by: §1. [17] C. He, X. Xu, Z. Wang, J. Song, F. Shen, and H. T. Shen (2026-01) Causal-Inspired Fourier Representation Learning for Wearable IMUs and Egocentric Action Recognition. IEEE Transactions on Circuits and Systems for Video Technology, p. 1–1. External Links: Link, Document, ISSN 1051-8215 Cited by: §1, §1, §1, §2.2. [18] K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick (2021-11) Masked Autoencoders Are Scalable Vision Learners. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2022-June, p. 15979–15988. External Links: Link, ISBN 9781665469463, Document, ISSN 10636919 Cited by: §2.2, §3.7. [19] D. Hendrycks and K. Gimpel (2016-06) Gaussian Error Linear Units (GELUs). External Links: Link Cited by: §3.6, §3.7. [20] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner, and G. Deepmind (2017-02) beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International conference on learning representations (ICLR), Cited by: §1, §3.5, §4.2, §5.1. [21] A. Hyvärinen, H. J., and H. P.O. ((2001)) Independent component analysis. In Natural Image Statistics: A Probabilistic Approach to Early Computational Vision, p. 151–175. Cited by: §1, §4.2, §5.1, 2(a). [22] L. Jing, P. Vincent, Y. LeCun, and Y. Tian (2021-10) Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022 - 10th International Conference on Learning Representations. External Links: Link Cited by: §3.3. [23] S. Kang, W. Choi, C. Y. Park, N. Cha, A. Kim, A. H. Khandoker, L. J. Hadjileontiadis, H. Kim, Y. Jeong, and U. Lee (2023-06) K-EmoPhone: A Mobile and Wearable Dataset with In-Situ Emotion, Stress, and Attention Labels. Scientific Data 2023 10:1 10 (1), p. 1–21. External Links: Link, Document, ISSN 2052-4463 Cited by: §1. [24] S. Karimi, A. J. Shah, G. D. Clifford, and R. Sameni (2025-12) Bidirectional Translation Between ECG and PCG. p. 1–5. External Links: Link, ISBN 9798331592080, Document Cited by: §2.3. [25] D. P. ;. Kingma and M. Welling (2014) Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §3.2, §3.5, §3.7, §4.2, §5.1, §5.1. [26] D. P. Kingma and J. L. Ba (2014-12) Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings. External Links: Link Cited by: §4.3. [27] v. d. M. L. and H. G. ((2008)) Visualizing data using t-sne. Journal of machine learning research 9, p. 2579–2605. Cited by: Figure 3, Figure 3. [28] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance (2018-07) EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of Neural Engineering 15 (5), p. 056013. External Links: Link, Document, ISSN 1741-2552 Cited by: §4.2, §5.1, 2(a), 2(a), 2(a), 2(a). [29] X. Liu, Y. Zhang, Z. Yu, H. Lu, H. Yue, and J. Yang (2024) RPPG-MAE: Self-Supervised Pretraining with Masked Autoencoders for Remote Physiological Measurements. IEEE Transactions on Multimedia 26, p. 7278–7293. External Links: Document, ISSN 19410077 Cited by: §1, §2.2. [30] B. Minor, C. Greeley, R. Holder, B. Thomas, L. B. Holder, and D. J. Cook (2026-01) A Feature-Augmented Transformer Model to Recognize Functional Activities From in-the-Wild Smartwatch Data. IEEE Journal of Biomedical and Health Informatics 30 (1), p. 256–265. External Links: Link, Document, ISSN 21682208 Cited by: §1. [31] D. A. Nguyen, C. Pham, and N. A. Le-Khac (2024) Virtual Fusion with Contrastive Learning for Single-Sensor-Based Activity Recognition. IEEE Sensors Journal 24 (15), p. 25041–25048. External Links: Link, Document, ISSN 15581748 Cited by: §2.1. [32] B. Schölkopf, A. Smola, and K. R. Müller (1997) Kernel principal component analysis. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 1327, p. 583–588. External Links: Link, ISBN 978-3-540-69620-9, Document, ISSN 1611-3349 Cited by: §1, §4.2, §5.1, 2(a), 2(a), 2(a). [33] P. Sriram Kumar, P. K. Govarthan, A. Aleem Shaik Gadda, N. Ganapathy, and J. Fredo Agastinose Ronickom (2024) Deep Learning-Based Automated Emotion Recognition Using Multimodal Physiological Signals and Time-Frequency Methods. IEEE Transactions on Instrumentation and Measurement 73. External Links: Link, Document, ISSN 15579662 Cited by: §2.1. [34] T. Su, C. Dai, X. Liu, and X. Jiang (2025) EMG Biometric Verification Via Disentangled Representations. IEEE Transactions on Industrial Informatics 21 (4), p. 3376–3385. External Links: Document, ISSN 19410050 Cited by: §1, §2.3. [35] W. T. et al. ((EMNLP,2020)) Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 38–45. Cited by: §4.3. [36] A. van den Oord DeepMind, Y. Li DeepMind, and O. Vinyals DeepMind (2018-07) Representation Learning with Contrastive Predictive Coding. External Links: Link Cited by: §1. [37] Y. Wu, M. Daoudi, and A. Amad (2024-01) Transformer-Based Self-Supervised Multimodal Representation Learning for Wearable Emotion Recognition. IEEE Transactions on Affective Computing 15 (1), p. 157–172. External Links: Link, Document, ISSN 19493045 Cited by: §1, §2.1, §2.2. [38] Z. Xiao, H. Tong, R. Qu, H. Xing, S. Luo, Z. Zhu, F. Song, and L. Feng (2025) CapMatch: Semi-Supervised Contrastive Transformer Capsule with Feature-Based Knowledge Distillation for Human Activity Recognition. IEEE Transactions on Neural Networks and Learning Systems 36 (2), p. 2690–2704. External Links: Link, Document, ISSN 21622388 Cited by: §1, §2.2. [39] Y. Zhang, H. Li, J. Liu, and X. Yue (2025) Scaling Omni-modal Pretraining with Multimodal Context: Advancing Universal Representation Learning Across Modalities. External Links: Link Cited by: §1. [40] Z. Zhang, H. Lu, Q. Zhao, K. Wen, and B. Liu (2025) PPG Sensor-Based Biometric Identification and Physiological Analysis via Temporal-Frequency Disentanglement With Liquid Neural Networks. IEEE Sensors Letters 9 (8). External Links: Link, Document, ISSN 24751472 Cited by: §1, §2.3. [41] X. Zhou, J. Yuan, L. Fan, X. Niu, K. Zha, and X. Liu (2025) MSMFT: Multi-Stream Multimodal Factorized Transformer for Human Activity Recognition. IEEE Sensors Journal 25 (6), p. 10402–10416. External Links: Link, Document, ISSN 15581748 Cited by: §1. [42] I. Ziogas, L. J. Hadjileontiadis, A. H. Khandoker, and A. Al Shehhi (2025) Self-Supervised Graph Representation Learning for In-The-Wild Wearable and Smartphone based Emotion Recognition. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. External Links: Link, ISBN 9798350368741, Document, ISSN 15206149 Cited by: §1. [43] I. Ziogas, A. A. Shehhi, A. H. Khandoker, and L. J. Hadjileontiadis (2026-01) Variational decomposition autoencoding improves disentanglement of latent representations. Preprint at https://arxiv.org/abs/2601.06844. External Links: Link Cited by: §1, §3.1, §3.2, §3.2, §3.2, §3.3, §3.3, §3.5, §3.5, §3.5, 1st item, §4.2, §5.3.3, 2(a), 2(a), 7(a).