Paper deep dive
Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning
Diyar Altinses, Andreas Schwung
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/27/2026, 1:32:15 AM
Summary
The paper introduces a mathematically grounded framework for fault-tolerant multimodal representation learning, utilizing layer-specific Lipschitz modulation and Jacobian-based criteria to balance anomaly detection sensitivity with error correction stability. By analyzing perturbation propagation in dense versus convolutional layers, the authors propose a two-stage self-supervised training scheme that improves robustness against sensor failures and signal degradation in industrial environments.
Entities (5)
Relation Signals (3)
Diyar Altinses → authored → Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning
confidence 100% · Diyar Altinses and Andreas Schwung... This Article is funded by...
Layer-Specific Lipschitz Modulation → improves → Anomaly Detection
confidence 90% · Experimental results on multimodal fault datasets demonstrate that the proposed approach improves both anomaly detection accuracy
Multimodal Convolutional Autoencoder → utilizes → Lipschitz Regularization
confidence 90% · we introduce layer-specific Lipschitz modulation and gradient clipping as principled mechanisms
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern multimodal systems deployed in industrial and safety-critical environments must remain reliable under partial sensor failures, signal degradation, or cross-modal inconsistencies. This work introduces a mathematically grounded framework for fault-tolerant multimodal representation learning that unifies self-supervised anomaly detection and error correction within a single architecture. Building upon a theoretical analysis of perturbation propagation, we derive Lipschitz- and Jacobian-based criteria that determine whether a neural operator amplifies or attenuates localized faults. Guided by this theory, we propose a two-stage self-supervised training scheme: pre-training a multimodal convolutional autoencoder on clean data to preserve localized anomaly signals in the latent space, and expanding it with a learnable compute block composed of dense layers for correction and contrastive objectives for anomaly identification. Furthermore, we introduce layer-specific Lipschitz modulation and gradient clipping as principled mechanisms to control sensitivity across detection and correction modules. Experimental results on multimodal fault datasets demonstrate that the proposed approach improves both anomaly detection accuracy and reconstruction under sensor corruption. Overall, this framework bridges the gap between analytical robustness guarantees and practical fault-tolerant multimodal learning.
Tags
Links
- Source: https://arxiv.org/abs/2603.25103v1
- Canonical: https://arxiv.org/abs/2603.25103v1
Trouble viewing inline? Open PDF directly →
Full Text
95,332 characters extracted from source content.
Expand or collapse full text
Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning Diyar Altinses and Andreas Schwung Department of Automation Technology and Learning Systems South Westphalia University of Applied Sciences Soest, Germany altinses.diyar@fh-swf.de, schwung.andreas@fh-swf.de This Article is funded by the Open Access Publication Fund of South Westphalia University of Applied Sciences. Submitted to Information Fusion Abstract Modern multimodal systems deployed in industrial and safety-critical environments must remain reliable under partial sensor failures, signal degradation, or cross-modal inconsistencies. This work introduces a mathematically grounded framework for fault-tolerant multimodal representation learning that unifies self-supervised anomaly detection and error correction within a single architecture. Building upon a theoretical analysis of perturbation propagation, we derive Lipschitz- and Jacobian-based criteria that determine whether a neural operator amplifies or attenuates localized faults. Guided by this theory, we propose a two-stage self-supervised training scheme: pre-training a multimodal convolutional autoencoder on clean data to preserve localized anomaly signals in the latent space, and expanding it with a learnable compute block composed of dense layers for correction and contrastive objectives for anomaly identification. Furthermore, we introduce layer-specific Lipschitz modulation and gradient clipping as principled mechanisms to control sensitivity across detection and correction modules. Experimental results on multimodal fault datasets demonstrate that the proposed approach improves both anomaly detection accuracy and reconstruction under sensor corruption. Overall, this framework bridges the gap between analytical robustness guarantees and practical fault-tolerant multimodal learning. Keywords Fault-tolerant multimodal learning ⋅· self-supervised representation learning ⋅· Lipschitz regularization ⋅· anomaly detection and correction ⋅· robustness analysis 1 Introduction Unplanned downtime represents a critical failure mode and a major source of operational inefficiency across industrial sectors [34, 17]. This unscheduled termination of production leads to substantial, multi-faceted economic losses, consistently ranking as a primary financial burden. These costs are derived fundamentally from three main categories: lost production volume, which directly impacts revenue, the significant expense of emergency maintenance and specialized repairs, and harmful effects on supply chain stability and contractual obligations [38]. Traditional strategies for mitigating system failure have historically relied heavily on hardware redundancy. This approach, while effective against component-specific malfunctions like random failures or wear-out, offers limited protection against common-cause failures [3]. These failures originate from shared environmental effects, such as excessive heat, high humidity, or mechanical vibration [22]. When these conditions impact multiple components simultaneously, even redundant ones, the safety margin is eliminated [4]. A modern approach uses multimodal machine learning to overcome the limitations of traditional, single-sensor diagnostic methods and redundancy. Machine learning-based methodologies enable the capability to fuse diverse, heterogeneous sensor data, sourced from spatially distributed and often disparate information streams, into one general model [2]. This fusion process integrates variables such as temporal sensor readings and camera data, resulting in a richer and more accurate estimation of the current state. Crucially, this advanced diagnostic capability enables compensating for the weakness of one modality with the strength of another. This creates an inherently fault-tolerant and adaptive production system [10]. A complete fault tolerance mechanism fundamentally requires a sequential three-stage process: fault injection, detection, and correction (or recovery). Existing research and industrial implementations often exhibit a fragmented focus, addressing these stages individually rather than as an integrated system [42, 6]. While some state-of-the-art approaches attempt to incorporate all three steps, they frequently encounter a critical performance trade-off: improvements in the correction mechanism often lead to a degradation in the detection stage [6]. Building upon the identified limitations, this research presents a theoretical analysis of how various perturbations impact different neural network layer structures. We extend this analysis by investigating the dynamic behavior of neural network sensitivity during the optimization. Informed by these theoretical insights, we propose a self-supervised, dual-regularized deep learning architecture. This novel approach is specifically designed to address the detection-correction trade-off, aiming to increase network sensitivity for robust anomaly detection while simultaneously reducing sensitivity for a stable and accurate correction process. This integrated framework represents a significant advancement toward highly efficient industrial fault tolerance. The contributions of this paper are summarized as follows: (i) We present a theoretical framework analyzing the effect of diverse operational perturbations on the structural and dynamic sensitivity of deep neural network architectures. (i) We developed, based on theory, a dual regularization mechanism that selectively increases sensitivity for anomaly detection tasks while simultaneously decreasing it for the fault correction process, effectively resolving the detection-correction performance trade-off. (i) We validated the proposed approach on three multimodal heterogeneous industrial datasets, demonstrating a unified fault-tolerance system that achieves superior performance compared to existing fragmented methodologies. The remainder of this paper is structured to systematically present our research and findings. Section 2 provides a detailed overview of the related work, focusing on advances in multimodal self-supervised learning and existing fault-tolerant deep learning methodologies. Following this, Section 3 establishes the foundational theoretical analysis concerning perturbation effects and the dynamic sensitivity of neural networks. The practical application of these theoretical insights is presented in Section 4, which introduces and fully details our proposed self-supervised, dual-path, and dual-regularized model. Section 6 then validates our approach through a comprehensive experimental evaluation and discussion of results. Finally, Section 7 summarizes the key findings of this work and offers concluding remarks. 2 Related Work Research on robust multimodal representation learning and fault-tolerant neural architectures has evolved along several directions. Three particularly relevant fields related to our approach are: multimodal self-supervised learning, fault-tolerant multimodal learning, and Lipschitz-based sensitivity regulation. The following subsections review the most pertinent literature within each domain. 2.1 Multimodal Self-Supervised Learning Multimodal self-supervised learning has matured rapidly, driven by architectures and objectives that exploit cross-view (or cross-modal) correspondences to learn rich joint representations without manual labels. Early audio-visual methods established the utility of cross-modal signals: Arandjelović and Zisserman showed that audio-visual correspondence yields strong cross-modal embeddings and localization cues [12] and later extended this to object-sound grounding [13]. Temporal synchronization and audio–visual temporal self-supervision were shown to be effective for learning temporally aligned features from video [23, 32]. Contrastive and multiview objectives generalized these ideas. Tian et al. propose the contrastive multiview coding, which includes a principled noise-contrastive objective for arbitrary sets of views and demonstrated strong transfer performance [37]. Deep multimodal frameworks have extended contrastive training to multiple modalities in video, showing that versatile multimodal representations can be learned end-to-end [41]. Transformer-based multimodal pretraining further scaled self-supervision to raw audio, video, and text with contrastive/fusion objectives [1][14]. In vision–language modeling, researchers demonstrate that two-stream and fusion transformers with large-scale pretraining, masked modeling, and matching objectives yield powerful joint embeddings for downstream tasks [28, 36, 18]. More recently, Radford et al. showed that large contrastive language–image models, that pair images with natural-language descriptions, provide a scalable source of multimodal supervision that transfers broadly [33]. These lines of work provide the foundations for our approach. However, the literature largely focuses on representation quality and transfer. Comparatively little work studies fault-tolerance and latent correction mechanisms in a principled, Lipschitz-aware manner. We close this gap by proposing a two-stage, self-supervised pipeline that combines contrastive pretraining (to preserve anomaly salience) with latent-space correction under Lipschitz-controlled regularization to enable both detection and correction. 2.2 Fault-tolerant Multimodal Learning Fault tolerance in multimodal learning sits at the intersection of robust fusion, missing-modality inference, and anomaly/resilience mechanisms. Several recent works have studied the robustness properties of multimodal models and proposed architectures and training schemes to mitigate the effects of partial or corrupted modalities. Ma et al. conduct a systematic study of Transformer-based multimodal models and show that fusion strategy and model design critically affect robustness to missing modalities [29]. Complementary approaches explicitly model missing modalities via imagination or imputation networks that predict absent modality embeddings from available views [43]. Several recent papers propose principled strategies for robust fusion. Representation decoupling reduces interference across modality combinations, thereby improving robustness under incomplete inputs [40]. Wang et al. explicitly separate common and modality-unique latent factors to enhance resilience to missing or corrupted inputs [39]. Other works design modality reliability scoring or uncertainty-aware fusion to down-weight unreliable streams and improve end-to-end robustness [20, 21]. A growing literature focuses on the problem of incomplete multimodal learning under uncertain missingness, offering distributional and generative remedies to recover missing modalities or construct robust joint representations [24, 26]. Contrastive and sampling-based strategies have also been adapted to create modality-agnostic latent spaces that are less sensitive to perturbations [40, 27]. A significant portion of recent literature focuses on the performance and robustness of multimodal autoencoders. One technique is the fuzzy regularization approach, which employs a specialized regularizer to amplify the magnitudes of correct features while actively attenuating corrupted ones [4]. Further elaborating on the latent space, researchers in [10] proposed extending the multimodal autoencoders to a variational version, implementing a probabilistic fusion mechanism to better model uncertainty and variability during the combination of latent representations. Concurrently, other work has concentrated on dynamic feature selection, exemplified by the concept fusion developed in [6]. This block enables the network to learn selective integration, dynamically choosing between joint and marginal information from the modalities. Despite progress, most methods evaluate robustness empirically. Few provide principled, layerwise, or Lipschitz-aware design rules that jointly address anomaly localization and correction in latent space. Our work fills this gap by linking convolutional locality to preserved perturbation salience, proposing linear compute blocks to disperse and correct latent-encoded faults, and introducing a layer-specific Lipschitz regularization strategy that explicitly trades off detectability and correctability. 2.3 Lipschitz-based Sensitivity Regulation Research on regulating model sensitivity via Lipschitz continuity and Jacobian control has grown rapidly, driven by concerns of robustness, generalization, and responsible model behavior. Early work on Lipschitz-constrained networks showed that bounding the operator norm of weight matrices improves adversarial robustness and calibration [11]. Miyato et al. introduced Jacobian regularization for semi-supervised learning to improve smoothness and generalization [31]. Spectral normalization appeared as a practical mechanism to bound layer Lipschitz constants in GANs and classification networks [30]. Fazlyab et al. analysed contraction-based stability in feedback systems via Lipschitz bounds and offered semidefinite relaxations for certified robustness [19]. More recently, Jacobian‐based penalties have been applied to large-scale pretraining: Chen et al. demonstrated that minimizing the norm of the Jacobian improves out-of-distribution robustness in massive transformer models [16]. On the generative modelling front, Rhodes et al. proposed controlling the Lipschitz to balance sample quality and latent stability [35]. From a theoretical standpoint, Lewandowski et al. derived generalization bounds in deep learning via bounds on the spectral norm of the Jacobian of the network mapping [25]. Moreover, Chen et al. established a convergence guarantee for gradient-clipped SGD by relating clipping thresholds to implicit Lipschitz constraints on the network and loss [15]. Finally, Altinses et al. provided a theoretical analysis of the Lipschitz continuity associated with standard fusion strategies in multimodal autoencoders, highlighting the stability limitations of fusion methods. Building on these gradient dynamics, they proposed a regularized attention-based fusion mechanism that explicitly constrains the Lipschitz constant to ensure robust training stability in industrial applications [8]. These works provide the foundation for our layer-specific Lipschitz modulation strategy: we adapt Lipschitz regularization not uniformly but selectively across layers with distinct roles (encoder vs. compute block) to simultaneously support anomaly localization via higher sensitivity and error correction via contraction. 3 Propagation of Localized Perturbations In multimodal learning architectures, distinct modalities such as image data and sensor data are often processed via different neural network components. Typically, convolutional layers are employed for image modalities to leverage their inherent spatial structure, whereas fully connected (dense) layers are used for sensor modalities due to their tabular nature. Understanding how localized perturbations, such as faults or anomalies injected into the input, propagate through these differing architectures is critical for interpreting model robustness and the effectiveness of fault detection mechanisms. Intuitively, convolutional layers tend to preserve spatial locality by restricting the impact of perturbations to a small subset of output features, whereas dense layers distribute perturbations across all output dimensions, potentially diluting localized signals. In this section, we rigorously formalize and prove this intuition for the case of multiple corrupted input entries. 3.1 Additive Perturbation Model Additive perturbations are the most fundamental and widely used abstraction for modeling noise and local faults in both theoretical analysis and engineering practice. They correspond to the intuitive notion that a clean signal x is affected by an independent disturbance δ, which may arise from sensor errors, environmental noise, or transmission corruption. In this setting, the observed input is written as x~=x+δ, x=x+δ, (1) where the perturbation vector δ has support supp(δ)=S⊆1,…,Nsupp(δ)=S \1,…,N\, representing the (possibly small) subset of coordinates where the fault actually occurs. Having established the additive perturbation framework, we now examine how such perturbations propagate through the two most common linear mappings in deep learning: fully connected (dense) layers and convolutional layers. Dense layers form the backbone of many sensor-processing or tabular-data subsystems and are characterized by their global receptive field: each output neuron aggregates contributions from every input coordinate. Consequently, any local perturbation in the input potentially affects all output dimensions. Our goal is to make this intuition mathematically explicit by quantifying the expected perturbation energy at the layer output under random weights. Therefore, we begin with the dense case. Lemma 3.1. Let W∈ℝM×NW ^M× N have independent entries Wj,iW_j,i with [Wj,i]=0E[W_j,i]=0 and Var[Wj,i]=σW2Var[W_j,i]= _W^2. Let δ be supported on S. Then the expected squared output perturbation satisfies [‖Wδ‖22]=MσW2‖δS‖22.E [\|Wδ\|_2^2 ]\;=\;M\, _W^2\,\| _S\|_2^2. Proof. Write ‖Wδ‖22=δ⊤W⊤Wδ=∑i,i′δiδi′(W⊤W)i,i′.\|Wδ\|_2^2=δ W Wδ= _i,i _i _i (W W)_i,i . Take expectations. For the matrix W⊤W W the entries are (W⊤W)i,i′=∑j=1MWj,iWj,i′.(W W)_i,i = _j=1^MW_j,iW_j,i . Thus [(W⊤W)i,i′]=∑j=1M[Wj,iWj,i′].E[(W W)_i,i ]= _j=1^ME[W_j,iW_j,i ]. Because entries in distinct columns are independent and zero-mean, for i≠i′i≠ i we have [Wj,iWj,i′]=[Wj,i][Wj,i′]=0E[W_j,iW_j,i ]=E[W_j,i]E[W_j,i ]=0. For i=i′i=i , [Wj,i2]=Var[Wj,i]=σW2E[W_j,i^2]=Var[W_j,i]= _W^2. Hence [W⊤W]=MσW2IN.E[W W]=M _W^2I_N. Therefore ‖Wδ‖22=δ⊤[W⊤W]δ=δ⊤(MσW2I)δ=MσW2‖δ‖22,E\|Wδ\|_2^2=δ E[W W]δ=δ (M _W^2I)δ=M _W^2\|δ\|_2^2, and restricting to the support S yields the stated formula. ∎ This result shows that each input perturbation component contributes equally (in expectation) to every output coordinate. The total perturbation energy grows linearly with the number of outputs M, but the per-output expected energy equals σW2‖δS‖22 _W^2\| _S\|_2^2, which is independent of M. Thus, a localized input perturbation appears as many small perturbations across outputs, a global spread which reduces per-coordinate salience. Note that this analysis primarily applies to the initial state (He/Xavier Initialization) or under the assumption of centered data/weights, which is often approximated by batch normalization. While Lemma 3.1 reveals that dense layers act as global mixing operators that eliminate the spatial structure of additive perturbations, convolutional layers behave fundamentally differently. Their defining property is weight sharing combined with local receptive fields, meaning that each output neuron only depends on a small neighborhood of input coordinates. This architectural locality strongly constrains how far a perturbation can spread through the layer. Therefore, to contrast with the global propagation observed in the dense case, we now analyze the expected perturbation energy in a convolutional layer under the same additive perturbation model. Therefore, we model a convolutional layer C∈ℝM′×NC ^M × N as a sparse matrix where each column C⋅,iC_·,i has nonzero entries only at output indices in SiS_i (the receptive field positions for input index i); assume |Si|≤K|S_i|≤ K. Nonzero entries (kernel positions) are independent, zero-mean, and have the variance σk2 _k^2. Lemma 3.2. Under the stated model, [‖Cδ‖22]=σk2∑i∈S|Si|δi2≤σk2rK‖δS‖22,E [\|Cδ\|_2^2 ]\;=\; _k^2 _i∈ S|S_i|\, _i^2\;≤\; _k^2\,rK\,\| _S\|_2^2, where r=|S|r=|S| and s:=∑i∈S|Si|≤rKs:= _i∈ S|S_i|≤ rK is the multiplicity-corrected number of affected outputs. Proof. As before, ‖Cδ‖22=δ⊤C⊤Cδ=∑i,i′δiδi′(C⊤C)i,i′,\|Cδ\|_2^2=δ C Cδ= _i,i _i _i (C C)_i,i , and (C⊤C)i,i′=∑j=1M′Cj,iCj,i′.(C C)_i,i = _j=1^M C_j,iC_j,i . For fixed j, independence and zero-mean of the nonzero kernel entries imply [Cj,iCj,i′]=0E[C_j,iC_j,i ]=0 when i≠i′i≠ i (unless by model design both entries correspond to the same kernel parameter, which we excluded by independence). For i=i′i=i and j∈Sij∈ S_i, [Cj,i2]=σk2E[C_j,i^2]= _k^2; for j∉Sij ∈ S_i, Cj,i=0C_j,i=0. Therefore [(C⊤C)i,i′]=σk2|Si|,i=i′,0,i≠i′.E[(C C)_i,i ]= cases _k^2|S_i|,&i=i ,\\ 0,&i≠ i . cases Hence ‖Cδ‖22=∑i∈Sδi2σk2|Si|,E\|Cδ\|_2^2= _i∈ S _i^2 _k^2|S_i|, ∑i∈Sδi2σk2|Si|≤σk2(maxi∈S|Si|)∑i∈Sδi2≤σk2rK‖δS‖22, _i∈ S _i^2 _k^2|S_i|≤ _k^2 ( _i∈ S|S_i| ) _i∈ S _i^2≤ _k^2rK\| _S\|_2^2, which proves the lemma. ∎ Unlike the dense case, each perturbed input coordinate i affects only |Si||S_i| outputs. Thus, the total affected output count s scales with rKrK and may be much smaller than the dense output dimension M. The convolution preserves spatial locality and keeps the perturbation energy concentrated into a bounded neighborhood instead of being globally spread. 3.2 Multiplicative perturbations While the additive model provides a mathematically convenient and widely applicable framework for studying noise propagation, it assumes that perturbations are independent of the underlying signal. In many real-world systems, however, this assumption does not hold: the magnitude or direction of a disturbance often depends on the local signal amplitude itself. Typical examples include gain fluctuations in sensors, feature-dependent dropout, fading in communication channels, and multiplicative calibration errors in imaging systems. To capture such effects, we now turn to the multiplicative perturbation model, which introduces a signal-dependent scaling of each input dimension. Formally, we represent the perturbed input as x~=x⊙(1+ϵ), x=x (1+ε), (2) where ⊙ denotes the elementwise (Hadamard) product and ϵ∈ℝNε ^N is a random or deterministic perturbation vector supported on a subset S⊆1,…,NS \1,…,N\. For linear layers, this is precisely an additive perturbation with δ=diag(x)εδ=diag(x) . Equivalently, we can express the perturbation as x~=x+(x⊙ϵ)=x+δ, x=x+(x ε)=x+δ, (3) so that the effective additive perturbation is δi=xiϵi _i=x_i _i. This formulation highlights that the multiplicative model generalizes the additive one: the additive case corresponds to ϵi=δi/xi _i= _i/x_i for nonzero xix_i, i.e., perturbations that are independent of the input magnitude. Lemma 3.3. Let W as in Lemma 3.1 and let ε be supported on S. Then [‖W(diag(x)ε)‖22]=MσW2∑i∈Sxi2εi2.E [\|W(diag(x) )\|_2^2 ]=M\, _W^2 _i∈ Sx_i^2 _i^2. Proof. Set δ:=diag(x)εδ:=diag(x) , i.e. δi=xiεi _i=x_i _i. Apply Lemma 3.1 to δ. The result follows directly. ∎ Lemma 3.4. Let C as in Lemma 3.2 and ε supported on S. Then [‖C(diag(x)ε)‖22]=σk2∑i∈S|Si|xi2εi2≤σk2rK∑i∈Sxi2εi2.E [\|C(diag(x) )\|_2^2 ]= _k^2 _i∈ S|S_i|\,x_i^2 _i^2≤ _k^2\,rK _i∈ Sx_i^2 _i^2. Proof. Same substitution: put δ=diag(x)εδ=diag(x) and apply Lemma 3.2. ∎ Multiplicative perturbations scale with the local signal amplitude xix_i: areas with larger xix_i amplify relative errors εi _i into bigger absolute distortions. Structurally, convolution still confines these amplified distortions to small neighborhoods, whereas dense layers spread the signal-dependent distortion across all outputs. 3.3 Comparison of Propagation Dynamics Having established the separate propagation characteristics of additive and multiplicative perturbations through dense and convolutional layers, we now synthesize these results into a unified comparative analysis. While the individual lemmas quantified the expected perturbation energy in isolation, a direct comparison reveals the fundamental structural difference between global and local mappings. The following theorem formalizes this distinction by comparing the expected perturbation energies in both architectures under equivalent statistical assumptions. The subsequent corollary interprets these findings in terms of architectural design principles, linking the theoretical analysis to practical implications for anomaly detection and fault correction. Theorem 3.5. Under the assumptions of Lemma 3.1 and Lemma 3.2 and their multiplicative analogues, the expected squared output energies satisfy [‖Wδ‖22]=MσW2‖δS‖22,[‖Cδ‖22]=σk2∑i∈S|Si|δi2.E [\|Wδ\|_2^2 ]=M _W^2\| _S\|_2^2, [\|Cδ\|_2^2 ]= _k^2 _i∈ S|S_i| _i^2. Define s=∑i∈S|Si|≤rKs= _i∈ S|S_i|≤ rK. If σW2≈σk2 _W^2≈ _k^2 and s≪Ms M, then the convolutional output concentrates the perturbation energy into far fewer coordinates than the dense output. Consequently, the average per-affected-output expected energy is larger for the convolutional mapping: ‖Cδ‖22s≥‖Wδ‖22M, E\|Cδ\|_2^2s≥ E\|Wδ\|_2^2M, under σk2=σW2 _k^2= _W^2 and if the δi2 _i^2 are distributed appropriately. Proof. The equalities are restatements of Lemma 3.1 and Lemma 3.2. Dividing both sides by the number of affected outputs yields the per-output averages. If σk2=σW2 _k^2= _W^2 and s≪Ms M, then the denominator on the left is much smaller, so the typical per-output energy is larger for the convolutional mapping. ∎ Under the conditions of Theorem 3.5, suppose s≪Ms M and the variances σW2 _W^2 and σk2 _k^2 are of comparable order. Then, for a fixed perturbation energy ‖δS‖22\| _S\|_2^2, the convolutional layer concentrates this energy into a significantly smaller number of output coordinates than the dense layer. This concentration implies that, on average, the perturbation amplitude per affected output coordinate is higher for the convolutional layer, thereby increasing the salience of the anomaly signal in latent representations and enhancing its detectability. Remark 1. The independence and disjoint-support assumptions in Lemma 3.1 and Lemma 3.2 are idealized. In practical convolutional networks, receptive fields often overlap, so Si∩Si′≠∅S_i∩ S_i ≠ for distinct perturbed indices i and i′i , and the corresponding weight entries may not be perfectly independent (e.g., due to learned structure after training). Overlapping supports introduce additional cross terms in the expectation, which can either increase or decrease the total output perturbation energy depending on the correlation pattern. Nonetheless, as long as the effective number of affected outputs s remains much smaller than M, the qualitative conclusion of Theorem 1, that convolutional mappings localize perturbations relative to dense mappings, still holds. Quantifying the precise effect in the correlated and overlapping case requires extending the analysis to account for the joint covariance structure of the weights and receptive fields. This difference in perturbation propagation has practical implications for fault and anomaly detection in multimodal systems. The localized preservation of perturbation magnitude in convolutional networks enables more precise identification and localization of faults in image data, as the perturbations remain distinct and spatially confined in the latent representations. In contrast, dense layers used for 1-dimensional data tend to diffuse such perturbations throughout the latent space, potentially making fault signals less distinguishable. This means that architectural choices, specifically the use of convolutional versus dense layers, play a fundamental role in how faults manifest in latent representations. For effective multimodal fault detection, understanding and leveraging these propagation characteristics can guide the design of network components and training strategies that enhance anomaly sensitivity and robustness. 3.4 Training Dynamics and Noise-Dependent Lipschitz Sensitivity The previous analysis demonstrated that the architectural choice determines the spatial support of perturbation propagation. However, the magnitude of this propagation and the model’s overall stability depend not only on the connectivity pattern but also on the global sensitivity of the mapping fθf_θ. While our structural analysis was conducted in the static case with fixed parameters, in practice, these parameters evolve during optimization. Therefore, it is essential to understand how the training dynamics interact with the model’s sensitivity, and how these sensitivities can be optimized to either stabilize learning for correction or amplify discriminative features for anomaly detection. The measure of sensitivity is the Lipschitz constant of the network function with respect to its input. Let fθ:ℝn→ℝmf_θ:R^n ^m be a neural network parameterized by weights θ. We denote by Jf(x)=∂fθ(x)∂xJ_f(x)= ∂ f_θ(x)∂ x the Jacobian of fθf_θ with respect to its input x. The global Lipschitz constant is then defined as Lf=supx≠y‖fθ(x)−fθ(y)‖2‖x−y‖2, L_f= _x≠ y \|f_θ(x)-f_θ(y) \|_2\|x-y\|_2, (4) which measures the maximal amplification of input perturbations through the network. The target is to understand how training regimes and input noise modify LfL_f. This provides a quantitative measure of how stable or sensitive the network is, and thus how it may be guided by gradient clipping, Jacobian regularization, or noise-based training strategies. The following Theorem 3.6 presents the clean input Lipschitz bound. Theorem 3.6. Let fθ(x)f_θ(x) be an L-layer feed-forward network fθ(x)=WLϕL−1(WL−1ϕL−2(…ϕ1(W1x)…))f_θ(x)=W_L _L-1(W_L-1 _L-2(… _1(W_1x)…)) with weight matrices WℓW_ and elementwise activation functions ϕℓ _ that are KϕK_φ-Lipschitz continuous. Then, for all x,y∈ℝnx,y ^n, it holds that ‖fθ(x)−fθ(y)‖2≤(∏ℓ=1L‖Wℓ‖2Kϕ)‖x−y‖2.\|f_θ(x)-f_θ(y)\|_2≤ ( _ =1^L\|W_ \|_2K_φ )\,\|x-y\|_2. Hence, the global Lipschitz constant satisfies Lf(clean)≤KϕL∏ℓ=1L‖Wℓ‖2.L_f^(clean)≤ K_φ^L _ =1^L\|W_ \|_2. Proof. The proof follows directly from the submultiplicativity of operator norms and the chain rule for composition of Lipschitz functions. For each layer hℓ(x)=Wℓϕℓ−1(⋅)h_ (x)=W_ _ -1(·), we have ‖hℓ(x1)−hℓ(x2)‖2≤‖Wℓ‖2Kϕ‖x1−x2‖2.\|h_ (x_1)-h_ (x_2)\|_2≤\|W_ \|_2K_φ\|x_1-x_2\|_2. Applying this inequality recursively over L layers yields the stated bound. ∎ Remark 2. Connection to Architecture: The product ∏‖Wℓ‖2Π\|W_ \|_2 in Theorem 3.6 highlights the synergy with the findings in Section 3. While convolutional layers restrict the Jacobian’s sparsity structure, keeping the spectral norm ‖Wℓ‖2\|W_ \|_2 low ensures that the active paths do not amplify the signal excessively. Thus, robustness is a dual objective: structural localization via architecture and spectral control via training. This theorem describes the static sensitivity of the network to clean inputs. The Lipschitz constant grows multiplicatively with layer depth and the spectral norms of the weights. If each layer’s operator norm is tightly controlled, e.g., using spectral normalization or weight decay, the overall sensitivity LfL_f remains small, implying stable propagation of input differences. Conversely, large layer norms or strong activations lead to high LfL_f, which makes the network highly sensitive to small input changes, which is useful for detection tasks but detrimental to correction stability. We now consider the case where the input is corrupted by additive noise, x′=x+δx =x+δ, with δ∼(0,Σ)δ (0, ) or ‖δ‖≤ϵ\|δ\|≤ε. The following Theorem 3.7 shows how this noise affects the Lipschitz constant and the effective sensitivity of the network. Theorem 3.7. Under the same assumptions as Theorem 3.6, let x′=x+δx =x+δ and assume the Jacobian of fθf_θ is locally LJL_J-Lipschitz, i.e. ‖Jf(x)−Jf(y)‖2≤LJ‖x−y‖2∀x,y.\|J_f(x)-J_f(y)\|_2≤ L_J\|x-y\|_2 ∀ x,y. Then, the difference between the noisy and clean network outputs satisfies ‖fθ(x+δ)−fθ(x)‖2≤Lf(clean)‖δ‖2+12LJ‖δ‖22.\|f_θ(x+δ)-f_θ(x)\|_2≤ L_f^(clean)\|δ\|_2+ 12L_J\|δ\|_2^2. Consequently, the effective Lipschitz constant in the presence of input noise is Lf(noisy)≤Lf(clean)+12LJ‖δ‖2.L_f^(noisy)≤ L_f^(clean)+ 12L_J\|δ\|_2. Proof. Using the second-order Taylor expansion around x: f(x+δ)=f(x)+Jf(x)δ+R(x,δ),f(x+δ)=f(x)+J_f(x)δ+R(x,δ), where the remainder R(x,δ)R(x,δ) satisfies ‖R(x,δ)‖2≤12LJ‖δ‖22\|R(x,δ)\|_2≤ 12L_J\|δ\|_2^2 by Lipschitz continuity of the Jacobian. Taking norms and applying the bound for Jf(x)J_f(x) gives ‖f(x+δ)−f(x)‖2 \|f(x+δ)-f(x)\|_2 ≤‖Jf(x)‖‖δ‖2+12LJ‖δ‖22 ≤\|J_f(x)\|\|δ\|_2+ 12L_J\|δ\|_2^2 ‖f(x+δ)−f(x)‖2 \|f(x+δ)-f(x)\|_2 ≤Lf(clean)‖δ‖2+12LJ‖δ‖22. ≤ L_f^(clean)\|δ\|_2+ 12L_J\|δ\|_2^2. Dividing by ‖δ‖2\|δ\|_2 and extending by x yields: ‖f(x+δ)−f(x)‖2‖δ‖2≤Lf(clean)+12LJ‖δ‖2, \|f(x+δ)-f(x)\|_2\|δ\|_2≤ L_f^(clean)+ 12L_J\|δ\|_2, and therefore the effective Lipschitz constant under noisy input. ∎ This result quantifies how input noise modifies the effective sensitivity of the network. The clean Lipschitz constant Lf(clean)L_f^(clean) acts as the baseline, while the additional term 12LJ‖δ‖2 12L_J\|δ\|_2 represents the curvature-induced sensitivity due to local nonlinearity. Hence, as the input noise amplitude ‖δ‖2\|δ\|_2 increases, the local Lipschitz constant grows linearly, indicating that noise amplifies sensitivity in regions where the Jacobian varies rapidly. Practically, this has two crucial implications: (i) For correction or denoising tasks: one should minimize both Lf(clean)L_f^(clean) and LJL_J. This ensures that noise does not destabilize training and that reconstruction remains smooth. (i) For anomaly detection: A controlled increase of LJL_J in critical layers can enhance sensitivity to small perturbations, making anomalous deviations more detectable. However, excessive curvature can make training unstable and lead to gradient explosion. Thus, by controlling Lf(clean)L_f^(clean) and LJL_J through appropriate training strategies, one can design networks that either suppress or amplify the propagation of noise, a fundamental mechanism behind robust correction versus sensitive detection architectures. 4 Multimodal alignment-based failure correction The mathematical results from Theorem 3.5 directly translate into architectural design. For effective anomaly detection, the perturbation must remain localized. Convolutional, graph-based, or local-attention layers are ideal for this purpose, as they maintain the geometric or topological coherence of anomalies. For correction or robust averaging, the opposite property is beneficial: spreading the perturbation reduces local intensity and allows recovery by averaging. Dense or strongly mixing architectures act as perturbation diffusers, approximating an isotropic error field where localized anomalies are removed. In the following sections, we introduce our proposed fault-tolerant two-staged self-supervised approach presented in Figure 1. Figure 1: Overview of the proposed fault-tolerant approach for robotic systems. The architecture includes encoding, correction with contrastive learning and detection modules, and decoding stages to reconstruct the original signal. 4.1 Problem Definition Let =[i]i=1MX=[X_i]_i=1^M denote the multimodal input space, where each modality iX_i corresponds to a distinct sensory or data domain. The model consists of modality-specific encoders Ei:i→ℝziE_i:X_i ^z_i and decoders Di:ℝzi→iD_i:R^z_i _i, which map the latent spaces back to the input. The latent codes are aggregated into a joint representation via a learnable fusion operator C:⨄i=1Mℝzi→ℝz,z~=C([Ei(xi)]i=1M), C: _i=1^MR^z_i ^z, z=C([E_i(x_i)]_i=1^M), (5) which is designed to detect latent inconsistencies and correct modality-specific faults. To model corruptions, each modality may undergo a stochastic perturbation f(i):i→if^(i):X_i _i such that xifail=f(i)(xi,δi)x_i^fail=f^(i)(x_i, _i), where δi _i represents the disturbance. The objective is to learn parameters (θi,ϕi,ψ)( _i, _i,ψ) of (Ei,Di,C)(E_i,D_i,C) minimizing the expected multimodal reconstruction loss under corrupted observations, while enforcing contrastive consistency in the latent space: minθ,ϕ,ψx,δ[∑i=1M _θ,φ,ψE_x,δ [ _i=1^M ‖xi−Di(z~)‖22 \|x_i-D_i( z)\|_2^2 +λ1ℒcon(E(x),E(f(x))) + _1L_con(E(x),E(f(x))) (6) +λ2ℒsim(C(E(f(x))),E(x))]. + _2L_sim(C(E(f(x))),E(x)) ]. Here, ℒconL_con enforces separation between clean and corrupted representations for anomaly detection, whereas ℒsimL_sim promotes post-correction alignment for reconstruction consistency. 4.2 Pre-trained Multimodal Autoencoder Our proposed method in Figure 1 is a two-stage framework that utilizes a pre-trained multimodal autoencoder consisting of encoder E and decoder D, a corrector (f(z)≈f(z+ϵ)f(z)≈ f(z+ε)) for robust data correction, and an anomaly detection. The core components are defined as follows: (i) Encoder E:ℝN→ℝdE:R^N ^d: Maps the high-dimensional input space ℝNR^N to a low-dimensional latent space ℝdR^d. (i) Decoder (D:ℝd→ℝND:R^d ^N): Reconstructs the input from the latent code. (i) Corrector (C:ℝd→ℝdC:R^d ^d): Corrects corrupted inputs in the latent space. (iv) Detector (M:ℝd→[0,1]dM:R^d→[0,1]^d): Measures the level of anomaly in the latent representation and outputs a score. Pre-training the autoencoder as a denoising one exposes it to perturbations. While this improves reconstruction robustness, it fundamentally contradicts the requirements for anomaly detection derived in subsection 3.4. As shown in Theorem 3.7, denoising forces the encoder to minimize its sensitivity to input perturbations. However, explicitly suppressing perturbations in E would diminish the anomaly signal in the latent representation z, rendering the subsequent detector M ineffective. Therefore, we pretrain the multimodal autoencoder exclusively on clean data. This preserves the encoder’s sensitivity to out-of-distribution shifts, ensuring that localized perturbations propagate into the latent space, consistent with the structural analysis in Theorem 3.5. After pre-training, the weights of encoder E and decoder D are frozen, establishing a fixed, deterministic mapping. With (E,D)(E,D) fixed, the objective shifts to training the corrector C to invert the effect of a perturbation δ. This decouples the conflicting goals of sensitivity assigned to E and robustness assigned to C. 4.3 Self-supervised Compute for Failure Detection and Correction At the core of the proposed architecture lies the compute block C:⨄i=1Mℝzi→ℝzC: _i=1^MR^z_i ^z, which aggregates modality-specific latent representations into a unified, fault-corrected embedding. For each modality i, let zi=Ei(xi),zifail=Ei(f(i)(xi)), z_i=E_i(x_i), z_i^fail=E_i(f^(i)(x_i)), (7) denote the clean and corrupted latent codes, respectively. The clean and corrupted aggregated embeddings are thus zc=C([zi]i=1M),zf=C([zifail]i=1M). z_c=C([z_i]_i=1^M), z_f=C([z_i^fail]_i=1^M). (8) The function C serves a dual objective: (i) before correction, the representations zifail\z_i^fail\ should remain separable from their clean counterparts to enable fault detection, and (i) after correction, the fused representation zfz_f should align closely with zcz_c to ensure consistency and recovery. These objectives are enforced via two competing regularizers. To encourage distinguishability of clean and corrupted states, a contrastive margin loss is applied in the pre-compute space: ℒcon=[max(0,m−‖zc−zf‖2)], _con=E [ (0,\,m-\|z_c-z_f\|_2) ], (9) where m>0m>0 denotes a minimum separation margin. The loss function, ℒconL_con, actively maximizes the ℓ2 _2-distance between clean (zcz_c) and corrupted (zfz_f) embeddings, driving this separation to be greater than the margin (m). Therefore, minimizing ℒconL_con forces the system to satisfy the constraint ‖zc−zf‖2≥m\|z_c-z_f\|_2≥ m. Under a global Lipschitz constraint, the encoder satisfies ‖zc−zf‖2≤L‖x−(x+)‖2=L‖2. \|z_c-z_f\|_2≤ L\|x-(x+n)\|_2=L\|n\|_2. (10) This means that when L is small, the embeddings of x and x+x+n remain close, indicating robustness to the perturbation n. In contrast, a margin-based contrastive loss encourages distinguishability by enforcing a minimal separation m: ‖zc−zf‖2≥m(ℒcon→0). \|z_c-z_f\|_2≥ m (L_con→ 0). (11) If the encoder is indeed constrained to be L-Lipschitz, these two conditions combine to give m≤‖zc−zf‖2≤L,‖2⇒L≥m‖2. m≤\|z_c-z_f\|_2≤ L,\|n\|_2 L≥ m\|n\|_2. (12) Thus, under a Lipschitz constraint, the contrastive loss effectively forces the model to allocate a sufficiently large local Lipschitz constant for the specific perturbation pair (x,x+)(x,x+n). In other words, even if ‖2\|n\|_2 is small, the representation must change by at least m, causing the encoder to become locally sensitive along the direction defined by n. Importantly, this effect applies only when a global Lipschitz bound is enforced. Without such a constraint, the network can achieve the separation m without implying any particular Lipschitz behavior. After aggregation, the compute block should contract corrupted representations toward the clean manifold: ℒsim=[‖C([zifail])−C([zi])‖22], _sim=E [\|C([z_i^fail])-C([z_i])\|_2^2 ], (13) which minimizes the distortion introduced by perturbations. This term enforces stability and correction capability. The overall loss governing the computational dynamics is ℒC=λconℒcon+λsimℒsim, _C= _conL_con+ _simL_sim, (14) with weights λcon,λsim>0 _con, _sim>0 balancing detection sensitivity and correction fidelity. Let LCL_C denote the Lipschitz constant of C. If λcon≫λsim _con _sim, optimization emphasizes representation separation, increasing LCL_C and thus sensitivity to perturbations, which is beneficial for anomaly detection but prone to overreacting to minor noise. Conversely, if λsim≫λcon _sim _con, C is trained as a contraction mapping (LC<1L_C<1), ensuring robustness and fault correction but reducing detectability of small anomalies. The optimal trade-off occurs when C maintains a piecewise Lipschitz behavior: expansive in directions aligned with anomaly subspaces, and contractive along nominal data manifolds. This duality formalizes the balance between contrastive and similarity alignment, allowing the compute block to serve simultaneously as a fault detector and corrector within the multimodal latent space. 4.4 Layer-Specific Lipschitz Regularization Strategy From Theorem 3.7, the local Taylor expansion under input perturbations reveals that the deviation |f(x+δ)−f(x)||f(x+δ)-f(x)| decomposes into two fundamentally different contributions: a first-order term governed by the Jacobian norm Lf(clean)=‖Jf(x)‖L_f^(clean)=\|J_f(x)\|, and a second-order term controlled by the Jacobian variation LJ=‖Jf(x+δ)−Jf(x)‖L_J=\|J_f(x+δ)-J_f(x)\|. The former determines the layer’s intrinsic sensitivity to small input changes, while the latter quantifies curvature and noise amplification. These effects play opposing functional roles in anomaly detection, where high sensitivity is beneficial, and correction or reconstruction, where strong contraction and low curvature are required. Therefore, a uniform Lipschitz constraint across the entire network is suboptimal. Instead, the Theorem 3.7 motivates a layer-specific regularization strategy in which encoder layers are allowed, and even encouraged, to maintain higher Jacobian norms for enhanced perturbation sensitivity, while compute and reconstruction layers are regularized to enforce low LfL_f and LJL_J, promoting stability and contraction. Formally, let the network be partitioned into NEN_E encoder layers E(ℓ)ℓ=1NE\E^( )\_ =1^N_E and NCN_C compute block layers C(m)m=1NC\C^(m)\_m=1^N_C. Each layer f(ℓ)f^( ) is characterized by its local Jacobian J(ℓ)=∇xf(ℓ)(x)J^( )= _xf^( )(x) and corresponding local Lipschitz quantities Lf(ℓ) L_f^( ) =‖J(ℓ)‖2, =\|J^( )\|_2, (15) LJ(ℓ) L_J^( ) =x,δ[‖J(ℓ)(x+δ)−J(ℓ)(x)‖F]. =E_x,δ [\|J^( )(x+δ)-J^( )(x)\|_F ]. (16) The first term measures the local sensitivity of clean inputs, while the second quantifies the curvature or Jacobian variation under noise. Controlling these constants enables architectural tuning between stability and sensitivity. We assign different regularization objectives depending on the layer role: ℒreg _reg =∑ℓ∈ℰ(λcon(ℓ)‖J(ℓ)(x+δ)−J(ℓ)(x)‖F2) = _ ( _con^( )\|J^( )(x+δ)-J^( )(x)\|_F^2 ) (17) +∑m∈(λcorr(m)‖J(m)(x)‖F2), + _m ( _corr^(m)\|J^(m)(x)\|_F^2 ), where ℰE and C denote the encoder and compute-layer index sets, respectively. The weighting coefficients λcon(ℓ) _con^( ) and λcorr(m) _corr^(m) are selected such that λcon(ℓ)≈λcorr(m) _con^( )≈ _corr^(m), encouraging similar curvature in the encoder and the compute block. Therefore, the overall loss for optimizing the compute block results in: ℒ=ℒC+ℒreg. =L_C+L_reg. (18) Practically, this analysis leads to two complementary but spatially separated regimes. For the correction and denoising regime, the objective is to minimize both Lf(clean)L_f^(clean) and LJL_J to enforce contraction mappings. This is achieved through spectral norm regularization, Jacobian penalties, or gradient clipping in layers beyond the anomaly detection. Formally, gradient clipping constrains the local Jacobian norm via g←g⋅min(1,τ‖g‖2), g← g· \! (1, τ\|g\|_2 ), (19) where τ>0τ>0 is the clipping threshold, effectively bounding LJ≤τL_J≤τ. This stabilizes training and ensures smooth reconstruction in the latent correction phase. To enhance the sensitivity of specific layers to localized perturbations for the anomaly detection regime without inducing gradient explosion, we adopt a selective gradient amplification strategy combined with global renormalization. Let θ denote all network parameters and θa⊂θ _a⊂θ the subset associated with the amplification region. During backpropagation, the gradients corresponding to θa _a are scaled by an amplification factor α>1α>1, while the overall gradient norm is constrained by a clipping threshold τ. Formally, for a given loss function ℒL, we compute g~=clip(g[θ∖θa]∪ga,τ), g=clip (g[θ _a]∪ g_a,τ ), (20) with g=∇θℒ,ga=αg[θa], g= _θL, g_a=α g[ _a], where clip(g,τ)=g⋅min(1,τ‖g‖2)clip(g,τ)=g· \! (1, τ\|g\|_2 ). This operation amplifies directional gradients in targeted layers to increase the local Lipschitz constant, thereby enhancing the model’s sensitivity to small perturbations. Meanwhile, renormalization ensures that the overall gradient energy remains bounded, ensuring training stability. The corresponding pseudocode is presented in Algorithm 1. Practically, this mechanism allows the model to emphasize anomaly-relevant features without compromising convergence or introducing instability. Algorithm 1 Selective Gradient Scaling and Renormalization. 1:amplification factor α>1α>1, clipping threshold τ>0τ>0, learning rate η 2:for each minibatch (x,y)(x,y) do 3: Compute loss ℒ=ℒtask(x,y)L=L_task(x,y) 4: Compute full gradient g=∇θℒg= _θL 5: Identify target parameter subset θa⊂θ _a⊂θ 6: Amplify gradients on θa _a: g[θa]←α⋅g[θa]g[ _a]←α· g[ _a] 7: Compute total norm: n=‖g‖2n=\|g\|_2 8: if n>τn>τ then 9: Normalize gradients: g←g⋅τng← g· τn 10: Update parameters: θ←θ−ηgθ←θ-η\,g This layer-wise Lipschitz modulation establishes a directional flow of sensitivity across the architecture: early and middle convolutional encoder layers amplify and preserve perturbations for detection, while the subsequent compute block acts as a Lipschitz contraction, globally smoothing and correcting these perturbations. The resulting gradient field ∇xf(x) _xf(x) transitions from expansive to contractive along the network depth, realizing an effective detect–correct framework that is mathematically grounded in controlled Lipschitz geometry. 5 Industrial Deployment and Practical Application In this section, we outline the integration of the proposed MMSSL framework within a real-world industrial production environment. The deployment architecture, illustrated in Figure 2, streamlines industrial integration through a three-stage pipeline comprising data ingestion, intelligent routing, and downstream execution. The system continuously ingests N heterogeneous modalities via dedicated encoders (E1,…,EnE_1,…,E_n) that map high-dimensional signals into a unified latent feature space while preserving perturbation sensitivity. At the core of the framework lies a conditional logic gate driven by a lightweight anomaly detector. Unlike traditional autoencoders that indiscriminately reconstruct all inputs, our approach dynamically routes the data based on its health. If the detector classifies the embedding as clean, the representation bypasses the correction block entirely, minimizing inference latency and preserving original signal fidelity. Conversely, upon detection of faults such as sensor corruption or occlusion, the embedding is routed through the Lipschitz-regularized correction module, which projects the distorted features back onto the learned clean manifold. This mechanism ensures that downstream applications receive a guaranteed valid representation, effectively decoupling high-level process control from low-level sensor reliability. Modality 1(Camera)Modality 2(Sensor)⋮ N(Audio/Log)Encoder 1Encoder 2⋮ N Fusion & Latent Space Failure Detected? Correction Module(Lipschitz Reg.) Downstream Tasks ∙ Quality Control ∙ KPI Estimation ∙ Process Control zzYESzcorrectedz_correctedzcleanz_cleanPhysical LayerEncoding LayerReliability LayerNOApplication Layer Figure 2: Operational pipeline for robust industrial deployment. Heterogeneous inputs are encoded into a joint latent space. The representation is analyzed for faults: clean data bypasses the correction block for efficiency, while faulty data is rectified via the Lipschitz-controlled module before executing downstream tasks. 6 Evaluation This section details the evaluation of our proposed fault tolerance framework. We begin by introducing the multimodal datasets leveraged for training and testing, along with a description of the experimental setup, including key hyperparameter choices. We then present and discuss the corresponding quantitative results, which validate the efficacy of our approach in achieving superior detection and correction capabilities. 6.1 Datasets Our experiments utilize three distinct multimodal datasets sourced from the industrial robotics domain and are based on the work [7]. These datasets, which are visually introduced in Figure 3, are crucial because they act as digital twins of real-world industrial machinery. They are built on authentic operating data and precisely model the physics of actual sensors and systems. Crucially, they include varying levels of task difficulty and structural diversity. Formally, each dataset is composed of N total data samples. A single data sample at index j is a collection of M observations, (j(1),…,j(M))(x_j^(1),…,x_j^(M)), taken at the same time. The data from each of the M different sensor types has its own space, ℳi⊆ℝdiM_i ^d_i, with the specific sensor reading denoted as j(i)∈ℳix_j^(i) _i. (a) MuJoCo (b) ABB single robot (c) ABB dual robot Figure 3: Three image modality samples of the three distinct multimodal datasets [7]. In this work, we focus on combining and reconstructing two specific types of paired multimodal data: spatial image data S and kinematic signals K. Each of these two data modalities is indexed by m∈s,km∈\s,k\. The kinematic sequences K, which represent motion or force, are straightforwardly encoded as matrices in ℝdkR^d_k, where dkd_k is the number of sensor dimensions recorded at each point in time. In contrast, the visual observations S are structured as standard, static RGB images. We use a fixed spatial resolution where the row and column sizes are equal, sr=sc=256s_r=s_c=256, meaning the images are represented as tensors in ℝ3×sr×scR^3× s_r× s_c. 6.2 Failure Injection The fault injection process is designed based on the methodologies established in the work by [5], which simulate real-world degradation mechanisms. Specifically, the resulting failures for the image modality are visually presented in Figure 4. In industrial applications, such degradation patterns often correspond to occlusions or mechanical wear, where faults manifest globally and locally with minimum one clean modality. This approach ensures that the network is exposed to representative multimodal failure modes, enabling it to learn fault dependencies during training. Figure 4: Several randomly applied augmentation techniques on one random camera sample. 6.3 Experimental Setup To validate the MMSSL framework for robotic sensory data, we implement the architecture in Figure 1 using visual and proprioceptive branches connected via a correction manifold. Preserving anomaly topology as described in Section 4, the spatial encoder E1E_1 employs four 2D convolutional layers (channels 32-256-128-64-32-32-32, kernel 5×55× 5, stride 2, padding 1). The sensor encoder E2E_2 uses a 1D convolutional network to maintain locality, processing dimension n0n_0 through five layers with kernel size of 3 and stride of 1 to the depth of 288. Latent vectors are concatenated and reconstructed by decoders D1,D2D_1,D_2, mirroring the encoders. The compute block comprises four fully connected layers with size of 576 and ReLU activations, but with the last layer transforming back to the shape of 288. Training proceeds in two isolated stages as described in subsection 3.4. First, the multimodal autoencoder is pre-trained on unperturbed data for 200 epochs using Adam optimizer and the configuration of ηAE=10−4 _AE=10^-4, β1=0.9,β2=0.999 _1=0.9, _2=0.999. Second, the compute block C is trained to map corrupted to clean embeddings for 200 epochs with batchsize of 16 to minimize ℒtotal=ℒrec+λ1ℒcon+λ2ℒsimL_total=L_rec+ _1L_con+ _2L_sim, with λ1=0.1,λ2=1.0 _1=0.1, _2=1.0, and margin m=0.5m=0.5. Lipschitz constraints enforce contraction via clipping τ=1.0τ=1.0 and sensitivity via amplification α=3.0α=3.0. 6.4 Results This section reports the experimental results of our proposed framework, providing a comparative analysis with state-of-the-art denoising models, including multimodal autoencoder (MMAE) [10], multimodal variational autoencoder (MMVAE) [10], multimodal fuzzy regularization (MMFR) [4], multimodal concept fusion (MMCF) [6], multimodal GAN correction (MMGAN) [9], and multimodal U-GAN correction (MMUGAN) [9]. Our evaluation is structured into four subsections. We begin by assessing the overall performance, followed by an in-depth investigation of the convergence behavior. Then, we discuss the fault detection capabilities, a metric that is intrinsic to our model’s architecture, and finally, we prove empirically the theory from Section 3. 6.4.1 Failure Correction Performance We begin our analysis by examining the general reconstruction fidelity of the models under failure conditions, with a specific focus on benchmarking our proposed self-supervised learning framework. Unlike generative adversarial approaches that rely on complex discriminator dynamics or variational methods that optimize a probabilistic lower bound, our approach leverages self-supervised geometric constraints to guide the correction process. Table 1 details the training and testing performance on the MuJoCo dataset, comparing these distinct paradigms against other state-of-the-art approaches. The table explicitly decomposes the error contributions into high-dimensional camera data and low-dimensional sensor data, providing a detailed view of how effectively the self-supervised signal competes with the state-of-the-art in multimodal failure correction. Table 1: Train and test performances (in 10−210^-2) of the failure correction using the mean over the final 250 batches of 10 trials. The reconstruction performance of the camera, the sensor, and both combined within the MuJoCo dataset. Model Camera Sensor Combined Training MMAE 0.114 ± 0.063 0.815 ± 0.562 0.929 ± 0.584 MMVAE 0.137 ± 0.026 0.694 ± 0.435 0.831 ± 0.442 MMFR 0.129 ± 0.033 0.973 ± 0.500 1.102 ± 0.504 MMCF 0.113 ± 0.017 0.165 ± 0.077 0.279 ± 0.079 MMGAN 0.113 ± 0.025 0.158 ± 0.097 0.209 ± 0.101 MMUGAN 0.123 ± 0.027 0.257 ± 0.182 0.313 ± 0.186 MMSSL 0.098 ± 0.023 0.307 ± 0.302 0.405 ± 0.310 Testing MMAE 0.145 ± 0.060 1.453 ± 0.214 1.599 ± 0.183 MMVAE 0.164 ± 0.011 1.395 ± 0.136 1.560 ± 0.131 MMFR 0.142 ± 0.023 1.318 ± 0.121 1.461 ± 0.123 MMCF 0.154 ± 0.006 0.894 ± 0.122 1.048 ± 0.121 MMGAN 0.134 ± 0.005 0.894 ± 0.055 0.950 ± 0.056 MMUGAN 0.125 ± 0.002 0.779 ± 0.018 0.835 ± 0.019 MMSSL 0.118 ± 0.005 0.405 ± 0.040 0.523 ± 0.042 The quantitative evaluation on the MuJoCo dataset in Table 1 demonstrates the superior generalization capability of the proposed MMSSL framework. While generative baselines like MMGAN achieve competitive training fits, they struggle to sustain this performance during testing, likely due to overfitting on the noise distribution. In contrast, MMSSL achieves a state-of-the-art combined test error of 0.523, reducing the error of the closest competitor (MMUGAN, 0.8350.835) by approximately 37%. Our method dominates in the sensor modality (0.4050.405 compared to 0.7790.779 for MMUGAN), indicating that the model capture underlying kinematic constraints more effectively. Furthermore, the minimal standard deviation (± 0.042) confirms that the Lipschitz-constrained correction acts as a stable contraction mapping, ensuring deterministic recovery and avoiding the training volatility and mode collapse often observed in the adversarial baselines. To validate this improvement and verify the geometric alignment of the corrected states, we visualize the topology of the learned latent manifold. The following t-SNE embedding in Figure 5 illustrates the distribution of clean, faulty, and corrected representations, demonstrating the model’s ability to map perturbed samples back onto the nominal data manifold. (a) Clean - Corrupted (b) Clean - Corrected Figure 5: Visualization of the learned latent feature space using t-SNE. (a) Distribution of clean (purple) versus corrupted (yellow) samples before correction. (b) Distribution of clean (purple) versus corrected (yellow) samples. As illustrated, the corrected representations in 5(b) exhibit a near-perfect distributional overlap with the clean baseline, effectively covering the nominal manifold. This confirms that the MMSSL framework successfully projects faulty inputs back onto the valid data distribution. Crucially, the distinct cluster structure of the clean data is fully preserved during this projection. This indicates that the correction process restores physical consistency without collapsing the latent space or discarding semantic information, effectively acting as a precise geometric projection operator. To validate the scalability and robustness of the proposed framework, we extend our evaluation to a more complex and realistic scenario: the single-robot welding station dataset. This dataset introduces higher variability in sensor noise and more intricate kinematic constraints compared to MuJoCo. Table 2 summarizes the reconstruction performance across all baseline models for this setup. Table 2: Train and test performances (in 10−210^-2) of the failure correction using the mean over the final 250 batches of 10 trials. The reconstruction performance of the camera, the sensor, and both combined within the single-robot welding station dataset. Model Camera Sensor Combined Training MMAE 0.131 ± 0.121 1.022 ± 1.203 1.153 ± 1.282 MMVAE 0.158 ± 0.093 0.961 ± 1.131 1.119 ± 1.180 MMFR 0.142 ± 0.081 0.927 ± 1.092 1.069 ± 1.133 MMCF 0.081 ± 0.022 0.602 ± 0.651 0.683 ± 0.643 MMGAN 0.074 ± 0.035 0.563 ± 0.997 0.637 ± 1.004 MMUGAN 0.073 ± 0.016 0.540 ± 0.531 0.613 ± 0.538 MMSSL 0.079 ± 0.032 0.478 ± 1.044 0.557 ± 1.048 Testing MMAE 0.141 ± 0.120 1.223 ± 0.681 1.364 ± 0.802 MMVAE 0.159 ± 0.092 1.050 ± 0.561 1.209 ± 0.653 MMFR 0.138 ± 0.083 1.071 ± 0.582 1.209 ± 0.661 MMCF 0.091 ± 0.011 0.702 ± 0.120 0.793 ± 0.122 MMGAN 0.084 ± 0.031 0.589 ± 0.476 0.673 ± 0.478 MMUGAN 0.081 ± 0.007 0.598 ± 0.072 0.679 ± 0.074 MMSSL 0.079 ± 0.003 0.496 ± 0.085 0.575 ± 0.088 The results on the single-robot welding dataset confirm the robustness of the MMSSL framework in complex industrial environments. Our self-supervised approach achieves a state-of-the-art combined reconstruction error of 0.575, significantly outperforming the best generative baseline (MMGAN: 0.673) by a margin of approximately 15%. MMSSL demonstrates superior reliability in the sensor modality again, reducing the error to 0.496 compared to 0.589 for MMGAN. While adversarial methods achieve competitive pixel-level camera errors, they exhibit significant volatility in the combined metrics (e.g., MMGAN ± 0.478), indicating a failure to consistently capture multimodal correlations. In contrast, MMSSL enforces semantic consistency through its geometric objective, ensuring that the model prioritizes physical reliability. This results in a stable contraction mapping (± 0.088) that effectively filters heavy industrial noise without the stochastic hallucinations typical of generative approaches. Finally, we scale the evaluation to the most complex scenario: the dual-robot welding station. This dataset introduces intricate inter-agent dependencies and doubles the sensor dimensionality, significantly increasing the difficulty of modeling kinematic correlations. Table 3 summarizes the results, revealing a critical failure mode in baseline architectures. Table 3: Train and test performances (in 10−210^-2) of the failure correction using the mean over the final 250 batches of 10 trials. The reconstruction performance of the camera (C), the sensor (S), and both combined (B) within the dual-robot welding station dataset. Model Camera Sensor Combined Training MMAE 0.031 ± 0.032 1.122 ± 1.241 1.153 ± 1.273 MMVAE 0.030 ± 0.011 0.962 ± 1.092 0.992 ± 1.103 MMFR 0.051 ± 0.041 1.121 ± 1.240 1.172 ± 1.281 MMCF 0.021 ± 0.010 0.872 ± 1.011 0.893 ± 1.021 MMGAN 0.078 ± 0.024 0.767 ± 0.562 0.844 ± 0.564 MMUGAN 0.077 ± 0.023 0.779 ± 0.729 0.856 ± 0.731 MMSSL 0.076 ± 0.024 0.683 ± 2.063 0.759 ± 2.069 Testing MMAE 0.032 ± 0.031 1.251 ± 0.322 1.283 ± 0.353 MMVAE 0.030 ± 0.001 1.082 ± 0.162 1.112 ± 0.163 MMFR 0.054 ± 0.031 1.274 ± 0.341 1.328 ± 0.372 MMCF 0.020 ± 0.001 0.981 ± 0.211 1.001 ± 0.212 MMGAN 0.080 ± 0.006 0.866 ± 0.142 0.945 ± 0.143 MMUGAN 0.077 ± 0.003 0.907 ± 0.167 0.984 ± 0.165 MMSSL 0.078 ± 0.002 0.777 ± 0.154 0.855 ± 0.152 The evaluation on the dual-robot dataset reveals a critical insight regarding multimodal learning dynamics. Baselines such as MMCF and MMVAE achieve superior pixel-level reconstruction (e.g., MMCF: 0.0200.020), yet this comes at the cost of poor sensor fidelity (0.9810.981). This disparity suggests a modality collapse, where the model minimizes the global loss by overfitting to the high-dimensional visual stream while neglecting the lower-dimensional, yet physically critical, sensor data. In contrast, MMSSL prioritizes consistency over superficial visual sharpness. While its camera reconstruction error is higher, it achieves the lowest sensor error (0.7770.777), resulting in the best overall combined performance (0.8550.855). By outperforming the closest generative baseline (MMGAN) by approximately 10% and the best autoencoder baseline (MMCF) by 15%, MMSSL demonstrates that its self-supervised objective successfully fuses the complex dual-robot kinematic streams, preventing visual dominance and ensuring physically grounded fault correction. 6.4.2 Convergence and Lipschitz behaviour Having established the superior failure correction performance of MMSSL across multiple datasets, we now turn our attention to the underlying optimization dynamics that enable this performance. While the previous section quantified the final model performance, this section analyzes the convergence stability and the spectral properties of the learned networks. (a) Contrastive vs. Similarity (b) Sensor vs. Image (c) Anomaly Loss (d) Lipschitz Constraints Figure 6: Training dynamics and spectral properties of MMSSL. (a) Both contrastive and similarity losses converge rapidly, with test losses (dashed) closely tracking training curves. (b) Reconstruction errors for both modalities decrease monotonically, confirming stable learning. (c) The anomaly detection loss stabilizes after an initial adaptation phase. (d) The spectral norm of the Corrector (green) is strictly bounded (<0.03<0.03), ensuring a contractive mapping, while the Detector (red) maintains high sensitivity (>2.0>2.0). To verify the optimization stability, we compare the training loss trajectories in Figure 6. As illustrated in 6(a) and 6(b), our self-supervised objective yields a smooth and fast convergence profile. The contrastive and similarity losses in 6(a) decrease monotonically, indicating that the encoder successfully learns a coherent multimodal latent space. Crucially, the test losses (dashed lines) closely track the training curves without divergence, demonstrating robust generalization and the absence of overfitting. Similarly, the reconstruction losses for both sensor and image modalities in 6(b) exhibit stable decay, reaching magnitudes of 10−310^-3 and 10−210^-2 respectively. Furthermore, we analyze the spectral properties of the network to validate our theoretical stability guarantees. 6(d) plots the evolution of the estimated Lipschitz constants for the detection and correction modules. The results confirm our layer-specific regularization strategy. The detector maintains a high Lipschitz value (L>2.0L>2.0), preserving the high-frequency sensitivity required to identify subtle anomalies. In stark contrast, the correction is effectively constrained to a low-Lipschitz regime (L≈0.01L≈ 0.01), empirically satisfying the contraction mapping condition L<1L<1. This spectral gap ensures that while the system remains highly sensitive to faults, the correction process dampens perturbations, preventing error amplification during the recovery phase. 6.4.3 Failure detection While robust correction is the primary objective, the reliability of the system hinges on its ability to accurately flag anomalous states. Table 4 summarizes the fault detection performance aggregated over 250 batches. Table 4: Fault detection performance aggregated over 250 batches. Metrics are macro-averaged by class. Class Precision Recall F1-Score Clean 0.929 0.999 0.963 Sensor Failure 0.997 0.839 0.911 Camera Failure 0.996 0.860 0.923 Macro Avg 0.974 0.900 0.933 The proposed framework achieves a high macro F1-Score of 0.933, driven by near-perfect precision across all classes (>99%>99\% for faults). This indicates that the system is highly conservative. It generates virtually no false alarms, a critical property for autonomous industrial monitors where false positives can lead to costly downtime. To understand the optimization stability, we analyze the metric evolution in Figure 7. (a) F1 & Accuracy (b) Class-wise Recall Figure 7: Training dynamics of the failure detector. (a) The model converges rapidly to a stable high-performance regime. (b) While clean states are identified immediately, the model progressively improves its sensitivity to sensor and camera faults, eventually plateauing at robust recall levels. The global accuracy and F1-score in 7(a) exhibit rapid convergence, stabilizing within the first 3,000 batches with decreasing variance. 7(b) decompose this performance by class and reveals the expected hierarchy of difficulty. The detection of the clean state (blue) saturates almost immediately, whereas the fault classes require a longer adaptation phase. Notably, the Sensor Fault recall (red) and Camera Fault recall (green) rise competitively, eventually stabilizing at ≈85%≈ 85\%. The initial volatility in the fault curves highlights the challenge of learning the separation margin m in the latent space during the early contrastive phase. The aggregated confusion matrix in Figure 8 provides insight into the remaining error modes. Figure 8: Normalized confusion matrix aggregated over 250 batches. The dominant diagonal confirms high classification accuracy. Off-diagonal errors are almost exclusively false negatives, identifying the detection threshold for subtle anomalies as the limiting factor. The model exhibits perfect cross-modal separation. The confusion between sensor faults and camera faults is negligible (<0.2%<0.2\%). This confirms that the encoder successfully maps these modalities to distinct subspaces as designed. The primary error source is the misclassification of faults as clean (False Negatives: 15.9%15.9\% for sensor, 13.9%13.9\% for camera). This suggests that certain low-magnitude perturbations fall within the natural variance of the clean data manifold or the enforced margin m. However, given the high precision (>99%>99\%), when the model does signal a fault, the prediction is exceptionally reliable. 6.4.4 Failure Propagation Analysis To empirically validate the theoretical analysis regarding perturbation propagation derived in Section 3, we analyzed the latent space activation patterns under localized input faults. Lemma 3.1 and Lemma 3.2 suggest a fundamental structural difference: dense layers are expected to diffuse localized faults globally across the feature space, whereas convolutional layers should preserve spatial locality, resulting in a sparse error profile. To verify this, we injected localized block perturbations into the input and tracked the magnitude of the latent deviation |δz|=|zfault−zclean|| _z|=|z_fault-z_clean| across feature dimensions. Figure 9 visualizes the raw error magnitude across 150 test samples. (a) Fully connected (b) Convolutional Figure 9: Latent error heatmaps for dense (a) and convolutional (b) encoders. The dense sensor encoder exhibits a blur effect, characterized by continuous vertical lines where a single input fault activates nearly all latent dimensions simultaneously. This confirms the global mixing property of fully connected layers. In contrast, the convolutional camera encoder displays a distinct block-sparse pattern. Active error regions are separated by large zones of zero magnitude (black), indicating that the perturbation remains spatially confined to specific feature maps, preserving the geometric structure of the anomaly. To quantify this behavior, we computed the distribution of absolute error magnitudes in the latent space, shown in Figure 10. Figure 10: Log-histogram of latent error magnitudes. The convolutional encoder (orange) exhibits a sharp peak at zero, whereas the dense encoder (blue) shows a broad distribution, confirming global diffusion. The log-scale histogram reveals a decisive difference in the error topology. The convolutional distribution (orange) is heavily zero-inflated, featuring a dominant peak at |δz|≈0| _z|≈ 0 and a heavy tail. This sparsity implies that for any given localized fault, the majority of latent features remain uncorrupted, maximizing the signal-to-noise ratio for the subsequent detector. Conversely, the dense distribution (blue) lacks this zero-peak and instead follows a broad, bell-shaped distribution shifted away from zero. This confirms that the fault energy is distributed entropically across the entire latent vector, diluting the local anomaly signal. Collectively, these results empirically confirm Lemma 3.1 and Lemma 3.2. 6.5 Ablation studies To examine the contribution of individual components within the MMSSL framework, we conducted a series of ablation studies. Specifically, we isolate the impact of the proposed layer-wise Lipschitz regularization strategy on reconstruction stability, and we analyze the sensitivity of the model to the weighting of the self-supervised loss terms. First, we investigate the spectral constraints imposed on the compute block. Table 5 compares the performance of the full MMSSL model against the baseline (w/o Reg) where the spectral normalization, gradient clipping, and amplification mechanisms described in Section 4 are removed. Table 5: Train and test performances (in 10−210^-2) of the proposed SSL approach with and without regularizer. Values represent the mean over the final 250 batches of 10 trials for camera, sensor, and both combined correction performance. Modality w/o Reg MMSSL Training Camera 0.127 ± 0.024 0.098 ± 0.023 Sensor 0.362 ± 0.349 0.307 ± 0.302 Combined 0.490 ± 0.357 0.405 ± 0.310 Testing Camera 0.147 ± 0.003 0.118 ± 0.005 Sensor 0.499 ± 0.047 0.405 ± 0.040 Combined 0.645 ± 0.046 0.523 ± 0.042 The unregularized model consistently converges to a suboptimal plateau, yielding a significantly higher combined test error (0.6450.645) compared to MMSSL (0.5230.523). This ≈19%≈ 19\% reduction in error demonstrates that the Lipschitz constraints do more than just stabilize training. They actively shape the optimization landscape. By enforcing a contractive mapping, the regularizer guides the network away from easy-to-reach but poor local minima and towards a more physically accurate correction manifold. This effect is consistent across modalities, with the camera error dropping from 0.1470.147 to 0.1180.118 and the sensor error from 0.4990.499 to 0.4050.405, proving that the geometric constraints are essential for maximizing reconstruction fidelity. To quantify the trade-off between the two self-supervised objectives, Figure 11 presents the impact of the weighting ratio λsim/λcon _sim/ _con on system performance. The dual-axis plot visualizes the detection accuracy (orange, right axis) against the correction loss (blue, left axis). Figure 11: Dual-axis plot showing the Correction Loss (left y-axis, blue, scale ×10−3× 10^-3) and Detection Accuracy (right y-axis, orange, scale ×10−1× 10^-1) as a function of the λsim/λcon _sim/ _con ratio (x-axis, logarithmic scale). The blue line with error bars represents the correction loss, and the orange line with square markers represents the detection accuracy. The results reveal a clear inverse relationship between the competing regularizers. In the contrastive-dominant regime (λsim≪λcon _sim _con), detection accuracy peaks at 98%98\% as the model enforces strong latent separation, but this distorts the manifold geometry, resulting in higher correction errors. Conversely, as the ratio increases to favor similarity (λsim≫λcon _sim _con), the correction loss drops to its minimum (5.1×10−35.1× 10^-3) due to stronger contraction, but this smoothing collapses the anomaly boundary, degrading detection accuracy to 84.5%84.5\%. A balanced ratio (≈100≈ 10^0) offers the optimal compromise, maintaining high discriminative power (>91%>91\%) while stabilizing the reconstruction. To visualize the internal optimization dynamics of the self-supervised framework, Figure 12 decomposes the total objective into its constituent parts. The dual-axis plot tracks the raw, unweighted values of the similarity loss (blue, left axis) and the contrastive loss (red, right axis) as a function of their weighting ratio λsim/λcon _sim/ _con. Figure 12: Pareto trade-off between the unweighted self-supervised loss components. As the weighting ratio λsim/λcon _sim/ _con increases, the similarity loss (blue) decreases, indicating better reconstruction, while the contrastive loss (red) rises, indicating reduced latent separability. The dashed lines illustrate the opposing linear trends of these competing objectives. The empirical results demonstrate a fundamental opposition between the two geometric objectives. As the ratio increases (moving right), the system prioritizes similarity, driving the unweighted similarity loss down from a peak of ≈3.8×10−2≈ 3.8× 10^-2 to a minimum of ≈2.7×10−2≈ 2.7× 10^-2. This confirms that stronger regularization effectively forces the latent manifold to contract. However, this contraction comes at a direct cost to separability: the unweighted contrastive Loss rises steadily from ≈1.5≈ 1.5 to ≈1.8×10−1≈ 1.8× 10^-1 (red trend line), indicating that the embeddings are becoming less distinguishable. The intersection of the trend lines near λsim/λcon≈100 _sim/ _con≈ 10^0 highlights a critical equilibrium point where neither objective dominates, allowing the model to minimize reconstruction error without completely collapsing the latent boundaries required for detection. To specifically assess the robustness of the anomaly detector under varying regularization regimes, Figure 13 details the evolution of classification metrics. The dual-axis plot displays the overall accuracy (blue, left axis) and the class-balanced F1 score (green, right axis) as a function of the loss weighting ratio λsim/λcon _sim/ _con. Figure 13: Dependence of detection performance on the loss weighting ratio. Both accuracy (blue) and F1 score (green) degrade as the similarity weight increases, confirming that strong contrastive regularization is essential for maintaining high discriminative power. The results exhibit a consistent downward trend that mirrors the contrastive loss behavior observed in the previous analyses. Performance is maximized in the contrastive-dominant regime (λsim=10−2 _sim=10^-2), where the F1 score reaches nearly 0.950.95 and accuracy peaks at 97%97\%. This confirms that a strong margin constraint is the primary driver for separability. As the system shifts towards a balanced weighting (λsim/λcon≈100 _sim/ _con≈ 10^0), there is a slight dip, yet the model maintains robust performance with an F1 score of ≈0.88≈ 0.88 and accuracy >90%>90\%. This stability plateau at unity is critical, as it allows the system to support accurate failure correction (as seen in the previous figures) without catastrophic degradation in detection. However, as the similarity term dominates (λsim≥10 _sim≥ 10), the F1 score plummets below 0.800.80, indicating that the detector can no longer reliably distinguish between the heavily smoothed anomaly representations and the clean manifold. 7 Conclusion This paper presented a theoretically motivated approach for robust multimodal learning under partial or corrupted input conditions. By linking perturbation propagation in convolutional and dense layers to their expected Lipschitz behavior, we established a formal basis for selecting architectural components that either localize or diffuse input anomalies. Building on these findings, we developed a two-stage self-supervised model consisting of a pre-trained multimodal autoencoder and a corrective compute block operating in latent space. The method integrates contrastive alignment for anomaly identification with linear reconstruction for correction, while employing selective Lipschitz regularization and gradient clipping to control sensitivity at different depths of the network. The results demonstrate that balancing sensitivity and contraction across layers yields architectures that can both detect and correct faults, a key requirement for reliable multimodal systems in industrial and autonomous settings. Future research will focus on extending this framework in three directions. First, integrating adaptive Lipschitz control through dynamic spectral normalization could enable online sensitivity modulation during deployment. Second, incorporating probabilistic uncertainty estimation into the correction block may improve interpretability and safety. References [1] H. Akbari, L. Yuan, R. Qian, W. Chuang, S. Chang, Y. Cui, and B. Gong (2021) Vatt: transformers for multimodal self-supervised learning from raw video, audio and text. Advances in neural information processing systems 34, p. 24206–24221. Cited by: §2.1. [2] U. Ali (2025) A multimodal lightweight approach to fault diagnosis of induction motors in high-dimensional dataset. arXiv preprint arXiv:2501.03746. Cited by: §1. [3] S. Alizadeh and S. Sriramula (2017) Reliability modelling of redundant safety systems without automatic diagnostics incorporating common cause failures and process demand. ISA transactions 71, p. 599–614. Cited by: §1. [4] D. Altinses and A. Schwung (2023) Deep multimodal fusion with corrupted spatio-temporal data using fuzzy regularization. In IECON 2023-49th Annual Conference of the IEEE Industrial Electronics Society, p. 1–7. Cited by: §1, §2.2, §6.4. [5] D. Altinses and A. Schwung (2023) Multimodal synthetic dataset balancing: a framework for realistic and balanced training data generation in industrial settings. In IECON 2023-49th, p. 1–7. Cited by: §6.2. [6] D. Altinses and A. Schwung (2025) Fault-tolerant multimodal representations learning: fusion of intermodal and intramodal correlations. IEEE Transactions on Artificial Intelligence (), p. 1–13. External Links: Document Cited by: §1, §2.2, §6.4. [7] D. Altinses and A. Schwung (2025) Performance benchmarking of multimodal data-driven approaches in industrial settings. Machine Learning with Applications, p. 100691. Cited by: Figure 3, §6.1. [8] D. Altinses and A. Schwung (2025) Stabilizing multimodal autoencoders: a theoretical and empirical analysis of fusion strategies. arXiv preprint arXiv:2512.20749. Cited by: §2.3. [9] D. Altinses and A. Schwung (2026) Generative adversarial fault-tolerance for robust multimodal sensor fusion in industrial systems. Mechanical Systems and Signal Processing, p. 131270. Cited by: §6.4. [10] D. Altinses, D. O. S. Torres, and A. Schwung (2025) Enhancing fault tolerance in multimodal learning: a vae-based approach with probabilistic fusion. In 2025 11th International Conference on Control, Decision and Information Technologies (CoDIT), p. 1714–1720. Cited by: §1, §2.2, §6.4. [11] C. Anil, J. Lucas, and R. Grosse (2019) Sorting out lipschitz function approximation. In International conference on machine learning, p. 291–301. Cited by: §2.3. [12] R. Arandjelovic and A. Zisserman (2017) Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, p. 609–617. Cited by: §2.1. [13] R. Arandjelovic and A. Zisserman (2018) Objects that sound. In Proceedings of the European conference on computer vision (ECCV), p. 435–451. Cited by: §2.1. [14] R. Chen, C. Li, L. Wang, M. Liu, S. Chen, J. Yang, and X. Zeng (2025) Pretraining graph transformer for molecular representation with fusion of multimodal information. Information Fusion 115, p. 102784. Cited by: §2.1. [15] X. Chen, S. Z. Wu, and M. Hong (2020) Understanding gradient clipping in private sgd: a geometric perspective. Advances in Neural Information Processing Systems 33, p. 13773–13782. Cited by: §2.3. [16] Y. Chen, W. Huang, K. Zhou, Y. Bian, B. Han, and J. Cheng (2023) Understanding and improving feature learning for out-of-distribution generalization. Advances in Neural Information Processing Systems 36, p. 68221–68275. Cited by: §2.3. [17] A. Diez-Olivan, J. Del Ser, D. Galar, and B. Sierra (2019) Data fusion and machine learning for industrial prognosis: trends and perspectives towards industry 4.0. Information Fusion 50, p. 92–111. Cited by: §1. [18] M. U. Din, W. Akram, L. S. Saoud, J. Rosell, and I. Hussain (2025) Multimodal fusion with vision-language-action models for robotic manipulation: a systematic review. Information Fusion, p. 104062. Cited by: §2.1. [19] M. Fazlyab, T. Entesari, A. Roy, and R. Chellappa (2023) Certified robustness via dynamic margin maximization and improved lipschitz regularization. Advances in Neural Information Processing Systems 36, p. 34451–34464. Cited by: §2.3. [20] Z. Gao, X. Jiang, X. Xu, F. Shen, Y. Li, and H. T. Shen (2024) Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26876–26885. Cited by: §2.2. [21] J. Hong, M. Kim, J. Choi, and Y. M. Ro (2023) Watch or listen: robust audio-visual speech recognition with visual corruption modeling and reliability scoring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18783–18794. Cited by: §2.2. [22] H. W. Jones (2025) Common cause failures dominate and defeat redundancy. In 2025 Annual Reliability and Maintainability Symposium (RAMS), p. 1–5. Cited by: §1. [23] B. Korbar, D. Tran, and L. Torresani (2018) Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems 31. Cited by: §2.1. [24] G. Lan, Y. Du, and Z. Yang (2024) Robust multimodal representation under uncertain missing modalities. ACM Transactions on Multimedia Computing, Communications and Applications 21 (1), p. 1–23. Cited by: §2.2. [25] A. Lewandowski, M. Bortkiewicz, S. Kumar, A. György, D. Schuurmans, M. Ostaszewski, and M. C. Machado (2024) Learning continually by spectral regularization. arXiv preprint arXiv:2406.06811. Cited by: §2.3. [26] M. Li, D. Yang, Y. Liu, S. Wang, J. Chen, S. Wang, J. Wei, Y. Jiang, Q. Xu, X. Hou, et al. (2024) Toward robust incomplete multimodal sentiment analysis via hierarchical representation learning. Advances in Neural Information Processing Systems 37, p. 28515–28536. Cited by: §2.2. [27] R. Lin and H. Hu (2023) Missmodal: increasing robustness to missing modality in multimodal sentiment analysis. Transactions of the Association for Computational Linguistics 11, p. 1686–1702. Cited by: §2.2. [28] J. Lu, D. Batra, D. Parikh, and S. Lee (2019) Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32. Cited by: §2.1. [29] M. Ma, J. Ren, L. Zhao, D. Testuggine, and X. Peng (2022) Are multimodal transformers robust to missing modality?. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18177–18186. Cited by: §2.2. [30] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida (2018) Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. Cited by: §2.3. [31] T. Miyato, S. Maeda, M. Koyama, and S. Ishii (2018) Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence 41 (8), p. 1979–1993. Cited by: §2.3. [32] P. Morgado, Y. Li, and N. Nvasconcelos (2020) Learning representations from audio-visual spatial alignment. Advances in Neural Information Processing Systems 33, p. 4733–4744. Cited by: §2.1. [33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §2.1. [34] C. Rahman, M. Hoque, and S. M. Uddin (2014) Assessment of total productive maintenance implementation through downtime and mean downtime analysis (case study: a semi-automated manufacturing company of bangladesh). Assessment 4 (09). Cited by: §1. [35] T. Rhodes and D. Lee (2021) Local disentanglement in variational auto-encoders using jacobian L_1L\_1 regularization. Advances in Neural Information Processing Systems 34, p. 22708–22719. Cited by: §2.3. [36] H. Tan and M. Bansal (2019) LXMERT: learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 5100–5111. Cited by: §2.1. [37] Y. Tian, D. Krishnan, and P. Isola (2020) Contrastive multiview coding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, p. 776–794. Cited by: §2.1. [38] X. Wang, Y. Wang, J. Yang, X. Jia, L. Li, W. Ding, and F. Wang (2024) The survey on multi-source data fusion in cyber-physical-social systems: foundational infrastructure for industrial metaverses and industries 5.0. Information Fusion 107, p. 102321. Cited by: §1. [39] Y. Wang, C. M. Albrecht, N. A. A. Braham, C. Liu, Z. Xiong, and X. X. Zhu (2024) Decoupling common and unique representations for multimodal self-supervised learning. In European Conference on Computer Vision, p. 286–303. Cited by: §2.2. [40] S. Wei, Y. Luo, Y. Wang, and C. Luo (2024) Robust multimodal learning via representation decoupling. In European Conference on Computer Vision, p. 38–54. Cited by: §2.2, §2.2. [41] J. Xiang, X. Zhu, and E. Cambria (2025) Integrating audio–visual text generation with contrastive learning for enhanced multimodal emotion analysis. Information Fusion, p. 103809. Cited by: §2.1. [42] L. Zhang and S. Sun (2025) Distributed h fusion filters of joint fault and state for multi-sensor networked systems with two-channel probability-uncertain deception attacks. Information Fusion 120, p. 102984. Cited by: §1. [43] J. Zhao, R. Li, and Q. Jin (2021) Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 2608–2618. Cited by: §2.2.