Paper deep dive
Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance
Fabian A. Mikulasch, Friedemann Zenke
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised learning (SSL) by predicting in latent space, without generating the input data itself, learns highly abstract, useful representations. Intuitively, this success is often attributed to its ability to discard nuisance information that is irrelevant to prediction. However, this poses a conundrum: both stochastic variation in a prediction-relevant latent signal and true nuisance make observations partly unpredictable; how could they be distinguished? Surprisingly, we prove that common SSL methods can achieve exactly this, by implicitly instantiating a latent-variable model with stochastic dynamics and observation-private nuisance. We trace their ability to recover the stochastic signal to two complementary principles: Predictive mutual information maximization ensures that representations retain the information needed for prediction, while latent distribution matching constrains how this information is encoded, thereby making the retained signal identifiable. We confirm this identifiability result in simulations for Gaussian predictors, which recover the true signal up to an affine transformation even in dynamic, nuisance-laden environments.
Tags
Links
- Source: https://arxiv.org/abs/2609.37789v1
- Canonical: https://arxiv.org/abs/2609.37789v1
Trouble viewing inline? Open PDF directly →
Full Text
63,018 characters extracted from source content.
Expand or collapse full text
Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance Fabian A. Mikulasch & Friedemann Zenke Affiliation: Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland Affiliation: Faculty of Science, University of Basel, Switzerland Email: firstname.lastname@fmi.ch Abstract SSL (SSL) by predicting in latent space, without generating the input data itself, learns highly abstract, useful representations. Intuitively, this success is often attributed to its ability to discard nuisance information that is irrelevant to prediction. However, this poses a conundrum: both stochastic variation in a prediction-relevant latent signal and true nuisance make observations partly unpredictable; how could they be distinguished? Surprisingly, we prove that common SSL methods can achieve exactly this, by implicitly instantiating a latent-variable model with stochastic dynamics and observation-private nuisance. We trace their ability to recover the stochastic signal to two complementary principles: Predictive MI maximization ensures that representations retain the information needed for prediction, while DM constrains how this information is encoded, thereby making the retained signal identifiable. We confirm this identifiability result in simulations for Gaussian predictors, which recover the true signal up to an affine transformation even in dynamic, nuisance-laden environments. 1 Introduction SSL (SSL) has become a central paradigm for representation learning, enabling models to extract useful structure from unlabeled data across vision, language, and audio (Noroozi and Favaro, 2016; Oord et al., 2018; Dawid and LeCun, 2024; Gui et al., 2024). Its success is often attributed to a simple intuition: different observations in the same context share a context-specific signal while also containing observation-specific variation that should not be represented. For example, image augmentations alter image details without changing object identity, different modalities can provide different views of the same signal, and temporal observations combine an evolving signal with non-predictive nuisance. By learning what is shared or predictable across observations, non-generative modeling is thought to retain the underlying signal while nuisance variables are not represented but left implicit in the generative model. This nuisance-based explanation is pervasive, but it is rarely made explicit in statistical models. For example, non-generative modeling has been derived from the idea that representations should preserve the MI (MI) shared between observations, while unpredictable pixel-level nuisance might be ignored (Oord et al., 2018; Tian et al., 2020; Shwartz-Ziv et al., 2023; Rodríguez Gálvez et al., 2023; Shwartz Ziv and LeCun, 2024). Although these information-theoretic approaches provide an intuitive view of how nuisance is treated in non-generative models, they do not formally specify what exactly constitutes nuisance, and how the remaining signal is represented. One route from the intuitive view to formal insights is given by identifiability theory. Results in nonlinear ICA (ICA) and non-generative modeling show that latent variables can be recovered up to restricted transformations when the learned and true conditional distributions have suitable structure (Hyvarinen et al., 2019). This analysis has been extended to prove that non-generative models can identify the signal even under the presence of nuisance (von Kügelgen et al., 2021). These results require that the model is provided with different views of the same data point, where nuisance is private to each view, and the signal variable is exactly the same for each view. However, this does not cover the more general setting where the signal can evolve stochastically itself. This raises the question: is it possible to distinguish noise in otherwise predictable dynamics from completely unpredictable nuisance? Here we formalize implicit nuisance in stochastic latent variable models and study when non-generative modeling recovers the nuisance-free signal. Our main contributions are: Disentangling the roles of MI maximization and distribution matching. We show that in non-generative SSL, predictive MI maximization and DM (DM) play distinct roles: MI maximization guarantees that representations retain all predictable information, while DM alone is responsible for identification of the retained signal. This resolves prior ambiguity about the function of the terms in non-generative SSL objectives (Tschannen et al., 2020; Mikulasch and Zenke, 2026), showing that MI maximization and DM are not sufficient on their own, but complement each other. Identifiability under stochastic, private nuisance. We introduce a latent-variable model in which nuisance is private to each observation given a stochastically evolving signal, relaxing the deterministic shared-context assumption of prior identifiability results (von Kügelgen et al., 2021; Daunhawer et al., 2023). For exponential-family predictive models we prove that exact DM together with predictive-information saturation recovers the true sufficient statistic as an affine readout of the learned statistic. Affine recovery for Gaussian predictors. We specialize our general result to the practically relevant case of Gaussian predictive models, showing that the signal is recovered as an affine transformation of the learned representation whenever the latent dimensionality is large enough. We confirm this in simulations with stochastic latent dynamics and unpredictable nuisance, and in a stochastic variant of the Causal3DIdent benchmark, demonstrating recovery of causally coupled signal factors from high-dimensional images. 2 Related work Theory of non-generative latent models. There has been significant interest in understanding non-generative modeling through theoretical analysis (Saunshi et al., 2019; Wang and Isola, 2020; Ben-Shaul et al., 2023, e.g.,). A line of work closely related to this article aims to connect contrastive methods (e.g., CPC) to latent variable models (von Kügelgen et al., 2021; Zimmermann et al., 2021; Aitchison and Ganev, 2024; Nakamura et al., 2023; Bizeul et al., 2024; Mikulasch and Zenke, 2026). Others proposed information-theoretic interpretations of regularization-based methods (Shwartz-Ziv et al., 2023, e.g., VICReg, ) and clustering/contrastive methods (Rodríguez Gálvez et al., 2023). Identifiable latent variable models without nuisance. Identifiability is a core concept in linear ICA Hyvärinen and Oja (2000), which guarantees the recovery of ”true” underlying variables up to trivial transformations. The concept was later extended to nonlinear ICA using assumptions such as temporal structure or auxiliary variables (Sprekeler et al., 2014; Khemakhem et al., 2020; Hyvarinen et al., 2019; Roeder et al., 2021). More recently, these insights also enabled proving identifiability for non-generative models (Zimmermann et al., 2021; Laiz et al., 2025; Mikulasch and Zenke, 2026). While these models study identification under stochastic dynamics, they do not include nuisance in their generative model. Identifiable latent variable models with nuisance. A related line of work studies identifiability in the presence of view-specific nuisances. Gresele et al. (2020) showed that shared sources can be identified from multiple nonlinear views despite view-specific corruptions. von Kügelgen et al. (2021) and Lyu et al. (2022) extended this approach to show that common SSL methods might be used to extract the shared components between views. Daunhawer et al. (2023) transported these insights to multimodal settings, proving recovery of factors shared across modalities in the presence of modality-specific latent variation. These works establish that identifiability is possible when nuisance variables are present, but require deterministic relationships of the recovered variables between views. 3 Theory Before we develop our theory of predictive SSL under nuisance we briefly review the concepts of predictive MI maximization and latent distribution matching that lie at the core of our work. The specific loss function we are going to analyze has been used in several previous works. Initially, Oord et al. (2018) proposed InfoNCE, to maximize predictive MI I[z;zc]I[z;z_c] between representations z and predictive variables zcz_c. Later work has emphasized that the objective that is actually maximized by many of the common SSL algorithms, including InfoNCE (Oord et al., 2018), SimCLR (Chen et al., 2020), and VICReg (Shwartz-Ziv et al., 2023), has the general form (Aitchison and Ganev, 2024; Mikulasch and Zenke, 2026) ℱ(f,fc,θ)=−DKL[qf(z,zc)∥pθ(z,zc)]+Iqf[z;zc]=⟨logpθ(z,zc)⟩qf(z,zc)+Hqf[z]+Hqf[zc]. splitF(f,f_c,θ)=&-D_KL[q_f(z,z_c)\|p_θ(z,z_c)]+I_q_f[z;z_c]\\ =& p_θ(z,z_c) _q_f(z,z_c)+H_q_f[z]+H_q_f[z_c]\;. split (1) Here qf(z,zc)q_f(z,z_c) is the pushforward distribution of observations p(x,xc)p(x,x_c) through the observation encoder z=f(x)z=f(x) and condition encoder zc=fc(xc)z_c=f_c(x_c), and pθ(z,zc)p_θ(z,z_c) is a latent model that is learned alongside. Different entropy estimators lead to different SSL algorithms—for example, KDE and log-determinant of covariance estimators can be related to SimCLR (Chen et al., 2020) and VICReg (Shwartz-Ziv et al., 2023), respectively (Mikulasch and Zenke, 2026). By this goal function, representations are not only required to be mutually informative, they are also required to follow a simple learned latent model pθ(z,zc)p_θ(z,z_c) via DM. In the following we will therefore refer to Equation 1 as the InfoLDM goal function. Commonly, but not necessarily, this goal function is simplified by ignoring the marginal of the conditioning variable and setting pθ(zc)=qf(zc)p_θ(z_c)=q_f(z_c), leading to ℱ(f,fc,θ)= (f,f_c,θ)= ⟨logpθ(z∣zc)⟩qf(z,zc)+Hqf[z]. p_θ(z z_c) _q_f(z,z_c)+H_q_f[z]\;. (2) This is a well-known lower bound on the MI between latent variables (Poole et al., 2019), but we here analyze it with the clear framing that maximizing it performs both MI maximization and DM. 3.1 Exponential family statistical recovery The main theorem specifies when maximizing the InfoLDM goal function (Equation 1) recovers the signal s in the learned representation z up to simple transformations. For this we make the following assumptions, which are specified more rigorously in Appendix A.1.1. Generative model. Let s∈⊆ℝdss ^d_s be a semantic signal variable, n∈ℝdnn ^d_n a noisy nuisance variable, and let c denote the true condition (or cause, context). Let x=g(s,n)x=g(s,n) be the observation, and xcx_c the observed condition. As before, let z=f(x)∈⊆ℝdzz=f(x) ^d_z be the learned representation, and zc=fc(xc)∈c⊆ℝdzcz_c=f_c(x_c) _c ^d_z_c the learned encoded condition. Finally, let qf(z,zc)q_f(z,z_c) be the pushforward distribution of observations, and pθ(z,zc)p_θ(z,z_c) the learned model (Figure 1). Generative model Learned modelccssnnxcx_cxxzcz_czzConditioning variables(e.g., history, actions, context)Signal variablesNuisance variablesEncoded conditionEncoded observationggfcf_cpθp_θf Figure 1: Graphical model for observation-private nuisance. The three layers show (from left to right) unobserved latent variables, observations, and learned representations. Square nodes denote deterministically defined variables. Assumption 1. The nuisance is private to the current observation in the sense that n⟂⟂c|s.n \!\!\! c s. (3) Assumption 2. The true predictive distribution is exponential family p(s∣c)=hs(s)exp(α(c)⊤τ⋆(s)−ψ(α(c))),p(s c)=h_s(s) \! (α(c) _ (s)-ψ(α(c)) ), (4) where τ⋆:→ℝrs _ :S ^r_s is a sufficient statistic, hsh_s a fixed carrier function, and α(c)α(c) encodes the dependency on the condition c. Similarly, the learned latent model is exponential family pθ(z∣zc)=hz(z)exp(β(zc)⊤τz(z)−ϕ(β(zc))),p_θ(z z_c)=h_z(z) \! (β(z_c) _z(z)-φ(β(z_c)) ), (5) where the predictor β(zc)β(z_c) is learned, and τz:→ℝrz _z:Z ^r_z. Assumption 3. By optimizing the InfoLDM goal function Equation 1 via qf(z,zc)q_f(z,z_c) and pθ(z,zc)p_θ(z,z_c) it is possible to reach the global optimum where DKL[qf(z,zc)∥pθ(z,zc)]=0,Iqf[z;zc]=I[(s,n);c]=I[s;c]<∞.D_KL[q_f(z,z_c)\|p_θ(z,z_c)]=0, I_q_f[z;z_c]=I[(s,n);c]=I[s;c]<∞. (6) Assumption 4. It is possible to choose c0,c1,…,crsc_0,c_1,…,c_r_s from the support of p(c)p(c) such that D:=[(α(c1)−α(c0))⊤(α(crs)−α(c0))⊤]∈ℝrs×rsD:= bmatrix(α(c_1)-α(c_0)) \\ \\ (α(c_r_s)-α(c_0)) bmatrix ^r_s× r_s (7) is invertible. Assumption 5. For every condition c the random vector τ⋆(s) _ (s) with s∼p(s∣c)s p(s c) is not contained in a proper affine hyperplane of ℝrsR^r_s. Most of the presented assumptions are standard in the context of nonlinear identification theory. Assumption 1 can be understood as a definition of nuisance factors, similar to the definition by von Kügelgen et al. (2021). Assumptions 2, 4 and 5 are standard assumptions in the theory of exponential family identification (Khemakhem et al., 2020, e.g.,). Chiefly, 4 and 5 ensure that all latent dimensions of the probabilistic model, not only a subset, are well constrained. Assumption 3 is the strongest. It implicitly replaces common invertibility assumptions on the generative model. To see this relation consider that the generative model (Figure 1) together with the data processing inequality imply Iqf[z;zc]≤I[x;xc]≤I[s;c]=I[(s,n);c].I_q_f[z;z_c]≤ I[x;x_c]≤ I[s;c]=I[(s,n);c]. At the optimum these inequalities become equalities, and in particular information has to be preserved in the observations I[x;xc]=I[s;c]I[x;x_c]=I[s;c], restricting g(s,n)g(s,n) and p(xc∣c)p(x_c c) to be information-preserving. Additionally, this restriction implicitly requires that the latent representation is sufficiently high-dimensional to capture the required information, as will be demonstrated by Theorem 1. With these assumptions in place we are ready to state our main result. Theorem 1 (Exponential-family statistical recovery through InfoLDM). Suppose Assumptions 1, 2, 3, 4, and 5 hold, and the InfoLDM goal function (Equation 1) is fully maximized. Then the dimension of the model sufficient statistic rz≥rsr_z≥ r_s, and there is a full-row-rank matrix A∈ℝrs×rzA ^r_s× r_z and a vector b∈ℝrsb ^r_s such that τ⋆(s)=Aτz(z)+balmost surely. _ (s)=A _z(z)+b surely. (8) If τ⋆ _ is injective, then s is almost surely a measurable function of z. If rz=rsr_z=r_s, then A is invertible and the two sufficient statistics are related by an invertible affine transformation. The full proof is given in Appendix A.1.2. In broad strokes, the first step is to use successful MI maximization to show that zcz_c retains all information about s from c, and similarly z about c. This can be leveraged to directly relate the true and learned conditional probability distributions of s and z. From there the result follows via well established identifiability theory approaches for exponential family models (Khemakhem et al., 2020, e.g.,). 3.2 Specialization to Gaussian prediction Theorem 1 is fairly general; here we aim to understand the behavior of commonly used non-generative models, which rely on Gaussianity assumptions. The goal function of many of these can be understood as approximately maximizing ℱ∝⟨−12σ2∥z−μθ(zc)∥2⟩qf(z,zc)+Hqf[z], - 12σ^2 z- _θ(z_c) ^2 _q_f(z,z_c)+H_q_f[z], (9) where μθ _θ is a learned prediction function (cf. von Kügelgen et al., 2021; Mikulasch and Zenke, 2026). This expression follows directly from the simplified goal function (Equation 2) by choosing pθp_θ to be a Gaussian with fixed variance Σθ=σ21l _θ=σ^21 -2.5ptl (Shwartz-Ziv et al., 2023; Mikulasch and Zenke, 2026). Further asserting p(s∣c)=(s,μ⋆(c),Σ⋆),pθ(z∣zc)=(z,μθ(zc),Σθ),Σ⋆≻0,Σθ≻0,p(s c)=N(s; _ (c), _ ), p_θ(z z_c)=N(z; _θ(z_c), _θ), _ 0, _θ 0, (10) Theorem 1 can be specialized to give the stronger result of affine recovery of s. Corollary 1 (Gaussian affine recovery). Under the assumptions of Theorem 1 and Equation 10, necessarily dz≥dsd_z≥ d_s, and there is a full-row-rank matrix A∈ℝds×dzA ^d_s× d_z and b∈ℝdsb ^d_s such that s=Az+balmost surely.s=Az+b surely. (11) If dz=dsd_z=d_s, then A is invertible. This result follows immediately from Theorem 1 by the fact that the statistic of the fixed variance Gaussian distribution is the identity τ⋆(s)=s _ (s)=s and τz(z)=z _z(z)=z, and ds=rsd_s=r_s, dz=rzd_z=r_z. Intuitively, affine recovery fully relies on noise in the dynamics, as described in previous analyses (Zimmermann et al., 2021; Laiz et al., 2025; Mikulasch and Zenke, 2026): Noise in the true latent variables has Gaussian structure, and forcing learned variables to be Gaussian locally straightens latent coordinates around the predicted mean, which globally leads to an affine relation between true and recovered variables. Information sufficiency on the other hand supplies the model with a guarantee that all predictable signal variables are recovered. Note that nothing forces the model to discard nuisance variables entirely, except if the latent dimensionality of the model is restricted to match the dimensionality of the true signal. Similar results might be obtained for other exponential family distributions, such as von Mises-Fisher for variables on the sphere. 4 Simulation experiments To numerically verify our proof, we performed a range of control experiments. First, we followed a common recovery experimental setup (Zimmermann et al., 2021, e.g.,). Specifically, we generated length-five sequences with a signal following st+1=ρRst+1−ρ2ϵt,ϵt∼(0,I),s_t+1=ρ Rs_t+ 1-ρ^2\, _t, _t (0,I), where ρ=0.9ρ=0.9 and R is a fixed random orthogonal matrix. At every step, we independently sampled a 15-dimensional Gaussian nuisance variable and map the combined signal and nuisance through a fixed injective 3-layer MLP (MLP) with leaky ReLU activation to obtain a 100-dimensional observation. We varied the signal dimension dSd_S from 5 to 20. To recover the signal variables we trained a 5-layer MLP mapping the input to a latent space with same dimension as the signal, dZ=dSd_Z=d_S. Predictions were generated through an LSTM (LSTM) taking all previous timesteps and a linear prediction head. Training maximized the InfoLDM loss for Gaussian predictors (Equation 9), where the entropy was estimated through either kNN, KDE, or log-determinant of covariance (logdet) entropy estimators, as specified in Mikulasch and Zenke (2026). Simulations showed good affine signal recovery up to ten-dimensional latents, except for KDE entropy estimation, which showed slightly worse performance in this scenario (Table 1). Table 1: R2R^2 of the learned representations with the true factors of variation for the numerical test, averaged over 5 runs plus minus standard deviations. We highlight well learned factors (R2>0.9R^2>0.9). Entropy estimator ds=5d_s=5 ds=10d_s=10 ds=15d_s=15 ds=20d_s=20 KDE 0.92±0.160.92± 0.16 0.81±0.040.81± 0.04 0.60±0.080.60± 0.08 0.48±0.080.48± 0.08 kNN 0.98±0.000.98± 0.00 0.95±0.040.95± 0.04 0.76±0.010.76± 0.01 0.64±0.020.64± 0.02 logdet 0.99±0.000.99± 0.00 0.95±0.040.95± 0.04 0.78±0.020.78± 0.02 0.68±0.010.68± 0.01 4.1 Identifying stochastic causal relations between images As second example we tested signal recovery in high-dimensional images with strong, structured signal stochasticity, and structured, but independent nuisance. For this purpose we constructed image pairs in which a designated subset of generative factors was causally related. We based this on the Causal3DIdent dataset (von Kügelgen et al., 2021), which provides 11 image factors (Figure 2). After mapping these factors to standard-normal coordinates, we sampled s2=ρs1+1−ρ2ϵs_2=ρ s_1+ 1-ρ^2\,ε with ϵ∼(0,I)ε (0,I) and ρ=0.9ρ=0.9. Since Causal3DIdent provides a finite set of rendered images, we selected the second image uniformly from the k=16k=16 nearest neighbors of s2s_2. All remaining factors, including object class, are thus random and determined by the selected image. We considered two settings: an object signal comprising object position, rotation, and hue, and an environment signal comprising spotlight position, spotlight hue, and background hue (Figure 2). For signal recovery we used the same architecture and losses as in the numerical examples (Section 4), except for switching to a ResNet-18 as the image encoder. To give models the option to encode also nuisance we chose the latent dimension to be 10, similar to the total number of signal and nuisance factors. The simulations showed that learning consistent resulted in affine recovery of most of the causal factors, while performance depended on the entropy estimator, with KDE performing best in this scenario (Table 2). We further found that the entropy estimator also influenced whether or not nuisance variables were retained in the latent representation. Especially object position tended to be partly retained, which is sensible given the convolutional encoder; but also other variables were partly retained, an effect that became even more pronounced when computing nonlinear R2R^2 (Appendix, Table 3). Table 2: R2R^2 of the learned representations with the true factors of variation for the stochastic Causal3DIdent dataset, averaged over 5 runs. Standard deviations are not displayed here but are mostly small (<0.02<0.02) except for unidentified signal / identified nuisance variables, indicating intermittent optimization failures. We highlight well learned factors (R2>0.9R^2>0.9) in bold and signal factors with grey background. Scenario Entr. est. pobj-xp_obj-x pobj-yp_obj-y pobj-zp_obj-z θobj-α _obj-α θobj-β _obj-β θobj-γ _obj-γ hobjh_obj pspotp_spot hspoth_spot hbackh_back Env. signal KDE 0.200.20 0.470.47 0.370.37 0.050.05 0.080.08 0.120.12 0.410.41 0.970.97 0.950.95 0.970.97 Env. signal kNN 0.050.05 0.550.55 0.390.39 0.080.08 0.050.05 0.070.07 0.110.11 0.970.97 0.950.95 0.970.97 Env. signal logdet −0.01-0.01 −0.02-0.02 −0.00-0.00 −0.01-0.01 −0.00-0.00 −0.01-0.01 −0.01-0.01 0.980.98 0.760.76 0.990.99 Object signal KDE 0.980.98 0.980.98 0.970.97 0.950.95 0.960.96 0.940.94 0.920.92 −0.01-0.01 0.010.01 0.130.13 Object signal kNN 0.970.97 0.960.96 0.960.96 0.870.87 0.920.92 0.260.26 0.910.91 −0.01-0.01 −0.00-0.00 0.560.56 Object signal logdet 0.980.98 0.970.97 0.970.97 0.920.92 0.950.95 0.910.91 0.870.87 0.150.15 −0.00-0.00 −0.00-0.00 Environment signal Examples Img. 1iobji_objpobjp_objθobj _objhobjh_objpspotp_spothspoth_spothbackh_backImg. 2iobji_objpobjp_objθobj _objhobjh_objpspotp_spothspoth_spothbackh_back Object signal Examples Img. 1iobji_objpobjp_objθobj _objhobjh_objpspotp_spothspoth_spothbackh_backImg. 2iobji_objpobjp_objθobj _objhobjh_objpspotp_spothspoth_spothbackh_back Figure 2: Causal3DIdent image-pairs. The left diagrams show the true latent factors which are: object identity (iobji_obj), position (pobj-xp_obj-x, etc., here displayed grouped), rotation (θobj-α _obj-α, etc.) and hue (hobjh_obj), spotlight position (pspotp_spot), hue (hspoth_spot), and background hue (hbackh_back). Arrows mark the factors causally coupled across image pairs. Right panels show example rendered image pairs. Even though factors between images are causally connected, there still exists significant variation due to noisy coupling. 4.2 Latent world model with actions and nuisance In a last experiment we tested recovery and nuisance removal in a physical system with controlled actions. To that end we extended the MuJoCo suite (Todorov et al., 2012). We chose the hopper environment, where actions were defined by a pretrained controller (Nikulin et al., 2025), and for simulation diversity we added small Gaussian noise on both environment dynamics and controller actions. To introduce nuisance variables we sampled 12-dimensional independent Gaussian nuisance vectors per timestep that were then translated into nuisances such as color changes of the physical body and background contrast, as well as additional high-dimensional background colored noise (Figure 3A,B). We expected that latent representations would capture hopper state variables that are causally related through the simulation. We used the same setup for learning as for the image dataset (Section 4.1) with a 16-dimensional latent state, while replacing the prediction head with an MLP that received the LSTM output and the encoded action. All models learned representations that allowed to linearly decode the hopper state (Figure 3D), while nonlinear decoding further improved R2R^2, depending on the entropy estimator (Appendix, Figure 4). The nonlinear recovery probe is reasonable given that the environment does not explicitly follow the Gaussian dynamics of Corollary 1. We also trained models with an image encoder with attached gradient, resulting in representations that did not allow to decode hopper state, but consistently encoded parts of the nuisance variables (Appendix, Figure 4). As additional control we trained a model with the same predictor setup and an image decoder to predict the next observation, which did not result in better performance than direct decoding. This discrepancy can also be seen in the different topological structure of the latent representations (Figure 3C). Figure 3: Learning a world model for the controlled MuJoCo hopper. (A) Dataset generation. (B) Example timeseries. (C) UMAP of true signal (hopper state) and learned representations. Colors denote variable values. (D) R2R^2 of latent representations with hopper variables from linear readout. 5 Conclusion and Limitations We proved that common SSL learning methods recover signal variables even in the presence of high-dimensional nuisance and stochastic latent dynamics, closing the gap between prior identifiability results, which considered either but not both. This answers the question posed at the outset: representations can formally distinguish unpredictable nuisance from mere stochastic variation in the predictable signal, because predictive MI maximization and DM jointly constrain what is retained and how. Our result requires only few assumptions, the strongest of which are exponential family conditional distributions in the true latent variables, and an algorithm that successfully maximizes the InfoLDM loss; otherwise, the latent dynamics can be almost arbitrarily complicated, and nuisance can affect observations nontrivially. Future work should explore the viability and real-world relevance of predictive exponential family distributions beyond the Gaussian scenario tested here. Our theory does not preclude that nuisance is encoded in the learned representation (orthogonally to the signal) if there is capacity to accommodate it, and entropy maximization in principle should encourage this (von Kügelgen et al., 2021; Mikulasch and Zenke, 2026). Empirically we found that nuisance variables are often not decodable even if the latent dimensionality would permit it, either because they collapse, or because they are represented in a highly nonlinear manner that is not easily inverted. We also found that both signal recovery performance and nuisance leakage depended on the choice of entropy estimator, and the data it was applied to. This suggest that improved entropy estimation is one of the most pressing open problems towards robust and generally applicable SSL methods. AI use statement In this work, we used generative AI tools to assist in the writeup of proofs, correcting proofs, and to create or edit software code. We have not used generative AI tools to help develop theoretical models or conceptual frameworks, formulate mathematical claims, propose or refine hypotheses, design or provide feedback on research methodology or experiments, implement methods and generate synthetic data sets, or to interpret results. We have reviewed all AI-assisted work. LLM-generated code was verified and tested for correctness, proofs were checked and formalized in Lean and the Lean statements critically reviewed. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Ethics statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Reproducibility statement For reproducibility we provide all simulation and evaluation code, as well as Lean proof formalizations at https://github.com/fmi-basel/identifiable-stochastic-nuisance. Acknowledgments We thank Michael Hauri and all Zenke Lab members for their input and discussions. This project was supported by the Swiss National Science Foundation (Grant Number PCEFP3_202981) and the Novartis Research Foundation. References Aitchison and Ganev (2024) L. Aitchison and S. K. Ganev InfoNCE is variational inference in a recognition parameterised model. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §2, §3. Ben-Shaul et al. (2023) I. Ben-Shaul, R. Shwartz-Ziv, T. Galanti, S. Dekel, and Y. LeCun Reverse engineering self-supervised learning. In Advances in Neural Information Processing Systems, Vol. 36, p. 58324–58345. External Links: Link Cited by: §2. Bizeul et al. (2024) A. Bizeul, B. Schölkopf, and C. Allen A probabilistic model behind self- supervised learning. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §2. Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119, p. 1597–1607. External Links: Link Cited by: §3, §3. Daunhawer et al. (2023) I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt Identifiability results for multimodal contrastive learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: item Identifiability under stochastic, private nuisance., §2. Dawid and LeCun (2024) A. Dawid and Y. LeCun Introduction to latent variable energy-based models: a path toward autonomous machine intelligence. Journal of Statistical Mechanics: Theory and Experiment 2024 (10), p. 104011. External Links: Link Cited by: §1. Gresele et al. (2020) L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf The incomplete rosetta stone problem: identifiability results for multi-view nonlinear ica. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, Vol. 115, p. 217–227. External Links: Link Cited by: §2. Gui et al. (2024) J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao A survey on self-supervised learning: algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12), p. 9052–9071. External Links: Link Cited by: §1. Hyvärinen and Oja (2000) A. Hyvärinen and E. Oja Independent component analysis: algorithms and applications. Neural Networks 13 (4), p. 411–430. External Links: ISSN 0893-6080, Link Cited by: §2. Hyvarinen et al. (2019) A. Hyvarinen, H. Sasaki, and R. Turner Nonlinear ica using auxiliary variables and generalized contrastive learning. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, Vol. 89, p. 859–868. External Links: Link Cited by: §1, §2. Khemakhem et al. (2020) I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen Variational autoencoders and nonlinear ica: a unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Vol. 108, p. 2207–2217. External Links: Link Cited by: §A.1.2, §2, §3.1, §3.1. Laiz et al. (2025) R. G. Laiz, T. Schmidt, and S. Schneider Self-supervised contrastive learning performs non-linear system identification. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2, §3.2. Lyu et al. (2022) Q. Lyu, X. Fu, W. Wang, and S. Lu Understanding latent correlation-based multiview learning and self-supervision: an identifiability perspective. In International Conference on Learning Representations, External Links: Link Cited by: §2. Mikulasch and Zenke (2026) F. A. Mikulasch and F. Zenke Understanding self-supervised learning via latent distribution matching. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §A.3, item Disentangling the roles of maximization and distribution matching., §2, §2, §3.2, §3.2, §3, §3, §4, §5. Nakamura et al. (2023) H. Nakamura, M. Okada, and T. Taniguchi Representation uncertainty in self-supervised learning as variational inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 16484–16493. External Links: Link Cited by: §2. Nikulin et al. (2025) A. Nikulin, I. Zisman, D. Tarasov, L. Nikita, A. Polubarov, I. Kiselev, and V. Kurenkov Latent action learning requires supervision in the presence of distractors. In Proceedings of Machine Learning Research, Vol. 267, p. 46427–46447. External Links: Link Cited by: §A.3, §4.2. Noroozi and Favaro (2016) M. Noroozi and P. Favaro Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. In Computer Vision – ECCV 2016, Lecture Notes in Computer Science, p. 69–84 (en). External Links: ISBN 978-3-319-46466-4, Link Cited by: §1. Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §1, §1, §3. Poole et al. (2019) B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97, p. 5171–5180. External Links: Link Cited by: §3. Rodríguez Gálvez et al. (2023) B. Rodríguez Gálvez, A. Blaas, P. Rodriguez, A. Golinski, X. Suau, J. Ramapuram, D. Busbridge, and L. Zappella The role of entropy and reconstruction in multi-view self-supervised learning. In Proceedings of the 40th International Conference on Machine Learning, Vol. 202, p. 29143–29160. External Links: Link Cited by: §1, §2. Roeder et al. (2021) G. Roeder, L. Metz, and D. Kingma On linear identifiability of learned representations. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, p. 9030–9039. External Links: Link Cited by: §2. Saunshi et al. (2019) N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97, p. 5628–5637. External Links: Link Cited by: §2. Shwartz Ziv and LeCun (2024) R. Shwartz Ziv and Y. LeCun To compress or not to compress—self-supervised learning and information theory: a review. Entropy 26 (3), p. 252. External Links: Link Cited by: §1. Shwartz-Ziv et al. (2023) R. Shwartz-Ziv, R. Balestriero, K. Kawaguchi, T. G. J. Rudner, and Y. LeCun An information theory perspective on variance-invariance-covariance regularization. In Advances in Neural Information Processing Systems, Vol. 36, p. 33965–33998. External Links: Link Cited by: §1, §2, §3.2, §3, §3. Sprekeler et al. (2014) H. Sprekeler, T. Zito, and L. Wiskott An extension of slow feature analysis for nonlinear blind source separation. Journal of Machine Learning Research 15 (26), p. 921–947. External Links: Link Cited by: §2. Tian et al. (2020) Y. Tian, D. Krishnan, and P. Isola Contrastive multiview coding. In European conference on computer vision, p. 776–794. External Links: Link Cited by: §1. Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa MuJoCo: a physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. External Links: Link Cited by: §4.2. Tschannen et al. (2020) M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic On mutual information maximization for representation learning. In International Conference on Learning Representations, External Links: Link Cited by: item Disentangling the roles of maximization and distribution matching.. von Kügelgen et al. (2021) J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, Vol. 34, p. 16451–16467. External Links: Link Cited by: §A.3, §A.3, item Identifiability under stochastic, private nuisance., §1, §2, §2, §3.1, §3.2, §4.1, §5. Wang and Isola (2020) T. Wang and P. Isola Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119, p. 9929–9939. External Links: Link Cited by: §2. Zimmermann et al. (2021) R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel Contrastive learning inverts the data generating process. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, p. 12979–12990. External Links: Link Cited by: §2, §2, §3.2, §4. Appendix A Appendix A.1 Mathematical appendix A.1.1 Detailed assumptions Assumption 1 (Private nuisance). The stochastic part of the model factorizes as p(n,s,xc,c)=p(n∣s)p(s∣c)p(xc∣c)p(c),p(n,s,x_c,c)=p(n s)p(s c)p(x_c c)p(c), (12) with x=g(s,n)x=g(s,n), z=f(x)z=f(x), and zc=fc(xc)z_c=f_c(x_c). More specifically, the nuisance is private to the current observation in the sense that n⟂⟂c|s.n \!\!\! c s. (13) The privacy condition has the equivalent information-theoretic form I[(s,n);c]=I[s;c]+I[n;c∣s]=I[s;c].I[(s,n);c]=I[s;c]+I[n;c s]=I[s;c]. (14) Thus the nuisance carries no information about the condition beyond the signal. Assumption 2 (Exponential-family prediction). Let τ⋆:→ℝrs _ :S ^r_s be a sufficient statistic, and let hs(s)h_s(s) be a fixed carrier function on S. For each condition c, let α(c)∈ℝrsα(c) ^r_s be its natural parameter, and let ψ be the corresponding log-partition function. The true predictive family is p(s∣c)=hs(s)exp(α(c)⊤τ⋆(s)−ψ(α(c)))p(s c)=h_s(s) \! (α(c) _ (s)-ψ(α(c)) ) (15) for almost every condition c. Similarly, let τz:→ℝrz _z:Z ^r_z be a sufficient statistic, and let hz(z)h_z(z) be a fixed carrier function on Z. For each encoded condition zcz_c, let β(zc)∈ℝrzβ(z_c) ^r_z be its learned natural parameter, and let ϕφ be the corresponding log-partition function. The learned predictive model is pθ(z∣zc)=hz(z)exp(β(zc)⊤τz(z)−ϕ(β(zc)))p_θ(z z_c)=h_z(z) \! (β(z_c) _z(z)-φ(β(z_c)) ) (16) for almost every encoded condition zcz_c. As mentioned before, it is not necessary to assume any particular marginal distribution of the encoded conditioning variables zcz_c. In the joint formulation, one may set pθ(zc)=qf(zc)p_θ(z_c)=q_f(z_c), so the KL term compares only the predictive conditionals (Equation 2). Assumption 3 (Distribution matching and information saturation). Let qf(z,zc)q_f(z,z_c) be the encoded joint distribution and let pθ(z,zc)p_θ(z,z_c) be a learned predictive model. Information-augmented predictive LDM uses the population objective ℱ(f,fc,θ)=−DKL[qf(z,zc)∥pθ(z,zc)]+Iqf[z;zc].F(f,f_c,θ)=-D_KL[q_f(z,z_c)\|p_θ(z,z_c)]+I_q_f[z;z_c]. (17) We assume that an ideal population optimum is realizable through optimizing the model pθp_θ, encoder f, and the condition encoder fcf_c, at which DKL[qf(z,zc)∥pθ(z,zc)]=0,Iqf[z;zc]=I[(s,n);c]=I[s;c]<∞.D_KL[q_f(z,z_c)\|p_θ(z,z_c)]=0, I_q_f[z;z_c]=I[(s,n);c]=I[s;c]<∞. (18) As discussed in the main text, this assumption implicitly replaces common invertibility assumptions. It also implies that latent distributions match, and for the proof we define p(z,zc):=qf(z,zc)=pθ(z,zc).p(z,z_c):=q_f(z,z_c)=p_θ(z,z_c). (19) Assumption 4 (Condition diversity). Even after excluding an arbitrary zero-probability subset of conditions, one can choose c0,c1,…,crsc_0,c_1,…,c_r_s such that D:=[(α(c1)−α(c0))⊤(α(crs)−α(c0))⊤]∈ℝrs×rsD:= bmatrix(α(c_1)-α(c_0)) \\ \\ (α(c_r_s)-α(c_0)) bmatrix ^r_s× r_s (20) is invertible. Invertability requires the condition variation to span all rsr_s natural-parameter directions. In the proof, these directions supply rsr_s independent likelihood-ratio equations that recover every coordinate of the source sufficient statistic. Assumption 5 (No affine redundancy in the source statistic). For every condition c for which Equation 15 holds, the random vector τ⋆(s) _ (s) with s∼p(s∣c)s p(s c) is not almost surely contained in a proper affine hyperplane of ℝrsR^r_s. This assumption complements Assumption 4 and says that the sufficient statistic also genuinely uses all rsr_s coordinates. A.1.2 Full Proof of Theorem 1 Theorem 1 (Restated) (Exponential-family readout in InfoLDM). Suppose Assumptions 1, 2, 3, 4, and 5 hold, and the InfoLDM goal function (Equation 1) is fully maximized. Then the dimension of the model sufficient statistic rz≥rsr_z≥ r_s, and there is a full-row-rank matrix A∈ℝrs×rzA ^r_s× r_z and a vector b∈ℝrsb ^r_s such that τ⋆(s)=Aτz(z)+balmost surely. _ (s)=A _z(z)+b surely. (21) If τ⋆ _ is injective, then s is almost surely a measurable function of z. If rz=rsr_z=r_s, then A is invertible and the two sufficient statistics are related by an invertible affine transformation. Proof. The proof proceeds in four steps. In Step 1, we show that MI maximization implies that the learned context zcz_c and true context c are interchangeable for latent prediction. In Step 2, we show that it also implies that it is possible to equate the likelihood ratios of the learned representation z and the of true signal s with respect to some reference condition c0c_0. This is the key identity that explains why representations have to recover the signal. The following two steps are closely related to previous exponential family identifiability proofs (Khemakhem et al., 2020). In Step 3 we use the above likelihood ratio, the exponential family assumption, and the condition diversity assumption to establish that there is an affine relation between the statistics τz(z) _z(z) and τ⋆(s) _ (s) when z and s are sampled under c0c_0. Finally, we extend this result in Step 4 to the entire p(c)p(c), and derive the full-rank and invertibility relating z and s. Step 1: Information saturation and transfer. The chain rule of mutual information lets us compute I[s;c]−I[z;zc]=(I[s;c]−I[z;c])+(I[z;c]−I[z;zc])=(I[s,z;c]−I[z;c])+(I[z;c,zc]−I[z;zc])=I[s;c∣z]+I[z;c∣zc]. splitI[s;c]-I[z;z_c]&= (I[s;c]-I[z;c] )+ (I[z;c]-I[z;z_c] )\\ &= (I[s,z;c]-I[z;c] )+ (I[z;c,z_c]-I[z;z_c] )\\ &=I[s;c z]+I[z;c z_c]. split (22) The second step follows, since from the generative model (Figure 1) and Assumption 1 we know that z⟂⟂c|sz \!\!\! c s and z⟂⟂zc|cz \!\!\! z_c c. Equation 22, together with Assumption 3 implies s⟂⟂c|zs \!\!\! c z and z⟂⟂c|zcz \!\!\! c z_c. Consequently, for almost every jointly occurring pair (c,zc)(c,z_c), p(z∣c)=p(z∣c,zc)=p(z∣zc).p(z c)=p(z c,z_c)=p(z z_c). (23) Thus p(z∣c)p(z c) is a member of the learned exponential family for almost every condition c. Step 2: Likelihood-ratio equality. The two conditional independences involving s tell us that the conditional distributions relating s and z are independent of c, i.e., p(z∣s,c)=p(z∣s)p(z s,c)=p(z s) and p(s∣z,c)=p(s∣z)p(s z,c)=p(s z). Therefore, the following two factorizations hold for almost every condition c: p(s,z∣c)=p(z∣s)p(s∣c)=p(s∣z)p(z∣c).p(s,z c)=p(z s)p(s c)=p(s z)p(z c). (24) Because the conditional distributions between s and z in Equation 24 do not depend on c, they cancel when the distributions at c and some reference c0c_0 are compared p(s∣c)p(s∣c0)=p(z∣c)p(z∣c0)for almost every (s,z) generated under c0. p(s c)p(s c_0)= p(z c)p(z c_0) almost every $(s,z)$ generated under $c_0$. (25) The ratios are well defined because all selected family members are strictly positive wherever their common carrier functions are positive. Step 3: Locally affine relation. We now want to select a number of rsr_s different contexts cic_i that constrain the relation between s and z. By using Equation 25 for any cic_i we get p(s∣ci)p(s∣c0)=p(z∣ci)p(z∣c0)for almost every (s,z) generated under c0. p(s c_i)p(s c_0)= p(z c_i)p(z c_0) almost every $(s,z)$ generated under $c_0$. (26) Taking logarithms, and using the two exponential-family forms (Assumption 2) we get (α(ci)⊤−α(c0)⊤)τ⋆(s)−[ψ(α(ci))−ψ(α(c0))]=(βi−β0)⊤τz(z)−[ϕ(βi)−ϕ(β0)].(α(c_i) -α(c_0) ) _ (s)-[ψ(α(c_i))-ψ(α(c_0))]=( _i- _0) _z(z)-[φ( _i)-φ( _0)]. (27) Here we chose parameters βi _i for which (by the result of Step 1) p(z∣ci)=hz(z)exp(βi⊤τz(z)−ϕ(βi)).p(z c_i)=h_z(z) \! ( _i _z(z)-φ( _i) ). (28) We now define di:=α(ci)−α(c0)d_i:=α(c_i)-α(c_0), and ei:=βi−β0e_i:= _i- _0. With this notation Equation 27 becomes di⊤τ⋆(s)=ei⊤τz(z)+hi,hi:=ψ(α(ci))−ψ(α(c0))−ϕ(βi)+ϕ(β0).d_i _ (s)=e_i _z(z)+h_i, h_i:=ψ(α(c_i))-ψ(α(c_0))-φ( _i)+φ( _0). (29) Let D have rows di⊤d_i , E have rows ei⊤e_i , and let h=(h1,…,hrs)⊤h=(h_1,…,h_r_s) . Thus, stacking Equation 29 gives Dτ⋆(s)=Eτz(z)+hfor almost every (s,z) generated under c0.D _ (s)=E _z(z)+h almost every $(s,z)$ generated under $c_0$. (30) Finally, Assumption 4 lets us choose conditions c0,c1,…,crsc_0,c_1,…,c_r_s for which D is invertible. This proves the readout for samples generated under c0c_0, with A:=D−1E,b:=D−1h.A:=D^-1E, b:=D^-1h. (31) Step 4: Globalization, rank, and reconstruction. What remains is to extend this reference-condition identity to the full joint distribution, prove that A has full row rank, and derive the reconstruction conclusions. For globalization, we note that for almost every condition c, the distribution p(s∣c)p(s c) has the same zero-probability sets as p(s∣c0)p(s c_0) because both are strictly positive on the common carrier. Multiplying these distributions with the same p(z∣s)p(z s) preserves this property for the conditional joint distributions of p(s,z∣c)p(s,z c), i.e., any set of probability zero under p(s,z∣c0)p(s,z c_0) has probability zero under p(s,z∣c)p(s,z c). In particular, the set (s,z):τ⋆(s)≠Aτz(z)+b\(s,z): _ (s)≠ A _z(z)+b\ has probability zero under p(s,z∣c0)p(s,z c_0) and hence under p(s,z∣c)p(s,z c). The readout therefore holds conditionally for almost every condition c. Averaging over p(c)p(c) gives Equation 8 under the original joint distribution. To show that A is full rank, assume it were not, i.e., rank(A)<rsrank(A)<r_s. In this case the reference-condition identity (Equation 30) places τ⋆(s) _ (s) in the proper affine subspace b+im(A)b+im(A) under p(s∣c0)p(s c_0). Because p(s∣c0)p(s c_0) is strictly positive on the common carrier, this contradicts Assumption 5. Thus rank(A)=rsrank(A)=r_s and rz≥rsr_z≥ r_s. When rz=rsr_z=r_s, the full-row-rank matrix A is square and hence invertible. To reconstruct s from z we have to invert τ⋆ _ . If τ⋆ _ is injective, the Lusin–Souslin theorem supplies a measurable inverse on its image, which may be extended measurably by a fixed value outside that image, so s=τ⋆−1(Aτz(z)+b)almost surely.s= _ ^-1\! (A _z(z)+b ) surely. (32) ∎ A.2 Lean formalization To verify the proof we translated it to compiling Lean code using generative AI. The declaration statisticReadout_of_exponentialFamily_nuisanceLDM formalizes Theorem 1 after the mutual-information argument. Below is the header of the main function which includes the main assumptions and the conclusion. The line numbers refer to the actual Lean source, the full declaration and proof can be found in the source code at Lean/ExponentialFamilyReadout.lean. 758 theorem statisticReadout_of_exponentialFamily_nuisanceLDM759 Ω H V X Y : Type*760 [MeasurableSpace Ω ] [StandardBorelSpace Ω ]761 [MeasurableSpace H] [StandardBorelSpace H] [Nonempty H]762 [MeasurableSpace V] [StandardBorelSpace V] [Nonempty V]763 [MeasurableSpace X] [StandardBorelSpace X] [Nonempty X]764 [MeasurableSpace Y] [StandardBorelSpace Y] [Nonempty Y]765 (P : Measure Ω ) [IsProbabilityMeasure P]766 (history : Ω → H) (view : Ω → V)767 (signal : Ω → X) (code : Ω → Y)768 (hhistory : Measurable history) (hview : Measurable view)769 (hsignal : Measurable signal) (hcode : Measurable code)770 (signal_history_given_code : signal ⟂ [code, hcode; P] history)771 (code_history_given_signal : code ⟂ [signal, hsignal; P] history)772 (code_history_given_view : code ⟂ [view, hview; P] history)773 (code_view_given_history : code ⟂ [history, hhistory; P] view)774 (sourceCarrier : Measure X) (codeCarrier : Measure Y)775 (sourceStatistic : X → EuclideanSpace ℝR ι )776 (codeStatistic : Y → EuclideanSpace ℝR κ)777 (hsourceStatistic : Measurable sourceStatistic)778 (hcodeStatistic : Measurable codeStatistic)779 (sourcePartition : EuclideanSpace ℝR ι → ℝR)780 (codePartition : EuclideanSpace ℝR κ → ℝR)781 (sourceParameter : H → EuclideanSpace ℝR ι )782 (learnedParameter : V → EuclideanSpace ℝR κ)783 (hsourceConditional :784 condDistrib signal history P =m[P.map history]785 fun c ↦ statisticNaturalExpMeasure sourceCarrier sourceStatistic786 sourcePartition (sourceParameter c))787 (hlearnedConditional :788 condDistrib code view P =m[P.map view]789 fun v ↦ statisticNaturalExpMeasure codeCarrier codeStatistic790 codePartition (learnedParameter v))791 (hhistoryDiversity :792 ∀ good : Set H, (∀ c ∂ .map history, c ∈ good) →793 ∃ (c0 : H) (c : ι → H),794 c0 ∈ good ∧ 795 (∀ i, c i ∈ good) ∧ 796 Submodule.span ℝR797 (Set.range fun i ↦ sourceParameter (c i) - sourceParameter c0) = ⊤ )798 (hsourceNoAffineRedundancy :799 ∀ (w : EuclideanSpace ℝR ι ) (a : ℝR),800 (fun x ↦ inner ℝR w (sourceStatistic x)) =m[sourceCarrier]801 (fun _ ↦ a) →802 w = 0) :803 ∃ (A : EuclideanSpace ℝR κ →L[ℝR] EuclideanSpace ℝR ι )804 (b : EuclideanSpace ℝR ι ),805 (fun ω ↦ sourceStatistic (signal ω)) =m[P]806 (fun ω ↦ A (codeStatistic (code ω)) + b) ∧ 807 Function.Surjective A ∧ 808 Module.finrank ℝR (EuclideanSpace ℝR ι ) ≤ 809 Module.finrank ℝR (EuclideanSpace ℝR κ) ∧ 810 (Function.Injective sourceStatistic →811 ∃ decoder : Y → X, Measurable decoder ∧ 812 signal =m[P] fun ω ↦ decoder (code ω)) ∧ 813 (Module.finrank ℝR (EuclideanSpace ℝR κ) =814 Module.finrank ℝR (EuclideanSpace ℝR ι ) →815 Function.Bijective A) := by This declaration reads in blocks as follows. Lines 759–769: probability space and variables. Omega is the underlying probability space, and H, V, X, and Y are the state spaces of c,zc,s,zc,z_c,s,z. The standard-Borel and measurability assumptions are the formal version of the measurable setup in Section 1. The finite types ι and κ index the coordinates of τ⋆ _ and τz _z, so their finranks are rsr_s and rzr_z. Lines 770–773: the four conditional independences. In the order shown, these are s⟂⟂c∣z,z⟂⟂c∣s,z⟂⟂c∣zc,z⟂⟂zc∣c.s \!\!\! c z, z \!\!\! c s, z \!\!\! c z_c, z \!\!\! z_c c. (33) The second and fourth are the Markov consequences of Assumption 1. The first and third follow in the paper from I[s;c]−I[z;zc]=I[s;c∣z]+I[z;c∣zc]=0.I[s;c]-I[z;z_c]=I[s;c z]+I[z;c z_c]=0. (34) Thus Lean starts at the exact conditional-independence consequences of MI saturation, rather than formalizing mutual information and the optimization problem. Lines 774–790: the two exponential families. The names sourceCarrier, sourceStatistic, sourcePartition, and sourceParameter correspond to hs,τ⋆,ψ,αh_s, _ ,ψ,α. Their code-side counterparts correspond to hz,τz,ϕ,βh_z, _z,φ,β. The two hypotheses hsourceConditional and hlearnedConditional are the conditional families p(s∣c) p(s c) =hs(s)exp(α(c)⊤τ⋆(s)−ψ(α(c))), =h_s(s) \! (α(c) _ (s)-ψ(α(c)) ), (35) p(z∣zc) p(z z_c) =hz(z)exp(β(zc)⊤τz(z)−ϕ(β(zc))), =h_z(z) \! (β(z_c) _z(z)-φ(β(z_c)) ), (36) where statisticNaturalExpMeasure is the Lean implementation of the exponential-family distribution in Equation 15 and Equation 16. Lean’s eventual-equality notation means that the displayed equality may fail only on a zero-probability set. Lines 791–802: diversity and nonredundancy. The hypothesis hconditionDiversity is Assumption 4 written without coordinates: after removing any zero-probability set of conditions, it selects c0c_0 and a family (ci)(c_i) whose differences α(ci)−α(c0)α(c_i)-α(c_0) span the whole source-statistic space. The hypothesis hsourceNoAffineRedundancy is Assumption 5: if ⟨w,τ⋆(s)⟩ w, _ (s) is almost surely constant on the carrier, then w=0w=0. Lines 803–815: conclusion. Lean returns a continuous linear map A:ℝrz→ℝrsA:R^r_z ^r_s and an offset b with τ⋆(s)=Aτz(z)+balmost surely. _ (s)=A _z(z)+b surely. (37) Function.Surjective A is the full-row-rank conclusion rank(A)=rsrank(A)=r_s, and the finrank inequality is rs≤rzr_s≤ r_z. The last two lines add the two corollaries in Theorem 1: an injective τ⋆ _ gives a measurable decoder z↦sz s, and equal statistic dimensions make A bijective. The proof first establishes the likelihood-ratio identity at a reference condition and then extends it to the full model. A.3 Simulation details Numerical evaluation. We generated 100000100000 training and 1000010000 test sequences of length five. The fixed observation map consisted of three affine layers with leaky-ReLU activations and mapped the concatenated signal and nuisance variables to 100100-dimensional observations. The encoder was a five-layer MLP with 200200 hidden units per layer and ReLU activations. Its output dimension matched the signal dimension, which was varied over dS∈5,10,15,20d_S∈\5,10,15,20\. A single-layer LSTM with 1010 hidden units summarized the observation history, and a linear prediction head predicted the next latent state. Models were trained for 2020 epochs using Adam and a linearly decaying learning rate. Batch size was 256256 for the kNN and log-determinant estimators. In this and the following experiments, entropy estimation was implemented as described by Mikulasch and Zenke (2026). We found that KDE performed poorly in this experiment and increased batchsize to 50965096 (von Kügelgen et al., 2021, similar to ) and the number of samples to 5×1065× 10^6 to improve performance. Causal3DIdent. We based our experiment on the data of von Kügelgen et al. (2021) but resized images to 224×224224× 224 pixels. We used a ResNet-18 encoder producing a 1010-dimensional representation, followed by a single-layer LSTM with 6464 hidden units and a linear prediction head. Models were trained for 2525 epochs with Adam. Batch sizes were 256256 for the kNN and KDE estimators and 6464 for the log-determinant estimator. MuJoCo Hopper. We generated 100000100000 training and 10001000 test rollouts from the DM-Control Hopper environment. Each sequence contained eight 128×128128× 128 rendered frames, separated by three control steps. Actions were produced by a pretrained controller (Nikulin et al., 2025) and perturbed with Gaussian exploration noise of standard deviation 0.10.1. Additional unobserved actuator noise with standard deviation 0.50.5 was applied before executing each action, making the controlled dynamics stochastic. Note that adding Gaussian noise every simulation step, not every render step, results in a system that does not fully follow the assumptions of Theorem 1. At every rendered timestep, independent nuisance variables modified the body and accent colors, camera position and field of view, illumination, and a procedurally generated colored background. The physical signal used for evaluation consisted of seven position variables and seven velocity variables. Images were encoded by a ResNet-18 into a 1616-dimensional latent state. A single-layer LSTM with 6464 hidden units summarized the latent history. The three controller commands between consecutive frames were encoded by a GRU with 1616 hidden units and supplied, together with the LSTM state, to a one-hidden-layer prediction MLP. For image decoding we used 4-layer upsampling CNNs trained with L2 loss. With attached decoder loss we approximately balanced latent prediction and image prediction loss via a Lagrange multiplier. For the observation prediction model we used the exact same setup without latent prediction loss and the decoder attached to the prediction module. Models were trained for 4040 epochs using Adam. Batch sizes were 128128, 512512, 3232, and 128128 for the kNN, KDE, log-determinant estimators, and next observation prediction, respectively. A.4 Additional results A.4.1 Causal3DIdent Table 3: Nonlinear R2R^2 for the Causal3DIdent experiment (Table 2), computed with trained MLP predictor on held-out dataset. Scenario Entr. est. pobj-xp_obj-x pobj-yp_obj-y pobj-zp_obj-z θobj-α _obj-α θobj-β _obj-β θobj-γ _obj-γ hobjh_obj pspotp_spot hspoth_spot hbackh_back Environment signal kde 0.350.35 0.520.52 0.340.34 0.070.07 0.100.10 0.160.16 0.530.53 0.990.99 0.970.97 0.990.99 Environment signal knn 0.210.21 0.470.47 0.340.34 0.080.08 0.060.06 0.080.08 0.160.16 0.980.98 0.960.96 0.980.98 Environment signal logdet −0.02-0.02 −0.02-0.02 −0.03-0.03 −0.02-0.02 −0.01-0.01 −0.00-0.00 −0.01-0.01 0.990.99 0.570.57 0.980.98 Object signal kde 0.980.98 0.980.98 0.970.97 0.940.94 0.970.97 0.850.85 0.910.91 0.140.14 0.030.03 0.360.36 Object signal knn 0.970.97 0.960.96 0.950.95 0.870.87 0.920.92 0.290.29 0.910.91 0.140.14 −0.04-0.04 0.690.69 Object signal logdet 0.990.99 0.990.99 0.980.98 0.950.95 0.970.97 0.960.96 0.920.92 0.160.16 −0.04-0.04 −0.04-0.04 A.4.2 Hopper Figure 4: Linear (Top) and nonlinear (Bottom) R2R^2 with physical (signal) and nuisance variables for the hopper experiment. Velocity is read out from LSTM state. Background pattern nuisance is not included.