Paper deep dive

Identifiability of Potentially Degenerate Gaussian Mixture Models With Piecewise Affine Mixing

Danru Xu, Sébastien Lachapelle, Sara Magliacane

Year: 2026Venue: arXiv preprintArea: stat.MLType: PreprintEmbeddings: 206

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 92%

Last extracted: 4/18/2026, 1:25:25 AM

Summary

The paper addresses the identifiability of latent variables in Causal Representation Learning (CRL) when these variables follow a potentially degenerate Gaussian Mixture Model (pdGMM) and are observed through a piecewise affine mixing function. The authors provide theoretical results proving identifiability up to affine transformations within components (ATwC), global affine transformations (AT), and permutation and scaling (PS), by leveraging sparsity regularization and specific geometric assumptions on the component supports.

Entities (4)

Causal Representation Learning · field-of-study · 100%pdGMM · statistical-model · 100%Identifiability · theoretical-property · 95%Piecewise Affine Mixing · mathematical-function · 95%

Relation Signals (2)

pdGMM → issubjectto → Piecewise Affine Mixing

confidence 90% · latent variables that follow a potentially degenerate Gaussian mixture distribution and that are only observed through the transformation via a piecewise affine mixing function

Causal Representation Learning → utilizes → pdGMM

confidence 85% · We study this problem for latent variables that follow a potentially degenerate Gaussian mixture distribution

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Causal representation learning (CRL) aims to identify the underlying latent variables from high-dimensional observations, even when variables are dependent with each other. We study this problem for latent variables that follow a potentially degenerate Gaussian mixture distribution and that are only observed through the transformation via a piecewise affine mixing function. We provide a series of progressively stronger identifiability results for this challenging setting in which the probability density functions are ill-defined because of the potential degeneracy. For identifiability up to permutation and scaling, we leverage a sparsity regularization on the learned representation. Based on our theoretical results, we propose a two-stage method to estimate the latent variables by enforcing sparsity and Gaussianity in the learned representations. Experiments on synthetic and image data highlight our method's effectiveness in recovering the ground-truth latent variables.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

205,462 characters extracted from source content.

Expand or collapse full text

Identifiability of Potentially Degenerate Gaussian Mixture Models With Piecewise Affine Mixing Danru Xu Sébastien Lachapelle Sara Magliacane University of Amsterdam Samsung AI Lab, Montreal Saarland University & University of Amsterdam Abstract Causal representation learning (CRL) aims to identify the underlying latent variables from high-dimensional observations, even when variables are dependent with each other. We study this problem for latent variables that follow a potentially degenerate Gaussian mixture distribution and that are only observed through the transformation via a piecewise affine mixing function. We provide a series of progressively stronger identifiability results for this challenging setting in which the probability density functions are ill-defined because of the potential degeneracy. For identifiability up to permutation and scaling, we leverage a sparsity regularization on the learned representation. Based on our theoretical results, we propose a two-stage method to estimate the latent variables by enforcing sparsity and Gaussianity in the learned representations. Experiments on synthetic and image data highlight our method’s effectiveness in recovering the ground-truth latent variables. 1 INTRODUCTION Recovering latent variables from high-dimensional observations, such as images, videos, or text, is essential for building reliable and interpretable models (Bengio et al., 2013). While traditional approaches, e.g., (nonlinear) ICA (Comon, 1994; Hyvärinen, 2013; Hyvarinen and Morioka, 2016, 2017), identify independent or conditionally independent variables, in many real-world settings latent variables often exhibit complex dependencies, e.g., because they are causally related. Causal representation learning (CRL) (Schölkopf et al., 2021) aims to identify latent variables with arbitrary dependencies from observations. Many CRL works consider additional data or information to identify causal variables, e.g., interventions (Brehmer et al., 2022; Lippe et al., 2022; Varici et al., 2023; von Kügelgen et al., 2023; Buchholz et al., 2023; Zhang et al., 2023; Ahuja et al., 2023a), actions or temporal structure (Lippe et al., 2023; Lachapelle et al., 2022, 2024), multi-view structures (von Kügelgen et al., 2021; Ahuja et al., 2023b; Yao et al., 2024; Morioka and Hyvärinen, 2024), grouping of observations by sparsity patterns (Xu et al., 2024), hierarchical structures (Kong et al., 2023, 2024) or conditions on the causal graphs (Zhang et al., 2024; Dong et al., 2024; Song et al., 2024). In our case, we do not assume any additional data or information besides the observations, but we instead consider a set of parametric assumptions on the latent variables Z and mixing function f through which we get the observation =()X=f(Z). In particular, we focus on latent variables that follow a mixture of Gaussian distributions with a piecewise affine mixing function. Similarly to us, Kivva et al. (2022) studies the identifiability of non-degenerate Gaussian Mixture Model (GMM) latent variables with piecewise affine mixing, but to achieve element-wise identifiability they require that all pairs of latent variables must be conditionally independent given an (unobserved) auxiliary variable. We achieve the same identifiability result without requiring this auxiliary variable, while extending the setting to latent variables that follow a potentially degenerate Gaussian Mixture Model (pdGMM), i.e., a GMM in which the components can be potentially degenerate Gaussians, but we instead require a sparsity assumption. This is inspired by the success of sparse (low-rank, i.e., degenerate) representations. In particular, in many real-world settings, mixtures of low-dimensional subspaces provide a more accurate model than full-rank representations (Buchanan et al., 2025). We provide a series of progressively stronger identifiability results under reasonable assumptions in this challenging setting, in which the probability density functions are ill-defined and hence previous results cannot be applied. As a final step, we leverage a sparsity regularization on the learned representation, inspired by multi-task learning (Lachapelle et al., 2023b) and partially observable CRL (Xu et al., 2024) to prove identifiability up to permutation, scaling and translation, i.e., that we learn a disentangled representation of the latent variables. We highlight our contributions: • We provide an identifiability result for pdGMMs, showing that the full distribution can be uniquely identified from an open subset that intersects the support of every component (Thm. 3.2). This is a critical step in our other results, but is also of independent interest due to its proof strategy; • We introduce a series of identifiability results for recovering latent variables that follow a pdGMMs distribution from observations (Thm. 3.5, 3.7, 3.9), showing that under progressively stronger assumptions, latent variables can be recovered up to (i) affine transformation within components (ATwC), (i) global affine transformation (AT), and finally, (i) permutation and scaling (PS), without interventions or supervision; • Finally, we propose a two-stage algorithm that implements these theoretical results and evaluate it on numerical and image datasets. 2 PROBLEM SETTING AND BACKGROUND We consider a setting with latent random variables =(Z1,…,Zn)∈⊆ℝnZ=(Z_1,...,Z_n) ^n that have arbitrary dependences with each other. Given only a set of observations ∈⊆ℝdX ^d generated by an unknown mixing function :→f:Z , our goal is to recover the latent variables Z. In general, we cannot fully recover these latent variables, but we can identify them up to an equivalence class. We first summarize two types of identifiability presented in previous work (Comon, 1994; Khemakhem et al., 2020; Kivva et al., 2022; Lachapelle et al., 2023b). Definition 2.1. Given an observation =()X=f(Z), a learned representation ()g(X) identifies Z up to permutation and scaling (PS) when there exists a permutation matrix P and an invertible diagonal matrix D such that ()=PDg(X)=PDZ almost surely. A weaker notion of identifiability is up to a global affine transformation (AT), which instead means that there exists a global invertible affine transformation h such that ()=()g(X)=h(Z) almost surely. In this paper, we assume that the observations X are generated by an injective continuous piecewise affine mixing function f (Def. B.10). We also assume that Z follows a potentially degenerate Gaussian mixture model (pdGMM) with unknown parameters, i.e., a Gaussian mixture model that explicitly allows for degenerate components (with singular covariance matrices). Gaussian Mixture Models (GMM) are a popular method in clustering. While the standard definition also allows for degenerate components, in practice there is often the assumption that the components are not degenerate. Since in this paper we focus on this setting, we make this choice explicit by slightly modifying their name and defining them as follows: Definition 2.2. A variable Z follows a potentially degenerate Gaussian mixture model (pdGMM) if its distribution is P=∑j=1JλjN(j,j)P= _j=1^J _jN( μ_j, _j), where for each component j∈[J]j∈[J] the mean is j∈ℝn μ_j ^n and the covariance matrix is j∈ℝn×n _j ^n× n, such that determinant |j|≥0| _j|≥ 0. We assume that the weights λj>0 _j>0 and ∑j=1Jλj=1 _j=1^J _j=1. If each component is distinct, i.e., for every i≠j∈[J]i≠ j∈[J] we have (i,i)≠(j,j)( μ_i, _i)≠( μ_j, _j), then we say that the pdGMM is in a reduced form. Other works have already studied latent variables that follow a non-degenerate GMM distribution and considered the problem of identifying these variables from observation. In particular, Kivva et al. (2022) provide the identifiability up to AT of non-degenerate GMMs latent variables with piecewise linear mixing functions. For a stronger notion of identifiability, identifiability up to PS, Kivva et al. (2022) require that all pairs of latent variables must be conditionally independent given an auxiliary variable U, which is an assumption we will not make. Crucially, these results rely on the analyticity of the probability density function (pdf) of Gaussian distributions, which does not hold for degenerate Gaussians, since their pdf is no longer well-defined. This means we cannot apply these results directly. In this challenging setting, we instead leverage results from a different line of work focusing on sparsity (Lachapelle et al., 2023b; Xu et al., 2024) to prove the identifiability up to PS and a constant translation for piecewise affine mixing functions. Lachapelle et al. (2023b) focus on multi-task learning with sparse task-specific predictors, while Xu et al. (2024) focus on partially observable causal representation learning settings, where each observation only captures a small number of “active” latent variables. While these works focus on sparsity, their motivation is related to our choice to allow for potential degenerate representations, since sparse (i.e., low-rank) representations can be seen as representations in which some of the components are degenerate. In many real-world settings we have (nearly)degenerate structures, such as high collinearity or strong dependencies among features, or intrinsically low-rank manifolds. This is typical for high-dimensional data that require large numbers of sparse features such as language models (Cunningham et al., 2024; Gao et al., 2025; Marks et al., 2025). 3 IDENTIFIABILITY OF PDGMMS Figure 1: Example of a 3-dimensional pdGMM with 2 components: a rank-2 component with support represented by the yellow plane, and a rank-1 component with support represented by the red line. Knowing the distribution on the open set E (in brown), intersecting both components, is sufficient to identify the pdGMM. In this section, we provide a series of theoretical results that prove progressively stronger notions of identifiability for latent variables following a potentially degenerate Gaussian Mixture Model (pdGMM) distribution with a piecewise affine mixing function. We provide all proofs in Appendix˜B. As a first step, we focus on the case in which the pdGMM variables are observed and prove that if two pdGMM variables have the same distribution on an open set, then they have the same distribution over all of ℝnR^n, i.e., they are identified. The intuition is illustrated in the example in Fig. 1. In addition to being of independent interest, this result is an important step for our other results. 3.1 Identifiability from an open subset Identifiability results for (pd)GMMs from the whole domain are well-established (Yakowitz and Spragins, 1968), proving that if two (pd)GMMs are the same on their domain, then they have the same components and weights up to a permutation. These results include the potentially degenerate case. On the other hand, in our quest to prove the identifiability of latent variables that follow a pdGMM only from observations, we need an identifiability result for observed pdGMMs on a subdomain, and more precisely, an open subset. We highlight this result in this subsection, because it is of independent interest beyond the CRL setting. Since we cannot rely on equality of pdfs because of potential degeneracy, we first need a definition of when two distributions are considered equal on a subdomain. Definition 3.1. Let μ:ℱ→[0,1]μ:F→[0,1] and μ′:ℱ→[0,1]μ :F→[0,1] be the probability measures induced by X and ′X , respectively, where ℱF is the Borel sigma-algebra over ℝnR^n. Given a subset E∈ℱE , we define the restriction of μ to E, denoted by μE:ℱ→[0,1] _E:F→[0,1], as μE(A):=μ(A∩E),∀A∈ℱ _E(A):=μ(A∩ E),∀ A . We say that X and ′X are equal in distribution over a subdomain E, denoted as =′X d=X on E, when μE=μE′ _E=μ _E. We show that the equality of two pdGMMs on an open subset of ℝnR^n that intersects the support of every Gaussian component in a pdGMM implies the equality of pdGMMs on the whole domain. Theorem 3.2 (Identifiability of pdGMMs from open set). Consider two pdGMMs in ℝnR^n in reduced form, ∼∑j=1JλjN(j,j)and′∼∑j=1J′λj′N(j′,j′),X _j=1^J _jN( μ_j, _j) _j=1^J _j N( μ_j , _j ), (1) where rank(j)>0,∀j∈[J]rank( _j)>0,∀ j∈[J] and rank(j′)>0,∀j∈[J′]rank( _j)>0,∀ j∈[J ]. Let jX_j be the supports of N(j,j)N( μ_j, _j). If there exists an open set E such that for all j∈[J]j∈[J], j∩E≠∅X_j∩ E≠ and =′X d=X on E, then =′X d=X on the whole domain, J=J′J=J , and there exists a permutation π:[J]→[J′]π:[J]→[J ] such that (λj,j,j)=(λπ(j)′,π(j)′,π(j)′)( _j, μ_j, _j)=(λ _π(j), μ _π(j), _π(j)). We provide the proof in App. B.1 and describe a proof sketch here. The main challenge is the ill-defined pdf of pdGMMs. Kivva et al. (2022) provides similar results for non-degenerate GMM, but their proof strategy relies on the analyticity of the pdf. However, in the degenerate case, the density does not exist on ℝnR^n, so their proof cannot be applied. We overcome this issue by projecting a pdGMM in ℝnR^n into a sequence of lower-dimensional spaces ℝkR^k, where k∈[n]k∈[n], such that each degenerate Gaussian component becomes non-degenerate in at least one image space. This enables us to apply classical identifiability results to recover the parameters of Gaussian components grouped by their ranks (Yakowitz and Spragins, 1968; Kivva et al., 2022). To make this strategy work, the projections must satisfy two key properties. First, they must preserve the rank of each component’s covariance matrix, thereby maintaining the intrinsic structure and information of the components. Second, a single projection is insufficient to fully recover the parameters of a pdGMM from lower-dimensional spaces; instead, a sufficiently rich set of rank-preserving projections is required to capture all distinguishable parameterizations. We therefore choose to work with a dense subset of the space of projections, and prove that such set of projections always exists. 3.2 Identifiability of latent pdGMMs In this section, we introduce our main results (Thm. 3.5, 3.7, and 3.9), which form a set of increasingly stronger assumptions on the distribution of Z, each yielding a correspondingly stronger identifiability. As a first step we introduce a weak notion of identifiability for mixture-based latent variables (i.e., the distribution is a mixture of multiple components), identifiability up to affine transformations within components (ATwC). This notion resembles the identifiability up to an affine transformation (AT) from literature, but it provides even less guarantees, since the recovered variables identify the ground truth latent variables only up to an affine function h within the support of each component, but without any guarantee that h is affine across all components, thus sacrificing the interpretability of the learned representations. We provide Example˜E.3 to illustrate the difference between ATwC and AT. Formally, we define identifiability up to ATwC as: Definition 3.3. The ground truth mixture-based representation vector Z is said to be identified up to affine transformations within components (ATwC) by a learned representation vector ()g(X) when there exists an invertible transformation h, such that ()=()g(X)=h(Z) almost surely and h is affine on each jZ_j, where jZ_j is the support of j-th component. To prove identifiability up to ATwC, we introduce a genericity condition on pdGMMs. We consider pdGMMs for which the supports of components with the same rank k may overlap, but there is at least one common point in the intersection where the components can be distinguished, i.e., their projections onto the non-degenerate subspace do not have the same norm. This rules out special cases where two components are geometrically indistinguishable in their overlapping region, which we prove in App. E.2 is a measure zero set in the parameter space under consideration. Intuitively, this means that this assumption only rules out highly symmetric or finetuned cases. We formalize it as follows: Assumption 3.4 (Genericity of pdGMMs). Consider a pdGMM in reduced form ∼∑j=1JλjN(j,j)Z _j=1^J _jN( μ_j, _j). Let k:=j∈[J]:rank(j)=k)J_k:=\j∈[J]:rank( _j)=k)\ be the indices of the components with rank k. For any k≤nk≤ n and any subset k0⊆kJ_k^0 _k, suppose that the intersection of the supports of the components jZ_j is non-empty, i.e., ∩j∈k0j≠∅ _j _k^0Z_j≠ . We assume there exists at least one point 0∈∩j∈k0jz_0∈ _j _k^0Z_j s.t. the Mahalanobis distance 111Mahalanobis distance of a point 0z_0 from a distribution N(j,j)N( μ_j, _j) is (0−j)Tj−1(0−j) (z_0- μ_j)^T _j^-1(z_0- μ_j). When j _j is singular, we use the Moore-Penrose inverse for j−1 _j^-1 of 0z_0 to the distributions of any two distinct Gaussian components i,j∈k0i,j _k^0 is not the same. Based on Ass. 3.4, we show that we can learn a pdGMM representation up to ATwC with a perfect reconstruction and enforcing Gaussianity on representations. Theorem 3.5 (Identifiability up to ATwC of pdGMM latent variables). Assume that Z is a pdGMM and the observation =()X=f(Z) follows the data-generating process in Sec. 2 for an injective continuous piecewise affine mixing function f, and Z satisfies Ass. 3.4. Let :→ℝng:X ^n be an invertible continuous piecewise affine function and ^:ℝn→ℝd f:R^n ^d be an invertible piecewise affine function onto its image. If both conditions hold, ‖−^(())‖22=0,and - f(g(X)) ^2_2=0\,,and (2) ()∼∑j=1J′λj′N(j′,j′), (X) _j=1^J λ _jN( μ _j, _j), (3) for appropriate values of j′,j′ μ _j, _j (in reduced form), then J’=J and Z is identified by ()g(X) up to affine transformation within components, i.e., ∘g is an invertible affine function on jZ_j, j∈[J]j∈[J]. We provide a proof in App. B.2.1. While Thm 3.5 obtains the identifiability up to ATwC, this result does not guarantee that there is a single global affine transformation h such that ()=()g(X)=h(Z) for all components supports. To achieve this stronger identifiability, the identifiability up to AT, we need an additional condition on Z. The following assumption requires that the supports of all components intersect at at least one point 0z^0 and that each of these subspaces is spanned by a subset of a shared global basis. Assumption 3.6 (Common basis and translation vector). Let 1,…,J⊆ℝnZ_1,…,Z_J ^n be affine spaces such that there exists 0∈⋂j=1Jjz^0∈ _j=1^JZ_j. We assume there exists a basis of ℝnR^n, denoted by 1,…,n\z^1,…,z^n\, such that for all j∈[J]j∈[J], there exists j⊆[n]K_j [n] s.t. k∣k∈j\z^k k _j\ is a basis for the vector space j−0Z_j-z^0. This assumption always holds for non-degenerate components. For degenerate components, a violation can happen because the components are not intersecting, or if their individual bases do not form a global shared basis. In App. E.3 we discuss a concrete scenario in which this assumption holds, where only a subset of latent variables are “active” in each observation sample, e.g., they are captured in an image, while others are “inactive”, i.e., they are masked by a constant. In this setting, the intersection of the supports is the vector of all the constant values for each variable, which will form a well-defined global shared basis. If this assumption only holds for a subset of components, then we can achieve identifiability up to AT for this subset, while for the rest can only achieve identifiability up to ATwC using the previous result. We provide an example to show the necessity of this assumption in App. E.3 and use it to achieve identifiability up to AT: Theorem 3.7 (Identifiability up to AT of pdGMM latent variables). We assume the same conditions as Thm. 3.5, and that the supports of components 1,…,JZ_1,…,Z_J satisfy the common basis assumption (Ass. 3.6), then the representation ()g(X) identifies Z up to a single global affine transformation across components, i.e., ∘g is affine on all of Z. We provide a proof in App. B.2.2. Identifiability up to an affine transformation (AT) still allows for arbitrary mixing of dimensions through general affine transformations. In many applications, we prefer representations where individual dimensions have separate and interpretable meanings. This motivates us to move to identifiability up to permutation and scaling (PS). Permutation and scaling are fundamental indeterminacies that typically cannot be solved without supervision or auxiliary information (Hyvärinen et al., 2024). To get this stronger result, we require a stricter assumption on Z. Assumption 3.8. Let 1,…,J⊆ℝnZ_1,…,Z_J ^n be the supports of the components of Z. We assume: • [Common Standard Basis] For all j∈[J]j∈[J], each jZ_j is a vector space, and there exists an index set j⊆[n]K_j [n] such that k∣k∈j\e^k k _j\ forms a basis for jZ_j, where 1,…,n\e^1,…,e^n\ denotes the standard basis of ℝnR^n (one-hot vectors). • [Sufficient Support Basis Index Variability] For every i∈[n]i∈[n], the union of the support index sets jK_j that do not include i covers all other standard basis directions: ⋃j∈[J]∣i∉j=[n]∖i. _j∈[J]\, \,i _jK_j=[n] \i\. This setting is consistent with the sparsity principle proposed by Xu et al. (2024), which shows that by enforcing sparsity of the transformed variables (as expressed in Eq. 4), the corresponding transformation is an element-wise affine function. The Common Standard Basis assumption is a special case of the earlier Common Basis Assumption (Ass. 3.6) where the global basis is fixed to the standard basis and zero translation vector is applied. This implies that the realizations of Z will have some coordinates equal to 0 with probability greater than zero. For example, assuming n=3n=3 with 1=1K_1=\1\ and 2=2,3K_2=\2,3\, we have that samples from the component j=1j=1 will have the form (1,0,0)(z_1,0,0) and samples from the component j=2j=2 will have the form (0,2,3)(0,z_2,z_3). In other words, the realizations of Z are sparse, with nonzero coordinates given by jK_j. The Sufficient Support Basis Index Variability was originally proposed by Lachapelle et al. (2023b) in the context of sparse multitask learning. Here, we adapt it into our context and use it to avoid cases in which certain latent variables are always degenerate together across all components, since then we would not be able to disentangle them from the observations. As we show in App. E.5, even if this assumption does not hold strictly in a setting, our framework still supports a weaker form of identifiability: block-wise identifiability, which allows disentanglement across blocks of latent variables, even if variables within each block may remain entangled. This relaxation still enables meaningful interpretability and structure in learned representations. We can now prove our final theoretical result that shows when we can disentangle, or identify pdGMM latent variables up to PS. Theorem 3.9 (Identifiability up to PS of pdGMM latent variables). We assume the same conditions as Thm. 3.5 and that the supports of components 1,…,JZ_1,…,Z_J satisfy Ass. 3.8. If the following holds: ‖()‖0≤‖0, (X) _0 _0, (4) where ∥⋅∥0 · _0 means L0 norm, then the representation ()g(X) identifies Z up to PST, i.e., ∘g is a permutation combined with an element-wise linear transformation on Z. We provide a proof in App. B.2.3. Interestingly, a special case of this result also solves an open question in previous work on partially observed causal representation learning (CRL) (Xu et al., 2024). In that setting the identifiability results required that the component index j is observed for each observation (that is the group index), which is a strong assumption. Modelling this partially observable CRL problem with mixture models allows one to consider settings in which we do not know how to group the observations, i.e., a more realistic partially observable CRL setting. 4 IMPLEMENTATION We implement our theoretical results in two stages: the identifiability up to AT in Thm. 3.7 serves as a prerequisite for achieving the identifiability up to PS in Thm. 3.9. In the first stage, we use an autoencoder structure to estimate the latent variables directly from observations ii∈[N]\x^i\_i∈[N] by minimizing the reconstruction error (Eq. 2), where ψ1g_ _1 and ^θ1 f_ _1 denote the encoder and decoder, respectively. To prevent the latent codes from drifting arbitrarily and keep them clustered around the Gaussian prior, we include the L2L_2 norm in the loss function. According to Thm. 3.7, the representations learned in this stage correspond to an affine transformation of the ground truth latent variables. In the second stage, we freeze the autoencoder model trained in the first stage and apply a second, inner autoencoder with affine transformations, where ψ2g_ _2 and ^θ2 f_ _2 denote the affine encoder and decoder. Following Xu et al. (2024), we approximate the sparsity constraint in Eq. 4 of Thm. 3.9 by replacing the non-differentiable L0L_0 norm with the differentiable L1L_1 norm except at zero. As the ground truth sparsity level is not known, we use a tunable hyperparameter ϵε to control it (we provide sensitivity analysis of ϵε in App. D.5). Combined with the reconstruction loss, this encourages the model to converge to an appropriate sparsity level. To solve the optimization problem, we use Cooper (Gallego-Posada and Ramirez, 2022), a Lagrangian-based optimization toolkit. The two-stage problems are solved using Adam (Kingma and Ba, 2015) and extra-gradient variant (Gidel et al., 2019). Stage 1: min(θ1,ψ1)⁡1N∑i∈[N]‖i−^θ1(ψ1(i))‖22+ _( _1, _1) 1N _i∈[N] ^i- f_ _1(g_ _1(x^i)) ^2_2+ (5) log(p(ψ1(i);,I)) log(p(g_ _1(x^i);0,I)) Stage 2: min(θ2,ψ2)⁡1N∑i∈[N]‖~i−^θ2(ψ2(~i))‖22 _( _2, _2) 1N _i∈[N] x^i- f_ _2(g_ _2( x^i)) ^2_2 (6) subject to: 1N∑i∈[N]‖ψ2(~i)‖1≤ϵ, subject to: 1N _i∈[N] _ _2( x^i) _1≤ε, where ~i:=ψ1(i) x^i:=g_ _1(x^i). We provide all details in App. C. 5 Experimental results Numerical Experiments Setup. We generate simulations with variations in the underlying causal model, the rank of Gaussian components, and mixing function f. We average results for each setup over 5 random seeds. We consider n=5,10,20,40n=\5,10,20,40\ latent variables Z. For each experiment, we sample a directed acyclic graph D from an Erdös-Rényi (ER)-k model with k∈0,1,2,3k∈\0,1,2,3\, where each ER-k graph contains n⋅kn· k directed edges. In particular, ER-0 corresponds to an empty graph, which implies independent latent variables. Given D, we simulate a linear Gaussian Structural Causal Model where edge weights are sampled uniformly from [−1,−0.2]∪[0.2,1][-1,-0.2]∪[0.2,1], and standard Gaussian noise is added at each node. To simulate pdGMMs, we define the number of mixture components J=5nJ=5n and introduce the ratio of non-degenerate dimensions ρ∈1var,50%,75%ρ∈\1 var,50\%,75\%\, where 1var1 var indicates only one non-degenerate dimension, and 50%50\% or 75%75\% are the proportion of non-degenerate dimensions relative to n. Then, we randomly sample J different ρnρ n-hot vectors to define a set of basis index sets jK_j, which determines the subspace (i.e., the support) where the component has non-degenerate variance. For dimensions not in jK_j (i.e., those with a 0 in the multi-hot vector), we set their value to the corresponding entry in a predefined translation vector 0z^0. To further show the different results for Ass. 3.6 and Ass. 3.8, we vary the translation vector 0=+δ⋅σz^0= μ+δ·σ, where ,σ∈ℝn μ,σ ^n are the mean and standard deviation of Z, and δ∈0,1,2,3δ∈\0,1,2,3\. In addition, we apply a n×n× n rotation matrix on Z, parameterized by a rotation angle θ∈0∘,15∘,30∘,45∘θ∈\0 ,15 ,30 ,45 \, to control the deviations from standard basis. l settings satisfy Ass. 3.6, which means we can always achieve identifiability up to AT, but Ass. 3.8 is only satisfied in the setting with δ=0δ=0 and θ=0θ=0, and thus we expect identifiability up to PS to only be possible there. To simulate the observed variables X, we apply an invertible piecewise affine mixing function f to the latent variables Z. We parameterize f by a multi-layer perceptron (MLP) with m∈3,10,20m∈\3,10,20\ hidden layers, (m−1)(m-1) Leaky-ReLU activation functions (with slope α∼Unif(0.5,1.5)α (0.5,1.5)) and a final affine layer. The number of layers m is a proxy for the non-linearity and complexity of the mixing function; the larger the m, the more complex the transformation from Z to X. n k m ρ δ θ 2R^2 ↑ Stage1 MCC Stage1 MCC ↑ Stage2 MCC ↑ VaDE 5 1 10 50% 0 0 0.93 0.75 0.96 0.45 10 1 10 50% 0 0 0.94 0.56 0.97 0.32 20 1 10 50% 0 0 0.93 0.46 0.92 0.23 40 1 10 50% 0 0 0.94 0.37 0.94 0.25 10 0 10 50% 0 0 0.94 0.51 0.97 0.33 10 1 10 50% 0 0 0.94 0.56 0.97 0.32 10 2 10 50% 0 0 0.93 0.58 0.95 0.31 10 3 10 50% 0 0 0.92 0.56 0.91 0.27 10 1 3 50% 0 0 0.99 0.50 0.96 0.33 10 1 10 50% 0 0 0.94 0.56 0.97 0.32 10 1 20 50% 0 0 0.88 0.50 0.93 0.35 10 1 10 1 var 0 0 0.75 0.46 0.45 0.28 10 1 10 50% 0 0 0.94 0.56 0.97 0.32 10 1 10 75% 0 0 0.95 0.57 0.93 0.31 10 1 10 50% 0 0 0.94 0.56 0.97 0.32 10 1 10 50% 1 0 0.95 0.59 0.88 0.30 10 1 10 50% 2 0 0.96 0.59 0.61 0.27 10 1 10 50% 3 0 0.96 0.58 0.49 0.28 10 1 10 50% 0 0 0.94 0.56 0.97 0.32 10 1 10 50% 0 15 0.93 0.58 0.83 0.32 10 1 10 50% 0 30 0.93 0.57 0.60 0.31 10 1 10 50% 0 45 0.93 0.57 0.62 0.32 Table 1: Results for the numerical experiments. The bold font highlights the parameters that vary in each block. 2R^2(Stage 1) shows identifiability up to AT after Stage 1 MCC(Stage 1) reflects the necessity of sparsity to achieve identifiability up to PS; MCC(Stage 2) presents the identifiability up to PS after Stage 2; MCC(VaDE) reports the results of Kivva et al. (2022). Full results with Std. Dev. are in Tab. 3. Results for Stage 1 (Thm. 3.7). For the encoder ψ1g_ _1 and the decoder ^θ1 f_ _1 in the first stage, we employ 5-layer MLPs configured with [50,100,100,50]×n[50,100,100,50]× n units per layer, where n is the number of latent variables. Each layer uses Leaky-ReLU activations, except for the final one. Batch normalization is applied to control the norm of ψ1()g_ _1(X). To validate the identifiability up to AT presented in Thm. 3.7, the metric we use is the R2R^2 of the linear regression model with Z as explanatory variable and ψ1()g_ _1(X) as response. A high R2R^2 indicates that the ground truth latent variables can be linearly mapped to the learned representation with low error, consistent with our identifiability result. We show the average R2R^2 over five random seeds in Tab. 1. Our method remains robust with large n, dense causal graph k, varying translation vectors 0(δ)z^0(δ) and basis direction θ. Performance decreases slightly only with increased nonlinearity (m) and lower proportions of nondegenerate variables (ρ). This is expected: higher nonlinearity increases the complexity of achieving affine identifiability, while lower ρ allows greater flexibility in how representations are distributed in ℝnR^n, making it more challenging to enforce Gaussianity. However, a high R2R^2 does not imply disentanglement. Following prior work (Khemakhem et al., 2020; Kivva et al., 2022), we also evaluate identifiability up to PS via Mean Correlation Coefficient (MCC), a metric detailed in App. D.1. Across all row groups, we observe that, as expected, Stage 1 alone does not identify the latent variables up to PS accurately, as reflected by the low MCC values reported in the third right column of Tab. 1. This motivates us to move to Stage 2. Results for Stage 2 (Thm. 3.9). For the affine encoder ψ2g_ _2 and the decoder ^θ2 f_ _2 in the second stage, we employ 7-layer MLPs configured with [10,50,50,50,50,10]×n[10,50,50,50,50,10]× n units per layer, where n is the number of latent variables. Batch normalization is applied to control the norm of ψ2()g_ _2(X). Since we generate a sufficient number of basis indices, our representation should be able to have a one-to-one correspondence with the ground truth latent variables. The last two columns of Tab. 1 reports the average MCC scores for our method and VaDE (Kivva et al., 2022). This is a misspecified setting for VaDE, because VaDE requires non-degeneracy of the data, but we report the results nevertheless as a baseline, and show that as expected we outperform it in all settings. The results indicate consistently good performance across a range of n, k, and m. However, the overall performance is influenced by the quality of representations from Stage 1, with a noticeable decrease under a low nondegeneracy ratio ρ. This is expected, since large translation vectors 0(δ)z^0(δ) and non-standard basis (θ) violate Ass. 3.8. Ablations. We provide an evaluation on independent latent variables in App. D.3, showing comparable results, and an empirical complexity analysis in App. D.4, showing that we can scale our method up to n=50n=50 and partially to n=100n=100 variables. This is in line with other CRL works, which mostly focus on 10−2010-20 latent variables. On the other hand, the inference time increases exponentially with the number of latents. In App. D.5 we report a sensitivity analysis on our hyperparameters, in particular the sparsity level, which we set to ϵ=0.01ε=0.01 for the main experiments, and the learning rate, which we set to lr=1e−4lr=1e-4. In App. D.6 we report an ablation on the sparsity constraint, showing its importance for the second stage. We also compare with a related method (Xu et al., 2024) that requires additional information (i.e., knowing the label of the component) and show that with the same information, our method provides comparable results. For completeness, we also provide results in App. D.7 with a setting that is fair to the baseline VaDE (Kivva et al., 2022), where we generate the latent with a non-degenerate GMM, showing that our method outperforms VaDE even in these settings. Similarly to other work in CRL, our theoretical results assume that we know the ground truth n, but empirically our method performs well even if we overestimate it, as shown in App. D.8. As an additional measure of the accuracy of our reconstruction, in App. D.9 we evaluate the Structural Hamming Distance (SHD) of the causal graph that we learn with a standard causal discovery method, PC (Spirtes et al., 2000), on our recovered latent variables, showing that the SHD is only slightly worse than the SHD for the ground truth variables. Settings when the assumptions are violated. In App. E we discuss the intuition, implications and practical applicability of our assumptions, providing also ablations for their violations. In particular, in App. E.1 we show empirically that our method is often still robust in a misspecified setting, i.e., when the variables are not Gaussian (but instead they follow and exponential or Gumbel distribution) or the mixing function is not piecewise affine (but instead is a smooth Leaky-ReLU or sigmoid). On the other hand, we do need all other assumptions. We provide evaluation showing the necessity of Ass. 3.4 (genericity of pdGMMs) in App. E.2, showing that when it does not hold the results are much worse. Similarly, results are much worse when Ass. 3.6 does not hold, so there does not exists a unique shared basis for all Gaussian components. To show a violation of this, we generate the data with a mixture of two different bases: a standard basis and one with rotations, and provide results in App. E.3, showing the necessity of this assumption. Table 1 already shows violations of Ass. 3.8a when δ (the masking value) and θ (the rotation of the standard basis) are non-zero, showcasing a decrease in performance when the assumption does not hold. When we have violations of Ass. 3.8b, we can still achieve block-identifiability, as discussed and shown empirically in App. E.5. Image dataset: Multiple Balls. To show potential applications of our approach in partially observable causal representation learning, we evaluate our method on an image dataset from Ahuja et al. (2022); Xu et al. (2024), which consists of b moving balls rendered in a 2D space, as shown in App. D.10. The latent causal variables correspond to the (x,y)(x,y) positions of each ball, which are modeled as Gaussian. We focus on a fixed position setting, where the balls usually move freely within the frame but occasionally remain stationary at unknown fixed positions. When a ball is stationary, the corresponding latent dimensions are degenerate. We generate datasets with varying numbers of balls: =2,4,6b=2,4,6. The (x,y)(x,y) coordinates of each ball i∈[]i∈[b] are sampled independently from a truncated 2-dimensional normal distribution (i,i)N( μ_i, _i) bounded within the square (0.05,0.95)2(0.05,0.95)^2, where i∼Unif(0.3,0.7)2 μ_i (0.3,0.7)^2, and i=((0.01,0.00)(0.00,0.01)) _i=((0.01,0.00)(0.00,0.01)). To avoid the high ratio of occlusion, we use rejection sampling to sample i μ_i to ensure the Euclidean distance between any pair of i μ_i values is at least 0.20.2. Each ball has a probability p=0.1p=0.1 to be at a fixed position i μ_i. Figure 2: R2R^2 of the linear regression between ball positions b1,…,b4b_1,…,b_4 and learned representations z^1,…,z^8 z_1,…, z_8. We use two variables for each (x,y)(x,y) position. Before we train the model described in Sec. 4, we pre-train a CNN model to reduce the input image dimension from 64×6464× 64 to a vector of size 16×116b× 1. More details about the network architecture are provided in App. D.10. Instead of a fixed-size training dataset, we generate images online until convergence. Since the variables describing the x and y position of the same ball are always degenerate together, they cannot be disentangled in our method (as sufficient variability in Ass. 3.8 does not hold). To evaluate the performance, we perform regression from each representation separately to the corresponding ground-truth position pairs, as illustrated in Fig. 2. The results show that the position of each ball can be recovered through pairs of representations, which is consistent with our theoretical results. Additional results are in App. D.10. 6 RELATED WORK Our work is inspired by previous work in nonlinear independent component analysis (ICA), causal representation learning (CRL) and sparsity-based methods. ICA (Comon, 1994; Hyvärinen, 2013) is a seminal approach that identifies latent variables under the assumption that the latent variables are independent and that the mixing function is linear. In the non-linear ICA setting, Hyvarinen and Morioka (2016, 2017) reformulate this standard ICA independence assumption as conditional independence given observed auxiliary variables. Khemakhem et al. (2020) show the identifiability of the VAE model with a conditionally independent prior. Willetts and Paige (2021) show that the auxilliary variables can be unobservable and learned; however, this still requires the conditional independence assumption. Causal representation learning (CRL) extends (nonlinear) ICA to the case in which latent variables have causal relations between them and hence arbitrary dependencies. Our approach fits in this line of work. Many CRL works consider additional data or information to identify causal variables, e.g., interventions (Brehmer et al., 2022; Lippe et al., 2022; Varici et al., 2023; von Kügelgen et al., 2023; Buchholz et al., 2023; Zhang et al., 2023; Ahuja et al., 2023a), actions or temporal structure (Lippe et al., 2023; Lachapelle et al., 2022, 2024), multi-view settings (von Kügelgen et al., 2021; Ahuja et al., 2023b; Yao et al., 2024; Morioka and Hyvärinen, 2024), multiple environments (Liu et al., 2024), grouping of observations by sparsity patterns (Xu et al., 2024), hierarchical structures (Kong et al., 2023, 2024) or conditions on causal graphs (Zhang et al., 2024; Dong et al., 2024; Song et al., 2024). In our case, we do not assume any additional data or information besides the observations, but instead consider parametric assumptions on the latent variables and mixing function. Similarly to us, Kivva et al. (2022) studies the identifiability of non-degenerate GMM latents with piecewise affine mixing. For PS identifiability, they require that all pairs of latent variables must be conditionally independent given an auxiliary variable. Other works consider restrictions on the mixing function, e.g., Squires et al. (2023); Bing et al. (2024) assume a linear mixing function, Ahuja et al. (2023a) assumes a finite-degree polynomial and either independent supports or perfect interventions, and Lachapelle et al. (2023a) assumes an additive mixing function. Our results do not require conditional independence, independent support or interventions and allow mixing functions composed of an infinite number of affine pieces, allowing the potential possibility to approximate any nonlinear functions up to an arbitrary precision. Our focus on low-rank representations is also inspired by prior work that leverages sparsity in (causal) representation learning, e.g., structural sparsity in Jacobians of nonlinear mixing functions (Zheng et al., 2022; Zheng and Zhang, 2023), sparsity constraints on affine mixing (Ng et al., 2023), sparsity on causal graphs (Zhang et al., 2024; Song et al., 2024), sparse decoders (Moran et al., 2022) and sparsity task-specific predictors (Lachapelle et al., 2023b; Fumero et al., 2023). For time-series data, Lachapelle et al. (2022, 2024) propose sparse actions and sparse temporal dependencies, while Li et al. (2025) introduce latent process sparsity, referring to sparse connections between latents both within and across time. Differently from these approaches, which impose sparsity on the model structure or latent interactions, we are enforcing sparsity of the representation itself. This aligns with prior work (Tonolini et al., 2020), as well as classical methods sparse component analysis (Gribonval and Lesage, 2006), and sparse dictionary learning (Mairal et al., 2009). Most works show empirical success on benchmark tasks and do not provide guarantees for identifiability, which is our main focus. Moran et al. (2022) focuses on independent latent variables, while Hu and Huang (2023) studies the identifiability of dictionary learning, but they assume affine mixing. Instead, we focus on piecewise-affine mixing for dependent latent variables. 7 CONCLUSIONS AND LIMITATIONS In this work, we focus on identifying pdGMM latent variables with piecewise affine mixing functions. We first show that the entire pdGMM can be identified via an open set. This result is a critical step for a series of increasingly stronger identifiability results, ultimately proving identifiability up to permutation and element-wise transformations, i.e. completely disentangled representations. While our experiments validate our results, there are still several limitations. As discussed in App. E, some of our assumptions might not hold in some settings, potentially leading to partial or no theoretical guarantees. In particular, the Gaussian assumption for mixture components may not hold, so extending our approach to other parametric models is an exciting avenue for future research. Finally, although our method encourages Gaussianity on representations, ensuring this in practice remains challenging. Acknowledgements We thank the anonymous reviewers for their helpful and constructive comments. The research of DX and SM was supported by the Air Force Office of Scientific Research under award number FA8655-22-1-7155. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force. We also thank SURF for the support in using the Dutch National Supercomputer Snellius. References K. Ahuja, J. S. Hartford, and Y. Bengio (2022) Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems 35, p. 15516–15528. Cited by: Appendix C, Figure 10, §5. K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio (2023a) Interventional causal representation learning. In International Conference on Machine Learning, p. 372–407. Cited by: §1, §6, §6. K. Ahuja, A. Mansouri, and Y. Wang (2023b) Multi-domain causal representation learning via weak distributional invariances. In Causal Representation Learning Workshop at NeurIPS 2023, Cited by: §1, §6. Y. Bengio, A. Courville, and P. Vincent (2013) Representation learning: a review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35 (8), p. 1798–1828. Cited by: §1. S. Bing, U. Ninad, J. Wahl, and J. Runge (2024) Identifying linearly-mixed causal representations from multi-node interventions. In Causal Learning and Reasoning, p. 843–867. Cited by: §6. J. Bona-Pellissier, F. Bachoc, and F. Malgouyres (2023) Parameter identifiability of a deep feedforward relu neural network. Machine Learning 112 (11), p. 4431–4493. Cited by: §B.2.1, §B.2.1, §B.2.1, Definition B.10, Definition B.11, Definition B.9. J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen (2022) Weakly supervised causal representation learning. Advances in Neural Information Processing Systems 35, p. 38319–38331. Cited by: §1, §6. S. Buchanan, D. Pai, P. Wang, and Y. Ma (2025) Learning deep representations of data distributions. Online. Note: https://ma-lab-berkeley.github.io/deep-representation-learning-book/. Cited by: §1. S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar (2023) Learning linear causal representations from interventions under general nonlinear mixing. In Advances in Neural Information Processing Systems, Cited by: §1, §6. R. Caron and T. Traynor (2005) The zero set of a polynomial. WSMR Report, p. 05–02. Cited by: §B.1, §B.1, §B.1, §B.1. P. Comon (1994) Independent component analysis, a new concept?. Signal Processing. Cited by: §1, §2, §6. H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey (2024) Sparse autoencoders find highly interpretable features in language models. International Conference on Learning Representations. Cited by: §E.3, §2. X. Dong, I. Ng, B. Huang, Y. Sun, S. Jin, R. Legaspi, P. Spirtes, and K. Zhang (2024) On the parameter identifiability of partially observed linear causal models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §1, §6. M. Fumero, F. Wenzel, L. Zancato, A. Achille, E. Rodolà, S. Soatto, B. Schölkopf, and F. Locatello (2023) Leveraging sparse and shared feature activations for disentangled representation learning. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: §6. J. Gallego-Posada and J. Ramirez (2022) Cooper: a toolkit for lagrangian-based constrained optimization. Cited by: §4. L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu (2025) Scaling and evaluating sparse autoencoders. International Conference on Learning Representations. Cited by: §E.3, §2. G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien (2019) A variational inequality perspective on generative adversarial networks. International Conference on Learning Representations. Cited by: §4. R. Gribonval and S. Lesage (2006) A survey of sparse component analysis for blind source separation: principles, perspectives, and new challenges. In ESANN’06 proceedings-14th European Symposium on Artificial Neural Networks, p. 323–330. Cited by: §6. J. Hu and K. Huang (2023) Global identifiability of $ _1$-based dictionary learning via matrix volume optimization. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: §6. A. Hyvärinen, I. Khemakhem, and R. Monti (2024) Identifiability of latent-variable and structural-equation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics 76 (1), p. 1–33. Cited by: §3.2. A. Hyvarinen and H. Morioka (2016) Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems 29. Cited by: §1, §6. A. Hyvarinen and H. Morioka (2017) Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, p. 460–469. Cited by: §1, §6. A. Hyvärinen (2013) Independent component analysis: recent advances. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 371 (1984), p. 20110534. Cited by: §1, §6. I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen (2020) Variational autoencoders and nonlinear ica: a unifying framework. In International Conference on Artificial Intelligence and Statistics, p. 2207–2217. Cited by: §2, §5, §6. D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization.. In International Conference on Learning Representations, Y. Bengio and Y. LeCun (Eds.), Cited by: §4. B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam (2022) Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems 35, p. 15687–15701. Cited by: §B.1, §B.1, §B.1, §B.1, §B.2.1, Theorem B.12, Theorem B.2, §D.6, Table 8, §1, §2, §2, §3.1, §5, §5, §5, Table 1, §6. L. Kong, G. Chen, B. Huang, E. P. Xing, Y. Chi, and K. Zhang (2024) Learning discrete concepts in latent hierarchical models. Advances in Neural Information Processing Systems. Cited by: §1, §6. L. Kong, B. Huang, F. Xie, E. Xing, Y. Chi, and K. Zhang (2023) Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems 36, p. 2010–2032. Cited by: §1, §6. S. Lachapelle, D. Mahajan, I. Mitliagkas, and S. Lacoste-Julien (2023a) Additive decoders for latent variables identification and cartesian-product extrapolation. In Advances in Neural Information Processing Systems, Cited by: §6. S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y. Bengio, S. Lacoste-Julien, and Q. Bertrand (2023b) Synergies between disentanglement and sparsity: generalization and identifiability in multi-task learning. In International Conference on Machine Learning, p. 18171–18206. Cited by: §B.2.3, Lemma B.20, §1, §2, §2, §3.2, §6. S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien (2024) Nonparametric partial disentanglement via mechanism sparsity: sparse actions, interventions and sparse temporal dependencies. External Links: 2401.04890 Cited by: §1, §6, §6. S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julien (2022) Disentanglement via mechanism sparsity regularization: a new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, p. 428–484. Cited by: Appendix C, §1, §6, §6. Z. Li, Y. Shen, K. Zheng, R. Cai, X. Song, M. Gong, G. Chen, and K. Zhang (2025) On the identification of temporal causal representation with instantaneous dependence. In The Thirteenth International Conference on Learning Representations, Cited by: §6. P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves (2023) BISCUIT: causal representation learning from binary interactions. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence. Cited by: §1, §6. P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves (2022) Citris: causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, p. 13557–13603. Cited by: §1, §6. Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. van den Hengel, K. Zhang, and J. Q. Shi (2024) Identifiable latent polynomial causal models through the lens of change. In The Twelfth International Conference on Learning Representations, Cited by: §6. J. Mairal, F. Bach, J. Ponce, and G. Sapiro (2009) Online dictionary learning for sparse coding. In Proceedings of the 26th annual international conference on machine learning, p. 689–696. Cited by: §6. S. Marks, C. Rager, E. J. Michaud, Y. Belinkov, D. Bau, and A. Mueller (2025) Sparse feature circuits: discovering and editing interpretable causal graphs in language models. International Conference on Learning Representations. Cited by: §E.3, §2. B. Mityagin (2020) The zero set of a real analytic function. Mathematical Notes 107 (3), p. 529–530. Cited by: §E.2, §E.2. G. Moran, D. Sridhar, Y. Wang, and D. Blei (2022) Identifiable deep generative models via sparse decoding. Transactions on machine learning research. Cited by: §6. H. Morioka and A. Hyvärinen (2024) Causal representation learning made identifiable by grouping of observational variables. International Conference on Machine Learning. Cited by: §1, §6. J. R. Munkres (2000) Topology. 2 edition, Prentice Hall, Inc.. Cited by: §B.2.1. I. Ng, Y. Zheng, X. Dong, and K. Zhang (2023) On the identifiability of sparse ica without assuming non-gaussianity. Advances in Neural Information Processing Systems 36. Cited by: §6. B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio (2021) Toward causal representation learning. Proceedings of the IEEE 109 (5), p. 612–634. Cited by: §1. X. Song, Z. Li, G. Chen, Y. Zheng, Y. Fan, X. Dong, and K. Zhang (2024) Causal temporal representation learning with nonstationary sparse transition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §1, §6, §6. P. Spirtes, C. N. Glymour, and R. Scheines (2000) Causation, prediction, and search. MIT press. Cited by: §D.9, §5. C. Squires, A. Seigal, S. Bhate, and C. Uhler (2023) Linear causal disentanglement via interventions. In 40th International Conference on Machine Learning, Cited by: §6. F. Tonolini, B. S. Jensen, and R. Murray-Smith (2020) Variational sparse coding. In Uncertainty in Artificial Intelligence, p. 690–700. Cited by: §6. B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer (2023) Score-based causal representation learning with interventions. Causal Representation Learning Workshop at NeurIPS 2023. Cited by: §1, §6. J. von Kügelgen, M. Besserve, W. Liang, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf (2023) Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing 36, Cited by: §1, §6. J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello (2021) Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems 34, p. 16451–16467. Cited by: §1, §6. M. Willetts and B. Paige (2021) I don’t need u: identifiable non-linear ica without side information. arXiv preprint arXiv:2106.05238. Cited by: §6. D. Xu, D. Yao, S. Lachapelle, P. Taslakian, J. von Kügelgen, F. Locatello, and S. Magliacane (2024) A sparsity principle for partially observable causal representation learning. International Conference on Machine Learning. Cited by: §B.2.1, §B.2.1, §B.2.1, §B.2.1, §B.2.3, Theorem B.17, Lemma B.18, Lemma B.20, §D.5, §D.6, Table 8, §E.3, §1, §1, §2, §3.2, §3.2, §4, §5, §5, §6. S. J. Yakowitz and J. D. Spragins (1968) On the identifiability of finite mixtures. The Annals of Mathematical Statistics 39 (1), p. 209–214. Cited by: Theorem B.1, Appendix B, §3.1, §3.1. D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Locatello (2024) Multi-view causal representation learning with partial observability. International Conference on Learning Representations. Cited by: §1, §6. J. Zhang, K. Greenewald, C. Squires, A. Srivastava, K. Shanmugam, and C. Uhler (2023) Identifiability guarantees for causal disentanglement from soft interventions. In Advances in Neural Information Processing Systems, Cited by: §1, §6. K. Zhang, S. Xie, I. Ng, and Y. Zheng (2024) Causal representation learning from multiple distributions: a general setting. International Conference on Machine Learning. Cited by: §1, §6, §6. X. Zheng, B. Aragam, P. K. Ravikumar, and E. P. Xing (2018) Dags with no tears: continuous optimization for structure learning. Advances in neural information processing systems 31. Cited by: Appendix C. Y. Zheng, I. Ng, and K. Zhang (2022) On the identifiability of nonlinear ica: sparsity and beyond. Advances in Neural Information Processing Systems 35, p. 16411–16422. Cited by: §6. Y. Zheng and K. Zhang (2023) Generalizing nonlinear ica beyond structural sparsity. Advances in Neural Information Processing Systems 36, p. 13326–13355. Cited by: §6. Identifiability of potentially degenerate Gaussian Mixture Models with piecewise affine mixing: Supplementary Materials Appendix A Notation Throughout this paper, we adopt the following conventions for mathematical notation: We use lowercase italic letters such as x to denote scalars, bold lowercase letters such as x to denote deterministic vectors, bold uppercase letters such as X to denote random vectors, and uppercase italic letters such as X to denote matrices, and calligraphic uppercase letters such as X to denote sets (e.g., a set of inputs or a domain). We list all specific notations in the following table: Symbol Description Z Causal latent variables X Observations f Mixing function n Latent space dimension d Observation space dimension J Number of components in mixture models Z Support of pdGMM latent jZ_j Support of j-th component of pdGMM latent X Support of observation jK_j Basis index for j-th component 0z^0 Translation vector Appendix B Proofs Our work focuses on Gaussian Mixture Models (GMMs), which are frequently used in machine learning models and possess a number of nice theoretical properties like closure under affine transformation etc. Before presenting our main results, we recall one of the fundamental properties of GMMs, stated in Theorem˜B.1, which will serve as a recurring tool in our derivations. This theorem builds a one-to-one correspondence-commonly referred to as identifiability of distributions between the probability measure and the parameterization of a GMM. Notably, this property holds even when the mixture includes potentially degenerate components (i.e., components with singular covariance matrices), which we refer to as potentially degenerate GMMs (pdGMMs). To proceed, we begin by reporting again the formal definition of pdGMM in a reduced form, i.e., with distinct components. Note that N(,)N( μ, ) stands for the Gaussian probability measure associated to parameters , μ, . See 2.2 We now report a classic result on the identifiability of Gaussian Mixture Models (GMMs) on the whole domain by Yakowitz and Spragins (1968). While this result holds also for pdGMMs, it only proves identifiability when two GMMs are equal in distribution on the whole domain, while in this paper we will focus on identifiability from an open subset of the domain. Theorem B.1 (Identifiable Gaussian Mixture Models - Proposition 2 in Yakowitz and Spragins (1968)). Consider a pair of finite GMMs (in reduced form) in ℝnR^n, P=∑j=1JλjN(j,j)andP′=∑j=1J′λj′N(j′,j′).P= _j=1^J _jN( μ_j, _j) P = _j=1^J _j N( μ_j , _j )\,. (7) we have that P=P′P=P if and only if the number of mixture components are the same, i.e., J=J′J=J , and for some permutation π for all j∈[J]j∈[J] the mixture components have the same parameters, i.e., λj=λπ(j) _j= _π(j) and (j,j)=(π(j)′,π(j)′)( μ_j, _j)=( μ_π(j) , _π(j) ). B.1 Identifiability of pdGMMs from an open subset In this section, we address the following question: Can a potentially degenerate Gaussian Mixture Model (pdGMM) be identified using only information from an open subset of its domain? Inspired by Kivva et al. (2022), which show that a non-degenerate GMM (so a GMM with |j|>0| _j|>0 for all j) can be uniquely identified from its restriction to a single open subset of the domain (Theorem˜B.2), we show that a similar identifiability result also holds for pdGMMs (Theorem˜3.2). However, the extension from GMMs to pdGMMs is nontrivial. The main challenge arises from the fact that the proof technique in Kivva et al. (2022) critically relies on the analyticity of the probability density function (w.r.t. the Lebesgue measure) of a non-degenerate Gaussian mixture. However, mixtures of degenerate Gaussians do not have a density function, which means this strategy cannot be applied. This necessitates the development of a new proof strategy to overcome the lack of density function. As a first step, we introduce a formal definition of equivalence in distribution over a constrained subdomain. See 3.1 We then recall the results from Kivva et al. (2022), which show that for non-degenerate GMMs, we can identify them via an open subdomain. Theorem B.2 (Identifiability of non-degenerate GMMs from open set (Kivva et al., 2022)). Consider a pair of finite non-degenerate GMMs in ℝnR^n in reduced form, ∼∑j=1JλjN(j,j)and′∼∑j=1J′λj′N(j′,j′),X _j=1^J _jN( μ_j, _j) _j=1^J _j N( μ_j , _j ), (8) where |det(j)|>0,∀j∈[J]| ( _j)|>0,∀ j∈[J] and |det(j′)|>0,∀j∈[J′]| ( _j)|>0,∀ j∈[J ]. If =′X d=X on E, where E⊆ℝnE ^n is an open set, then, =′X d=X . Next, we focus on extending these results to pdGMMs, which we will formalize in the following theorem. See 3.2 As previously mentioned, proving the above theorem is challenging because, in the degenerate case, the probability density function no longer exists, making the techniques used in Kivva et al. (2022) inapplicable. To overcome this difficulty, we first provide a weaker result, Theorem B.3, which is requiring all supports intersect at one point at least, as a middle step to get Theorem˜3.2. Theorem B.3 (Identifiability of pdGMMs from open set with common intersection). Under the same conditions as Theorem˜3.2, and further assuming that there exists a point 0z_0, such that 0∈∩j∈[J]jz_0∈ _j∈[J]X_j, then, the same results stated in Theorem˜3.2 hold. To prove Theorem B.3, we build upon four key technical lemmas that we introduce and prove below. We now provide a brief overview of our proof strategy. The core idea is to project a pdGMM in ℝnR^n into a sequence of lower-dimensional spaces ℝkR^k, where k∈[n]k∈[n]. Intuitively, the projection serves to “resolve” the degeneracy by embedding each low-rank component into a subspace where it appears non-degenerate, thus enabling the application of classical identifiability arguments in Theorem B.2. To make this strategy work, the projections must satisfy two key properties. First, to ensure that no critical information is lost during projection (intuitively, the intrinsic structure of the component is retained under projection), the rank of the covariance matrix for the targeted Gaussian component must be preserved. Second, a single projection is insufficient to fully recover the parameters of a pdGMM; instead, we require a sufficiently rich set of rank-preserving projections to capture all distinguishable parameterizations. The existence of such a family of rank-preserving mappings is guaranteed by Lemma B.4. Lemma B.4. Suppose a positive semi-definite and symmetric matrix ∈ℝn×n ^n× n has rank k≤nk≤ n. Let ℬ:=A∈ℝn×m:rank(ATA)=k, :=\A ^n× m:rank(A^T A)=k\, (9) where k≤m≤nk≤ m≤ n. Then, we have that i) the Lebesgue measure of its complement ℬcB^c, denoted by ℒ(ℬc)L(B^c) is zero, and i) ℬB is open. Proof. The fact that rank(AAT)=krank(A A^T)=k means there exists at least one k×k× k submatrix with full rank k. Consider the sets I,J:=A∈ℝn×m:det((ATA)I,J)=0 _I,J:=\A ^n× m: ((A^T A)_I,J)=0\ (10) which are the sets of matrices such that AjATA _jA^T has a submatrix with index sets I,J⊆[m]I,J [m] with |I|=|J||I|=|J| that is full rank. Then the complement of ℬB, denoted as ℬcB^c, can be rewritten as ℬc=∩I,J⊆[m]:|I|=|J|=kI,J. ^c= _\I,J [m]:|I|=|J|=k\A_I,J. (11) This implies ℒ(ℬc)≤ℒ(I,J)L(B^c) (A_I,J), ∀(I,J)∈I,J⊆[n]:|I|=|J|=k∀(I,J)∈\I,J [n]:|I|=|J|=k\. Therefore, if we can find one I,JA_I,J which is measure zero, ℬcB^c has Lebesgue measure zero as well. Note that det((ATA)I,J) ((A^T A)_I,J) is a polynomial in Ai,jA_i,j, where i∈I,j∈Ji∈ I,j∈ J are individual row and column indices. This is because det is a polynomial and (ATA)I,J(A^T A)_I,J is a polynomial and thus composition of polynomials is a polynomial. Additionally, I,JA_I,J is the set of the roots of this polynomial. By Caron and Traynor (2005), a polynomial from ℝnR^n to ℝR is either zero everywhere or non-zero almost everywhere. In our context, it means if the polynomial det((ATA)I,J) ((A^T A)_I,J) is not zero everywhere, then it is nonzero almost everywhere, i.e. its roots I,JA_I,J has zero Lebesgue measure in ℝn×mR^n× m. Thus, all we need to show is that the polynomial det((ATA)I,J) ((A^T A)_I,J) is not always zero. In other words, we must find a A0∈ℝn×mA_0 ^n× m, such that for one pair I,JI,J, det((ATA)I,J)≠0 ((A^T A)_I,J)≠ 0. Then in the following, we show how to construct such a A0A_0. Since rank()=k rank( )=k, there exists an invertible k×k× k submatrix I,J _I,J. Let ~:=[1⋯m]∈ℝn×m I:=[e_1\ ·s\ e_m] ^n× m where the vectors ke_k are the elements of the standard basis of ℝnR^n (one-hot vector). Notice that (~T~)I,J=(~T)I,⋅~⋅,J=(~⋅,I)T~⋅,J=I,J. ( I^T I)_I,J=( I^T)_I,· I_·,J=( I_·,I)^T I_·,J= _I,J\,. (12) So we found index sets I,J⊆[m]I,J [m] with |I|=|J|=k|I|=|J|=k and a matrix A0:=~∈ℝn×mA_0:= I ^n× m such that det((A0TA0)I,J)=det((~T~)I,J)=det(I,J)≠0, ((A_0^T A_0)_I,J)= (( I^T I)_I,J)= ( _I,J)≠ 0\,, as desired. Hence, the polynomial function p(A):=det((ATA)I,J)p(A):= ((A^T A)_I,J) is not everywhere zero and thus, by Caron and Traynor (2005), it is nonzero almost everywhere. In other words, I,JA_I,J has zero Lebesgue measure. We can conclude that ℬcB^c has zero Lebesgue measure. We now show that ℬB is open. Take the complement set on Equation˜11, we have ℬ=∪I,J⊆[n]:|I|=|J|=kI,Jc, = _\I,J [n]:|I|=|J|=k\A^c_I,J, (13) I,Jc=A∈ℝn×m:det((ATA)[I,J])≠0. ^c_I,J=\A ^n× m: ((A^T A)_[I,J])≠ 0\. (14) Since det((ATA)[I,J]) ((A^T A)_[I,J]) is a continuous function and I,JcA^c_I,J is the preimage of open set ℝ\0R \0\, I,JcA^c_I,J is open as well. As the union of finite open sets, we can conclude ℬB is open. ∎ The following lemma will be useful many times in what follows. For a probability measure μ and a function f, (μ)f(μ) designates the pushforward of μ under f. The pushforward measure is defined as (μ)(E):=μ(−1(E))f(μ)(E):=μ(f^-1(E)). We recall that as defined in Def. 3.1, μE _E is a restriction of μ to E. Lemma B.5. Let μ:ℱ(n)→ℝμ:F^(n) be a measure on ℱ(n)F^(n), the Borel sigma-algebra of ℝnR^n. Let :ℝn→ℝmf:R^n ^m be an injective function for any set E∈ℱ(n)E ^(n), we have (μE)=(μ)(E)f( _E)=f(μ)_f(E). Proof. Take an arbitrary set B∈ℱ(m)B ^(m), the Borel sigma-algebra over ℝmR^m. We have (μE)(B)=μE(−1(B))=μ(−1(B)∩E) ( _E)(B)= _E(f^-1(B))=μ(f^-1(B)∩ E) (15) Since f is injective, we have that E=−1((E))E=f^-1(f(E)). This means (μE)(B) ( _E)(B) =μ(−1(B)∩−1((E))) =μ(f^-1(B) ^-1(f(E))) (16) =μ(−1(B∩(E))) =μ(f^-1(B (E))) (17) =(μ)(B∩(E)) =f(μ)(B (E)) (18) =(μ)(E)(B), =f(μ)_f(E)(B)\,, (19) where the second equality uses the general fact that taking the preimage preserves intersections. ∎ The next lemma is a slightly different version from the one showed above where the function is required to be injective only on the support of the measure μ. Lemma B.6. Let μ:ℱ(n)→ℝμ:F^(n) be a measure on ℱ(n)F^(n), the Borel sigma-algebra of ℝnR^n. Let ⊆ℝnX ^n be the support of μ, and let :ℝn→ℝmf:R^n ^m be such that its restriction to X is injective, then for any set E∈ℱ(n)E ^(n), we have (μE)=(μ)(E∩)f( _E)=f(μ)_f(E ). Proof. Take an arbitrary set B∈ℱ(m)B ^(m), the Borel sigma-algebra over ℝmR^m. We have (μE)(B) ( _E)(B) =μE(−1(B)) = _E(f^-1(B)) (20) =μ(−1(B)∩E) =μ(f^-1(B)∩ E) (21) =μ(−1(B)∩E∩)+μ(−1(B)∩E∩c) =μ(f^-1(B)∩ E )+μ(f^-1(B)∩ E ^c) (22) =μ(−1(B)∩(E∩)), =μ(f^-1(B)∩(E ))\,, (23) where the last equality used the fact that μ(c)=0μ(X^c)=0. Since f is injective on X and (E∩)⊆(E ) , we have that (E∩)=−1((E∩))∩(E )=f^-1(f(E )) . This means (μE)(B) ( _E)(B) =μ(−1(B)∩−1((E∩))∩) =μ(f^-1(B) ^-1(f(E )) ) (24) =μ(−1(B)∩−1((E∩))∩)+μ(−1(B)∩−1((E∩))∩c)⏟=0 =μ(f^-1(B) ^-1(f(E )) )+ μ(f^-1(B) ^-1(f(E )) ^c)_=0 (25) =μ(−1(B)∩−1((E∩))) =μ(f^-1(B) ^-1(f(E ))) (26) =μ(−1(B∩(E∩))) =μ(f^-1(B (E ))) (27) =(μ)(B∩(E∩)) =f(μ)(B (E )) (28) =(μ)(E∩)(B), =f(μ)_f(E )(B)\,, (29) where the second equality uses the fact that μ(c)=0μ(X^c)=0 and fourth equality uses the general fact that taking the preimage preserves intersections. ∎ The previous two lemmas show that the injectivity can ensure the exchange order of pushforward and restricted measures. In the following lemma, we show that the rank-preserving projections are injective on the domain we are interested in, and hence we can apply these lemmas. Lemma B.7. For A∈ℝn×mA ^n× m, where m≤nm≤ n, if rank()=rank(ATA)=k≤mrank( )= rank(A^T A)=k≤ m, then, ATA^T is injective on col()+col( )+ μ, ∀∈ℝn∀ μ ^n. Proof. Since rank(ATA)≤rank(AT)rank(A^T A)≤ rank(A^T ), we get k≤rank(AT)≤mk≤ rank(A^T )≤ m. Now we consider the null space for ATA^T and . First, we know dim(null(AT))∈[n−m,n−k](null(A^T ))∈[n-m,n-k] and dim(null())=n−k(null( ))=n-k. Second, ∀x∈ℝn∀ x ^n, if x=0 x=0, then ATx=0A^T x=0. This means null()⊆null(AT)null( ) null(A^T ). Combined with the first point, we can get dim(null(AT))=n−k(null(A^T ))=n-k. Since both of them are affine subspaces with the same dimension and one is the subset of the other, we get null()=null(AT)null( )=null(A^T ). This means ∀y∈ℝn, if ATy=0⇒y=0 ∀ y ^n, if A^T y=0 y=0 (30) Denote x:=y∈col()x:= y∈ col( ), then we have ∀x∈col(), if ATx=0⇒x=0. ∀ x∈ col( ), if A^Tx=0 x=0. (31) Now consider two distinct points x1,x2∈col()+x_1,x_2∈ col( )+ μ. As x1−x2∈col()x_1-x_2∈ col( ) but not equal to 0, by Equation˜31, we have AT(x1−x2)≠0⇒ATx1≠ATx2, A^T(x_1-x_2)≠ 0 A^Tx_1≠ A^Tx_2, (32) which means ATA^T is injective on col()+col( )+ μ. ∎ In the following lemma, we show that only the components with well-defined density can be equal to each other, and the same for the components without density. Intuitively, we can group the Gaussian components by rank and analyze each group separately. To discuss the existence of well-defined density, we recall the Radon-Nikodym theorem: informally, if measure μ is absolutely continuous with respect to measure ℒL, denoted as μ≪ℒμ , then there exists a density function of μ w.r.t. ℒL. In other words, μ≪ℒμ means for any set A, if ℒ(A)=0L(A)=0, then μ(A)=0μ(A)=0. Lemma B.8. Let μ, μ~ μ, λ, λ~ λ be nonnegative finite measures on a common measure space such that μ+λ=μ~+λ~. μ+λ= μ+ λ. (33) Let Λ and Λ~ be the supports of λ and λ~ λ, respectively. If there is a third measure ℒL over the same space such that (i) μ,μ~≪ℒμ, μ , and (i) ℒ(Λ)=ℒ(Λ~)=0L( )=L( )=0, then μ=μ~μ= μ and λ=λ~λ= λ. Proof. First, we rewrite the equation equivalently as: μ−μ~=λ~−λ. μ- μ= λ-λ. (34) Let B be an arbitrary measurable set. We have (μ−μ~)(B) (μ- μ)(B) =(λ~−λ)(B) =( λ-λ)(B) (35) =(λ~−λ)(B∩(Λ∪Λ~))+(λ~−λ)(B∩(Λ∪Λ~)c) =( λ-λ)(B∩( ∪ ))+( λ-λ)(B∩( ∪ )^c) (36) Since B∩(Λ∪Λ~)cB∩( ∪ )^c has no overlap with the support of neither λ nor λ~ λ, we have (λ~−λ)(B∩(Λ∪Λ~)c)=0( λ-λ)(B∩( ∪ )^c)=0. This means (μ−μ~)(B) (μ- μ)(B) =(λ~−λ)(B∩(Λ∪Λ~)) =( λ-λ)(B∩( ∪ )) (37) =(μ−μ~)(B∩(Λ∪Λ~)) =(μ- μ)(B∩( ∪ )) (38) =μ(B∩(Λ∪Λ~))−μ~(B∩(Λ∪Λ~)), =μ(B∩( ∪ ))- μ(B∩( ∪ ))\,, (39) where the second equality holds by Equation˜34. Note that ℒ(B∩(Λ∪Λ~))≤ℒ(Λ∪Λ~))≤ℒ(Λ)+ℒ(Λ~)=0,L(B∩( ∪ )) ( ∪ )) ( )+L( )=0\,, which means ℒ(B∩(Λ∪Λ~))=0L(B∩( ∪ ))=0. Since μ,μ~≪ℒμ, μ , we must have μ(B∩(Λ∪Λ~))=μ~(B∩(Λ∪Λ~))=0μ(B∩( ∪ ))= μ(B∩( ∪ ))=0. This combined with Equation˜36 implies (μ−μ~)(B)=0(μ- μ)(B)=0, i.e. μ(B)=μ~(B)μ(B)= μ(B). This is true for any B, thus μ=μ~μ= μ. Plugging this last fact into Equation˜34 yields, λ~=λ λ=λ. ∎ Now we are ready to prove our intermediate result, Theorem˜B.3, that proves the identifiability of pdGMMs from an open set with a common intersection. See B.3 Proof. Let kj:=rank(j)k_j:=rank( _j) and kj′:=rank(j′)k _j:=rank( _j). W.l.o.g., suppose that the components in each of the two pdGMM are ordered by rank in a decreasing fashion, i.e., k1≥k2≥…≥kJk_1≥ k_2≥...≥ k_J, k1′≥k2′≥…≥kJ′k _1≥ k _2≥...≥ k _J and that the rank of the first component of the first pdGMM X is larger or equal to the rank of the first component of the second pdGMM ′X , i.e., k1≥k1′k_1≥ k _1. We define k:=j∈[J]:kj≤k S_k:=\j∈[J]:k_j≤ k\ (40) as the index set of all components in the first pdGMM X, which have rank lower than or equal to k. Similarly we define the same index set for ′X as k′:=j∈[J′]:kj′≤k. S _k:=\j∈[J ]:k _j≤ k\. (41) We define the set of linear mappings that preserve the rank for the component j of ′X as ℬk,j:=A∈ℝn×k∣rank(ATjA)=kj. _k,j:=\A ^n× k rank(A^T _jA)=k_j\. (42) We now define the intersection of all the rank-preserving matrices for the components that have rank lower than or equal to k as ℬk:=∩j∈kℬk,j. _k:= _\j∈S_k\B_k,j. (43) Similarly for ′X , we define ℬk′:=∩j∈k′ℬk,j′ with ℬk,j′:=A∈ℝn×k∣rank(ATj′A)=kj′. _k:= _\j∈S _k\B _k,j with B _k,j:=\A ^n× k rank(A^T _jA)=k _j\. (44) ℬkB_k and ℬk′B _k involve the linear mappings that preserve the ranks for the components with rank no larger than k. Furthermore, to ensure the components in X are still different from each other after mapping, we first consider a pair of components in X and define k,i,j:=A∈ℝn×k∣AT(i−j)≠ or AT(i−j)A≠, _k,i,j:=\A ^n× k A^T( μ_i- μ_j) 0 or A^T( _i- _j)A 0\, (45) and then go through all possible pairs and define k:=∩i≠j∈k:ki,kj=kk,i,j. _k:= _\i≠ j∈S_k:k_i,k_j=k\C_k,i,j. (46) Similarly for ′X , we define k′ _k :=∩i≠j∈k′:ki′,kj′=kk,i,j′ := _\i≠ j∈S _k:k _i,k _j=k\C _k,i,j (47) with k,i,j′ with C _k,i,j :=A∈ℝn×k∣AT(i′−j′)≠ or AT(i′−j′)A≠. :=\A ^n× k A^T( μ _i- μ _j) 0 or A^T( _i- _j)A 0\. (48) We now analyze the set k⊆ℝn×kA_k ^n× k where k>0k>0. Define k:=ℬk∩ℬk′∩k∩k′. _k:=B_k _k _k _k. (49) We will show that i) ℝn×k ^n× k _k has Lebesgue measure zero, and i) A is an open set in ℝn×kR^n× k. We analyze each of the components Let ℒL be the Lebesgue measure. For ℬkB_k, by Lemma B.4, we know ℒ(ℬk,jc)=0L(B_k,j^c)=0 for all j∈[J]j∈[J]. Thus, we have 0≤ℒ(ℬkc)=ℒ(∪j∈kℬk,jc)≤∑j∈kℒ(ℬk,jc)=0 0 (B_k^c)=L( _j∈S_kB_k,j^c)≤ _j∈S_kL(B_k,j^c)=0 (50) which of course implies ℒ(ℬkc)=0L(B_k^c)=0. Next, from Lemma B.4, we also know ℬk,jB_k,j is open for all j∈[J]j∈[J]. As ℬkB_k is the finite intersection of ℬk,jB_k,j’s, we know ℬkB_k is open as well. We can apply the same arguments and obtain i) ℒ(ℬk′⁣c)=0L(B_k c)=0, and i) ℬk′B_k is an open set. For k,i,jC_k,i,j, we separate it into two cases: k,i,j=k,i,jμ∪k,i.jσ, _k,i,j=C_k,i,j^μ _k,i.j^σ, (51) where k,i,jμ:=A∈ℝn×k∣AT(i−j)≠, _k,i,j^μ:=\A ^n× k A^T( μ_i- μ_j) 0\, (52) and k,i,jσ:=A∈ℝn×k∣AT(i−j)A≠. _k,i,j^σ:=\A ^n× k A^T( _i- _j)A 0\. (53) If i≠j μ_i≠ μ_j, AT(i−j)A^T( μ_i- μ_j) can be regarded as m polynomial functions of A from ℝnR^n to ℝR, i.e. AT(i−j)=[A⋅1T(i−j)A⋅2T(i−j)⋮A⋅kT(i−j)]. A^T( μ_i- μ_j)= bmatrixA^T_· 1( μ_i- μ_j)\\ A^T_· 2( μ_i- μ_j)\\ \\ A^T_· k( μ_i- μ_j) bmatrix. (54) Each A⋅i(i−j)A_· i( μ_i- μ_j), i∈[k]i∈[k] is not zero everywhere as (i−j)≠( μ_i- μ_j) 0. By Caron and Traynor (2005), a polynomial from ℝnR^n to ℝR is either zero everywhere or non-zero almost everywhere. Apply to our functions, it means they are all nonzero almost everywhere. Denote the zero set as μO_μ and we know ℒ(μ)=0L(O_μ)=0. Since these polynomial functions all share the same coefficients, we have (k,i,jμ)c=μ×⋯×μ⏟m=μm, (C_k,i,j^μ)^c= O_μ×·s×O_μ_m=O_μ^m, (55) which means ℒ((k,i,jμ)c)=ℒ(μm)=ℒ(μ)m=0. ((C_k,i,j^μ)^c)=L(O_μ^m)=L(O_μ)^m=0. (56) If i≠j _i≠ _j, then consider the diagonal elements of AT(i−j)A^T( _i- _j)A as n polynomial functions of A, denote as pii:=A⋅iT(i−j)A⋅ip_i:=A_· i^T( _i- _j)A_· i, ∀i∈[m]∀ i∈[m]. Since i−j≠ _i- _j 0, piip_i is not zero everywhere. By Caron and Traynor (2005), they are all nonzero almost everywhere. Denote the zero set as σO_σ and we know ℒ(σ)=0L(O_σ)=0. Since they are only the diagonal elements, we have (k,i,jσ)c⊆σm⇒0≤ℒ((k,i,jσ)c)≤ℒ(σm)=ℒ(σ)m=0, (C_k,i,j^σ)^c _σ^m 0 ((C_k,i,j^σ)^c) (O_σ^m)=L(O_σ)^m=0, (57) which means ℒ((k,i,jσ)c)=0L((C_k,i,j^σ)^c)=0. By Equation˜51, we have k,i,jc=(k,i,jμ)c∩(k,i,jσ)c. _k,i,j^c=(C_k,i,j^μ)^c∩(C_k,i,j^σ)^c. (58) As we assume, X is in reduced form, which means i≠j μ_i≠ μ_j or i≠j _i≠ _j. This implies at least one of (ijμ)c(C_ij^μ)^c and (ijσ)c(C_ij^σ)^c is Lebesgue measure zero. Then, as the intersection of these two sets, we have ℒ(ijc)=0L(C_ij^c)=0. Since the union of measure zero sets is measure zero, we have ℒ(c)=0L(C^c)=0. Since AT(i−j)A^T( μ_i- μ_j) is continuous function and (k,i,jμ)(C_k,i,j^μ) is the preimage of open set ℝk\R^k \0, then (k,i,jμ)(C_k,i,j^μ) is open. Similarly, we have (k,i,jσ)(C_k,i,j^σ) is open as well. As the union of two open sets, k,i,jC_k,i,j is also open. Then as the finite intersection of k,i,jC_k,i,j, we obtain )kC)k is an open set. For k′C _k, we can apply the same arguments and obtain i) ℒ(k′⁣c)=0L(C c_k)=0, and i) k′C_k is an open set. To sum up, all the above results mean that ℒ(kc)=ℒ(ℬkc∪ℬk′⁣c∪kc∪k′⁣c)=0L(A^c_k)=L(B_k^c c_k ^c_k c_k)=0 and thus kA_k is not empty and, furthermore, k=ℬk∩ℬk′∩k∩k′A_k=B_k _k _k _k is an open set in ℝn×kR^n× k. Now, we have =′X d=X on E, by Def. 3.1, which means (∑j=1JλjN(j,j))E ( _j=1^J _jN( μ_j, _j) )_E =(∑j=1J′λj′N(j′,j′))E = ( _j=1^J _j N( μ_j , _j ) )_E (59) ⇒∑j=1JλjN(j,j)E _j=1^J _jN( μ_j, _j)_E =∑j=1J′λj′N(j′,j′)E. = _j=1^J _j N( μ_j , _j )_E. (60) We start from the highest rank k1k_1. Since k1A_k_1 is non-empty, it has an element A1∈k1A_1 _k_1. Apply A1A_1 on both sides of Equation˜60 and we get A1T∑j=1JλjN(j,j)E A_1^T _j=1^J _jN( μ_j, _j)_E =A1T∑j=1J′λj′N(j′,j′)E =A_1^T _j=1^J _j N( μ_j , _j )_E (61) ∑j=1JλjA1T(N(j,j)E) _j=1^J _jA_1^T(N( μ_j, _j)_E) =∑j=1J′λj′A1T(N(j′,j′)E) = _j=1^J _j A_1^T(N( μ_j , _j )_E) (62) ⇒∑j=1JλjA1T(N(j,j)j∩E) _j=1^J _jA_1^T(N( μ_j, _j)_X_j∩ E) =∑j=1J′λj′A1T(N(j′,j′)j′∩E). = _j=1^J _j A_1^T(N( μ_j , _j )_X _j∩ E). (63) Due to ℬk∩ℬk′B_k _k , we have rank(A1TjA1)=kjrank(A_1^T _jA_1)=k_j for all j∈[J]j∈[J] and rank(A1Tj′A1)=kj′rank(A_1^T _jA_1)=k _j for all j∈[J′]j∈[J ] . Thus, we can apply Lemma B.7, and obtain A1A_1 is injective on col(j)+jcol( _j)+ μ_j, which is exactly jX_j. Then, we can apply Lemma B.6 on Equation˜63 and get ∑j=1JλjA1T(N(j,j)j∩E) _j=1^J _jA_1^T(N( μ_j, _j)_X_j∩ E) =∑j=1J′λj′A1T(N(j′,j′)j′∩E) = _j=1^J _j A_1^T(N( μ_j , _j )_X _j∩ E) (64) ∑j=1Jλj(A1TN(j,j))A1T(j∩E) _j=1^J _j(A_1^TN( μ_j, _j))_A_1^T(X_j∩ E) =∑j=1J′λj′(A1TN(j′,j′))A1T(j′∩E) = _j=1^J _j (A_1^TN( μ_j , _j ))_A_1^T(X _j∩ E) (65) ∑j=1JλjN(A1Tj,A1TjA1)A1T(j∩E) _j=1^J _jN(A_1^T μ_j,A_1^T _jA_1)_A_1^T(X_j∩ E) =∑j=1J′λj′N(A1Tj′,A1Tj′A1)A1T(j′∩E). = _j=1^J _j N(A_1^T μ_j ,A_1^T _j A_1)_A_1^T(X _j∩ E). (66) Next, we split both sides into two terms by the rank value kjk_j and kj′k _j ∑j∈[J]:kj=k1λjN(A1Tj,A1TjA1)A1T(j∩E)+∑j∈[J]:kj<k1λjN(A1Tj,A1TjA1)A1T(j∩E) _\j∈[J]:k_j=k_1\ _jN(A_1^T μ_j,A_1^T _jA_1)_A_1^T(X_j∩ E)+ _\j∈[J]:k_j<k_1\ _jN(A_1^T μ_j,A_1^T _jA_1)_A_1^T(X_j∩ E) (67) = = ∑j∈[J′]:kj′=k1λj′N(A1Tj′,A1Tj′A1)A1T(j′∩E)+∑j∈[J′]:kj′<k1λj′N(A1Tj′,A1Tj′A1)A1T(j′∩E). _\j∈[J ]:k_j =k_1\ _j N(A_1^T μ_j ,A_1^T _j A_1)_A_1^T(X _j∩ E)+ _\j∈[J ]:k_j <k_1\ _j N(A_1^T μ_j ,A_1^T _j A_1)_A_1^T(X _j∩ E)\,. (68) For the terms in the first sum on both sides, we have det(ATjA)≠0 (A^T _jA)≠ 0, which implies non-singular covariance matrices after transformation. This means N(A1Tj,A1TjA1)≪ℒN(A_1^T μ_j,A_1^T _jA_1) and N(A1Tj′,A1Tj′A1)≪ℒN(A_1^T μ_j ,A_1^T _j A_1) , where ℒL is the Lebesgue measure222This is another of saying the distribution has a density w.r.t. ℒL, by the Radon-Nikodym theorem. In general, if λ≪μλ μ, we also have λE≪μ _E μ and thus we have N(A1Tj,A1TjA1)A1T(j∩E)≪ℒandN(A1Tj′,A1Tj′A1)A1T(j′∩E)≪ℒ N(A_1^T μ_j,A_1^T _jA_1)_A_1^T(X_j∩ E) N(A_1^T μ_j ,A_1^T _j A_1)_A_1^T(X _j∩ E) (69) For the terms in the second sum of both sides, we have rank(A1TjA1)=kj<k1,rank(A1Tj′A1)=kj′<k1, rank(A_1^T _jA_1)=k_j<k_1, rank(A_1^T _jA_1)=k _j<k_1, (70) which means after transformation, these components are still degenerate. Importantly, it means the support of N(A1Tj,A1TjA1)N(A_1^T μ_j,A_1^T _jA_1) and N(A1Tj′,A1Tj′A1)N(A_1^T μ_j ,A_1^T _j A_1) have zero Lebesgue measure. By restricting further, the support of N(A1Tj,A1TjA1)A1T(j∩E)N(A_1^T μ_j,A_1^T _jA_1)_A_1^T(X_j∩ E) and N(A1Tj′,A1Tj′A1)A1T(j′∩E)N(A_1^T μ_j ,A_1^T _j A_1)_A_1^T(X _j∩ E) still have measure zero. Therefore, we can apply Lemma B.8 and obtain the equality for the terms with density ∑j∈[J]:kj=k1λjN(A1Tj,A1TjA1)A1T(j∩E)=∑j∈[J′]:kj′=k1λj′N(A1Tj′,A1Tj′A1)A1T(j′∩E). _\j∈[J]:k_j=k_1\ _jN(A_1^T μ_j,A_1^T _jA_1)_A_1^T(X_j∩ E)= _\j∈[J ]:k _j=k_1\ _j N(A_1^T μ_j ,A_1^T _j A_1)_A_1^T(X _j∩ E). (71) Let U:=∩j∈[J]:kj=k1A1T(j∩E)U:= _\j∈[J]:k_j=k_1\A_1^T(X_j∩ E). Since A1T(j∩E)A_1^T(X_j∩ E) is an open set in ℝk1R^k_1 for all j∈[J]:kj=k1\j∈[J]:k_j=k_1\, and have common element A10A_1z_0 by condition. Thus U is an open and non-empty set. We further restrict the measure of Equation˜71 to U: ∑j∈[J]:kj=k1λjNU(A1Tj,A1TjA1) _\j∈[J]:k_j=k_1\ _jN_U(A_1^T μ_j,A_1^T _jA_1) =∑j∈[J]:kj′=k1λj′NU(A1Tj′,A1Tj′A1) = _\j∈[J]:k _j=k_1\ _j N_U(A_1^T μ_j ,A_1^T _j A_1) (72) (∑j∈[J]:kj=k1λjN(A1Tj,A1TjA1))U ( _\j∈[J]:k_j=k_1\ _jN(A_1^T μ_j,A_1^T _jA_1) )_U =(∑j∈[J]:kj′=k1λj′N(A1Tj′,A1Tj′A1))U. = ( _\j∈[J]:k _j=k_1\ _j N(A_1^T μ_j ,A_1^T _j A_1) )_U. (73) Since each component in Equation˜73 has density functions, and they are all different due to ∩′C . As we discussed before, U is an open and non-empty set, thus by Theorem˜B.2, we can get ∑j∈[J]:kj=k1λjN(A1Tj,A1TjA1)=∑j∈[J]:kj′=k1λj′N(A1Tj′,A1Tj′A1), _\j∈[J]:k_j=k_1\ _jN(A_1^T μ_j,A_1^T _jA_1)= _\j∈[J]:k _j=k_1\ _j N(A_1^T μ_j ,A_1^T _j A_1), (74) which means |j∈[J]:kj=k1|=|j∈[J′]:kj′=k1||\j∈[J]:k_j=k_1\|=|\j∈[J ]:k_j =k_1\|, and there exits a permutation π:j∈[J]:kj=k1→j∈[J′]:kj′=k1π:\j∈[J]:k_j=k_1\→\j∈[J ]:k_j =k_1\, such that (A1Tj,A1TjA1)=(A1Tπ(j)′,A1Tπ(j)′A1),λj=λπ(j)′,∀A1∈k1. (A_1^T μ_j,A_1^T _jA_1)=(A_1^T μ_π(j) ,A_1^T _π(j) A_1), _j=λ _π(j), ∀ A_1 _k_1. (75) Consider two functions: f1(A1):=(A1T(j−π(j)′))1=0,f2(A1):=(A1T(j−π(j)′)A1)1,1=0,∀A1∈k1. f_1(A_1):=(A_1^T( μ_j- μ _π(j)))_1=0, f_2(A_1):= (A_1^T( _j- _π(j))A_1 )_1,1=0, ∀ A_1 _k_1. (76) Since these hold for all A1∈k1A_1 _k_1, and k1A_k_1 is open and non-empty. Thus, we can take the first and second derivatives on f1f_1 and f2f_2, respectively, w.r.t. the elements in the first column of A1A_1. We get f1′(A1):=j−π(j)′=n×1,f2′(A1)=j−π(j)′=n×n,∀A1∈k1. f _1(A_1):= μ_j- μ _π(j)=0^n× 1, f _2(A_1)= _j- _π(j)=0^n× n, ∀ A_1 _k_1. (77) Thus we conclude ∃π:j∈[J]:kj=k1→j∈[J′]:kj′=k1,s.t.j=π(j)′,j=π(j)′,λj=λπ(j)′. ∃π:\j∈[J]:k_j=k_1\→\j∈[J ]:k _j=k_1\,s.t. μ_j= μ _π(j), _j= _π(j), _j=λ _π(j). (78) Now go back to Equation˜63, we can cancel out the terms with rank k1k_1 both sides and get ∑j∈[J]:kj<k1λjNE(j,j)=∑j∈[J′]:kj′<k1λj′NE(j′,j′) _\j∈[J]:k_j<k_1\ _jN_E( μ_j, _j)= _\j∈[J ]:k _j<k_1\ _j N_E( μ_j , _j ) (79) W.l.o.g, suppose k2≥k2′k_2≥ k _2, the same argument can apply to k2k_2. And continue progressing from the largest to the smallest rank to cover all elements in both X and ′X , and finally, we can get J=J′, and ∃π:[J]→[J′],s.t.j=π(j)′,j=π(j)′,λj=λπ(j)′, J=J , and ∃π:[J]→[J ],s.t. μ_j= μ _π(j), _j= _π(j), _j=λ _π(j), (80) and by Theorem˜B.1, we can conclude =′X d=X . ∎ We have concluded the proof of Theorem˜B.3, which proves the identifiability of two pdGMMs from an open set assuming a common non-empty intersection set for all components. Based on these results, we can prove our final goal of this section: Theorem˜3.2, which removes the assumption of the common non-empty intersection set for all components. The basic idea is to group the components by whether they intersect on the same point and apply Theorem˜B.3 on each group. See 3.2 Proof. We first focus on N(1,1)N( μ_1, _1). Pick up one point 1∈1∩Ez_1 _1∩ E, and define index set u1:=j∈[J]:1∈j,u1′:=j∈[J′]:1∈j′. u_1:=\j∈[J]:z_1 _j\, u _1:=\j∈[J ]:z_1 _j\. (81) Then, we can construct a open ball B(1,δ)⊂EB(z_1,δ)⊂ E, such that ∀j∈[J] 1∀ j∈[J] u_1, j∩B(1,δ)=∅X_j∩ B(z_1,δ)= . By condition we have =′X d=X on E, we restrict it on B(1,δ)B(z_1,δ) and eliminate the zero measures on both sides, then we get ∑j∈u1λjNB(j,j)=∑j∈u1′λj′NB(j′,j′) _j∈ u_1 _jN_B( μ_j, _j)= _j∈ u_1 _j N_B( μ_j , _j ) (82) (∑j∈u1λjN(j,j))B=(∑j∈u1′λj′N(j′,j′))B. ( _j∈ u_1 _jN( μ_j, _j) )_B= ( _j∈ u_1 _j N( μ_j , _j ) )_B. (83) By Theorem˜B.3, we have ∑j∈u1λjN(j,j)=∑j∈u1′λj′N(j′,j′), _j∈ u_1 _jN( μ_j, _j)= _j∈ u_1 _j N( μ_j , _j ), (84) also |u1|=|u1′||u_1|=|u_1 |, and there exists a permutation π1:u1→u1′ _1:u_1→ u_1 , such that (λj,j,j)=(λπ1(j)′,π1(j)′,π1(j)′)( _j, μ_j, _j)=(λ _ _1(j), μ _ _1(j), _ _1(j)). Then, we move to the rest t∈[J]\1t∈[J] \1\. Pick up one point t∈t∩Ez_t _t∩ E, and define index set ut:=j∈[J]\(∪j<tuj):t∈j,ut′:=j∈[J′]\(∪j<tuj′):t∈j′. u_t:=\j∈[J] ( _j<tu_j):z_t _j\, u _t:=\j∈[J ] ( _j<tu _j):z_t _j\. (85) If utu_t is not empty, this means we haven’t idenfity component t yet. Then as before, we construct a open ball B(t,δ)⊂EB(z_t,δ)⊂ E, such that ∀j∈[J] ∀ j∈[J] u_t, j∩B(t,δ)=∅X_j∩ B(z_t,δ)= . We restrict it on B(t,δ)B(z_t,δ) and apply Theorem˜B.3 to get ∑j∈utλjN(j,j)=∑j∈ut′λj′N(j′,j′), _j∈ u_t _jN( μ_j, _j)= _j∈ u_t _j N( μ_j , _j ), (86) also |ut|=|ut′||u_t|=|u_t |, and there exists a permutation πt:ut→ut′ _t:u_t→ u_t , such that (λj,j,j)=(λπt(j)′,πt(j)′,πt(j)′)( _j, μ_j, _j)=(λ _ _t(j), μ _ _t(j), _ _t(j)). By going through all t∈[J]t∈[J], we can combine all results to conclude that =′X d=X . Also J=J′J=J , and there exists a permutation π:J→J′π:J→ J , such that (λj,j,j)=(λπ(j)′,π(j)′,π(j)′)( _j, μ_j, _j)=(λ _π(j), μ _π(j), _π(j)). ∎ B.2 Proofs for the identifiability of pdGMM latent variables In this section, we consider the setting where latent variables follow pdGMMs and aim to identify the them from observations. In Section˜B.2.1 we provide the proof of Theorem˜3.5, which shows the identifiability up to affine transformation within components. In Section˜B.2.2 we provide the proof of Theorem˜3.7, which shows the identifiability up to affine transformation. Finally, in Section˜B.2.3 we provide the proof of Theorem˜3.9, which shows the identifiability up to permutation and element-wise affine transformation. B.2.1 Affine Identifiability within component of pdGMM latent variables mixed by piecewise affine function In this section, we consider a set of latent variables that follow pdGMM and are transformed into observations via a continuous, invertible, piecewise affine mixing function. Our objective is to recover the latent variables from the observed data, without any auxiliary information, except for knowing that the latent variables are pdGMM. First, following Bona-Pellissier et al. (2023), we define continuous piecewise affine functions by closed polyhedra as follows: Definition B.9 (Closed polyhedron (Bona-Pellissier et al., 2023)). We say P⊂ℝnP ^n is a closed polyhedron if and only if there exists q∈ℕq , 1,…,q∈ℝna_1,...,a_q ^n and b1,…,bq∈ℝb_1,...,b_q such that ∈P⇔1T+b1≤0⋮qT+bq≤0. ∈ P \ aligned &a_1^Tx+b_1≤ 0\\ & \\ &a_q^Tx+b_q≤ 0. aligned . (87) Definition B.10 (Continuous piecewise affine function (Bona-Pellissier et al., 2023)). We say that a function :ℝn→ℝmf:R^n ^m is a continuous piecewise affine function if there exists a finite set of closed polyhedra whose union is ℝnR^n and such that f is affine over each polyhedron. A simple application of the pasting lemma (see e.g. Munkres (2000)) shows that a continuous piecewise affine function is indeed continuous. For a continuous piecewise affine function f, there are infinitely many finite covers of ℝnR^n by closed polyhedra such that f is affine over each polyhedron. We will be interested into a specific type of covers called admissible. Definition B.11. [(Bona-Pellissier et al., 2023)] Let f be a continuous piecewise affine function. Let Π be a finite set of closed polyhedra of ℝnR^n. We say that Π is admissible with respect to f if and only if 1. ⋃P∈ΠP=ℝn _P∈ P=R^n; 2. for all P∈ΠP∈ , f is affine on P; and 3. for all P∈ΠP∈ , the interior of P, denoted by P∘P , is not empty. Bona-Pellissier et al. (2023, Proposition 22) show that such an admissible set of closed polyhedra exists for all continuous piecewise affine functions. We first present a result by Kivva et al. (2022) that studies the identifiability of non-degenerate GMM latent variables with piecewise affine mixing function, which serves as a primary inspiration for our approach. Theorem B.12 (Affine Identifiability of GMM latent variables (Kivva et al., 2022)). Assume the observation =()X=f(Z) follows the data-generating process in Sec. 2. Suppose Z follows a Gaussian mixture model ∼∑j=1JλjN(j,j)Z _j=1^J _jN( μ_j, _j) (88) with j∈ℝn μ_j ^n, j∈ℝn×n _j ^n× n and |j|>0| _j|>0 , λj>0 _j>0, and ∑j=1Jλj=1 _j=1^J _j=1. For every i≠j∈[J]i≠ j∈[J] we have (i,i)≠(j,j)( μ_i, _i)≠( μ_j, _j). Then Z is identifiable from X up to an affine transformation. However, the analysis of Theorem˜B.12 is restricted to non-degenerate Gaussian mixture models, assuming full-rank covariance matrices for all components. We begin by presenting an intermediate theorem (Thm. B.14), which shows that any continuous invertible piecewise affine function capable of transforming one pdGMM into another pdGMM must be affine on the support of each component in pdGMM. However, this property does not hold universally. We show now a counterexample in Example B.13, where f is not affine over the support of any Gaussian components but still preserves the same distribution. Example B.13. Consider a pdGMM with dimension n=2n=2 and Gaussian components J=2J=2, ∼0.5N((0,0),(1000))+0.5N((0,0),(0001))Z 0.5N((0,0), pmatrix1&0\\ 0&0 pmatrix)+0.5N((0,0), pmatrix0&0\\ 0&1 pmatrix) (89) Denote the two dimensions as 1z_1 and 2z_2. One can construct a function :→ℝ2f:Z ^2 that is clearly not affine on Z: ()=(1001), if 1<0 and 2=0,(−100−1), if 1=0 and 2>0,(0−110), if 1=0 and 2<0 or 1>0 and 2=0.f(z)= cases pmatrix1&0\\ 0&1 pmatrixz, if $ z_1<0$ and $z_2=0$,\\ pmatrix-1&0\\ 0&-1 pmatrixz, if $ z_1=0$ and $z_2>0$,\\ pmatrix0&-1\\ 1&0 pmatrixz, if $ z_1=0$ and $z_2<0$ or $z_1>0$ and $z_2=0$. cases (90) In other words, f switches the relative position of axes, while retaining the same distribution, i.e. ()=f(Z) d=Z. Figure 3: Visualization of Example B.13: f switches half axes of 1z_1 and 2z_2. The orange area refers to the support of component 1, and the green area refers to the support of component 2 before transformation. To avoid these special cases, we propose Ass. 3.4, which describes a general class of models. See 3.4 The idea is: the supports of components with the same rank k may overlap, but there is at least one common point in the intersection where the components can be distinguished, i.e., their projections onto the non-degenerate subspace do not have the same norm. We provide proof and example to illustrate the genericity of Ass. 3.4 in Appendix˜E. The following theorem shows that any continuous invertible piecewise affine function capable of transforming one pdGMM into another pdGMM must be affine on the support of each component in pdGMM. Theorem B.14 (Identifiability of pdGMMs from open set with common intersection). Consider a pair of finite pdGMMs in ℝnR^n in reduced form, ∼∑j=1JλjN(j,j)and′∼∑j=1J′λj′N(j′,j′),Z _j=1^J _jN( μ_j, _j) _j=1^J _j N( μ_j , _j ), (91) where |det(j)|≥0,∀j∈[J]| ( _j)|≥ 0,∀ j∈[J] and |det(j′)|≥0,∀j∈[J′]| ( _j)|≥ 0,∀ j∈[J ]. Suppose Z satisfies Assumption 3.4. If there exists a continuous, invertible, piecewise affine function :ℝn→ℝnf:R^n ^n satisfying ′=()Z d=f(Z), then for all j0∈[J]j_0∈[J], f is affine on j0Z_j_0, where j0Z_j_0 is the support of N(j0,j0)N( μ_j_0, _j_0). We introduce some useful theoretical results before we start the proof of Theorem˜B.14. The first lemma shows that the full-ranked matrices sharing the same Gram matrix can be converted to each other via rotations. Lemma B.15. Consider two matrices A,B∈ℝn×kA,B ^n× k with n≥kn≥ k and A is full-column rank with rank(A)=krank(A)=k. If AAT=BBTAA^T=B^T, then there exists an orthogonal matrix Q∈ℝk×kQ ^k× k such that A=BQA=BQ. Proof. Since AAT=BBTAA^T=B^T and A is full-column-rank, we can derive A=BBTA(ATA)−1A=B^TA(A^TA)^-1. Then, we aim to prove Q:=BTA(ATA)−1Q:=B^TA(A^TA)^-1 is an orthogonal matrix by showing QTQ=IQ^TQ=I, where I is a identity matrix: QTQ=[BTA(ATA)−1]TBTA(ATA)−1=(ATA)−1ATBBTA(ATA)−1=I.Q^TQ=[B^TA(A^TA)^-1]^TB^TA(A^TA)^-1=(A^TA)^-1A^TB^TA(A^TA)^-1=I. (92) ∎ In the following lemma, we show that if we can find an open cover, such that two probability measure are the same over any open set in this open cover, then they are the same measure. Lemma B.16. Let μ:ℱ→[0,1]μ:F→[0,1] and λ:ℱ→[0,1]λ:F→[0,1] be two probability measures on the Borel sigma-algebra ℱF over ℝnR^n. Let ⊆ℝnX ^n be the support of μ. Let O be an open cover of X. If for all O∈O , we have μO∩=λO∩, _O = _O \,, then μ=λμ=λ. Proof. Let B∈ℱB . Let BO_B be a cover of B. Since B is a subset of a second-countable space, one can find a countable open subcover O1,O2,…\O_1,O_2,…\ of B. We construct a partition B as follow: For all k∈ℕk , define B1:=B∩O1andBk:=B∩Ok∩(⋂i=1k−1Bic).B_1:=B∩ O_1 B_k:=B∩ O_k∩ ( _i=1^k-1B_i^c )\,. We now show that B1,B2,…\B_1,B_2,…\ is indeed a partition of B. For any x∈Bx∈ B, we show that there exists a unique BkB_k such that x∈Bkx∈ B_k. Let kx:=min⁡k∈ℕ∣x∈Okk_x:= \k x∈ O_k\. Clearly, x∉Bix ∈ B_i for i<kxi<k_x since x∉Oix ∈ O_i for i<kxi<k_x. Moreover, x∈Bkxx∈ B_k_x since x∈Okxx∈ O_k_x and x∈Bicx∈ B_i^c for all i<kxi<k_x. It is then clear that x∉Bix ∈ B_i for i>kxi>k_x since x∉Bkxcx ∈ B_k_x^c. We write μ(B) μ(B) =μ(B∩)=μ((⋃k≥1Bk)∩)=μ(⋃k≥1(Bk∩))=∑k=1∞μ(Bk∩) =μ(B )=μ( ( _k≥ 1B_k ) )=μ( _k≥ 1(B_k ))= _k=1^∞μ(B_k ) (93) =∑k=1∞μ(B∩Ok∩(⋂i=1k−1Bic)∩)=∑k=1∞μOk∩(B∩(⋂i=1k−1Bic)) = _k=1^∞μ(B∩ O_k∩ ( _i=1^k-1B_i^c ) )= _k=1^∞ _O_k (B∩ ( _i=1^k-1B_i^c )) (94) =∑k=1∞λOk∩(B∩(⋂i=1k−1Bic))=⋯=λ(B∩), = _k=1^∞ _O_k (B∩ ( _i=1^k-1B_i^c ))=·s=λ(B )\,, (95) where the ⋯·s means we apply the exact same steps as before in reverse order. By taking B=B=X, we get 1=μ()=λ(∩)=λ()1=μ(X)=λ(X )=λ(X), which means λ(c)=0λ(X^c)=0. This means μ(B)=λ(B∩)=λ(B∩)+λ(B∩c)=λ(B),μ(B)=λ(B )=λ(B )+λ(B ^c)=λ(B)\,, which is what we intended to show. ∎ Next, we report two results from Xu et al. (2024), which show the identifiability results of potentially degenerate multivariate normal distributions (pdMVNs), as we will use them as important final steps in the proof of Theorem˜B.14 later. Theorem B.17 (Affine Identifiability for pdMVNs with Piecewise Affine f (Xu et al. (2024))). Assume ,^:ℝn→ℝdf, f:R^n ^d are injective and piecewise affine. We assume Z and Z follow a (degenerate) multivariate normal distribution. If ()=^(^)f(Z) d= f( Z), then there exists an invertible affine transformation :ℝn→ℝnh:R^n ^n such that ()=^h(Z) d= Z. Lemma B.18 (Linearity of f for pdMVNs (Xu et al. (2024))). Let ∼N(,)Z N( μ, ), where ||≥0| |≥ 0, i.e. Z is potentially a degenerate multivariate normal. Assume that :ℝn→ℝnf:R^n ^n is a continuous piecewise affine function such that ()=f(Z) d=Z. Then f is affine over Z, the support of Z. Now we are ready to prove Theorem˜B.14. Below, we outline the structure of the proof. We begin by focusing on a fixed Gaussian component j0j_0 from Z and partitioning its support into polyhedral regions such that the function f is affine within each subdomain. Within each polyhedral region, by applying Theorem˜B.2, we show that there exists a component j0′j _0 in ′Z such that component j0j_0 is mapped to j0′j _0. Next, under Ass. 3.4, we show that f consistently assigns j0j_0 to the same j0′j _0 across all polyhedra. This successfully builds a one-to-one correspondence between the Gaussian components on both sides via f. Finally, using the results from Xu et al. (2024), we conclude that f must be affine on the support of each component. See B.14 Proof. Fix j0∈[J]j_0∈[J]. We must show that f is affine on j0Z_j_0, the support of the j0j_0th component. To achieve this, we will first show that, there exists an j0′∈[J′]j _0∈[J ], such that (N(j0,j0))=N(j0′,j0′).f(N( μ_j_0, _j_0))=N( μ _j _0, _j _0). To get this, we will show that for all 0∈j0z_0 _j_0, there exists an open ball B centered at 0z_0 such that ((N(j0,j0)))(B)=(N(j0′,j0′))(B). (f(N( μ_j_0, _j_0)) )_f(B)=(N( μ_j _0, _j _0))_f(B). Let us fix some 0∈j0z_0 _j_0 and let Π be an admissible set of closed polyhedra for f (Definition˜B.11). The existence of admissible set is ensured by Proposition 22 from Bona-Pellissier et al. (2023). We consider two cases: • Case 1. There exists P∈ΠP∈ such that 0∈P∘z_0∈ P , where P∘P is the interior part of P as defined in Def. B.11. • Case 2. For all P∈ΠP∈ , 0∉P∘z_0 ∈ P , where P∘P is the interior part of P. Case 1. Case 1 is simple. Since 0z_0 is in the interior of P, we can find an open set with interior point 0z_0, B, such that B⊂PB⊂ P. By definition of an admissible cover, f is affine on P, and thus on B, and thus on B∩jB _j. Since f is injective, we know there exists an invertible matrix H and weight vector b such that ()=H+f(z)=Hz+b, ∀∈B ∈ B. Define ϕ():=H+φ(z):=Hz+b for all ∈ℝnz ^n. We can thus constrain the probability measure to (B)f(B) on both sides of ()=′f(Z) d=Z and write ∑j∈Jλj(N(j,j)B)=∑j=1J′λj′N(j′,j′)(B), _j∈ J _jf (N( μ_j, _j)_B )= _j=1^J _j N( μ_j , _j )_f(B)\,, (97) ∑j∈Jλjϕ(N(j,j)B)=∑j=1J′λj′N(j′,j′)(B). _j∈ J _jφ (N( μ_j, _j)_B )= _j=1^J _j N( μ_j , _j )_f(B)\,. (98) By Lemma B.5 and f is injective, we have ∑j∈Jλj(N(Hj+,HjHT))(B)=∑j=1J′λj′N(j′,j′)(B). _j∈ J _j (N(H μ_j+b,H _jH^T) )_f(B)= _j=1^J _j N( μ_j , _j )_f(B)\,. (99) Since B is open set and f is an open map on B (all surjective linear maps are), (B)f(B) is an open set as well. Thus we can apply Theorem˜3.2 and get there must exists an j0′∈[J′]j _0∈[J ], such that (N(j0,j0))=N(j0′,j0′)f(N( μ_j_0, _j_0))=N( μ _j _0, _j _0) on (B)f(B). Case 2. Case 2 is much more technical and will thus be the focus of the rest of the proof. Since Π covers ℝnR^n, there must be at least one P1∈ΠP_1∈ such that 0∈P1z_0∈ P_1. Since 0z_0 is not in the interior of P1P_1, it must be in its boundary, ∂P1∂ P_1. We now argue that 0z_0 must also be in a distinct polyhedron. We argue by contradiction. Suppose that is false. This means 0z_0 is in the complement of all polyhedra P∈Π∖P1P∈ \P_1\. Since these polyhedra are closed, their complements are open, thus we can find an open set U containing 0z_0 that do not intersect with any P∈Π∖P1P∈ \P_1\. But since 0z_0 is on the boundary of P1P_1, this means a non-empty portion of U lies outside P1P_1. But this would mean this portion is outside of all P∈ΠP∈ , which is a contradiction since Π covers ℝnR^n. This means there must be at least one other polyhedron P2≠P1P_2 =P_1 containing 0z_0. Again, since 0z_0 is not in the interior of P2P_2, it must be on its boundary, ∂P2∂ P_2. To summarize, we found two distinct polyhedra, P1P_1 and P2P_2, such that 0∈P1∩P2z_0∈ P_1∩ P_2 and such that 0∈∂P1z_0∈∂ P_1 and 0∈∂P2z_0∈∂ P_2. In general, 0z_0 could be in more than two polyhedra. Let P1,…,PL∈ΠP_1,…,P_L∈ be the polyhedra that contain 0z_0. We know from the previous steps that L≥2L≥ 2. Define the set of all components that intersects the point 0z_0. J0:=j∈[J]:0∈j, J_z_0:=\j∈[J]:z_0 _j\, (100) Of course, j0∈J0j_0∈ J_z_0. Since the jZ_j are affine subspaces, they are closed sets. This means we can find an open set B′B containing 0z_0 small enough such that B′B intersects only the components j∈J0j∈ J_z_0. Furthermore, since all P∈ΠP∈ are closed, we can choose an open set B′B containing 0z_0 that is small enough so has to not intersect with the polyhedra other than P1,…,PLP_1,…,P_L. Take B0:=B′∩B′B_z_0:=B ∩ B . Since 0z_0 is on the boundaries of each P1,…,PLP_1,…,P_L, any open set must intersect with each PlP_l for l∈[L]l∈[L]. We can thus define the non-empty sets Sl:=B0∩PlS_l:=B_z_0∩ P_l for all l∈[L]l∈[L]. Define the index set Jl:=j∈[J]:Sl∩j≠∅, J_l:=\j∈[J]:S_l _j≠ \\,, (101) i.e. this is the set of components jZ_j that intersects the polyhedron PlP_l and the open set B0B_z_0. Notice that Jl⊆J0J_l J_z_0. Then our goal for the following part is to show that there exists an j0′∈[J′]j _0∈[J ], such that (N(j0,j0))=N(j0′,j0′)f(N( μ_j_0, _j_0))=N( μ _j _0, _j _0) on SlS_l. But we do not need to go through all l∈[L]l∈[L]. Instead, since we are focusing on component j0j_0 now, in Case 2, we first explore the one on which N(j0,j0)N( μ_j_0, _j_0) has positive probability, and later come back to discuss the rest of the polyhedra. For this reason, we introduce one more index notation: Lj0:=l∈[L]:N(j0,j0)(Sl)>0), L_j_0:=\l∈[L]:N( μ_j_0, _j_0)(S_l)>0)\\,, (102) where N(j0,j0)(Sl)N( μ_j_0, _j_0)(S_l) means the probability that measure N(j0,j0)N( μ_j_0, _j_0) take on SlS_l. Pick some polyhedron l∈[Lj0]l∈[L_j_0]. By condition we have ′=()Z d=f(Z), we restrict the measure on both sides to (Sl)f(S_l) and then we get ((∑j=1JλjN(j,j)))(Sl)=(∑j=1J′λj′N(j′,j′))(Sl) (f( _j=1^J _jN( μ_j, _j)) )_f(S_l)= ( _j=1^J _j N( μ_j , _j ) )_f(S_l) (103) Since f is injective, we can use Lemma B.5 to switch constrain and f on the left hand side and we get ((∑j=1JλjN(j,j))Sl) ( ( _j=1^J _jN( μ_j, _j) )_S_l ) =(∑j=1J′λj′N(j′,j′))(Sl), = ( _j=1^J _j N( μ_j , _j ) )_f(S_l), (104) (∑j=1JλjN(j,j)Sl) ( _j=1^J _jN( μ_j, _j)_S_l ) =∑j=1J′λj′N(j′,j′)(Sl), = _j=1^J _j N( μ_j , _j )_f(S_l), (105) ∑j=1Jλj(N(j,j)Sl) _j=1^J _jf (N( μ_j, _j)_S_l ) =∑j=1J′λj′N(j′,j′)(Sl), = _j=1^J _j N( μ_j , _j )_f(S_l)\,, (106) where the second line uses the fact that the restriction of a mixture is the mixture of the restriction and the third line uses the fact that the pushforward of a mixture is the mixture of the pushforwards. Notice that if j∩Sl=∅Z_j∩ S_l= , we have that N(j,j)Sl=0N( μ_j, _j)_S_l=0, i.e. it is the zero measure. The pushforward of the zero measure is also the zero measure, thus ∑j∈Jlλj(N(j,j)Sl)=∑j=1J′λj′N(j′,j′)(Sl) _j∈ J_l _jf (N( μ_j, _j)_S_l )= _j=1^J _j N( μ_j , _j )_f(S_l) (107) Since f is piecewise affine and injective, we know there exists an invertible matrix HlH^l and weight vector lb^l such that ()=Hl+lf(z)=H^lz+b^l, ∀∈Sl ∈ S_l. Define ϕl():=Hl+lφ^l(z):=H^lz+b^l for all ∈ℝnz ^n. We can thus write ∑j∈Jlλjϕl(N(j,j)Sl)=∑j=1J′λj′N(j′,j′)(Sl), _j∈ J_l _jφ^l (N( μ_j, _j)_S_l )= _j=1^J _j N( μ_j , _j )_f(S_l)\,, (108) since =ϕlf=φ^l on SlS_l. Since ϕlφ^l is injective, we can apply Lemma B.5 to get ∑j∈Jlλjϕl(N(j,j))ϕl(Sl) _j∈ J_l _jφ^l (N( μ_j, _j) )_φ^l(S_l) =∑j=1J′λj′N(j′,j′)(Sl) = _j=1^J _j N( μ_j , _j )_f(S_l) (109) ∑j∈JlλjN(Hlj+l,Hlj(Hl)T)(Sl) _j∈ J_l _jN(H^l μ_j+b^l,H^l _j(H^l)^T)_f(S_l) =∑j=1J′λj′N(j′,j′)(Sl). = _j=1^J _j N( μ_j , _j )_f(S_l)\,. (110) Define Jl′:=j∈[J′]:j′∩(Sl)≠∅J _l:=\j∈[J ]:Z _j (S_l)≠ \. This allows us to write ∑j∈JlλjN(Hlj+l,Hlj(Hl)T)(Sl)=∑j∈Jl′λj′N(j′,j′)(Sl). _j∈ J_l _jN(H^l μ_j+b^l,H^l _j(H^l)^T)_f(S_l)= _j∈ J _l _j N( μ_j , _j )_f(S_l). (111) Now we discuss the possible relations between N(j0,j0)N( μ_j_0, _j_0) and SlS_l. If N(j0,j0)N( μ_j_0, _j_0) takes probability zero on SlS_l, then we can skip it. If not, then we face two cases: • Case 2.1 j0Z_j_0 intersects with the interior of SlS_l, i.e. j0∩Sl∘≠∅Z_j_0∩ S_l ≠ , where Sl∘:=B0∩Pl∘S_l :=B_z_0∩ P _l. • Case 2.2 j0Z_j_0 only intersects with the boundary of SlS_l with positive probability, i.e. j0∩Sl∘=∅Z_j_0∩ S_l = and N(j0,j0)(∂Sl)>0N( μ_j_0, _j_0)(∂ S_l)>0, where ∂Sl:=B0∩∂Pl∂ S_l:=B_z_0∩∂ P_l. Case 2.1. For Case 2.1, define the index set that also intersects with the interior part of SlS_l: Jl∘:=j∈Jl:j∩Sl∘≠∅,Jl′⁣∘:=j∈Jl′:j′∩(Sl∘)≠∅.J_l :=\j∈ J_l:Z_j∩ S_l ≠ \, J_l :=\j∈ J _l:Z _j (S_l )≠ \. We further constrain Equation˜111 on (Sl∘)f(S_l ) and cancel out the purely zero measure terms on both sides, i.e. the terms are not in Jl∘J_l and Jl′⁣∘J_l ∑j∈Jl∘λjN(Hlj+l,Hlj(Hl)T)(Sl∘)=∑j∈Jl′⁣∘λj′N(j′,j′)(Sl∘). _j∈ J_l _jN(H^l μ_j+b^l,H^l _j(H^l)^T)_f(S_l )= _j∈ J_l _j N( μ_j , _j )_f(S_l ). (112) f is an open map on Sl∘S_l since it is affine on Sl∘S_l . Therefore, (Sl∘)f(S_l ) is an open set as well. By the definition of Jl∘J_l , we have ∀j∈Jl∘∀ j∈ J_l , j∩Sl∘≠∅Z_j∩ S_l ≠ , then we can derive (j∩Sl∘)≠∅,∀j∈Jl∘f(Z_j∩ S_l )≠ ,∀ j∈ J_l . Since f is invertible function, we can further derive ∀j∈Jl,(j)∩(Sl)≠∅∀ j∈ J_l,f(Z_j) (S_l)≠ . Therefore, we can use Theorem˜3.2 and get ∑j∈Jl∘λjN(Hlj+l,Hlj(Hl)T)=∑j∈Jl′⁣∘λj′N(j′,j′), _j∈ J_l _jN(H^l μ_j+b^l,H^l _j(H^l)^T)= _j∈ J_l _j N( μ_j , _j ), (113) which means |Jl∘|=|Jl′⁣∘||J_l |=|J_l | and there exists a permutation πl:Jl∘→Jl′⁣∘π^l:J_l → J_l such that Hlj(Hl)T=πl(j)′,Hlj+l=πl(j)′. H^l _j(H^l)^T= _π^l(j) , H^l μ_j+b^l= μ_π^l(j) . (114) Case 2.2. For Case 2.2, because of the definition of Lj0L_j_0 (Equation˜102) where we pick up the SlS_l, we know N(j0,j0)N( μ_j_0, _j_0) has positive probability on ∂Sl∂ S_l. Since we know j0Z_j_0 is an affine space, it can only intersect with one facet of ∂Sl∂ S_l. We denote this facet ∂Slj0∂ S_l^j_0. Define the index sets that the supports intersect with this facet with positive probability: Jlj0:=j∈Jl ∘:N(j,j)(∂Slj0)>0,Jl′⁣j0:=j∈Jl′ ′⁣∘:N(j′,j′)((∂Slj0))>0.J_l^j_0:=\j∈ J_l J_l :N( μ_j, _j)(∂ S_l^j_0)>0\, J_l j_0:=\j∈ J _l J_l :N( μ _j, _j)(f(∂ S_l^j_0))>0\. Constrain Equation˜111 further on (∂Slj0)f(∂ S_l^j_0) and eliminate the probability zero terms then we get ∑j∈Jlj0λjN(Hlj+l,Hlj(Hl)T)(∂Slj0)=∑j∈Jl′⁣j0λj′N(j′,j′)(∂Slj0). _j∈ J_l^j_0 _jN(H^l μ_j+b^l,H^l _j(H^l)^T)_f(∂ S_l^j_0)= _j∈ J_l j_0 _j N( μ_j , _j )_f(∂ S_l^j_0). (115) Since all the Gaussian components in Jlj0J_l^j_0 take a positive probability on ∂Slj0∂ S_l^j_0, which implies their rank is lower than or equal to the rank of ∂Slj0∂ S_l^j_0 and their support is parallel with ∂Slj0∂ S_l^j_0. Then it must have no intersection with the interior of all adjacent polyhedra. Thus, we can construct an open set E⊂ℝnE ^n, such that ∂Slj0⊂E∂ S_l^j_0⊂ E. And expand the constrained measure from (∂Slj0)f(∂ S_l^j_0) to (E)f(E) and then we have ∑j∈Jlj0λjN(Hlj+l,Hlj(Hl)T)(E)=∑j∈Jl′⁣j0λj′N(j′,j′)(E). _j∈ J_l^j_0 _jN(H^l μ_j+b^l,H^l _j(H^l)^T)_f(E)= _j∈ J_l j_0 _j N( μ_j , _j )_f(E). (116) f is an open map on E. Therefore, (E)f(E) is also an open set. Therefore, we can use Theorem˜3.2 and get ∑j∈Jlj0λjN(Hlj+l,Hlj(Hl)T)=∑j∈Jl′⁣j0λj′N(j′,j′), _j∈ J_l^j_0 _jN(H^l μ_j+b^l,H^l _j(H^l)^T)= _j∈ J_l j_0 _j N( μ_j , _j ), (117) which means |Jlj0|=|Jl′⁣j0||J_l^j_0|=|J_l j_0| and there exists a permutation πl:Jlj0→Jl′⁣j0π^l:J_l^j_0→ J_l j_0 such that Hlj(Hl)T=πl(j)′,Hlj+l=πl(j)′. H^l _j(H^l)^T= _π^l(j) , H^l μ_j+b^l= μ_π^l(j) . (118) To sum up, Equation˜114 holds for all SlS_l in Case 2.1, and Equation˜118 holds for all SlS_l in Case 2.2, which covers all j0∩B0Z_j_0∩ B_z_0. Thus, we can summarize it as for all l∈[Lj0]l∈[L_j_0] Hlj(Hl)T=πl(j)′,Hlj+l=πl(j)′. H^l _j(H^l)^T= _π^l(j) , H^l μ_j+b^l= μ_π^l(j) . (119) In the following part, we focus on the points 0z_0 satisfying Ass. 3.4, denote this set as S_Z. We aim to show that, consider 0∈z_0∈S_Z, for all l∈[Lj0]l∈[L_j_0], permutations πlπ^l map j0j_0 to the same index in ′Z . We prove the claim for all 0∈z_0∈S_Z by contradiction: suppose there exist l1,l2∈[Lj0]l_1,l_2∈[L_j_0], such that πl1(j0)≠πl2(j0)π^l1(j_0)≠π^l2(j_0). Denote πl1(j0):=j0′π^l1(j_0):=j_0 , this means, there exists a polyhedron l∗∈[Lj0]l*∈[L_j_0], such that (πl⁣∗)−1(j0′)≠(πl1)−1(j0′)(π^l*)^-1(j_0 )≠(π^l_1)^-1(j_0 ). Denote j∗:=(πl⁣∗)−1(j0′)j*:=(π^l*)^-1(j_0 ), we have j∗≠j0j*≠ j_0. We can derive this Hl1j0(Hl1)T=j0′,Hl1j0+l1=j0′,Hl⁣∗j⁣∗(Hl⁣∗)T=j0′,Hl⁣∗j⁣∗+l⁣∗=j0′. casesH^l_1 _j_0(H^l_1)^T= _j_0 , H^l_1 μ_j_0+b^l_1= μ_j_0 ,\\ H^l* _j*(H^l*)^T= _j_0 , H^l* μ_j*+b^l*= μ_j_0 . cases (120) ⇒ Hl1j0(Hl1)T=Hl⁣∗j⁣∗(Hl⁣∗)T,Hl1j0+l1=Hl⁣∗j⁣∗+l⁣∗. casesH^l_1 _j_0(H^l_1)^T=H^l* _j*(H^l*)^T,\\ H^l_1 μ_j_0+b^l_1=H^l* μ_j*+b^l*.\\ cases (121) Since 0z_0 is in both polyhedra l1l_1 and l∗l*, we get Hl10+l1 H^l_1z_0+b^l_1 =(0)=Hl⁣∗0+l⁣∗ =f(z_0)=H^l*z_0+b^l* (122) By taking the difference between Equation˜122 and Equation˜121, we have Hl1(0−j0)=Hl⁣∗(0−j⁣∗).H^l_1(z_0- μ_j_0)=H^l*(z_0- μ_j*). (123) Suppose rank(j0′)=krank( _j_0 )=k. By Equation˜121, since both Hl1H^l_1 and Hl2H^l_2 are invertible matrix, we have rank(j0′)=rank(j0)=rank(j⁣∗)=k.rank( _j_0 )=rank( _j_0)=rank( _j*)=k. (124) By the definition of degenerate Gaussian random variable, we can find a n×kn× k matrix A1A_1 where A1A1T=j0A_1A_1^T= _j_0 and rank(A1)=krank(A_1)=k, and have A1+j0∼N(j0,j0),A_1 + μ_j_0 N( μ_j_0, _j_0), where follows a standard k-dimensional normal distribution. Hence, there exists a unique value 1∈ℝk _1 ^k, such that 0=A11+j0z_0=A_1 _1+ μ_j_0. The same analysis for N(j⁣∗,j⁣∗)N( μ_j*, _j*), there exists a unique 2∈ℝk _2 ^k such that 0=A22+j⁣∗z_0=A_2 _2+ μ_j*. Substitute into Equation˜123, we get Hl1A11=Hl2A22.H^l_1A_1 _1=H^l_2A_2 _2. (125) Additionally, we have A1A1T=j0A_1A_1^T= _j_0 and A2A2T=j⁣∗A_2A_2^T= _j*. Substitute into Equation˜121, we get Hl1A1(Hl1A1)T=Hl2A2(Hl2)A2)T.H^l_1A_1(H^l_1A_1)^T=H^l_2A_2(H^l_2)A_2)^T. First we have rank(Hl1A1)=krank(H^l_1A_1)=k as Hl1H^l_1 is invertible. By Lemma B.15, this implies there exists an orthogonal matrix Q∈ℝk×kQ ^k× k, such that Hl1A1=Hl2A2QH^l_1A_1=H^l_2A_2Q. Substitute into Equation˜125, we have Hl1A1(1−QT2)=0.H^l_1A_1( _1-Q^T _2)=0. Then we can derive 1−QT2=0 _1-Q^T _2=0, which implies ‖1‖=‖2‖|| _1||=|| _2||.Note that ‖1‖=(0−j0)Tj0−1(0−j0)|| _1||= (z_0- μ_j_0)^T _j_0^-1(z_0- μ_j_0) and ‖2‖=(0−j∗)Tj∗−1(0−j∗)|| _2||= (z_0- μ_j_*)^T _j_*^-1(z_0- μ_j_*). This means for all 0∈z_0∈S_Z, the Mahalanobis distance of 0z_0 to N(j0,j0)N( μ_j_0, _j_0) and N(j∗,j∗)N( μ_j_*, _j_*) are the same. We can find it will be contradictory to Assumption 3.4 if πl1(j0)≠πl2(j0)π^l_1(j_0)≠π^l_2(j_0). Therefore, we can claim there exists j0′∈[J′]j _0∈[J ], such that πl(j0)=j0′π^l(j_0)=j _0 for all l∈[Lj0]l∈[L_j_0]. Due to Equation˜114 and the identifiability of pdMVN, we have N(Hlj0+l,Hlj0(Hl)T)=N(j0′,j0′),∀l∈[Lj0]. N(H^l μ_j_0+b^l,H^l _j_0(H^l)^T)=N( μ_j_0 , _j_0 ), ∀ l∈[L_j_0]. (126) Now we finished all the discussion of l∈[Lj0]l∈[L_j_0], i.e. all the SlS_l where N(j0,j0)N( μ_j_0, _j_0) can take positive probability. However, as our goal is to show (N(j0,j0))=N(j0′,j0′)f(N( μ_j_0, _j_0))=N( μ _j _0, _j _0) over j0∩BZ_j_0∩ B, there is a probability-zero subdomain missing in the above analysis. Next, we discuss the rest part l∈[L]\[Lj0]l∈[L] [L_j_0]. These cases can only happen if j0Z_j_0 intersects with the boundary of SlS_l only, i.e. j0∩Sl∘=∅Z_j_0∩ S_l = , j0∩∂Sl≠∅Z_j_0∩∂ S_l≠ and N(j0,j0)(∂Sl)=0N( μ_j_0, _j_0)(∂ S_l)=0. For this case, since j0∩∂Sl≠∅Z_j_0∩∂ S_l≠ and N(j0,j0)(∂Sl)=0N( μ_j_0, _j_0)(∂ S_l)=0, we know for every point ∗∈j0∩∂Slz* _j_0∩∂ S_l, there must exist an adjecent Sl⁣∗S_l* such that ∗∈Sl⁣∗z*∈ S_l* and l∗∈[Lj0]l*∈[L_j_0]. Therefore, we can conclude that j0∩B⊆∪l∈[Lj0]SlZ_j_0∩ B _l∈[L_j_0]S_l. We now define a partition of the set B0∩j0B_z_0 _j_0 by SlS_l, l∈[Lj0]l∈[L_j_0]. For any non-empty subset I⊆[Lj0]I [L_j_0], we define DI:=(∩l′∈ISl′)\(∪l′∉ISl′).D_I:= ( _l ∈ IS_l ) ( _l ∈ IS_l ). Note that all the affine functions Hl+lH^lz+b^l with l∈Il∈ I agree on DID_I since DI⊆∩l∈IPlD_I _l∈ IP_l. For simplicity, we can directly use lI:=min⁡Il_I:= I, i.e. the smallest index in I. When l=lIl=l_I we constrain the measures on both sides of Equation˜126 on subdomain (DI)f(D_I) and get N(HlIj0+lI,HlIj0(HlI)T)(DI)=N(j0′,j0′)(DI) N(H^l_I μ_j_0+b^l_I,H^l_I _j_0(H^l_I)^T)_f(D_I)=N( μ_j_0 , _j_0 )_f(D_I) (127) Sum up all the I⊆[Lj0]I [L_j_0], we have ∑I⊆[Lj0]N(HlIj0+lI,HlIj0(HlI)T)(DI)=∑I⊆[Lj0]N(j0′,j0′)(DI), _I [L_j_0]N(H^l_I μ_j_0+b^l_I,H^l_I _j_0(H^l_I)^T)_f(D_I)= _I [L_j_0]N( μ_j_0 , _j_0 )_f(D_I), (128) ∑I⊆[Lj0](N(j0,j0)DI)=∑I⊆[Lj0]N(j0′,j0′)(DI). _I [L_j_0]f (N( μ_j_0, _j_0)_D_I )= _I [L_j_0]N( μ_j_0 , _j_0 )_f(D_I). (129) Note that DID_I’s are disjoint from each other. Thus, we can switch sum into union and get (N(j0,j0)∪I⊆[Lj0]DI)=N(j0′,j0′)(∪I⊆[Lj0]DI). (N( μ_j_0, _j_0)_ _I [L_j_0]D_I )=N( μ_j_0 , _j_0 )_f( _I [L_j_0]D_I). (130) Since DID_I is partition set of B0∩j0B_z_0 _j_0, which means ∪I⊆[Lj0]DI=B0∩j0 _I [L_j_0]D_I=B_z_0 _j_0, substitute into Equation˜130 and then we have (N(j0,j0)B0∩j0)=N(j0′,j0′)(B0∩j0), (N( μ_j_0, _j_0)_B_z_0 _j_0 )=N( μ_j_0 , _j_0 )_f(B_z_0 _j_0), (131) N(j0,j0)B0∩j0=−1(N(j0′,j0′)(B0∩j0)). N( μ_j_0, _j_0)_B_z_0 _j_0=f^-1 (N( μ_j_0 , _j_0 )_f(B_z_0 _j_0) ). (132) Since f is bijective on ℝnR^n, we have −1f^-1 is injective. Thus by apply Lemma B.5, we can conclude Case 2: N(j0,j0)B0∩j0=−1(N(j0′,j0′))B0∩j0. N( μ_j_0, _j_0)_B_z_0 _j_0=f^-1 (N( μ_j_0 , _j_0 ) )_B_z_0 _j_0. (133) Note that Equation˜133 is true for all 0∈z_0∈S_Z, i.e. we can construct an open set B0B_z_0 containing 0z_0 for all 0∈z_0∈S_Z. Notice that B0∣0∈\B_z_0 _0∈S_Z\ is thus an open cover of the support of the j0Z_j_0. We can thus apply Lemma B.16 to get N(j0,j0) N( μ_j_0, _j_0) =−1(N(j0′,j0′)) =f^-1 (N( μ_j_0 , _j_0 ) ) (134) (N(j0,j0)) (N( μ_j_0, _j_0) ) =N(j0′,j0′). =N( μ _j _0, _j _0). (135) Then, since f is injective and piecewise affine, by Theorem˜B.17 from Xu et al. (2024), there exists an invertible affine transformation j0:ℝn→ℝnh_j_0:R^n ^n such that j0(N(j0,j0))=N(j0′,j0′)h_j_0(N( μ_j_0, _j_0))=N( μ _j _0, _j _0). Substitute into Equation˜135 then we have (N(j0,j0))=j0(N(j0,j0))⇒j0−1∘(N(j0,j0))=N(j0,j0). (N( μ_j_0, _j_0) )=h_j_0(N( μ_j_0, _j_0)) _j_0^-1 (N( μ_j_0, _j_0) )=N( μ_j_0, _j_0). (136) As j0−1∘h_j_0^-1 is a continuous piecewise affine function, by Lemma B.18 from Xu et al. (2024), we can conclude that j0−1∘h_j_0^-1 is affine on j0Z_j_0. Furthermore, we can conclude that f is affine on j0Z_j_0. This result holds for all j0∈[J]j_0∈[J]. ∎ See 3.5 Proof. Since =()X=f(Z), we can rewrite Equation (2) (perfect reconstruction) as ‖()−^((()))‖22=0. ||f(Z)- f(g(f(Z)))||^2_2=0\,. (137) This means that ()f(Z) and ^((())) f(g(f(Z))) are equal ℙP_Z-almost everywhere, which implies ()f(Z) and ^(())) f(g(X))) are equally distributed with the same image space. Therefore, we can define :=^−1∘h:= f^-1 on Z, which is a continuous and invertible piecewise affine function, and we have =−1(())Z d=h^-1(g(X)), where both Z and ()g(X) are pdGMMs. Since Z satisfies Ass. 3.4, and −1h^-1 is a continuous and invertible piecewise affine function, we can apply Theorem˜B.14 and get −1h^-1 is affine on j′Z _j, where j′Z _j is the support of N(j′,j′)N( μ _j, _j). One step more, we can conclude that h, i.e., ^−1∘ f^-1 , is affine on jZ_j. ∎ B.2.2 From affine on components to affine everywhere While Theorem˜3.5 proves that ∘g is affine within each individual component, it does not guarantee that there is a global affine function across all components. In this section, we seek conditions under which ∘g is globally affine, i.e., affine on the whole support for the pdGMM. In the following proposition (Prop. B.19), we show that if the supports of all components share a common basis (i.e., each support can be spanned by a subset of this common basis), then it is possible to construct a consistent affine function across the entire domain. See 3.6 Proposition B.19. Let 1,…,J⊆ℝnZ_1,…,Z_J ^n be affine spaces that satisfy Ass. 3.6, and let f:ℝn→ℝf:R^n be a function such that, for all j∈[J]j∈[J], the restriction of f to jZ_j is affine. Then the restriction of f to :=∪i∈[m]iZ:= _i∈[m]Z_i is affine. Proof. Our goal is to construct an affine function +baz+b with ∈ℝ1×na ^1× n and b∈ℝb such that this function agrees with f on Z. First notice that, since f is affine on each jZ_j, we have: for each j∈[J]j∈[J], there exists j∈ℝ1×na^j ^1× n and bj∈ℝb^j such that, for all ∈jz _j, f()=j+bjf(z)=a^jz+b^j. Since 0∈∩j∈[J]jz^0∈ _j∈[J]Z_j, we have that, for all j∈[J]j∈[J], f(0)=j0+bjf(z^0)=a^jz^0+b^j for all j∈[J]j∈[J]. Let us define :=[1⋯n]∈ℝn×nB:=[z^1\ ·s\ z^n] ^n× n, which is invertible since its columns form a basis by assumption. Fix j∈[J]j∈[J] and k∈jk _j. We know that f(k+0) f(z^k+z^0) =j(k+0)+bj =a^j(z^k+z^0)+b^j (138) =jk+j0+bj =a^jz^k+a^jz^0+b^j (139) =jk+f(0) =a^jz^k+f(z^0) (140) =j⏟~j:=k+f(0) = a^jB_ a^j:=e^k+f(z^0) (141) =~jk+f(0) = a^je^k+f(z^0) (142) =~kj+f(0) = a^j_k+f(z^0) (143) ⟹~kj a^j_k =f(k+0)−f(0)∀k∈j. =f(z^k+z^0)-f(z^0) ∀ k _j\,. (144) Let us define :=[(f(1+0)−f(0))⋯(f(n+0)−f(0))]−1∈ℝ1×nand := bmatrix (f(z^1+z^0)-f(z^0) )&·s& (f(z^n+z^0)-f(z^0) ) bmatrixB^-1 ^1× nand (145) b b :=f(0)−0. :=f(z^0)-az^0\,. (146) We must now show that, for all ∈∪j∈[J]jz∈ _j∈[J]Z_j, f()=+bf(z)=az+b. Fix some arbitrary ∈∪j∈[J]jz∈ _j∈[J]Z_j. We know there has to be one j0∈[J]j_0∈[J] s.t. ∈j0z _j_0. Hence f()=j0+bj0f(z)=a^j_0z+b^j_0. We showed earlier that bj=f(0)−j0b^j=f(z^0)-a^jz^0 for all j∈[J]j∈[J], hence we have f()=j0+f(0)−j00f(z)=a^j_0z+f(z^0)-a^j_0z^0. Since B is invertible, there exists ~∈ℝn z ^n such that −0=B~z-z^0=B z. Since k∣k∈j\z^k k _j\ forms a basis for j−0Z_j-z^0, we have that whenever k∉jk _j, ~k=0 z_k=0. We can thus write f() f(z) =j0+bj0 =a^j_0z+b^j_0 (147) =j0(B~+0)+bj0 =a^j_0(B z+z^0)+b^j_0 (148) =j0B~+j00+bj0 =a^j_0B z+a^j_0z^0+b^j_0 (149) =~j0~+f(0) = a^j_0 z+f(z^0) (150) =∑k∈j~kj0~k+∑k∉j~kj0~k⏟0+f(0) = _k _j a_k^j_0 z_k+ _k _j a_k^j_0 z_k_0+f(z^0) (151) =∑k∈j~kj0~k+f(0) = _k _j a_k^j_0 z_k+f(z^0) (152) =∑k∈j(f(k+0)−f(0))~k+f(0) = _k _j (f(z^k+z^0)-f(z^0) ) z_k+f(z^0) (153) =∑k∈j(f(k+0)−f(0))~k+∑k∉j(f(k+0)−f(0))~k⏟0+f(0) = _k _j (f(z^k+z^0)-f(z^0) ) z_k+ _k _j (f(z^k+z^0)-f(z^0) ) z_k_0+f(z^0) (154) =[(f(1+0)−f(0))⋯(f(n+0)−f(0))]~+f(0) = bmatrix(f(z^1+z^0)-f(z^0))&·s&(f(z^n+z^0)-f(z^0)) bmatrix z+f(z^0) (155) =[(f(1+0)−f(0))⋯(f(n+0)−f(0))]−1(−0)+f(0) = bmatrix(f(z^1+z^0)-f(z^0))&·s&(f(z^n+z^0)-f(z^0)) bmatrixB^-1(z-z^0)+f(z^0) (156) =(−0)+f(0) =a(z-z^0)+f(z^0) (157) =+f(0)−0⏟b:= =az+ f(z^0)-az^0_b:= (158) =+b. =az+b\,. (159) Since ∈∪j∈[J]jz∈ _j∈[J]Z_j was arbitrary, the above equality holds for all z in =∪j∈[J]jZ= _j∈[J]Z_j. This concludes the proof. ∎ It is easy to construct examples of collections of subspaces 1,…,JZ_1,…,Z_J satisfying the condition of Prop. B.19. One can simply choose some basis 1,⋯nz^1,·sz^n of ℝnR^n and construct each jZ_j as the span of some subset of this basis. In particular, one can use the standard basis 1,…,ne^1,…,e^n, which will yield axis-aligned subspaces of ℝnR^n. We provide a counter-example in Appendix˜E. Then by combining Ass. 3.6 with Theorem˜B.14, we can directly conclude the identifiability up to affine transformation in the following theorem. See 3.7 Proof. Due to Theorem˜3.5, we have ∘g is affine restricted on 1,…,JZ_1,...,Z_J individually. In addition, since we assume 1,…,JZ_1,...,Z_J satisfying Ass. 3.6, by applying Prop. B.19, we can conclude ∘g is affine on =∪j=1JjZ= _j=1^JZ_j. ∎ B.2.3 From affine identifiability to element-wise identifiability via sparsity principle In this section, we take a step further to investigate the possibility of achieving a stronger level of identifiability—namely, up to permutation and scaling indeterminacies only. We first introduce the following assumption, which is a special case of Ass. 3.6 that assumes zero translation vector and standard common basis of ℝnR^n. See 3.8 We then adapt a key argument due to Lachapelle et al. (2023b) in the context of sparse multitask learning and later adapted by Xu et al. (2024) in the context of partial observability. In our case, the masked latent variables exactly follow a piecewise-degenerate Gaussian mixture model (pdGMM) with supports that share a common standard basis. The adapted result is formalized as follows: Lemma B.20 (Element-wise Identifiability from sparsity (Lachapelle et al., 2023b; Xu et al., 2024)). Assume that the latent variables Z with support Z follows a pdGMM (Def. 2.2). The supports of its components satisfy Ass. 3.8. Let the function :→ℝnv:Z ^n be invertible and affine on Z, and ‖()‖0 (Z) _0 ≤‖0. _0. (160) Then v is a permutation composed with an element-wise invertible affine transformation on Z. Since from Theorem˜3.7 the mixing process is successfully reduced from the piecewise affine to affine, we can directly apply Lemma B.20 on this results under Ass. 3.8 to conclude the following theorem. See 3.9 Proof. Since satisfying Ass. 3.8 implies that Ass. 3.6 holds, by Theorem˜3.7, we have ∘g is affine and invertible on Z. By Lemma. B.20, we can conclude ∘g is a permutation composed with an element-wise invertible affine transformation on Z. ∎ Appendix C Implementation details and Licences This section provides further details about the experiment implementation in Section˜4. The implementation is built on the open-source code by Lachapelle et al. (2022) released under Apache 2.0 License; the code by Ahuja et al. (2022) released under MIT License; the code by Zheng et al. (2018) released under Apache 2.0 License. We list all the hyperparameters we used for training in Table˜2, and provide all the code for our method, and the experiments at GitHub repository. Table 2: Parameters for experiments results in Sec. 5 and App. D. Numerical Numerical Balls Balls Balls Stage 1 Stage 2 CNN Stage 1 Stage 2 Sparsity level ϵε 0.01 0.01 Optimizer Adam ExtraAdam Adam Adam ExtraAdam Primal optimizer learning rate 1e-4 1e-4 1e-3 1e-3 1e-4 Dual optimizer learning rate 5e-5 5e-5 Batch size 6144 6144 256 256 256 Group number J 5n5n 5n5n 22^b 22^b 22^b # Seeds 5 5 5 5 5 # Iterations 5000 80000 20000 10000 5000 Appendix D Full experimental results D.1 Metrics Metric for identifiability up to affine transformation R2R^2 metric is the coefficient of determination of the linear regression model with ground truth latents Z as explanatory and our learned representations ()g(X) as response. R2R^2 normally ranges from 0 to 1, and the closer it is to 1, the better its affine identifiability. Metric for identifiability up to permutation and scaling. MCC metric is based on the Pearson correlation matrix Corrn×nCorr^n× n between the learned representations ^=() Z=g(X) and ground truth latent variables Z. Since our results are up to permutation, we compute the MCC on the permutation π that maximizes the average of |Corri,π(i)||Corr_i,π(i)| for each index of a ground truth variable i∈[n]i∈[n], i.e. MCC=1nmaxπ∈perm([n])∑i=1n|Corri,π(i)| 1n _π∈ perm([n]) _i=1^n|Corr_i,π(i)|. We denote the correlation matrix with the permutation π as Corrπn×nCorr^n× n_π. MCC can take value in [0,1][0,1], and the closer it is to 1, the better element-wise affine identifiability. D.2 Numerical experiments We first show the full version (including standard deviation) of Table˜1. Table 3: Results for the numerical experiments (Average ± Standard Deviation). The bold font indicates which parameters are varying in each block of rows. 2R^2(Stage 1) shows the identifiability up to AT after Stage 1 (Thm. 3.7); MCC(Stage 1) reflects the neccesarity of sparsity to achieve PTS identifiability; MCC(Stage 2) presents the identifiability up to PTS after Stage 2 (Thm. 3.9). n k m ρ δ θ 2R^2 ↑ (Stage1) MCC (Stage1) MCC (Stage1) MCC ↑ (VaDE) 5 1 10 50% 0 0 0.93± 0.02 0.75± 0.03 0.96± 0.01 ‘ 0.45± 0.05 10 1 10 50% 0 0 0.94± 0.02 0.56± 0.03 0.97± 0.01 0.32± 0.03 20 1 10 50% 0 0 0.93± 0.02 0.46± 0.01 0.92± 0.05 0.23± 0.15 40 1 10 50% 0 0 0.94± 0.01 0.37± 0.01 0.94± 0.03 0.25± 0.11 10 0 10 50% 0 0 0.94± 0.02 0.51± 0.11 0.97± 0.01 0.33± 0.05 10 1 10 50% 0 0 0.94± 0.02 0.56± 0.03 0.97± 0.01 0.32± 0.03 10 2 10 50% 0 0 0.93± 0.01 0.58± 0.01 0.95± 0.01 0.31± 0.03 10 3 10 50% 0 0 0.92± 0.03 0.56± 0.02 0.91± 0.04 0.27± 0.03 10 1 3 50% 0 0 0.99± 0.01 0.50± 0.07 0.96± 0.03 0.33± 0.04 10 1 10 50% 0 0 0.94± 0.02 0.56± 0.03 0.97± 0.01 0.32± 0.03 10 1 20 50% 0 0 0.88± 0.02 0.50± 0.07 0.93± 0.01 0.35± 0.07 10 1 10 1 var 0 0 0.75± 0.03 0.46± 0.02 0.45± 0.04 0.28± 0.03 10 1 10 50% 0 0 0.94± 0.02 0.56± 0.03 0.97± 0.01 0.32± 0.03 10 1 10 75% 0 0 0.95± 0.02 0.57± 0.03 0.93± 0.04 0.31± 0.04 10 1 10 50% 0 0 0.94± 0.02 0.56± 0.03 0.97± 0.01 0.32± 0.03 10 1 10 50% 1 0 0.95± 0.01 0.59± 0.01 0.88± 0.04 0.30± 0.02 10 1 10 50% 2 0 0.96± 0.01 0.59± 0.03 0.61± 0.04 0.27± 0.03 10 1 10 50% 3 0 0.96± 0.01 0.58± 0.01 0.49± 0.02 0.28± 0.03 10 1 10 50% 0 0 0.94± 0.02 0.56± 0.03 0.97± 0.01 0.32± 0.03 10 1 10 50% 0 15 0.93± 0.02 0.58± 0.03 0.83± 0.02 0.32± 0.03 10 1 10 50% 0 30 0.93± 0.02 0.57± 0.03 0.60± 0.02 0.31± 0.04 10 1 10 50% 0 45 0.93± 0.02 0.57± 0.03 0.62± 0.01 0.32± 0.03 We show the Pearson Correlation matrix Corrπn×nCorr^n× n_π with the permutation π between ground truth latent variables Z and the learned representations ^=() Z=g(X). One figure represents one ablation study in Table˜1. We choose one random seed to plot for each setup. Ground truth Pear. Corr. matrix on the left shows the original linear correlation inside Z, compared with the estimator on the right-hand side. Figure 4: Ablation study on increasing the latent dimension n from 5 to 40 and fixing k=1k=1, m=10m=10, ρ=50%ρ=50\%, δ=0.0σδ=0.0σ, θ=0θ=0. The case with a higher n is more complicated to learn the latent variables. Figure 5: Ablation study on increasing the density of causal graphs k from 0 to 3 and fixing n=10n=10, m=10m=10, ρ=50%ρ=50\%, δ=0.0σδ=0.0σ, θ=0θ=0. In the case with a denser graph, it is more complicated to learn the latent variables. Figure 6: Ablation study on increasing the number of Leaky-ReLU layers (m−1)(m-1) from 3 to 20 and fixing n=10n=10, k=1k=1, ρ=50%ρ=50\%, δ=0.0σδ=0.0σ, θ=0θ=0. The case with a higher n is more complicated to learn the latent variables. The case with larger m is more complicated to learn the latent variables due to a greater extent of nonlinearity. Figure 7: Piecewise linear mixing function with linear causal relation and Gaussian noise: ablation study on increasing the ratio of active (unmasked) variables ρ from 1 variable only to 75% and fixing n=10n=10, k=1k=1, m=10m=10, δ=0.0σδ=0.0σ, θ=0θ=0. Learning the latent variables is more complicated in the case with a larger portion of active variables. Figure 8: Ablation study on increasing the distance between masked value and mean of latents from 0.0σ0.0σ to 3.0σ3.0σ, where σ is the standard deviation of latents, and fixing n=10n=10, k=1k=1, m=10m=10, ρ=50%ρ=50\%, θ=0θ=0. Disentangling the latent variables is more complicated in the case of a larger deviation from zero translation. Figure 9: Ablation study on increasing the rotation of common basis from 0.0∘0.0 to 45.0∘45.0 , and fixing n=10n=10, k=1k=1, m=10m=10, ρ=50%ρ=50\%, δ=0.0σδ=0.0σ. Disentangling the latent variables is more complicated in the case of a larger deviation from the standard basis. D.3 Test on independent variables To exclude the natural dependency between latent variables that potentially increases the value of metric MCC that might cause a false positive measurement, we provide additional experiment results with independent latent variables using five random seeds in Table˜4. As we can see, it keeps the consistent performance as those in Table˜1, where we repeat the results in the right column of Table˜4 for easier comparison. This provides evidence that the MCC obtained by our method does not come from the inherent dependencies among latents. Table 4: Results for the numerical experiments (Stage 2) tested on independent data. The bold font indicates which parameters are varying in each block of rows. n k m ρ δ θ MCC Independent MCC in Tab. 1 5 1 10 50 % 0 0 0.96± 0.01 0.96± 0.01 10 1 10 50 % 0 0 0.97± 0.01 0.97± 0.01 20 1 10 50 % 0 0 0.93± 0.05 0.92± 0.05 40 1 10 50 % 0 0 0.94± 0.03 0.94± 0.03 10 0 10 50 % 0 0 0.97± 0.01 0.97± 0.01 10 1 10 50 % 0 0 0.97± 0.01 0.97± 0.01 10 2 10 50 % 0 0 0.97± 0.01 0.95± 0.01 10 3 10 50 % 0 0 0.91± 0.04 0.91± 0.04 10 1 3 50 % 0 0 0.96± 0.03 0.96± 0.03 10 1 10 50 % 0 0 0.97± 0.01 0.97± 0.01 10 1 20 50 % 0 0 0.94± 0.02 0.93± 0.01 10 1 10 1var 0 0 0.46 ± 0.04 0.45± 0.04 10 1 10 50 % 0 0 0.97± 0.01 0.97± 0.01 10 1 10 75 % 0 0 0.93 ± 0.03 0.93± 0.04 10 1 10 50 % 0 0 0.97± 0.01 0.97± 0.01 10 1 10 50 % 1 0 0.87± 0.04 0.88± 0.04 10 1 10 50 % 2 0 0.61± 0.04 0.61± 0.04 10 1 10 50 % 3 0 0.49± 0.02 0.49± 0.02 10 1 10 50 % 0 0 0.96± 0.03 0.97± 0.01 10 1 10 50 % 0 15 0.84 ± 0.01 0.83± 0.02 10 1 10 50 % 0 30 0.61± 0.01 0.60± 0.02 10 1 10 50 % 0 45 0.63± 0.01 0.62± 0.01 D.4 Empirical study on scalability As shown in Tab. 2, Stage 1 only need 5000 epoch whereas Stage 2 need 80000. Consequently, the second stage of optimization constitutes the primary computational bottleneck of our method. To quantify scalability, we report both the number of epochs needed for convergence and the average execution time per 1000 epochs in seconds for the second stage. We can observe an exponential complexity increase in the table: Table 5: Number of epochs for convergence and training time n 5 10 20 40 50 Epoch 2,500 20,000 125,000 300,000 800,000 Execution time per 1000 epoch in Stage2 (s) 13.4 20.2 45.6 145.4 219.7 In particular we report results for n=50n=50 and limited results for n=100n=100. For comparisons, few works in the CRL literature are able to scale reliably to this number of latent variables, and most of the methods typically focus on 10-20 variables. For n=50n=50, we got MCC=0.93±0.03MCC=0.93± 0.03 averaged over 3 random seeds. For n=100n=100, full convergence of the second stage was not reached. However, after 15,000 epochs of the first stage, we observed an average R2=0.82±0.13R^2=0.82± 0.13, indicating a meaningful affine identifiability which validates Theorem˜3.7. D.5 Sensitivity Analysis of Hyper-parameters We performed a sensitivity analysis on two hyperparameters: the sparsity level ϵε in Equation (5), which controls the sparsity level, and the learning rate. Bold options in the tables are the ones we used in the main paper, where ϵ=1e−2ε=1e-2 is inspired by Xu et al. (2024), which shows that the results are not sensitive to ϵε, but they are sensitive to the learning rate. Table 6: Sensitivity analysis of sparsity level ϵε in Eq. (5): Average MCCs over 5 random seeds (Setup: n=10n=10, k=1k=1, m=10m=10, ρ=50ρ=50, δ=0δ=0, θ=0θ=0) ϵε 1000 100 10 1 1e-1 MCC 0.50± 0.01 0.50± 0.01 0.50± 0.01 0.50± 0.01 0.90± 0.08 ϵε 1e-2 1e-3 1e-4 1e-5 MCC 0.97± 0.01 0.94± 0.03 0.90± 0.07 0.88± 0.07 The final hyperparameters we used in the experiments are chosen based on the MCC and R2R^2 computed with the ground truth variables. If we did not know the ground truth, tuning the parameters would be more difficult, but in line with other DL approaches. For example, we can choose the learning rate as usual in DL by observing the behavior of the loss, while for the sparsity level ϵε we can choose it based on potential violations of our assumptions in the reconstructed variables, e.g., if the reconstructed variables are non-Gaussian. Table 7: Sensitivity analysis of learning rate: Average MCCs over 5 random seeds Learning rate 1e-2 1e-3 1e-4 1e-5 MCC 0.25± 0.03 0.53± 0.08 0.97± 0.01 0.92± 0.02 D.6 Ablation study on sparsity constraint and comparison with close works We ran the experiments on all settings without a sparsity constraint in the second stage. As summarized Table˜8, the significant drop in MCC shows the important role that sparsity plays in the disentanglement process. Additionally, we compared two works (Xu et al., 2024; Kivva et al., 2022) which are closely related to our work, where Xu et al. (2024) serves as an oracle since it provides both group information and degenerate dimension, which is learned in our method. Xu et al. (2024) can be seen as a special case which can be explained in our theorem and since it provides stronger information so we do not aim to outperform it. Kivva et al. (2022) only works for non-degenerate cases, which shows the importance of our method to solve the unexplored problem in the degenerate latent space. Table 8: Ablation study without sparsity constraints and baselines: Average MCCs over 5 random seeds. Ours = proposed method, WO = without sparsity constraints, Oracle (Xu et al., 2024), VaDE (Kivva et al., 2022) n 5 10 20 40 Ours 0.96± 0.01 0.97± 0.01 0.92± 0.05 0.94± 0.03 WO 0.57± 0.02 0.57± 0.04 0.44± 0.01 0.36± 0.01 Oracle 0.91± 0.05 0.98± 0.00 0.96± 0.01 0.94± 0.02 VaDE 0.45± 0.05 0.32± 0.03 0.23± 0.15 0.25± 0.11 (a) Varying n k 0 1 2 3 Ours 0.97± 0.01 0.97± 0.01 0.95± 0.01 0.91± 0.04 WO 0.56± 0.04 0.57± 0.04 0.54± 0.01 0.55± 0.02 Oracle 0.98± 0.01 0.98± 0.00 0.95± 0.04 0.95± 0.03 VaDE 0.33± 0.05 0.32± 0.03 0.31± 0.03 0.27± 0.03 (b) Varying k m 3 10 20 Ours 0.96± 0.03 0.97± 0.01 0.93± 0.01 WO 0.55± 0.01 0.57± 0.04 0.56± 0.02 Oracle 0.99± 0.00 0.98± 0.00 0.95± 0.01 VaDE 0.33± 0.04 0.32± 0.03 0.35± 0.07 (c) Varying m ρ 1 50 75 Ours 0.45± 0.04 0.97± 0.01 0.93± 0.04 WO 0.42± 0.03 0.57± 0.04 0.56± 0.03 Oracle 0.86± 0.04 0.98± 0.00 0.98± 0.01 VaDE 0.28± 0.03 0.32± 0.03 0.31± 0.04 (d) Varying ρ δ 0 1 2 3 Ours 0.97± 0.01 0.88± 0.04 0.61± 0.04 0.49± 0.02 WO 0.57± 0.04 0.54± 0.02 0.55± 0.04 0.55± 0.03 Oracle 0.98± 0.00 0.73± 0.16 0.41± 0.04 0.38± 0.02 VaDE 0.32± 0.03 0.30± 0.02 0.27± 0.03 0.28± 0.03 (e) Varying δ θ 0 15 30 45 Ours 0.97± 0.01 0.83± 0.02 0.60± 0.02 0.62± 0.01 WO 0.57± 0.04 0.56± 0.03 0.58± 0.04 0.56± 0.02 Oracle 0.98± 0.00 0.85± 0.01 0.62± 0.02 0.63± 0.01 VaDE 0.32± 0.03 0.32± 0.03 0.31± 0.04 0.32± 0.03 (f) Varying θ D.7 Comparison with Baseline VaDE without misspecification We provide results where we evaluate our method in a setting that is fair to the baseline VaDE, where we generate the latent with a non-degenerate GMM. In this setting, we can only compare the identifiability up to ATAT measured by metric R2R^2, but we cannot compare for PSPS for any of the two methods, because our method needs degeneracy, and VaDE needs an auxiliary variable and pairwise conditional independence. As shown below, our method outperforms VaDE for all settings. Table 9: Results for the numerical experiments (Stage 1) on nondegenerate GMM data. The bold font indicates which parameters are varying in each block of rows. n k m ρ δ θ MCC of VaDE MCC of Our Method 5 1 10 50 % 0 0 0.66± 0.09 0.94± 0.01 10 1 10 50 % 0 0 0.78± 0.05 0.94± 0.01 20 1 10 50 % 0 0 0.86± 0.04 0.93± 0.01 40 1 10 50 % 0 0 0.84± 0.07 0.93± 0.01 10 0 10 50 % 0 0 0.86± 0.05 0.95± 0.01 10 1 10 50 % 0 0 0.78± 0.05 0.94± 0.01 10 2 10 50 % 0 0 0.74± 0.12 0.94± 0.01 10 3 10 50 % 0 0 0.69± 0.11 0.93± 0.02 10 1 3 50 % 0 0 0.78± 0.10 0.99± 0.00 10 1 10 50 % 0 0 0.78± 0.05 0.94± 0.01 10 1 20 50 % 0 0 0.70± 0.10 0.89± 0.01 10 1 10 1var 0 0 0.77 ± 0.05 0.94± 0.01 10 1 10 50 % 0 0 0.78± 0.05 0.94± 0.01 10 1 10 75 % 0 0 0.75 ± 0.08 0.94± 0.02 10 1 10 50 % 0 0 0.78± 0.05 0.94± 0.01 10 1 10 50 % 0 15 0.77 ± 0.01 0.94± 0.01 10 1 10 50 % 0 30 0.78± 0.02 0.94± 0.02 10 1 10 50 % 0 45 0.78± 0.02 0.94± 0.01 D.8 Over-parameterization of latent dimension n In our previous analysis, the latent dimension n is treated as fixed. To the best of our knowledge, prior work typically assumes n is known in theoretical analysis, but then relies on over-parameterization strategies in empirical analysis. We also consider an over-parameterized choice and show empirically that our method performs well under these settings. While a formal proof of identifiability for remains open, our experimental findings suggest that moderate over-specification does not undermine identifiability guarantees in practice. We use our basic setup: n=10n=10, k=1k=1, m=10m=10, ρ=50ρ=50, δ=0δ=0, θ=0θ=0, varying estimation n^∈12,14,16,18,20 n∈\12,14,16,18,20\. Table 10: Average R2 and MCC over 5 random seeds n n 12(n+2) 14(n+4) 16(n+6) 18(n+8) 20(n+10) R2 0.91 0.89 0.88 0.88 0.83 MCC 0.96 0.96 0.93 0.96 0.96 D.9 Causal graph learning analysis The aim of causal representation learning (CRL) is to recover latent causal variables from high-dimensional observations. Once these causal variables are recovered, one can learn the equivalence class of the causal graph on these variables, e.g. by applying a standard causal discovery method on top of these recovered variables. We provide results in which we apply a standard causal discovery method, PC (Spirtes et al., 2000), on our representations. We consider the same setting as in Table˜3. We generate 50005000 samples for each graph. We use the implementation from the pcalg package and use partial correlation tests with a significance threshold α=0.01α=0.01. In Tab. 11, we report the average SHD on the CPDAG learned over our z w.r.t. the oracle CPDAG (the ground truth CPDAG with oracle tests). We report the average SHD on the CPDAG learned over the ground truth variables z w.r.t. the oracle CPDAG. This is the best result we could obtain with a perfect reconstruction of the ground truth variables. We report the delta between them as an alternative measure of the accuracy of our reconstruction of the causal variables. Table 11: The average SHD on the CPDAG learned over our z and z w.r.t. the oracle CPDAG. n k m ρ δ θ SHD (z) SHD ( z) Δ 5 1 10 50 % 0 0 1.4± 1.67 3.8± 1.79 2.4 10 1 10 50 % 0 0 1.8± 1.79 4.2± 1.10 2.4 20 1 10 50 % 0 0 7.8± 2.86 22± 12.90 14.2 40 1 10 50 % 0 0 13.6± 4.04 39.8± 17.75 26.2 10 1 10 50 % 0 0 1.8± 1.79 4.2± 1.10 2.4 10 2 10 50 % 0 0 13.2± 2.17 16± 2.64 2.8 10 3 10 50 % 0 0 25± 1.87 26.6± 3.85 1.6 10 1 3 50 % 0 0 1.8± 1.79 3± 1.22 1.2 10 1 10 50 % 0 0 1.8± 1.79 4.2± 1.10 2.4 10 1 20 50 % 0 0 1.8± 1.79 8.8± 6.05 7 10 1 10 1var 0 0 1.8± 1.79 21.4± 3.05 19.6 10 1 10 50 % 0 0 1.8± 1.79 4.2± 1.10 2.4 10 1 10 75 % 0 0 1.8± 1.79 7.4± 4.04 5.6 10 1 10 50 % 0 0 1.8± 1.79 4.2± 1.10 2.4 10 1 10 50 % 1 0 1.8± 1.79 16.8± 5.12 15 10 1 10 50 % 2 0 1.8± 1.79 21.8± 2.59 20 10 1 10 50 % 3 0 1.8± 1.79 22.2± 2.68 20.4 10 1 10 50 % 0 0 1.8± 1.79 4.2± 1.10 2.4 10 1 10 50 % 0 15 2.2± 1.24 18± 2.83 15.8 10 1 10 50 % 0 30 2.6± 2.41 20.8± 1.64 18.2 10 1 10 50 % 0 45 4.5± 0.71 21.4± 2.30 16.9 These results show that our learned causal graph is similar to the learned graphs on the ground truth variables. D.10 Image dataset: Multiple Balls Data generation process We use PyGame to render images with size 64×64×364× 64× 3 as shown in Figure˜10. The number of balls b varies from 2 to 6. In this setting, the latent variables we consider are the (x,y)(x,y) coordinates of each ball with support [0,1]2[0,1]^2. We sample the (x,y)(x,y) coordinates of each ball i∈[]i∈[b] independently from a truncated 2-dimensional normal distribution (i,i)N( μ_i, _i) bounded within the square (0.05,0.95)2(0.05,0.95)^2 with randomized i μ_i and fixed i _i: i∼Unif(0.3,0.7)2 μ_i (0.3,0.7)^2, and i=((0.01,0.00)(0.00,0.01)) _i=((0.01,0.00)(0.00,0.01)). To avoid the high ratio of occultation, we use rejection sampling to sample i μ_i to ensure the Euclidean distance between any pair of i μ_i values is at least 0.20.2. To model the degeneracy in this setting, we set each ball at a fixed position i μ_i with probability p=0.1p=0.1 by sampling a binomial variable to indicate the degeneracy status. Figure 10: Example images of multiple ball dataset, modified based on the rendering code provided by Ahuja et al. (2022). Balls can move freely inside the frame, and each ball’s coordinates can be fixed randomly to a constant value, which causes their latent dimensions to be degenerate; Varying the number of balls from 2 to 6. CNN architecture Before we train the model described in Sec. 4, we pre-train a CNN model to reduce the input image dimension from 64×64×364× 64× 3 to a vector of size 128×1128× 1. To control the information loss, we use a symmetric convolutional autoencoder to reconstruct the images. All convolutional layers use kernel size 3 with stride 1 and padding 1, and a ReLU activation follows each convolutional or transposed convolutional layer, except for the final output layer which uses a Sigmoid activation function. We list the concrete network structure in Table˜12. Table 12: Architecture of the convolutional autoencoder used in Sec. 5 (input size: 3×64×643× 64× 64). Layer Type Configuration Output Shape Input RGB image (3, 64, 64) Encoder Conv2D + ReLU 3 → 32, kernel=3, stride=1, pad=1 (32, 64, 64) MaxPool2D 2×2 (32, 32, 32) Conv2D + ReLU 32 → 64, kernel=3, stride=1, pad=1 (64, 32, 32) MaxPool2D 2×2 (64, 16, 16) Conv2D + ReLU 64 → 64, kernel=3, stride=1, pad=1 (64, 16, 16) MaxPool2D 2×2 (64, 8, 8) Conv2D + ReLU 64 → 128, kernel=3, stride=1, pad=1 (128, 8, 8) MaxPool2D 2×2 (128, 4, 4) Conv2D + ReLU 128 → 128, kernel=3, stride=1, pad=1 (128, 4, 4) MaxPool2D 4×4 (128, 1, 1) Decoder ConvTranspose2D + ReLU 128 → 128, kernel=2, stride=2 (128, 2, 2) ConvTranspose2D + ReLU 128 → 64, kernel=2, stride=2 (64, 4, 4) ConvTranspose2D + ReLU 64 → 64, kernel=2, stride=2 (64, 8, 8) ConvTranspose2D + ReLU 64 → 32, kernel=2, stride=2 (32, 16, 16) ConvTranspose2D + Sigmoid 32 → 3, kernel=4, stride=4 (3, 64, 64) To evaluate the performance, we perform linear regression from each representation separately to the corresponding ground-truth ball (x,y)-position. For =2,4,6b=2,4,6, we provide the average R2R^2 over 5 random seeds for each ball in Tables˜13, 14 and 15. The results show that for the settings =2,4b=2,4, the position of each ball can be well recovered through pairs of representations, which is consistent with our theoretical results. However, as shown in the example images, when =6b=6, even though we use rejection sampling to ensure the distance between the mean values of any pair of balls is at least 0.2, due to the limited space of the frame, a high portion of occultation cannot be avoided, which also cause a performance drop as shown in Table˜15. Table 13: Results for Multiple Balls with =2b=2: averaged R2R^2 of linear regression between ground truth ball positions and learned representations. Ball 1 Ball 2 MCC 0.958 ± 0.052 0.914 ± 0.148 Table 14: Results for Multiple Balls with =4b=4: averaged R2R^2 of linear regression between ground truth ball positions and learned representations. Ball 1 Ball 2 Ball 3 Ball 4 MCC 0.953 ± 0.033 0.853 ± 0.194 0.932 ± 0.041 0.880 ± 0.137 Table 15: Results for Multiple Balls with =6b=6: averaged R2R^2 of linear regression between ground truth ball positions and learned representations. Ball 1 Ball 2 Ball 3 MCC 0.743 ± 0.106 0.105 ± 0.046 0.293 ± 0.229 Ball 4 Ball 5 Ball 6 MCC 0.407 ± 0.375 0.474 ± 0.183 0.595 ± 0.241 D.11 Identify x−x- and y−y-positions with fined-grained masks we can define finer-grained sparsity patterns. In particular, our method can disentangle the x−x- and y−y-positions of each ball, if the dataset allows freezing x−x- and y−y-positions individually. We ran experiments on 5 random seeds with 2 balls (in total 4 latent variables), and got an average MCC = 0.97±0.030.97± 0.03. In Table˜16, we show that the R2R^2 of linear regression between ground truth ball positions 1xb_1^x, 1yb_1^y, 2xb_2^x, 2yb_2^y, and learned representations ^1 z_1,…,^4 z_4. As we can see, both x−x- and y−y- can be disentangled well. Table 16: Example results of z^i z_i assignments across blocks 1xb_1^x 1yb_1^y 2xb_2^x 2yb_2^y ^1 z_1 0.03 0.00 0.98 0.00 ^2 z_2 0.95 0.01 0.01 0.00 ^3 z_3 0.00 0.00 0.00 0.97 ^4 z_4 0.02 0.99 0.00 0.01 D.12 Compute information The experiments in Section˜5 are implemented on NVIDIA A100 GPU. The experiments in Section˜5 on multiple balls are implemented on NVIDIA A40 GPU. Each job used 1 GPU per node and 20 GB of memory. Appendix E Discussion on Assumptions In this section, we assess the applicability and robustness of our assumptions and discuss to what extent they hold or may fail in practical scenarios. We begin with empirical evidence on violations of Gaussianity and piecewise affine mixing, followed by a closer examination of each theoretical assumption, supported by proofs and illustrative examples. E.1 Parametric assumptions: Gaussian latent and piecewise affine mixing Mixture models capture a significantly broader range of latent structures than standard single-component (e.g., standard Gaussian) priors used in many prior works (e.g. VAE and β VAE). In fact, mixture models, especially GMMs, are commonly used to approximate real-world latent spaces in applications. Our use of a degenerate mixture is especially relevant in high-dimensional sparse regimes, where not all latent variables are active in every sample. The piecewise affine mixing function assumption aligns with a broad class of practical transformations (e.g., ReLU networks). Moreover, it can approximate any diffeomorphism functions up to arbitrary precision given a sufficient number of pieces. In addition, to test the robustness of our method, we report some results for violations of these assumptions in Table˜17. We experiment with the violation of 1) the Gaussian assumption (replaced by Exponential and Gumbel), and 2) the piecewise affine (replace LeakyReLU by 1. smooth LeakyReLU 2. Sigmoid ). Except for the Sigmoid setting, we did not observe a performance drop, indicating the robustness of our method under certain violations of our assumptions. Table 17: Results for the numerical experiments (Stage 2) on misspecified settings. n k m ρ MCC Exponential Gumbel Smooth LeakyReLU Sigmoid 5 1 10 50 % 0.92± 0.06 0.95± 0.01 0.97± 0.01 0.36± 0.02 10 1 10 50 % 0.96± 0.01 0.97± 0.01 0.98± 0.01 0.32± 0.06 20 1 10 50 % 0.94± 0.02 0.95± 0.02 0.94± 0.09 0.29± 0.04 10 0 10 50 % 0.95± 0.01 0.97± 0.01 0.95± 0.08 0.37± 0.06 10 1 10 50 % 0.96± 0.01 0.97± 0.01 0.98± 0.01 0.32± 0.06 10 2 10 50 % 0.93± 0.04 0.96± 0.02 0.97± 0.01 0.29± 0.06 10 3 10 50 % 0.91± 0.04 0.91± 0.06 0.93± 0.03 0.36± 0.09 10 1 3 50 % 0.98± 0.01 0.99± 0.01 0.99± 0.00 0.60± 0.03 10 1 10 50 % 0.96± 0.01 0.97± 0.01 0.98± 0.01 0.32± 0.06 10 1 20 50 % 0.91± 0.04 0.94± 0.02 0.97± 0.01 0.30± 0.08 10 1 10 1var 0.66± 0.08 0.59± 0.07 0.70 ± 0.07 0.36± 0.01 10 1 10 50 % 0.96± 0.01 0.97± 0.01 0.98± 0.01 0.32± 0.06 10 1 10 75 % 0.92± 0.04 0.94± 0.04 0.95 ± 0.03 0.32± 0.07 E.2 Assumption˜3.4: Genericity of pdGMMs In this section, we clarify the genericity of Assumption˜3.4. We begin by providing a proof that the set of parameters violating this assumption is a measure-zero subset of the parameter space under consideration, i.e., even when we consider only covariance matrices that are degenerate. Then, we introduce an example in two-dimensional space to support and illustrate the intuition behind it. We also extend this example to higher dimensions and conduct an ablation study to show the necessity of this assumption. Finally, we discuss the applicability of this assumption in practice. See 3.4 Lemma E.1. Consider a pdGMM in reduced form ∼∑j=1JλjN(j,j)Z _j=1^J _jN( μ_j, _j). The set of parameters (λj,j,j),J∈[J]( _j, μ_j, _j),J∈[J] violating Assumption˜3.4 is measure-zero in the parameter space under consideration, i.e., when we consider j _j to be degenerate covariance matrices. Proof. A violation of Assumption˜3.4 means for some 0<k≤n0<k≤ n, there exists an subset k0∈kJ_k^0 _k, such that all points 0∈∩j∈k0jz_0∈ _j _k^0Z_j, ∃i,j∈k0∃\,i,j _k^0 such that the Mahalonobis distance are the same, i.e., (0−i)Ti−1(0−i)=(0−j)Tj−1(0−j) (z_0- μ_i)^T _i^-1(z_0- μ_i)= (z_0- μ_j)^T _j^-1(z_0- μ_j). To constrain the parameter space of i _i and j _j to the ones with rank k, we parameterize i=AiAiT _i=A_iA_i^T and j=AjAjT _j=A_jA_j^T via full-column-rank decompositions with rank(Ai)=rank(Aj)=k rank(A_i)= rank(A_j)=k. Therefore, we have (0−i)Ti−1(0−i)=‖(AiTAi)−1AiT(0−i)‖ (z_0- μ_i)^T _i^-1(z_0- μ_i)=||(A_i^TA_i)^-1A_i^T(z_0- μ_i)|| (161) (0−j)Tj−1(0−j)=‖(AjTAj)−1AjT(0−j)‖. (z_0- μ_j)^T _j^-1(z_0- μ_j)=||(A_j^TA_j)^-1A_j^T(z_0- μ_j)||. (162) In this way, we can define the subset of parameter space that violates the assumption for each pair of components i and j. ijk0:=(Ai,Aj,i,j):‖(AiTAi)−1AiT(0−i)‖=‖(AjTAj)−1AjT(0−j)‖∀0∈∩j∈k0jV^J_k^0_ij:=\(A_i,A_j, μ_i, μ_j):||(A_i^TA_i)^-1A_i^T(z_0- μ_i)||=||(A_j^TA_j)^-1A_j^T(z_0- μ_j)|| ∀\,z_0∈ _j _k^0Z_j\ (163) The total set of violating parameters for the pdGMM is =∪k=1n∪k0⊆k∪i,j∈k0ijk0V= _k=1^n _J_k^0 _k _i,j _k^0V^J_k^0_ij. Since a finite union of measure zero sets is still measure zero, if we can show that the measure of ijk0V^J_k^0_ij is 0 for all pairs of components i,j∈k0i,j _k^0, the proof is complete. Let ij0:=‖(AiTAi)−1AiT(0−i)‖−‖(AjTAj)−1AjT(0−j)‖f^z_0_ij:=||(A_i^TA_i)^-1A_i^T(z_0- μ_i)||-||(A_j^TA_j)^-1A_j^T(z_0- μ_j)||. By Mityagin (2020), a real analytic function from ℝmR^m to ℝR for m≥1m≥ 1 is either zero everywhere or non-zero almost everywhere. We can apply this result to our setting and show that ij0f^z_0_ij, which is a real analytic function in (Ai,Aj,i,j)(A_i,A_j, μ_i, μ_j), is non-zero almost everywhere, which means that for each point 0z_0 we can distinguish each pair of components. This means that once we can find one nonzero solution, we can conclude that the function is all nonzero almost everywhere. We construct the nonzero solution in this way: we can put the first term in the definition of ij0f^z_0_ij to zero by taking i=0 μ_i=z_0 and using an arbitrary full-column matrix AiA_i. For the second term, we want to show that there exists a non-zero solution. We fix AjA_j and show that if 0−j∉Im((AjTAj)−1AjT)z_0- μ_j ∈ Im((A_j^TA_j)^-1A_j^T), then ij0f^z_0_ij is nonzero. Thus, we conclude based on the result by Mityagin (2020) that ij0f^z_0_ij is non-zero almost everywhere for any 0z_0. This means that the set of violations ijk0V^J_k^0_ij is a measure-zero set for any pair i,ji,j and therefore its finite union V is a measure-zero set. ∎ Example E.2. Consider n=2n=2 and J=2J=2 with components N(1,1)N( μ_1, _1) and N(2,2)N( μ_2, _2). Suppose 1=(00),1=(1000). μ_1= pmatrix0\\ 0 pmatrix, _1= pmatrix1&0\\ 0&0 pmatrix. (164) There are only two ways to decompose 1 _1 into the form A1A1TA_1A_1^T: A1=(1,0)TA_1=(1,0)^T or (−1,0)T(-1,0)^T. Let 2=A2A2T _2=A_2A_2^T where A2=(a1,a2)TA_2=(a_1,a_2)^T, and 2=(μ1,μ2) μ_2=( _1, _2). Consider 1=∈ℝ _1= , then 2= _2= or −- . To violate Ass. 3.4, (2,2)( μ_2, _2) can must satisfy: =a1+μ10=a2+μ2 or −=a1+μ10=a2+μ2 or =−a1+μ10=−a2+μ2 or −=−a1+μ10=−a2+μ2 cases =a_1 + _1\\ 0=a_2 + _2 cases or cases- =a_1 + _1\\ 0=a_2 + _2 cases or cases =-a_1 + _1\\ 0=-a_2 + _2 cases or cases- =-a_1 + _1\\ 0=-a_2 + _2 cases (165) By solving the equations, we can conclude if and only if (a1−1)μ2=a2μ1(a_1-1) _2=a_2 _1 or |(a1−1)μ2|=|a2μ1||(a_1-1) _2|=|a_2 _1|, Ass. 3.4 is not satisfied, which is a measure-zero subset of parameter space. We extend this example to dimensions n=2,5,10n=2,5,10 and show the results of our method on it in Table˜18. As you can see, these results are much worse than the comparable results in Table˜1 for n=5,10n=5,10, where the R2R^2 was ∼0.93 0.93, and the MCCMCC was ∼0.96 0.96, showing why this assumption is needed. Table 18: Average R2R^2 and MCC over 5 random seeds n 2 5 10 R2R^2 0.87 0.70 0.64 MCC 0.92 0.63 0.47 In summary, we have shown that the violations of this assumption occur only on a measure-zero subset, supporting its genericity. The two-dimensional example further illustrates this result by explicitly characterizing the exceptional cases, which correspond to highly fine-tuned parameter choices. Together, these arguments support the statement that Assumption˜3.4 is a mild and reasonable assumption, holding almost everywhere in real-world settings. E.3 Assumption˜3.6: Common basis and translation vector This assumption states that the supports of all components should intersect and that a shared basis can represent them. This always holds for non-degenerate GMMs. For degenerate components, a violation can happen because the components are not intersecting, or if their individual bases do not form a global shared basis. To show the necessity of Assumption˜3.6, we provide an example that violates this assumption, showing that such cases only local affine identifiability can be achieved, whereas global identifiability fails. Example E.3. Consider n=2n=2 with 1:=span1Z_1:= span\e^1\, 2:=span2Z_2:= span\e^2\ and 3=span1+2Z_3= span\e^1+e^2\. In this case, all three subspaces are one-dimensional and the issue is that the family of vectors 1,2,1+2\e^1,e^2,e^1+e^2\ is linearly dependent and thus do not form a basis. One can construct a function f:∪j=13j→ℝf: _j=1^3Z_j that is affine on each jZ_j individually, but not on the union: f():=0,if∈1∪21,if∈3. f(z):= cases0,\ if\ z _1 _2\\ z_1,\ if\ z _3 cases\,. (166) Clearly this function is not affine on ∪j=13j _j=1^3Z_j since f(1+2)=1≠0=f(1)+f(2). f(e^1+e^2)=1 =0=f(e^1)+f(e^2)\,. (167) Intuitively, Assumption˜3.6 requires that there exists a unique shared basis for all Gaussian components. We conduct an ablation study to show the necessity. To show a violation of this, we generate the data with a mixture of two different bases: a standard basis and one with rotations. As shown in Tab. 19, these results are worse than the comparable results in Tab. 1, which shows why this assumption is needed for AT. Table 19: Average R2R^2 over 5 random seeds k 0 1 2 3 R2R^2 0.83 0.82 0.84 0.85 One concrete setting in which this assumption holds is the partially observable scenario in Xu et al. (2024). In that setting, there are n causal variables, but only a subset of them are “active” in each observation sample, e.g., they are captured in an image, while others are “inactive”. The sets of “active” and “inactive” variables can change for each observation. For example, one could consider a camera that captures objects moving in and out of frame, or are occluded by another object. We consider the “inactive” objects as contributing to the observation with a specific constant value, i.e., their inputs to the mixing functions are constant (e.g. 0). In this setting, the intersection of the supports is the vector of all the constant values for each variable and since we are masking the inputs for the “inactive” variables, the individual basis of the supports will always be orthogonal and hence form a well-defined global shared basis, satisfying Assumption˜3.6. Other possible applications could include other high-dimensional applications that rely on sparse representations, such as language models (Cunningham et al., 2024; Gao et al., 2025; Marks et al., 2025), where we also have “active” and “inactive” concepts. If it does not hold strictly in a setting, e.g., it only holds for a subset of components, for this subset we can still achieve the same result as Theorem˜3.7, but for the other components we will learn a local linear identified representation individually as stated in Theorem˜3.5, but not a global one. E.4 Assumption˜3.8a: Common Standard Basis The only difference between the Common Standard Basis Assumption (Assumption˜3.8 (a)) and Assumption˜3.6 is the interception term, which can be canceled out easily by shifting the modeling by a constant in practice. Table 1 already shows violations of Ass. 3.8a when δ (the masking value) and θ (the rotation of the standard basis) are non-zero, showcasing a decrease in performance when the assumption does not hold. E.5 Assumption˜3.8b: Sufficient Support Basis Index Variability The Sufficient Support Basis Index Variability Assumption (Assumption˜3.8 (b)), it is more likely to hold when the sparsity patterns across samples are sufficiently diverse. For example, in the partially observable CRL setting, this would mean that two variables are not always “inactive” and “active” at the same time, since we would otherwise not be able to disentangle them. Even if this assumption does not hold strictly in a setting, our framework still supports a weaker form of identifiability: block-wise identifiability, which allows disentanglement across blocks of latent variables, even if variables within each block may remain entangled. This relaxation still enables meaningful interpretability and structure in learned representations. We ran experiments with different blocks of variables that were always active/inactive together. We use our basic setup: n=10n=10, k=1k=1, m=10m=10, ρ=50ρ=50, δ=0δ=0, θ=0θ=0 with three types of block to see whether the performance varies with different block sizes: [[0,1],[2,3],[4,5],[6,7],[8,9]][[z_0,z_1],[z_2,z_3],[z_4,z_5],[z_6,z_7],[z_8,z_9]], [[0],[1],[2],[3,4,5],[6,7,8,9]][[z_0],[z_1],[z_2],[z_3,z_4,z_5],[z_6,z_7,z_8,z_9]], [[0],[1],[2,3],[4,5,6,7,8,9]][[z_0],[z_1],[z_2,z_3],[z_4,z_5,z_6,z_7,z_8,z_9]], and we show R2R^2 of the linear regression between each block and learned representations ^1,…,^10 z_1,…, z_10. The results show that even though the variables are entangled within blocks, disentanglement can be achieved across blocks. Table 20: Example results of ^i z_i assignments across blocks [0,1][z_0,z_1] [2,3][z_2,z_3] [4,5][z_4,z_5] [6,7][z_6,z_7] [8,9][z_8,z_9] ^0 z_0 0.95 0.01 0.01 0.02 0.01 ^1 z_1 0.90 0.00 0.00 0.00 0.00 ^2 z_2 0.01 0.84 0.01 0.01 0.00 ^3 z_3 0.00 0.84 0.00 0.00 0.01 ^4 z_4 0.00 0.00 0.91 0.00 0.00 ^5 z_5 0.00 0.00 0.91 0.00 0.01 ^6 z_6 0.00 0.00 0.00 0.86 0.00 ^7 z_7 0.01 0.01 0.00 0.96 0.01 ^8 z_8 0.01 0.00 0.00 0.01 0.91 ^9 z_9 0.00 0.00 0.01 0.02 0.88 Table 21: Example results of ^i z_i assignments across blocks [0][z_0] [1][z_1] [2][z_2] [3,4,5][z_3,z_4,z_5] [6,7,8,9][z_6,z_7,z_8,z_9] ^0 z_0 0.95 0.00 0.00 0.01 0.03 ^1 z_1 0.00 0.89 0.00 0.00 0.00 ^2 z_2 0.01 0.00 0.92 0.01 0.02 ^3 z_3 0.00 0.00 0.00 0.91 0.02 ^4 z_4 0.01 0.00 0.00 0.94 0.01 ^5 z_5 0.00 0.00 0.00 0.92 0.00 ^6 z_6 0.00 0.00 0.00 0.00 0.89 ^7 z_7 0.01 0.00 0.01 0.01 0.96 ^8 z_8 0.01 0.00 0.00 0.01 0.92 ^9 z_9 0.00 0.00 0.00 0.02 0.93 Table 22: Example results of ^i z_i assignments across blocks [0][z_0] [1][z_1] [2,3][z_2,z_3] [4,5,6,7,8,9][z_4,z_5,z_6,z_7,z_8,z_9] ^0 z_0 0.92 0.00 0.01 0.05 ^1 z_1 0.00 0.89 0.00 0.01 ^2 z_2 0.01 0.00 0.91 0.04 ^3 z_3 0.00 0.00 0.88 0.01 ^4 z_4 0.01 0.00 0.00 0.94 ^5 z_5 0.01 0.00 0.01 0.91 ^6 z_6 0.02 0.00 0.00 0.92 ^7 z_7 0.00 0.00 0.00 0.88 ^8 z_8 0.01 0.00 0.00 0.92 ^9 z_9 0.00 0.00 0.00 0.82

Instant research discovery