Paper deep dive
MaDeL: Manifold-Decomposed Feature Losses for Generative Modeling
Beomsu Kim, Jong Chul Ye, Kwanyoung Kim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative models are often trained with isotropic objectives such as mean-squared error. For data concentrated near a low-dimensional manifold, however, such losses conflate displacement along the manifold, which may represent valid variation, with displacement away from it, which produces invalid samples. This mismatch is especially problematic in sparse, highly constrained domains, where ambient-space regression can encourage off-manifold interpolation. We ask whether a generative objective can distinguish manifold-parallel variation from manifold-orthogonal deviation directly from data, without explicitly estimating the manifold. We introduce a manifold-decomposed feature loss (MaDeL) that learns complementary representations from corrupted observations: one is trained to recover the clean sample, while the other is trained to recover the corruption. We show that, under a feature bottleneck, their Jacobians align with the tangent and normal spaces, exactly for linear manifolds and locally for smooth manifolds. Together, these representations define an anisotropic objective that separately measures intrinsic variation and off-manifold deviation. Across synthetic, Earth and climate science, and torsion-angle benchmarks, MaDeL improves support recovery and average angular $W_1$ under single-step sampling; on protein backbones, it reduces steric clashes across one- and few-step sampling budgets.
Tags
Links
- Source: https://arxiv.org/abs/2610.04419v1
- Canonical: https://arxiv.org/abs/2610.04419v1
Trouble viewing inline? Open PDF directly →
Full Text
74,495 characters extracted from source content.
Expand or collapse full text
MaDeL: Manifold-Decomposed Feature Losses for Generative Modeling Beomsu Kim Affiliation: Graduate School of AI, KAIST Email: beomsu.kim@kaist.ac.kr Jong Chul Ye †thanks: Co-corresponding authors. Affiliation: Graduate School of AI, KAIST Email: jong.ye@kaist.ac.kr Kwanyoung Kim22footnotemark: 2 Affiliation: Department of AI, GIST Email: k0.kim@gist.ac.kr Abstract Generative models are often trained with isotropic objectives such as mean-squared error. For data concentrated near a low-dimensional manifold, however, such losses conflate displacement along the manifold, which may represent valid variation, with displacement away from it, which produces invalid samples. This mismatch is especially problematic in sparse, highly constrained domains, where ambient-space regression can encourage off-manifold interpolation. We ask whether a generative objective can distinguish manifold-parallel variation from manifold-orthogonal deviation directly from data, without explicitly estimating the manifold. We introduce a manifold-decomposed feature loss (MaDeL) that learns complementary representations from corrupted observations: one is trained to recover the clean sample, while the other is trained to recover the corruption. We show that, under a feature bottleneck, their Jacobians align with the tangent and normal spaces, exactly for linear manifolds and locally for smooth manifolds. Together, these representations define an anisotropic objective that separately measures intrinsic variation and off-manifold deviation. Across synthetic, Earth and climate science, and torsion-angle benchmarks, MaDeL improves support recovery and average angular W1W_1 under single-step sampling; on protein backbones, it reduces steric clashes across one- and few-step sampling budgets. Figure 1: Overview of MaDeL. (a) A shared encoder splits a corrupted observation into manifold-parallel and manifold-orthogonal features, trained with x and δ-prediction. (b) These learned features define the decomposed generative loss. The resulting gradients produce complementary tangent and normal updates to the data manifold. 1 Introduction Generative models have achieved remarkable fidelity across images, video, molecules, and protein structures (Karras et al., 2022; Ho et al., 2022; Hoogeboom et al., 2022; Yim et al., 2023b). Despite substantial differences in architecture and formulation, many modern approaches share a common training principle: they learn a denoising, score, velocity, or reconstruction target using a largely isotropic regression objective. For example, diffusion and flow-based models commonly minimize an ℓ2 _2 discrepancy between predicted and target quantities (Ho et al., 2020; Song et al., 2021; Lipman et al., 2023; Liu et al., 2023). Such objectives measure how far a prediction is from its target, but are largely agnostic to the direction in which the prediction is wrong. This isotropic treatment is poorly matched to the geometry of natural data. High-dimensional observations are hypothesized to concentrate near a lower-dimensional, structured subset of the ambient space (Tenenbaum et al., 2000; Narayanan and Mitter, 2010; Fefferman et al., 2016; Pope et al., 2021). Locally, perturbations of equal magnitude can have very different consequences depending on their orientation relative to the data manifold (Rifai et al., 2011a; Alain and Bengio, 2014). A displacement parallel to the manifold may correspond to meaningful variation, such as a change in object pose or a valid protein conformation, whereas an orthogonal displacement can produce perceptual artifacts or physically invalid structures. An isotropic loss cannot distinguish between them. This distinction is especially important for generative modeling. Near the data manifold, a generator must both suppress deviations away from valid data and model variation along the manifold. The desired dynamics are therefore inherently anisotropic: orthogonal directions should restore samples toward the manifold, whereas parallel directions should describe variation within it. Recent analyses of diffusion models support this geometric perspective, showing that score functions recover directions associated with the normal bundle of the data manifold at low noise levels (Stanczuk et al., 2024). Ignoring this distinction encourages interpolation through off-manifold, low-density regions. Ideally, a generative objective would therefore measure prediction errors according to the local geometry of the data distribution rather than a fixed isotropic metric. Explicitly estimating tangent and normal spaces for complex high-dimensional data, however, is challenging. Instead, we use learned feature spaces to induce data-dependent notions of similarity, building on their successful use as perceptual losses for image synthesis (Johnson et al., 2016; Zhang et al., 2018). Concretely, given data x, we construct a corrupted observation ~=+ x= x+ δ and learn two complementary representations. Manifold-parallel features are learned by predicting x from ~ x ( x-prediction), encouraging them to preserve intrinsic variation while suppressing corruption. Manifold-orthogonal features are learned by predicting δ from the same observation ( δ-prediction), encouraging them to capture deviations away from the manifold. Together, these representations provide a learned decomposition of errors into manifold-parallel and orthogonal components. We use these representations to define a geometry-aware training objective for generative models: rather than penalizing all prediction errors uniformly, it measures them through feature spaces specialized for intrinsic variation and off-manifold deviation, and thus distinguishes predictions that are equally distant in ambient space but differ in their compatibility with the data distribution (Fig. 1). Specifically, our contributions are: • Theoretical characterization of learned geometry (Secs. 4.1 and 4.2). We show that, under suitable bottlenecks, optimal x-prediction and δ-prediction encoders have Jacobian row spaces contained in the tangent and normal spaces of the data manifold, respectively, exactly for linear manifolds and to leading order locally for smooth manifolds. Their feature losses yield complementary manifold-parallel and manifold-orthogonal gradients. • Manifold-decomposed generative loss (Sec. 4.3). We introduce MaDeL, a geometry-aware feature loss that separately measures manifold-parallel variation and manifold-orthogonal deviation using these representations, without access to the data manifold. • Validation across diverse manifolds (Sec. 5). We evaluate MaDeL on a helix, Earth and climate science data, protein and RNA torsion angles, and protein backbone generation. MaDeL improves support recovery and average angular W1W_1 on the single-step benchmarks and reduces steric clashes in protein backbones across one- and few-step budgets. 2 Background Isotropic regression objectives. Let x_ θ and targ x_targ denote a model prediction and its regression target. Diffusion and Consistency Models (Song et al., 2023; Song and Dhariwal, 2024), for example, are commonly trained with isotropic ambient-space objectives of the form ℒiso()≔[ρ(‖^−targ‖22)], _ ( θ) [ρ(\| x_ θ- x_targ\|_2^2)], (1) where ρ(u)=uρ(u)=u yields mean-squared error (MSE) and ρ(u)=u+c2−cρ(u)= u+c^2-c yields the pseudo-Huber loss. Although these methods construct targ x_targ differently, their losses depend only on the residual magnitude and therefore penalize equal-magnitude errors identically regardless of direction. Geometry-aware generative modeling. Natural data often concentrate near low-dimensional subsets of the ambient space, making their local geometry anisotropic: perturbations of equal magnitude can represent meaningful variation when tangent to the data manifold but invalid deviation when normal to it, a distinction ignored by isotropic regression. Method Geometry Tan. / Nor. Train. RFM (Chen and Lipman, 2024) given ✓/✗ ✓ RMF (Woo et al., 2026) given ✓/✗ ✓ Tango-DM (Liu et al., 2025) given ✓/✗ ✓ SDDM (Sun et al., 2023) pretrained ✓/✓ ✗ MPGD (He et al., 2024) pretrained ✗/✓ ✗ Saito & Matsubara (2026) pretrained ✓/✓ ✗ AYT (Kim et al., 2026) learned ✗/✓ ✓ MaDeL (Ours) learned ✓/✓ ✓ Table 1: Geometry-aware methods: source of geometry, tangent/normal modeling, and use in training. Tab. 1 summarizes how prior work accounts for this geometry. Existing approaches primarily incorporate the geometry of a known state space: Riemannian diffusion models (Huang et al., 2022) and Riemannian Flow Matching (RFM) (Chen and Lipman, 2024) extend generative dynamics to non-Euclidean manifolds, while MeanFlow (MF) (Geng et al., 2025) and Riemannian MeanFlow (RMF) (Woo et al., 2026) enable one-step generation in Euclidean and manifold-valued spaces, respectively, and Tango-DM trains only the tangential score component of a known manifold (Liu et al., 2025). Other work targets the geometry of the embedded data distribution, but extracts it from a pretrained model: low-noise diffusion scores approximate its normal bundle (Stanczuk et al., 2024), SDDM decomposes scores into tangent and normal components (Sun et al., 2023), MPGD constructs manifold-preserving guidance from pretrained autoencoders (He et al., 2024), and the score Jacobian yields a tangent-aware metric (Saito and Matsubara, 2026). In contrast, we learn complementary tangent- and normal-sensitive features from data and place them directly in the training objective, with neither a known manifold nor a pretrained models. Feature losses as learned directional metrics. Learned features introduce anisotropy without an explicit parameterization of the data manifold: perceptual losses measure errors in representations aligned with perceptual similarity (Johnson et al., 2016; Zhang et al., 2018), and regularized autoencoders capture local geometry through denoising, Jacobian contraction, or reconstruction fields (Vincent et al., 2008; Rifai et al., 2011b; Alain and Bengio, 2014). Features have also been incorporated directly into generative training through pretrained discriminator features (Sauer et al., 2021) and feature-based consistency objectives (Song et al., 2023), including manifold-aligned tangents (AYT) (Kim et al., 2026). These approaches, however, provide a single notion of similarity rather than complementary sensitivities to intrinsic variation and off-manifold deviation. Figure 2: A manifold-decomposed metric mitigates off-manifold interpolation. With sparse training samples (crosses), Cartesian MSE and Pseudo-Huber interpolate through the circle, whereas the oracle decomposed metric preserves its support. 3 Motivation Let ϕ:ℝd→ℝn φ:R^d ^n be a feature encoder and define ℒϕ(^,)≔12‖ϕ(^)−ϕ()‖22L_ φ( x, x) 12\| φ( x)- φ( x)\|_2^2. Its gradient is ϕ(^)⊤(ϕ(^)−ϕ()) J_ φ( x) ( φ( x)- φ( x)), so the encoder Jacobian determines the input-space directions to which the loss is sensitive. We learn complementary encoders ϕ∥ φ and ϕ⟂ φ for manifold-parallel variation and manifold-orthogonal deviation, respectively. Writing a∈∥,⟂a∈ , for the two branches, we define ℒ(^,) ( x, x) =∑a∈∥,⟂λaℒϕa(^,), = _a∈ , λ^aL_ φ^a( x, x), (2) ∇^ℒ(^,) _ xL( x, x) =∑a∈∥,⟂λaϕa(^)⊤(ϕa(^)−ϕa()). = _a∈ , λ^a J_ φ^a( x) ( φ^a( x)- φ^a( x) ). (3) If the encoders ϕ∥ φ and ϕ⟂ φ specialize as intended, these terms provide complementary update directions along and away from the data manifold, inducing a learned directional geometry rather than a generic perceptual metric. Sanity check on a toy example. We first isolate the effect of the training metric on a manifold that is known exactly, using an oracle decomposition that the learned features of Sec. 4 are designed to approximate. The target distribution is a mixture of four von Mises distributions on the unit circle in ℝ2R^2. For each dataset size N, identical generators are trained with the same OT-matched objective and differ only in the regression metric; the crosses in Fig. 2 denote the training samples, and N=∞N=∞ denotes training with fresh samples from the underlying distribution. As shown in Fig. 2 left, Cartesian MSE and Pseudo-Huber (PH) interpolate between sparse modes along Euclidean chords and produce off-manifold samples inside the circle. The oracle decomposed metric applies PH in polar coordinates with (λ∥,λ⟂)=(1,5)(λ ,λ )=(1,5), where angle and radius correspond to the manifold-parallel and manifold-orthogonal directions, and preserves the support. Fig. 2 right quantifies this behavior: the decomposed metric reduces the fraction of off-manifold samples to nearly zero across dataset sizes while attaining a comparable angular W1W_1 distance to the target distribution. The improvement therefore arises from the manifold-orthogonal geometry without loss of variation along the manifold, which underscores the role of the training metric. 4 Main Contribution: MaDeL Building on Sec. 3, we show that the two feature spaces arise from x-prediction and δ-prediction on corrupted data, and turn them into a two-stage training procedure. We characterize the optimal encoders exactly for linear manifolds (Sec. 4.1), extend the characterization locally to smooth manifolds (Sec. 4.2), connect the encoder Jacobians to the update directions of the decomposed loss, and describe the shared encoder and generator training (Sec. 4.3). 4.1 Theory: Linear Manifolds and Autoencoders Suppose the data x is supported on an r-dimensional linear subspace of ℝdR^d. Concretely, x =,∼(,),r≔rank()<d, = A z, z ( 0, I), r ( A)<d, (4) such that data x is Gaussian distributed with covariance ≔⊤=diag(λ1,…,λd)⊤, A A = U ( _1,…, _d) U , (5) with eigenvalues λ1≥⋯≥λr>0=λr+1=⋯=λd _1≥·s≥ _r>0= _r+1=·s= _d. We consider linear encoder-decoder pairs (,)( E, D) with bottleneck dimensions n∥n_ for x-prediction and n⟂n_ for δ-prediction, together with isotropic corruption ∼(,σ2) δ ( 0,σ^2 I). Proposition 1 ( x-prediction selects high-variance directions). Consider the denoising objective min,,‖−(+)‖22,n∥≤r. _ E, DE_ x, δ\| x- D E( x+ δ)\|_2^2, n_ ≤ r. (6) Assume there are no ties between positive eigenvalues, i.e., λn∥>λn∥+1 _n_ > _n_ +1, and let n∥ U_n_ contain the eigenvectors corresponding to the n∥n_ largest eigenvalues of . Then the optimal composite map is ⋆⋆=n∥∥n∥⊤,∥≔diag(λ1λ1+σ2,…,λn∥λn∥+σ2). D E = U_n_ _ U_n_ , _ ( _1 _1+σ^2,…, _n_ _n_ +σ^2). (7) Moreover, every optimal rank-n∥n_ factorization can be written as ⋆=n∥⊤ E = R U_n_ , ⋆=n∥∥−1 D = U_n_ _ R^-1 for an arbitrary invertible matrix ∈ℝn∥×n∥ R ^n_ × n_ . Proposition 1 shows that x-prediction allocates the limited representation capacity to directions of largest data variance. Importantly, the encoder is insensitive to every direction outside the selected principal subspace: ⋆= E v= 0 for all ⟂Span(n∥) v ( U_n_ ). For the linear data manifold, ker() ( ) is exactly its normal space; hence, all manifold-orthogonal directions are annihilated by the x-prediction encoder. The corruption in the input is essential for this result: unlike ordinary autoencoding of x from x, denoising exposes the encoder to off-manifold directions while requiring the output to remain unchanged. Proposition 2 ( δ-prediction selects low-variance directions). Consider instead the objective min,,‖−(+)‖22. _ E, DE_ x, δ\| δ- D E( x+ δ)\|_2^2. (8) Let −n⟂ U_-n_ contain eigenvectors corresponding to the n⟂n_ smallest eigenvalues λd−n⟂+1,…,λd _d-n_ +1,…, _d of . Then the optimal composite map is ⋆⋆=−n⟂⟂−n⟂⊤,⟂≔diag(σ2λd−n⟂+1+σ2,…,σ2λd+σ2), D E = U_-n_ _ U_-n_ , _ ( σ^2 _d-n_ +1+σ^2,…, σ^2 _d+σ^2), (9) and every optimal rank-n⟂n_ factorization can be written as ⋆=−n⟂⊤ E = R U_-n_ , ⋆=−n⟂⟂−1 D = U_-n_ _ R^-1 for an arbitrary invertible matrix ∈ℝn⟂×n⟂ R ^n_ × n_ . In contrast to x-prediction, δ-prediction prioritizes directions of smallest data variance. In particular, if n⟂≤d−rn_ ≤ d-r, the selected directions can lie entirely within ker() ( ), which is exactly the normal space of the data manifold. When n⟂=d−rn_ =d-r, the encoder spans the complete normal space. Thus, under a suitable bottleneck, the two objectives induce complementary representations: x-prediction selects manifold-parallel variation, whereas δ-prediction selects manifold-orthogonal variation. The distinction can also be seen directly from their direction-wise gains. For an eigen-direction with data variance λ, x-prediction and δ-prediction assign g∥(λ)=λ+σ2g_ (λ)= λ+σ^2 and g⟂(λ)=σ2λ+σ2g_ (λ)= σ^2λ+σ^2 which satisfy g∥(λ)+g⟂(λ)=1g_ (λ)+g_ (λ)=1. x-prediction therefore increasingly favors high-variance directions, while δ-prediction increasingly favors low-variance directions. 4.2 Theory: General Smooth Manifolds and Autoencoders We now extend the linear analysis to a general smooth data manifold with smooth autoencoder. Let ℳ⊂ℝdM ^d be an r-dimensional smooth embedded manifold and consider a point 0∈ℳ x_0 . We parameterize nearby clean samples on the manifold as ϵ=ι(exp0(ϵ)) x_ε= ( _ x_0(ε ξ)), where the random tangent vector ∈T0ℳ ξ∈ T_ x_0M possesses the first and second moments []=,[⊤]=T, [ ξ]= 0, [ ξ ξ ]= _T, (10) and ι:ℳ→ℝd :M ^d denotes the embedding. Let ι≔dι0 J_ d _ x_0 and T≔ιTι⊤=diag(λ1,…,λd)⊤,λ1≥⋯≥λr>0=λr+1=⋯=λd, C_T J_ _T J_ = U ( _1,…, _d) U , _1≥·s≥ _r>0= _r+1=·s= _d, (11) so that Im(T)=T0ℳ ( C_T)=T_ x_0M and ker(T)=N0ℳ ( C_T)=N_ x_0M whenever T _T has full rank on T0ℳT_ x_0M. As before, independent isotropic corruption σ∼(,σ2) δ_σ ( 0,σ^2 I) is added to ϵ x_ε. For each objective, let the encoder E and decoder D be twice continuously differentiable, with bottleneck dimension n∥n_ for x-prediction and n⟂n_ for δ-prediction. Denote their local Jacobians by 1≔d0,1≔d(0). E_1 d E_ x_0, D_1 d D_ E( x_0). (12) The key observation is that, after first-order expansion around 0 x_0, both objectives reduce exactly to the corresponding linear problems addressed in Sec. 4.1, with the local tangent covariance T C_T replacing the global covariance . Proposition 3 (Local behavior of x-prediction). Assume ((0))=0 D( E( x_0))= x_0. As ϵ,σ→0ε,σ→ 0, ,σ‖ϵ−((ϵ+σ))‖22=,σ‖ϵι−11(ϵι+σ)‖22+o(ϵ2+σ2). _ ξ, δ_σ\| x_ε- D( E( x_ε+ δ_σ))\|_2^2=E_ ξ, δ_σ\|ε J_ ξ- D_1 E_1(ε J_ ξ+ δ_σ)\|_2^2+o(ε^2+σ^2). (13) For n∥≤rn_ ≤ r, the leading-order objective is minimized by 1⋆1⋆=n∥∥locn∥⊤,∥loc≔diag(ϵ2λ1ϵ2λ1+σ2,…,ϵ2λn∥ϵ2λn∥+σ2), D_1 E_1 = U_n_ _ U_n_ , _ ( ε^2 _1ε^2 _1+σ^2,…, ε^2 _n_ ε^2 _n_ +σ^2), (14) with 1⋆=n∥⊤ E_1 = R U_n_ , 1⋆=n∥∥loc−1 D_1 = U_n_ _ R^-1 for an arbitrary invertible matrix ∈ℝn∥×n∥ R ^n_ × n_ . Thus, to leading order, the Jacobian of the x-prediction encoder lies in the local tangent space and is insensitive to normal perturbations: 1⋆=0 E_1 v=0 for all ∈N0ℳ v∈ N_ x_0M. Proposition 4 (Local behavior of δ-prediction). Assume ((0))= D( E( x_0))= 0. As ϵ,σ→0ε,σ→ 0, ,σ‖σ−((ϵ+σ))‖22=,σ‖σ−11(ϵι+σ)‖22+o(ϵ2+σ2). _ ξ, δ_σ\| δ_σ- D( E( x_ε+ δ_σ))\|_2^2=E_ ξ, δ_σ\| δ_σ- D_1 E_1(ε J_ ξ+ δ_σ)\|_2^2+o(ε^2+σ^2). (15) The leading-order objective is minimized by 1⋆1⋆=−n⟂⟂loc−n⟂⊤,⟂loc≔diag(σ2ϵ2λd−n⟂+1+σ2,…,σ2ϵ2λd+σ2), D_1 E_1 = U_-n_ _ U_-n_ , _ ( σ^2ε^2 _d-n_ +1+σ^2,…, σ^2ε^2 _d+σ^2), (16) with 1⋆=−n⟂⊤ E_1 = R U_-n_ , 1⋆=−n⟂⟂loc−1 D_1 = U_-n_ _ R^-1 for an arbitrary invertible matrix ∈ℝn⟂×n⟂ R ^n_ × n_ . Figure 3: Encoder Jacobians on a 2D spiral. For n⟂≤d−rn_ ≤ d-r, the vanishing eigenvalues of T C_T correspond to normal directions; hence, the local Jacobian of the δ-prediction encoder lies in the normal space and spans it when n⟂=d−rn_ =d-r. Propositions 3 and 4 thus provide complementary local sensitivities without access to the tangent or normal spaces. Fig. 3 verifies this on a two-dimensional spiral, where the x- and δ-prediction encoders have Jacobians parallel and orthogonal to the manifold. The linear setting is therefore not merely illustrative: after locally linearizing the manifold and encoder-decoder, the general problem reduces to the rank-constrained regression of Sec. 4.1. This motivates the construction below, where x- and δ-prediction encoders define the manifold-parallel and manifold-orthogonal losses. 4.3 From Autoencoders to Generative Learning We now connect these local Jacobians to the generative objective in Eq. (2). Recall from Eq. (3) that the gradient of a feature loss is obtained by mapping the feature residual back to the ambient space through the transpose of the feature Jacobian; the row space of the encoder Jacobian therefore determines the directions along which the loss can update a generated sample. Proposition 5 (Directional decomposition of the feature loss). Let 0∈ℳ x_0 , n⟂≤d−rn_ ≤ d-r, and ϕ∥ φ and ϕ⟂ φ denote encoders satisfying the locally optimal characterizations of Prop. 3 and 4. Then Im((ϕ∥(0))⊤)⊆T0ℳ,Im((ϕ⟂(0))⊤)⊆N0ℳ. (( J_ φ ( x_0)) ) T_ x_0M, (( J_ φ ( x_0)) ) N_ x_0M. (17) Thus, the gradient of the decomposed feature loss in Eq. (2), evaluated locally at 0 x_0, decomposes ∇^ℒ=λ∥∥+λ⟂⟂,∥∈T0ℳ,⟂∈N0ℳ. _ xL=λ g +λ g , g ∈ T_ x_0M, g ∈ N_ x_0M. (18) Proposition 5 connects the representation-learning objectives to the generative loss: the x-prediction encoder yields feature gradients confined to manifold-parallel directions and the δ-prediction encoder to manifold-orthogonal directions, so that their combination in Eq. (2) updates a generated sample along complementary directions. Algorithm 1 MaDeL: Learning Manifold-Decomposed Features and the Generative Model 1: Encoder ϕ φ, projections ∥,⟂ P_ , P_ onto its first n∥n_ and last n⟂n_ features, decoders ∥,⟂ D , D , generator θ G_θ; steps Nrep,NgenN_rep,N_gen; weights (λ,λ,λiso,λ∥,λ⟂)( _ x, _ δ, _ ,λ ,λ ) 2: Stage 1: Representation learning 3: for step n=1,…,Nrepn=1,…,N_rep do 4: Sample a clean sample x, corruption δ, and let ~←+ x← x+ δ 5: Compute representations ∥←(∥∘ϕ)(~) h ←( P_ φ)( x) and ⟂←(⟂∘ϕ)(~) h ←( P_ φ)( x) 6: Minimize λ⋅‖−∥(∥)‖22+λ⋅‖−⟂(⟂)‖22 _ x·\| x- D ( h )\|_2^2+ _ δ·\| δ- D ( h )\|_2^2 w.r.t. (ϕ,∥,⟂)( φ, D , D ) 7: end for 8: Freeze ϕ∥≔∥∘ϕ φ P_ φ and ϕ⟂≔⟂∘ϕ φ P_ φ 9: Stage 2: Generative model learning 10: for step n=1,…,Ngenn=1,…,N_gen do 11: Sample prior and generative model output ^θ←θ() x_θ← G_θ( ) 12: Assign a target targ x_ to ^θ x_θ by the underlying generative method 13: Minimize λiso‖^−targ‖22+λ∥‖ϕ∥(^θ)−ϕ∥(targ)‖22+λ⟂‖ϕ⟂(^θ)−ϕ⟂(targ)‖22 _ \| x_ θ- x_ \|_2^2+λ \| φ ( x_θ)- φ ( x_ )\|_2^2+λ \| φ ( x_θ)- φ ( x_ )\|_2^2 w.r.t. θ 14: end for Learning the encoders. We use a shared-encoder dual-decoder architecture. Given ~=+ x= x+ δ, the encoder ϕ:ℝd→ℝn φ:R^d ^n, with n=n∥+n⟂n=n_ +n_ , is split by coordinate projections ∥ P_ and ⟂ P_ into ϕ∥≔∥∘ϕ,ϕ⟂≔⟂∘ϕ. φ P_ φ, φ P_ φ. (19) These branch encoders correspond to Propositions 3 and 4. Under the assumptions of Sec. 4.2, n∥≤rn_ ≤ r restricts x-prediction features to dominant tangent directions, while n⟂≤d−rn_ ≤ d-r restricts δ-prediction features to the normal space, which they span when n⟂=d−rn_ =d-r. Because these dimensions are unknown, n∥n_ and n⟂n_ are treated as branch-capacity hyperparameters. Each feature subset is provided exclusively to its corresponding decoder: the parallel decoder ∥ D is trained for x-prediction, while the orthogonal decoder ⟂ D is trained for δ-prediction: ℒAE≔λ⋅,[‖−∥(ϕ∥(+))‖22]+λ⋅,[‖−⟂(ϕ⟂(+))‖22] _ _ x·E_ x, δ[\| x- D ( φ ( x+ δ))\|_2^2]+ _ δ·E_ x, δ[\| δ- D ( φ ( x+ δ))\|_2^2] (20) The partition prevents the two decoders from sharing task-specific features while retaining a common feature extractor. By the analysis of Sec. 4.1, x-prediction renders ϕ∥ φ insensitive to off-manifold corruption, whereas δ-prediction drives ϕ⟂ φ to capture deviations from the data manifold. These feature subsets enter the generative loss in Eq. (2) directly. Training the generator. Our method is two-stage: after training, the encoders are frozen and used to train the generator. Given from the prior, the generator produces ^θ≔θ() x_θ G_θ( ); the underlying method constructs a regression target targ x_ , and we modify only the loss comparing these quantities. For OT matching, targ x_ is assigned by minibatch OT between current outputs and real samples, following explicit Wasserstein minimization (Fatras et al., 2020; Chen et al., 2019). Given batches i\ _i\ and j\ x_j\, with ^θ,i=θ(i) x_θ,i= G_θ( _i), we compute π⋆≔argminπ∈Π(B)∑i,jπij∥^θ,i−j∥22. π _π∈ (B) _i,j _ij\| x_θ,i- x_j\|_2^2. (21) We then set targ=j x_ = x_j for the data sample assigned to ^θ,i x_θ,i, treating the assignment as fixed during the generator update. For a flow-map semigroup objective such as Riemannian MeanFlow (Woo et al., 2026), let θs→t(s) G_θ^s→ t( x_s) denote the model prediction obtained by transporting s x_s directly from time s to t. The regression target is constructed by composing predictions over two shorter intervals: ^θ=θs→t(s),targ=sg[θr→t(θs→r(s))]. x_θ= G_θ^s→ t( x_s), x_ = [ G_θ^r→ t ( G_θ^s→ r( x_s) ) ]. (22) Here, s≤r≤ts≤ r≤ t, and sg denotes stop-gradient. Additional terms required by the underlying method, such as the MeanFlow boundary loss, are retained. Given targ x_targ, we train the generator using ℒgen(θ)≔[λiso‖^−targ‖22+λ∥ℒϕ∥(^θ,targ)+λ⟂ℒϕ⟂(^θ,targ)]. _gen(θ) [ _ \| x_ θ- x_ \|_2^2+λ L_ φ ( x_θ, x_targ)+λ L_ φ ( x_θ, x_ )]. (23) Thus, the proposed metric is independent of how the regression target is constructed. The squared discrepancy in either ambient or feature space may be replaced by any differentiable isotropic loss; the gradient-decomposition analysis extends directly, up to a loss-dependent scalar weighting of each residual. Algorithm 1 summarizes the complete two-stage procedure. Figure 4: Single-step generation on manifold-valued benchmarks. 5 Experiments We evaluate MaDeL on four benchmark families spanning synthetic and structured real-world distributions. Full experimental details are provided in App. C. Benchmarks. Helix on sphere: a helix in 2S^2, embedded isometrically into 512⊂ℝ513S^512 ^513 by a fixed orthonormal map and perturbed by tangential Gaussian noise. Earth and climate science: geographic coordinates of volcanic eruptions, earthquakes, floods, and wildfires represented on 2S^2, following prior benchmarks for manifold-valued generation (Bortoli et al., 2022; Chen and Lipman, 2024). Protein and RNA torsion angles: the preprocessed benchmarks of Riemannian Flow Matching (Chen and Lipman, 2024), following Huang et al. (2022), comprising four Ramachandran-angle distributions (General, Glycine, Proline, and Pre-Pro) on 2T^2 and a seven-dimensional RNA torsion distribution on 7T^7. Protein backbones: SCOPe protein structures (Chandonia et al., 2022), following the preprocessing and evaluation protocol of FrameFlow (Yim et al., 2023a). Method Earth and Climate Science Datasets (W1↓W_1 ) Protein and RNA Torsion-Angle Datasets (W1↓W_1 ) Earthquake Fire Flood Volcano Avg. General Glycine Proline Pre-Pro RNA Avg. RFM 27.19∘27.19 33.04∘33.04 23.68∘23.68 28.40∘28.40 28.08∘28.08 57.79∘57.79 64.80∘64.80 60.74∘60.74 32.68∘32.68 63.55∘63.55 55.912∘55.912 MF 6.43∘6.43 9.22∘9.22 6.67∘6.67 14.95∘14.95 9.32∘9.32 17.27∘17.27 11.57∘11.57 16.25∘16.25 13.77∘13.77 35.62∘35.62 18.896∘18.896 RMF 9.91∘9.91 10.21∘10.21 8.39∘8.39 16.82∘16.82 11.33∘11.33 34.33∘34.33 24.63∘24.63 31.84∘31.84 15.21∘15.21 41.84∘41.84 29.570∘29.570 OTM 6.22∘6.22 6.25∘6.25 4.88∘4.88 6.22∘6.22 5.89∘5.89 4.81∘4.81 8.27∘8.27 4.97∘4.97 5.97∘5.97 23.68∘23.68 9.540∘9.540 + MaDeL 5.81∘5.81 5.93∘5.93 5.16∘5.16 5.46∘5.46 5.59∘5.59 4.51∘4.51 6.69∘6.69 4.39∘4.39 6.03∘6.03 22.52∘22.52 8.828∘8.828 Table 2: Single-step generation error on Earth and torsion-angle datasets. Angular W1W_1 distance (degrees) between generated and ground-truth distributions; lower is better. RFM: Riemannian Flow Matching; MF, RMF: (Riemannian) MeanFlow; OTM: coordinate-space OT matching; + MaDeL: OTM with our loss. All methods use single-step sampling. Baselines. We compare with Riemannian Flow Matching (RFM) (Chen and Lipman, 2024), MeanFlow (MF) (Geng et al., 2025), and Riemannian MeanFlow (RMF) (Woo et al., 2026), each trained with its native objective. We also compare with coordinate-space OT matching (OTM), which computes a minibatch OT coupling between generated samples and real samples and regresses each generated output toward its assigned real-data target. OTM and OTM + MaDeL use the same coupling, generator architecture, and sampling procedure, differing only in the regression loss. 5.1 Main Results Helix on sphere. In Fig. 4 top, RFM, MF, RMF, and OTM distribute samples broadly over the sphere, failing to concentrate on the helical support. The Riemannian methods preserve the geometry of the spherical state space but do not recover the lower-dimensional data manifold embedded within it. In contrast, OTM + MaDeL reconstructs the global helical structure and places more probability mass near the data support using the same single network evaluation. Earth and climate science. As shown in Fig. 4 bottom, RFM spreads samples broadly across the surface, while MF and RMF recover the major high-density regions but blur their finer structure. OTM captures much of the global support but retains noticeable dispersion away from the narrow data structures. Using the same OT assignments, augmenting the coordinate-space regression loss with MaDeL produces more sharply localized samples while preserving broad geographic coverage. In Tab. 2 left, OTM + MaDeL achieves the lowest angular distance on three of the four datasets and improves the Earth average from 5.89∘5.89 to 5.59∘5.59 ; OTM remains strongest on Flood. Protein and RNA torsion angles. In Tab. 2 right, OTM + MaDeL achieves the lowest angular W1W_1 distance on four of the five datasets and improves the mean distance from 9.540∘9.540 to 8.828∘8.828 over OTM. It remains essentially tied on Pre-Pro (6.03∘6.03 versus 5.97∘5.97 ), with the largest improvement on Glycine, the broadest Ramachandran distribution. Protein backbone generation. We train identical RMF models, with MaDeL adding a frozen decomposed-feature term to the standard semigroup objective. Although MaDeL does not improve designability, it improves local steric geometry. We measure the maximum van der Waals overlap between backbone NN, CαC_α, and CC atoms of residues separated by at least two positions; lower values indicate fewer steric clashes, and real SCOPe backbones score 0.280.28–0.300.30 Å. As shown in Fig. 5, across eight paired prior batches from the one-million-step checkpoints, MaDeL reduces the median maximum overlap at all ten tested lengths and all three sampling budgets. At L=128L=128, it reduces overlap from 1.3751.375 to 1.0111.011 Å at NFE=10NFE=10 (lower in 8/88/8 batches), from 1.7421.742 to 1.5031.503 Å at NFE=5NFE=5 (7/87/8), and from 2.7322.732 to 2.6812.681 Å at NFE=1NFE=1 (6/86/8). This is not explained by chain expansion: at L=60L=60, MaDeL produces backbones that are both more compact and less clashing in 66–77 of eight batches. Designability remains essentially unchanged (10/64010/640 versus 12/64012/640), as does median self-consistency RMSD (8.638.63 versus 8.668.66 Å at L=128L=128 and NFE=10NFE=10). Figure 5: Steric violation versus backbone length. Deepest backbone van der Waals overlap per structure (N/Cα/C, residues ≥2≥ 2 apart), median over 8 independent batches of 128 paired samples from each model’s 1M-step checkpoint; bands are the full batch range, dashed is real SCOPe. MaDeL is lower at every length and budget. 5.2 Ablation Study and Analysis Loss Earthquake Fire Flood Volcano Avg. OTM 6.22∘6.22 6.25∘6.25 4.88∘4.88 6.22∘6.22 5.89∘5.89 + Overparam. AE 5.89∘5.89 6.16∘6.16 5.08∘5.08 6.24∘6.24 5.84∘5.84 + ∥ only (λ⟂=0λ =0) 6.12∘6.12 6.94∘6.94 5.41∘5.41 6.04∘6.04 6.13∘6.13 + ⟂ only (λ∥=0λ =0) 6.59∘6.59 6.42∘6.42 5.17∘5.17 6.15∘6.15 6.08∘6.08 + MaDeL 5.81∘5.81 5.93∘5.93 5.16∘5.16 5.46∘5.46 5.59∘5.59 Table 3: Ablation of the decomposed loss. Angular W1W_1 under the same OT pairing. Ablation. We ablate the two feature branches under the same OT pairing (Tab. 3). While Flood is an exception, with all feature-based variants underperforming coordinate-space OTM, the average results show a clear trend: the overparameterized autoencoder yields only a modest improvement from 5.89∘5.89 to 5.84∘5.84 , either manifold-decomposed branch alone performs worse (6.13∘6.13 for ∥ and 6.08∘6.08 for ⟂ ), and combining both branches achieves the best average of 5.59∘5.59 . MaDeL also attains the lowest error on Earthquake, Fire, and Volcano, indicating that the complementary branches provide the strongest overall performance despite variation across datasets. σ=0.0σ=0.0 0.010.01 0.050.05 0.10.1 ϕ∥ φ 15.8815.88 16.7516.75 45.0445.04 33.2133.21 ϕ⟂ φ 0.020.02 0.020.02 0.040.04 0.030.03 Table 4: Jacobian tangent-to-normal energy ratio. Analysis. To test whether the encoders realize the decomposition predicted by Sec. 4.2, we use the tangent and normal spaces of the helix and measure the ratio of manifold-parallel to manifold-orthogonal Jacobian energy (Tab. 4). The ratio is evaluated at samples perturbed by noise with standard deviation σ. Across noise levels, ϕ∥ φ produces large ratios (15.8815.88–45.0445.04), whereas ϕ⟂ φ produces ratios near zero (0.020.02–0.040.04), confirming that energies concentrate in complementary directions on and near the manifold. 6 Conclusion We introduced MaDeL, a manifold-decomposed feature loss for generative modeling that distinguishes variation along the data manifold from deviation away from it. We showed that, under a feature bottleneck, x-prediction and δ-prediction learn encoders whose Jacobians are confined to the tangent and normal spaces of the manifold, exactly for linear manifolds and locally for smooth manifolds, so that the resulting feature loss yields complementary tangent- and normal-aligned generative updates without explicit access to the manifold. Across controlled manifolds and structured real-world domains, MaDeL better preserves data support and improves distributional or local geometric accuracy relative to isotropic regression, which suggests that explicitly shaping the geometry of the training signal is a practical alternative to parameterizing the data manifold itself, particularly in sparse and highly constrained generative problems. Limitations. Our analysis is local and holds to leading order in the corruption level, while the branch dimensions and loss weights must be chosen without knowledge of the intrinsic dimension. Our experiments cover a limited set of synthetic and structured scientific benchmarks, and do not yet establish scalability across models, datasets, or modalities. Future work will develop MaDeL for scalable manifold generative modeling; App. B discusses these limitations. References Alain and Bengio (2014) G. Alain and Y. Bengio What Regularized Auto-Encoders Learn from the Data-Generating Distribution. JMLR. Cited by: §1, §2. Bondi (1964) A. Bondi Van der Waals Volumes and Radii. The Journal of Physical Chemistry 68 (3), p. 441–451. Cited by: §C.5. Bortoli et al. (2022) V. D. Bortoli, E. Mathieu, M. Hutchinson, J. Thornton, Y. W. Teh, and A. Doucet Riemannian Score-Based Generative Modelling. In NeurIPS, Cited by: §C.4, §5. Chandonia et al. (2022) J. Chandonia, L. Guan, S. Lin, C. Yu, N. K. Fox, and S. E. Brenner SCOPe: Improvements to the Structural Classification of Proteins—Extended Database to Facilitate Variant Interpretation and Machine Learning. Nucleic Acids Research 50 (D1), p. D553–D559. Cited by: §C.5, §5. Chen and Lipman (2024) R. T. Q. Chen and Y. Lipman Flow Matching on General Geometries. In ICLR, Cited by: §C.4, §2, Table 1, §5, §5. Chen et al. (2019) Y. Chen, M. Telgarsky, C. Zhang, B. Bailey, D. Hsu, and J. Peng A Gradual, Semi-Discrete Approach to Generative Network Training via Explicit Wasserstein Minimization. In ICML, Cited by: §4.3. Dauparas et al. (2022) J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, et al. Robust Deep Learning-Based Protein Sequence Design Using ProteinMPNN. Science 378 (6615), p. 49–56. Cited by: §C.5. Fatras et al. (2020) K. Fatras, Y. Zine, R. Flamary, R. Gribonval, and N. Courty Learning with Minibatch Wasserstein: Asymptotic and Gradient Properties. In AISTATS, Cited by: §4.3. Fefferman et al. (2016) C. Fefferman, S. Mitter, and H. Narayanan Testing the Manifold Hypothesis. Journal of the American Mathematical Society 29 (4), p. 983–1049. Cited by: §1. Geng et al. (2025) Z. Geng, M. Deng, X. Bai, J. Z. Kolter, and K. He Mean Flows for One-Step Generative Modeling. In NeurIPS, Cited by: §2, §5. He et al. (2024) Y. He, N. Murata, C. Lai, Y. Takida, T. Uesaka, D. Kim, W. Liao, Y. Mitsufuji, J. Z. Kolter, R. Salakhutdinov, and S. Ermon Manifold Preserving Guided Diffusion. In ICLR, Cited by: §2, Table 1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising Diffusion Probabilistic Models. In NeurIPS, Cited by: §1. Ho et al. (2022) J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet Video Diffusion Models. In NeurIPS, Cited by: §1. Hoogeboom et al. (2022) E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling Equivariant Diffusion for Molecule Generation in 3D. In ICML, Cited by: §1. Huang et al. (2022) C. Huang, M. Aghajohari, A. J. Bose, P. Panangaden, and A. Courville Riemannian Diffusion Models. In NeurIPS, Cited by: §C.4, §2, §5. Johnson et al. (2016) J. Johnson, A. Alahi, and F. Li Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV, Cited by: §1, §2. Karras et al. (2022) T. Karras, M. Aittala, T. Aila, and S. Laine Elucidating the Design Space of Diffusion-Based Generative Models. In NeurIPS, Cited by: §1. Kim et al. (2026) B. Kim, B. Cha, and J. C. Ye Align Your Trajectory Tangent: Training Better Consistency Models via Manifold-Aligned Tangents. In ICML, Cited by: §2, Table 1. Lin et al. (2023) Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, and A. Rives Evolutionary-Scale Prediction of Atomic-Level Protein Structure with a Language Model. Science 379 (6637), p. 1123–1130. Cited by: §C.5. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow Matching for Generative Modeling. In ICLR, Cited by: §1. Liu et al. (2023) X. Liu, C. Gong, and Q. Liu Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In ICLR, Cited by: §1. Liu et al. (2025) Z. Liu, W. Zhang, and T. Li Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity. In NeurIPS, Cited by: §2, Table 1. Narayanan and Mitter (2010) H. Narayanan and S. Mitter Sample Complexity of Testing the Manifold Hypothesis. In NeurIPS, Cited by: §1. Pope et al. (2021) P. Pope, C. Zhu, A. Abdelkader, M. Goldblum, and T. Goldstein The Intrinsic Dimension of Images and Its Impact on Learning. In ICLR, Cited by: §1. Rifai et al. (2011a) S. Rifai, Y. N. Dauphin, P. Vincent, Y. Bengio, and X. Muller The Manifold Tangent Classifier. In NeurIPS, Cited by: §1. Rifai et al. (2011b) S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio Contractive Auto-Encoders: Explicit Invariance During Feature Extraction. In ICML, Cited by: §2. Saito and Matsubara (2026) S. Saito and T. Matsubara Be Tangential to Manifold: Discovering Riemannian Metric for Diffusion Models. In ECCV, Cited by: §2, Table 1. Sauer et al. (2021) A. Sauer, K. Chitta, J. Müller, and A. Geiger Projected GANs Converge Faster. In NeurIPS, Cited by: §2. Song et al. (2023) Y. Song, P. Dhariwal, M. Chen, and I. Sutskever Consistency Models. In ICML, Cited by: §2, §2. Song and Dhariwal (2024) Y. Song and P. Dhariwal Improved Techniques for Training Consistency Models. In ICLR, Cited by: §2. Song et al. (2021) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR, Cited by: §1. Stanczuk et al. (2024) J. P. Stanczuk, G. Batzolis, T. Deveney, and C. Schönlieb Diffusion Models Encode the Intrinsic Dimension of Data Manifolds. In ICML, Cited by: §1, §2. Sun et al. (2023) S. Sun, L. Wei, J. Xing, J. Jia, and Q. Tian SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation. In ICML, Cited by: §2, Table 1. Tenenbaum et al. (2000) J. B. Tenenbaum, V. de Silva, and J. C. Langford A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science. Cited by: §1. Vincent et al. (2008) P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol Extracting and Composing Robust Features with Denoising Autoencoders. In ICML, Cited by: §2. Woo et al. (2026) D. Woo, M. Skreta, S. Park, K. Neklyudov, and S. Ahn Riemannian MeanFlow. In ICML, Cited by: §C.5, §2, Table 1, §4.3, §5. Yim et al. (2023a) J. Yim, A. Campbell, A. Y. K. Foong, M. Gastegger, J. Jiménez-Luna, S. Lewis, V. G. Satorras, B. S. Veeling, R. Barzilay, T. Jaakkola, and F. Noé Fast Protein Backbone Generation with SE(3) Flow Matching. arXiv preprint arXiv:2310.05297. Cited by: §C.5, §5. Yim et al. (2023b) J. Yim, B. L. Trippe, V. D. Bortoli, E. Mathieu, A. Doucet, R. Barzilay, and T. Jaakkola SE(3) Diffusion Model with Application to Protein Backbone Generation. In ICML, Cited by: §1. Zhang et al. (2018) R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, Cited by: §1, §2. Appendix A Proofs A.1 Linear manifolds Proof of Proposition 1. Let ≔+ y x+ δ and ≔[⊤]=+σ2 C [ y y ]= +σ^2 I. Since σ>0σ>0, C is positive definite. The objective depends on the encoder and decoder only through ≔ F D E, where rank()≤n∥ ( F)≤ n_ . Using [⊤]=E[ x y ]= gives ‖−‖22 \| x- F y\|_2^2 =tr()−2tr()+tr(⊤) =tr( )-2tr( F )+tr( F C F ) (24) =∥1/2−1/2∥F2+tr()−tr(−1). =\| F C^1/2- C^-1/2\|_F^2+tr( )-tr( C^-1 ). (25) Hence, after defining ≔1/2 G F C^1/2, minimizing the objective is equivalent to minrank()≤n∥−1/2∥F2. _ ( G)≤ n_ \| G- C^-1/2\|_F^2. (26) Because =diag(λi+σ2)⊤ C= U ( _i+σ^2) U commutes with , −1/2=diag(λ1λ1+σ2,…,λdλd+σ2)⊤. C^-1/2= U \! ( _1 _1+σ^2,…, _d _d+σ^2 ) U . (27) The scalar map λ↦λ/λ+σ2λ λ/ λ+σ^2 is strictly increasing on [0,∞)[0,∞). The Eckart–Young–Mirsky theorem and the assumed eigengap therefore show that the unique optimal rank-n∥n_ approximation retains the first n∥n_ eigendirections: ⋆=n∥diag(λ1λ1+σ2,…,λn∥λn∥+σ2)n∥⊤. G = U_n_ \! ( _1 _1+σ^2,…, _n_ _n_ +σ^2 ) U_n_ . (28) Multiplying by −1/2 C^-1/2 on the right yields ⋆=n∥∥n∥⊤. F = U_n_ _ U_n_ . (29) Since n∥≤rn_ ≤ r, ∥ _ is invertible and rank(⋆)=n∥ ( F )=n_ . Thus, any factorization ⋆=⋆⋆ F = D E through an n∥n_ -dimensional bottleneck has rank(⋆)=rank(⋆)=n∥ ( D )= ( E )=n_ and Im(⋆)=Span(n∥) ( D )= ( U_n_ ). Write ⋆=n∥ D = U_n_ A for an invertible A. Then ⋆=−1∥n∥⊤ E = A^-1 _ U_n_ . Setting ≔−1∥ R A^-1 _ gives ⋆=n∥⊤,⋆=n∥∥−1, E = R U_n_ , D = U_n_ _ R^-1, (30) which characterizes every optimal factorization. ∎ Proof of Proposition 2. Again let =+ y= x+ δ, =+σ2 C= +σ^2 I, and = F= D E. Since [⊤]=σ2E[ δ y ]=σ^2 I, ‖−‖22 \| δ- F y\|_2^2 =dσ2−2σ2tr()+tr(⊤) =dσ^2-2σ^2tr( F)+tr( F C F ) (31) =∥1/2−σ2−1/2∥F2+dσ2−σ4tr(−1). =\| F C^1/2-σ^2 C^-1/2\|_F^2+dσ^2-σ^4tr( C^-1). (32) With ≔1/2 G F C^1/2, it remains to find the best rank-n⟂n_ approximation of σ2−1/2=diag(σ2λ1+σ2,…,σ2λd+σ2)⊤. σ^2 C^-1/2= U \! ( σ^2 _1+σ^2,…, σ^2 _d+σ^2 ) U . (33) These singular values decrease as λi _i increases. Thus, the Eckart–Young–Mirsky theorem selects the n⟂n_ smallest-variance eigendirections, giving ⋆=−n⟂diag(σ2λd−n⟂+1+σ2,…,σ2λd+σ2)−n⟂⊤. G = U_-n_ \! ( σ^2 _d-n_ +1+σ^2,…, σ^2 _d+σ^2 ) U_-n_ . (34) Therefore, ⋆=⋆−1/2=−n⟂−n⟂⊤. F = G C^-1/2= U_-n_ _ U_-n_ . (35) The diagonal entries of ⟂ _ are strictly positive, so rank(⋆)=n⟂ ( F )=n_ . The same factorization argument as in Proposition 1 then yields ⋆=−n⟂⊤,⋆=−n⟂⟂−1 E = R U_-n_ , D = U_-n_ _ R^-1 (36) for an arbitrary invertible R. If eigenvalues are tied at the truncation boundary, −n⟂ U_-n_ may be any orthonormal basis of an optimal subspace within the tied eigenspace. ∎ A.2 General smooth manifolds Proof of Proposition 3. Define the local autoencoder map AE≔∘ A_ D E. The manifold parameterization has the first-order expansion ϵ=0+ϵι+ϵ,ϵ=o(ϵ) x_ε= x_0+ε J_ ξ+ r_ε, r_ε=o(ε) (37) in mean square. Since AE A_ is twice continuously differentiable and AE(0)=0 A_ ( x_0)= x_0, the chain rule gives AE(ϵ+σ)=0+11(ϵι+σ)+AE, A_ ( x_ε+ δ_σ)= x_0+ D_1 E_1(ε J_ ξ+ δ_σ)+ r_ , (38) where the combined Taylor remainder satisfies ‖ϵ−AE‖22=o(ϵ2+σ2)E\| r_ε- r_ \|_2^2=o(ε^2+σ^2) and its cross term with the first-order residual is also o(ϵ2+σ2)o(ε^2+σ^2). Consequently, ,σ‖ϵ−((ϵ+σ))‖22 _ ξ, δ_σ\| x_ε- D( E( x_ε+ δ_σ))\|_2^2 (39) =,σ‖ϵι−11(ϵι+σ)‖22+o(ϵ2+σ2). =E_ ξ, δ_σ\|ε J_ ξ- D_1 E_1(ε J_ ξ+ δ_σ)\|_2^2+o(ε^2+σ^2). (40) The leading-order problem is precisely the reduced-rank denoising problem of Proposition 1, with clean covariance ϵ2Tε^2 C_T and corruption covariance σ2σ^2 I. Its eigendirections are those of T C_T, and the corresponding gains are ϵ2λiϵ2λi+σ2. ε^2 _iε^2 _i+σ^2. (41) Applying Proposition 1 therefore gives the stated 1⋆1⋆ D_1 E_1 , 1⋆ E_1 , and 1⋆ D_1 . ∎ Proof of Proposition 4. Use the same expansion ϵ=0+ϵι+ϵ x_ε= x_0+ε J_ ξ+ r_ε. Now AE(0)=((0))= A_ ( x_0)= D( E( x_0))= 0, so Taylor expansion and the chain rule yield ((ϵ+σ))=11(ϵι+σ)+AE. D( E( x_ε+ δ_σ))= D_1 E_1(ε J_ ξ+ δ_σ)+ r_ . (42) As in the preceding proof, the squared remainder and its cross term with the first-order residual are o(ϵ2+σ2)o(ε^2+σ^2). Hence, ,σ‖σ−((ϵ+σ))‖22 _ ξ, δ_σ\| δ_σ- D( E( x_ε+ δ_σ))\|_2^2 (43) =,σ‖σ−11(ϵι+σ)‖22+o(ϵ2+σ2). =E_ ξ, δ_σ\| δ_σ- D_1 E_1(ε J_ ξ+ δ_σ)\|_2^2+o(ε^2+σ^2). (44) This is the linear δ-prediction problem of Proposition 2 with data covariance ϵ2Tε^2 C_T. The optimal rank-n⟂n_ map therefore retains the n⟂n_ smallest eigenvalues of T C_T and assigns gain σ2ϵ2λi+σ2 σ^2ε^2 _i+σ^2 (45) to the eigendirection with eigenvalue λi _i. Proposition 2 then gives the stated composite map and all of its optimal factorizations. ∎ A.3 Directional decomposition of the generative loss Proof of Proposition 5. By Proposition 3, the Jacobian of a locally optimal manifold-parallel encoder has the form ϕ∥(0)=∥n∥⊤. J_ φ ( x_0)= R_ U_n_ . (46) The columns of n∥ U_n_ are positive-eigenvalue eigenvectors of T C_T and therefore lie in Im(T)=T0ℳ ( C_T)=T_ x_0M. Since ∥ R_ is invertible, Im((ϕ∥(0))⊤)=Span(n∥)⊆T0ℳ. (( J_ φ ( x_0)) )= ( U_n_ ) T_ x_0M. (47) Similarly, Proposition 4 gives ϕ⟂(0)=⟂−n⟂⊤. J_ φ ( x_0)= R_ U_-n_ . (48) Because n⟂≤d−rn_ ≤ d-r, the columns of −n⟂ U_-n_ can be chosen entirely from the zero-eigenspace of T C_T, which equals N0ℳN_ x_0M. Thus, Im((ϕ⟂(0))⊤)=Span(−n⟂)⊆N0ℳ. (( J_ φ ( x_0)) )= ( U_-n_ ) N_ x_0M. (49) For a target x, define the two local feature-gradient components at ^=0 x= x_0 by ∥ g ≔(ϕ∥(0))⊤(ϕ∥(0)−ϕ∥()), ( J_ φ ( x_0)) ( φ ( x_0)- φ ( x) ), (50) ⟂ g ≔(ϕ⟂(0))⊤(ϕ⟂(0)−ϕ⟂()). ( J_ φ ( x_0)) ( φ ( x_0)- φ ( x) ). (51) The two image-space inclusions above imply ∥∈T0ℳ g ∈ T_ x_0M and ⟂∈N0ℳ g ∈ N_ x_0M. Substituting these definitions into the gradient identity in Eq. (3) yields ∇^ℒ=λ∥∥+λ⟂⟂, _ xL=λ g +λ g , (52) which proves the claimed directional decomposition. ∎ Appendix B Scope of the Analysis and Limitations Our characterization is local and holds to leading order in the corruption level; finite corruption, strong curvature, manifold boundaries, and imperfect optimization can therefore mix the two components. The corruption distribution, the branch dimensions n∥n_ and n⟂n_ , and the loss weights λ∥λ and λ⟂λ must be selected without knowledge of the intrinsic dimension, and the feature geometry requires a separate representation-training stage for each domain. Our experiments concern sparse structured distributions and single-step generators; broader validation across architectures, modalities, and scale remains to be established. Appendix C Experimental Detail C.1 Shared Setup for the Helix, Earth, and Torsion Experiments The helix, Earth, and torsion-angle experiments use a common training and evaluation framework. RFM, MF, and RMF use the same network architecture, consisting of four hidden layers of width 256256 and a Fourier time embedding with four frequency octaves. Each method is independently initialized with seed 00, and evaluation uses exponential-moving-average weights with decay 0.9990.999. The flow-based methods are evaluated at NFE∈1,2,5,10NFE∈\1,2,5,10\, whereas OTM and OTM + MaDeL directly generate samples with one network evaluation. For OTM and OTM + MaDeL, we compute an exact Hungarian assignment between generated and real samples using ambient squared distance. Both methods therefore use identical generated-to-data pairings and differ only in their regression losses. For Earth and torsion angles, OTM + MaDeL uses ℒOTM+MaDeL=‖^−π‖22+0.1hc(ϕ∥(^)−ϕ∥(π))+0.1hc(ϕ⟂(^)−ϕ⟂(π)), _OTM+MaDeL=\| x- x_π\|_2^2+0.1\,h_c\! ( φ ( x)- φ ( x_π) )+0.1\,h_c\! ( φ ( x)- φ ( x_π) ), (53) where π x_π is the assigned real sample and hc()=‖22+c2−ch_c( u)= \| u\|_2^2+c^2-c with c=0.01c=0.01. The helix experiment omits the ambient term and uses feature weights λ∥=1λ =1 and λ⟂=5λ =5. The feature encoder is trained separately and frozen during generative training. Its parallel decoder predicts the clean sample, while its orthogonal decoder predicts the normalized corruption /rms() δ/rms( δ). Normalizing the corruption prevents the orthogonal representation from collapsing when the corruption magnitude is small. C.2 Helix on Sphere We construct a spherical helix on 2S^2 with winding parameter 5.05.0, latitude oscillation 0.50.5, and parameter range [−π,π][-π,π]. Clean samples are obtained by applying tangent Gaussian perturbations with standard deviation σsignal=0.01 _signal=0.01 and are then embedded into 512⊂ℝ513S^512 ^513 through a fixed random orthonormal map from ℝ3R^3 to ℝ513R^513. Encoder inputs are formed by applying an additional Gaussian perturbation with per-coordinate standard deviation σperturb=0.10 _perturb=0.10 in the ambient tangent space of 512S^512. The clean distribution therefore remains in the three-dimensional image of the embedding, while the corruption can leave this subspace. We use 50,00050,000 data samples. All generative models are trained for 50,00050,000 iterations using RAdam with learning rate 10−310^-3, batch size 256256, gradient clipping at 2.02.0, and EMA decay 0.9990.999. The encoder uses n∥=3n_ =3 parallel and n⟂=509n_ =509 orthogonal features and is trained with clean- and corruption-prediction weights 11 and 55, respectively. OTM and OTM + MaDeL use a three-dimensional Gaussian prior and exact Hungarian assignment. For MF, the prior standard deviation is 1/5131/ 513 so that its expected norm matches the unit-norm data. Evaluation uses 2,0482,048 generated samples. The qualitative comparison in Fig. 4 uses NFE=1NFE=1 and projects samples back to the underlying 2S^2 coordinates. We additionally monitor sliced W22W_2^2, energy distance, distance to the underlying helix, displacement outside the embedding subspace, and coverage along the helix. C.3 Earth and Climate Science The Earth benchmark contains event locations on 2S^2 for volcanoes, earthquakes, floods, and wildfires, with 827827, 6,1206,120, 4,8754,875, and 12,80912,809 records, respectively. We use deterministic 80/10/1080/10/10 train, validation, and test splits with seed 4242. Duplicate coordinates are retained during training because repeated events at one location encode the empirical event density. Clean locations are corrupted in the tangent space of 2S^2 using Gaussian noise with standard deviation 0.150.15. The encoder uses n∥=n⟂=2n_ =n_ =2 and equal clean- and corruption-prediction weights. It is trained for 10,00010,000 iterations with batch size 2,0482,048. Generative models are trained for 20,00020,000 iterations using Adam with learning rate 10−310^-3 and EMA decay 0.9990.999. Flow-based methods use batch size 2,0482,048, while the exact-Hungarian OTM variants use batch size 256256. We report geodesic W1W_1 distance between generated and real event distributions. The cost is the great-circle distance on 2S^2, and the empirical transport problem is solved by exact Hungarian assignment. Datasets with more than 2,0482,048 records use a fixed seeded reference subsample, shared across all methods and checkpoints. Distances are converted from radians to degrees for Tab. 2. The qualitative Earthquake comparison in Fig. 4 uses 4,0964,096 generated samples at NFE=1NFE=1. C.4 Protein and RNA Torsion Angles We use the torsion-angle benchmarks of Riemannian Flow Matching (Chen and Lipman, 2024), following the preprocessing of Riemannian diffusion models (Huang et al., 2022; Bortoli et al., 2022). The benchmark contains four two-dimensional Ramachandran distributions and one seven-dimensional RNA torsion distribution: Dataset Manifold Number of records General 2T^2 137,718137,718 Glycine 2T^2 13,18313,183 Proline 2T^2 7,6187,618 Pre-Pro 2T^2 6,9006,900 RNA 7T^7 9,4579,457 Each angle is represented by its cosine and sine, embedding dT^d isometrically into ℝ2dR^2d without an angular discontinuity. Corruption is applied independently to each angle using wrapped Gaussian noise with standard deviation 0.150.15 radians. All generative models use batch size 256256 and are trained for 20,00020,000 iterations using Adam with learning rate 10−310^-3 and EMA decay 0.9990.999. The encoder is trained separately for 10,00010,000 iterations with batch size 2,0482,048. Its parallel and orthogonal feature dimensions are: n∥=n⟂=2n_ =n_ =2 for 2T^2 and n∥=n⟂=7n_ =n_ =7 for 7T^7. We evaluate geodesic W1W_1 using exact Hungarian assignment under the flat-torus distance, defined as the Euclidean norm of the wrapped per-angle differences. The generated and test distributions are restricted to at most 2,0482,048 samples using a fixed seeded subsample, and the resulting distances are reported in degrees in Tab. 2. All values in the main table use NFE=1NFE=1. C.5 SCOPe Protein Backbone Generation Dataset and generator. We use 3,6733,673 SCOPe backbones containing between 6060 and 128128 residues (Chandonia et al., 2022), matching the dataset used by RMF (Woo et al., 2026). Each backbone is represented by residue-local rigid frames in SE(3)LSE(3)^L. The generator follows the FrameFlow architecture (Yim et al., 2023a), using node and edge dimensions 256256 and 128128, six invariant-point-attention blocks, eight attention heads, and self-conditioning. Both the baseline and MaDeL models use the RMF SplitMeanFlow objective, comprising a flow-matching anchor and a semigroup composition term. Rotational and translational losses are weighted by 11 and 22, respectively, with translations scaled by 0.10.1. The time floor is 0.010.01, and both objective terms are clipped at 1010 after 10,00010,000 iterations. Models are trained with AdamW using learning rate 10−410^-4, weight decay 0.010.01, gradient clipping at 1.01.0, and EMA decay 0.99990.9999. Batches contain at most 5454 proteins under a length-dependent memory budget. Although training runs for up to two million iterations, the reported steric evaluation uses the one-million-step checkpoints. Frozen protein encoder. The protein feature encoder is a split-latent U-Net over backbone frames with 6464 feature planes, encoder depths [4,4,4,4][4,4,4,4], and decoder depths [2,2,2][2,2,2]. It is trained for 250,000250,000 iterations using batch size 2,0482,048, learning rate 10−410^-4, and equal clean- and corruption-prediction weights. Structured corruptions include rigid segment and loop displacements, chain breaks, and atom-level bond, angle, and torsion perturbations. Each of the 1313 corruption operations is independently activated according to a base probability of 0.150.15, producing approximately two corruptions per sample. The EMA encoder checkpoint is frozen during RMF training. The baseline sets the feature-semigroup weight to zero. The MaDeL model adds the decomposed feature semigroup term with weight 0.00290.0029 while retaining the standard RMF objective. All other architecture, optimization, conditioning, and sampling settings are identical between the two runs. Designability and self-consistency. For each checkpoint, backbone length, and NFE, we generate 6464 backbones from identical priors for both models. Following standard protein-backbone evaluation, ProteinMPNN (Dauparas et al., 2022) generates eight sequences per backbone at temperature 0.10.1, which are refolded using ESMFold (Lin et al., 2023). Self-consistency RMSD is the minimum RMSD over the eight refolded sequences, and a backbone is considered designable when this value is below 2.02.0 Å. Steric validity. We evaluate the one-million-step checkpoints at lengths 60,68,76,84,92,100,108,116,124,128\60,68,76,84,92,100,108,116,124,128\ and NFE∈1,5,10NFE∈\1,5,10\. For each length and NFE, we draw eight independent batches of 128128 samples, using identical priors for the baseline and MaDeL within each batch. For every generated structure, we compute the maximum van der Waals overlap max(a,b)(ra+rb−‖a−b‖2) _(a,b) (r_a+r_b-\| x_a- x_b\|_2 ) (54) over backbone NN, CαC_α, and CC atoms belonging to residues separated by at least two sequence positions. We use Bondi radii of 1.551.55 Å for nitrogen and 1.701.70 Å for carbon (Bondi, 1964). Backbone oxygen atoms are excluded because their placement is determined by the PDB-writing convention rather than directly emitted by the model. The reported statistic is the median maximum overlap within each batch. Real SCOPe backbones within four residues of each target length provide the reference distribution and have median overlaps of 0.280.28–0.300.30 Å.