Paper deep dive
Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads
Zijian Zhao, Sen Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A multiplicative dual-encoder network computes a real-valued output for a pair of inputs as the inner product of their separate encodings. This architecture has been developed independently in operator learning, bipartite matching, contrastive vision-language models, retrieval, and other areas, yet no unified theory guides the basic design decisions: how many interaction modes to represent, how to normalize the encoders, and when the architecture should be avoided. We provide such a foundation by introducing the class of functions of low interaction rank, a class whose intrinsic complexity is measured by its interaction spectrum. Within this framework, approximation error decomposes into a spectral truncation term and an encoder-realization term; sample complexity is governed by the sum of the two encoder complexities rather than their product; and a usability criterion based on spectral decay determines when the architecture can succeed. The same framework exposes a central identifiability problem: the encoders are defined only up to a linear gauge symmetry that leaves the learned coordinates arbitrary. We show that normalization is gauge fixing and that whitening pins the interaction modes up to permutation and sign, thereby explaining the uninterpretability of contrastive dimensions and providing a constructive remedy. Experiments on synthetic kernels, operator learning, and CLIP models validate the theoretical predictions: spectral decay rates match the predicted scaling, whitening recovers the true modes, and independently trained CLIP models are related by a single rotation which, after removal by whitening, exposes interpretable concept axes. The code of this paper is provided at this https URL .
Tags
Links
- Source: https://arxiv.org/abs/2608.11661v1
- Canonical: https://arxiv.org/abs/2608.11661v1
Trouble viewing inline? Open PDF directly →
Full Text
151,221 characters extracted from source content.
Expand or collapse full text
Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads Zijian Zhao Affiliation: The Hong Kong University of Science and Technology Sen Li Thanks: Corresponding Author: Sen Li Affiliation: The Hong Kong University of Science and Technology Affiliation: The Hong Kong University of Science and Technology (Guangzhou) Abstract A multiplicative dual-encoder network computes a real-valued output for a pair of inputs as the inner product of their separate encodings. This architecture has been developed independently in operator learning, bipartite matching, contrastive vision-language models, retrieval, and other areas, yet no unified theory guides the basic design decisions: how many interaction modes to represent, how to normalize the encoders, and when the architecture should be avoided. We provide such a foundation by introducing the class of functions of low interaction rank, a class whose intrinsic complexity is measured by its interaction spectrum. Within this framework, approximation error decomposes into a spectral truncation term and an encoder-realization term; sample complexity is governed by the sum of the two encoder complexities rather than their product; and a usability criterion based on spectral decay determines when the architecture can succeed. The same framework exposes a central identifiability problem: the encoders are defined only up to a linear gauge symmetry that leaves the learned coordinates arbitrary. We show that normalization is gauge fixing and that whitening pins the interaction modes up to permutation and sign, thereby explaining the uninterpretability of contrastive dimensions and providing a constructive remedy. Experiments on synthetic kernels, operator learning, and CLIP models validate the theoretical predictions: spectral decay rates match the predicted scaling, whitening recovers the true modes, and independently trained CLIP models are related by a single rotation which, after removal by whitening, exposes interpretable concept axes. The code of this paper is provided at https://github.com/RS2002/Mul-Net. 1 Introduction A common architectural pattern has emerged independently across several machine learning communities: computing a real-valued output for a pair of inputs as the inner product of their separately encoded representations. Contrastive models use it for cross-modal similarity (17; 27); operator networks evaluate learned maps via branch-trunk inner products (14); retrieval systems rank queries against documents with two-tower encoders (11); goal-conditioned reinforcement learning and successor features represent values as inner products of encodings (21; 9; 3); and knowledge-graph completion, linear attention, factorization machines, and multiplicative matching networks all share the same underlying structure (24; 23; 12; 19; 30; 29). In each domain, this structure provides tangible computational and representational benefits. Yet the corresponding methods have been developed largely in isolation, even though they are all instances of the same fundamental architecture, differing only in the nature of the two inputs and how the output is consumed. Consequently, each community independently confronts the same design questions: how many interaction modes to retain, how to normalize the encoders to ensure stable training and meaningful representations, and when the architecture is fundamentally ill-suited. We now formalize this shared architecture: a multiplicative dual-encoder head computes F(u,v)≈⟨fθ(u),gφ(v)⟩=∑k=1dfk(u)gk(v) F(u,v)≈ f_θ(u),g_ (v) = _k=1^df_k(u)\,g_k(v) (1) where fθf_θ and gφg_ are learned encoders mapping the two inputs into a common space ℝdR^d. We study the class ℳd=F:F(u,v)=⟨f(u),g(v)⟩,f:U→ℝd,g:V→ℝdM_d=\\,F:\,F(u,v)= f(u),g(v) ,\;f:U ^d,\;g:V ^d\,\ (2) of functions representable by a rank-d head. The complexity of a target F within this class is measured by its interaction spectrum σkk≥1\ _k\_k≥ 1, the singular values of the integral operator whose kernel is F; its interaction rank is the number of nonzero singular values. Within this framework, we answer four questions about approximation error, encoder identifiability, sample complexity, and fundamental architectural limitations. • Approximation: The error of any rank-d head decomposes into a spectral truncation term, unavoidable by any encoder, and an encoder-realization term; target smoothness controls the decay rate. (Section 2) • Identifiability: The representation (1) is invariant under a linear gauge symmetry acting on the pair of encoders, so the encoders of any trained model are defined only up to this symmetry. We show that the normalizations used across these domains are precisely gauge-fixing choices, and we prove that whitening pins the interaction modes up to permutation and sign. This explains why the individual dimensions of contrastive models lack inherent semantic meaning and provides a constructive remedy. (Section 3) • Estimation: Sample complexity is governed by the sum of the two encoder complexities rather than their product; for smooth targets, the optimal rank grows only logarithmically in the sample budget. (Section 4) • Usability: A flat interaction spectrum forces every rank-d head to a relative error floor of 1−d/N1-d/N, where N is the size of the discrete input domain; this barrier is escaped by early-interaction networks and yields a practical criterion based on the measured spectrum. (Section 4) To further validate our conclusions, we conducted experiments on synthetic kernels, operator learning, and CLIP models, confirming the predicted spectral behavior, demonstrating unique mode recovery under whitening, and verifying the flat-spectrum error floor. (Section 5) 2 The Low-Interaction-Rank Class: Unification and Approximation Section 1 introduced the class ℳdM_d of rank-d heads. We now associate with every target a spectral object that measures its intrinsic complexity within this class, show that ten existing method families are instances of the same architecture, and develop an approximation theory that quantifies how well a rank-d head can represent a given target. 2.1 The class and the interaction spectrum Let U and V be compact metric spaces with probability measures μU _U, μV _V, and let F∈L2(μU⊗μV)F∈ L^2( _U _V) be the target function of two arguments. A rank-d head is a pair of encoders f:→ℝdf:U ^d, g:→ℝdg:V ^d with coordinate functions fk∈L2(μU)f_k∈ L^2( _U), gk∈L2(μV)g_k∈ L^2( _V). Definition 1 (Interaction spectrum). For F∈L2(μU⊗μV)F∈ L^2( _U _V), the interaction operator is the integral operator TF:L2(μV)→L2(μU)T_F:L^2( _V)→ L^2( _U) defined by (TFh)(u)=∫F(u,v)h(v)dμV(v).(T_Fh)(u)= _VF(u,v)\,h(v)\,d _V(v). (3) Since F∈L2(μU⊗μV)F∈ L^2( _U _V), this operator is Hilbert-Schmidt, and F admits the Schmidt decomposition F=∑k≥1σkak⊗bk,(ak⊗bk)(u,v)=ak(u)bk(v),F= _k≥ 1 _k\,a_k b_k, (a_k b_k)(u,v)=a_k(u)\,b_k(v), (4) with singular values σ1≥σ2≥⋯≥0 _1≥ _2≥·s≥ 0 and orthonormal systems ak⊂L2(μU)\a_k\⊂ L^2( _U), bk⊂L2(μV)\b_k\⊂ L^2( _V). The sequence σk\ _k\ is the interaction spectrum of F; the number i-rank(F)=#k:σk>0i-rank(F)=\#\k: _k>0\ is its interaction rank; and the functions aka_k, bkb_k are its interaction modes. The class introduced in Section 1 satisfies ℳd=F:i-rank(F)≤dM_d=\F:i-rank(F)≤ d\, and the spectrum depends on the reference measures. By the isometry between L2(μU⊗μV)L^2( _U _V) and the Hilbert-Schmidt class, a function is representable by a rank-d head if and only if rank(TS)≤d (T_S)≤ d; hence the parametric class (2) coincides with ℳdM_d (Lemma 1, Appendix B). The spectral truncation term ∑k>dσk2 _k>d _k^2 depends only on the target and the reference measures, not on the encoders: no head, regardless of its capacity, can represent F with error below this quantity. The bound is attained in L2L^2 by the truncated Schmidt series, and is unique when σd>σd+1 _d> _d+1 (Corollary 1, Appendix B). The decay rate of this tail, which is controlled by the smoothness of F (Theorem 3), determines how many interaction modes d a good head must retain. This is the operator-theoretic analogue of classical low-rank matrix approximation: the Schmidt decomposition is the infinite-dimensional Eckart-Young theorem (22; 7), and the truncation term is exactly the Hilbert-Schmidt error of the best rank-d approximation of TFT_F. The spectrum is relative to the reference measures μU,μV _U, _V, which fix the inner product in which the modes are orthonormal; the same target may have different spectra under different measures. All subsequent claims therefore hold for the measures under which a head is trained and evaluated. 2.2 Ten families as instances of the head Appendix A collects ten method families from different communities in Table 5. Each computes a real-valued output for a pair of inputs as the inner product of two separately learned encodings. The questions of how many interaction modes d to keep, how to normalize the two encoders (Section 3), and how many samples the head requires (Section 4) are answered independently by each community; within the present framework, they are the same questions. 2.3 Approximation and separation from single-tower models We now study how well ℳdM_d approximates a fixed target. Corollary 1 isolates the part of the error that no encoder can avoid. The remaining part is attributable to the encoders themselves. We assume that the encoder classes contain functions approaching the scaled modes σkak _ka_k and bkb_k with errors ηf,k,ηg,k _f,k, _g,k as in (20) (Assumption 1, Appendix B); for neural encoders, these errors follow standard ReLU approximation rates (25). Theorem 1 (Approximation error decomposition). Let F∈L2(μU⊗μV)F∈ L^2( _U _V) have interaction spectrum σk\ _k\. Under Assumption 1, with Bf=maxk‖f^k‖L2(μU)B_f= _k\| f_k\|_L^2( _U) and Bg=maxk‖g^k‖L2(μV)B_g= _k\| g_k\|_L^2( _V), inff∈ℱg∈‖F−⟨f,g⟩‖L2(μU⊗μV)2≤∑k>dσk2⏟truncation term+C∑k≤d(σk2ηg,k2+Bg2ηf,k2)⏟realization term, _ subarraycf \\ g subarray\|F- f,g \|_L^2( _U _V)^2\;≤\; _k>d _k^2_truncation term\;+\; C _k≤ d ( _k^2\, _g,k^2+B_g^2\, _f,k^2 )_realization term, (5) where C is an absolute constant. Proof idea. The error is split around the truncated Schmidt series, and the tensor-product product rule is applied term by term. Orthonormality of the ideal modes diagonalizes the two leading sums and avoids a factor of d; the remaining term is bounded by Cauchy-Schwarz. The full calculation, including the explicit constant, is deferred to Appendix B. ∎ The decomposition is tight up to the constant: the truncation term is intrinsic to the target, while the realization term depends on the model. Dense encoder classes therefore approach the lower bound of Corollary 1. Whether training actually finds the coordinate alignment required by (20) is precisely the identifiability question addressed in Section 3. Target smoothness controls the truncation term: an s-smooth target has spectrum σk=O(k−s/mV) _k=O(k^-s/m_V), while real-analytic targets decay exponentially. The precise tail bounds are given in Theorem 3 (Appendix B). The dual-encoder head separates from single-tower models on both the approximation and the computational axes. On the approximation axis, it avoids the curse of the joint dimension: approximating a rank-d target within ε with ReLU encoders requires Ndual=O(d(ε−mU/s+ε−mV/s))N_dual=O (d( ^-m_U/s+ ^-m_V/s) ) (6) parameters, whereas any single-tower model that treats F as a general (mU+mV)(m_U+m_V)-dimensional HsH^s function requires Ω(ε−(mU+mV)/s) ( ^-(m_U+m_V)/s) parameters in the worst case; hence Ndual/Nsingle→0N_dual/N_single→ 0 as ε→0 → 0 (Proposition 2). On the compute axis, evaluating the head on all pairs costs O((n+m)Cnet+nmd)O((n+m)\,C_net+nm\,d) against quadratic joint processing, recovering the complexity arguments of QK-attention matching (29), linear attention (12), and DeepONet (14) (Proposition 1, Appendix B). 3 Identifiability as Gauge-Fixing The theory of Section 2 characterizes what a rank-d head can represent, but it does not specify which representation a training algorithm will actually find. The representation is far from unique: for any invertible matrix A, the pair (Af,A−⊤g)(Af,\,A^- g) represents exactly the same function. Consequently, the encoders of any trained model are defined only up to a symmetry of dimension d2d^2. This redundancy is not purely formal. It renders the population risk constant along large orbits, so the Hessian at a minimizer is highly degenerate and optimization becomes ill-posed. At the same time, it leaves the learned coordinates essentially arbitrary. Normalizations were introduced precisely to control these instabilities (17; 30); a normalization makes the encoders meaningful exactly to the extent that it removes the symmetry. We now formalize this view. Each normalization is a gauge-fixing choice, a rule for selecting one representative from each equivalence class. The residual symmetry that survives a constraint is precisely what remains arbitrary under it. The interaction spectrum constitutes the invariant content of the symmetry, and the gaps in that spectrum control how much identifiability any normalization can provide. Definition 2 (Gauge action). For A∈GLdA _d, the gauge transformation acts on the encoders by ΦA(f,g)=(Af,A−⊤g), _A(f,g)= (Af,\;A^- g ), (7) where (Af)(u)=Af(u)(Af)(u)=A\,f(u) and (A−⊤g)(v)=A−⊤g(v) (A^- g )(v)=A^- g(v). The set of all such transformations is the gauge group of the head. Gauge transformations leave the represented function, and therefore the population risk, unchanged: ⟨Af(u),A−⊤g(v)⟩=⟨f(u),g(v)⟩. Af(u),A^- g(v) = f(u),g(v) . (8) The second moments Σf=u[ff⊤] _f=E_u[f ] and Σg=v[gg⊤] _g=E_v[g ] transform by congruence, so the eigenvalues of the product ΣfΣg _f _g are gauge invariants (Lemma 2, Appendix C). Unfixed gauge also makes optimization ill-posed. Along the orbit of a minimizer the risk is flat, so the Hessian has at least d2−dimSd^2- S zero directions; without normalization this is generically d2d^2 (Proposition 4). A normalization pins exactly the subgroup whose residual freedom it eliminates; a smaller residual group therefore leaves fewer degenerate directions and yields a better-conditioned problem. Normalizations as sections of the orbit. Two levels of normalization occur in practice, and only one affects identifiability. Embedding-level normalization constrains the encoders themselves and determines the residual gauge. Output-level normalization, such as the contrastive softmax in CLIP or top-k selection in retrieval, acts on the score matrix after the inner product; it does not constrain the encoders and leaves the embedding-level gauge untouched. Table 6 in Appendix C classifies the embedding-level schemes used in practice. What each normalization identifies. The residual symmetry of a normalization indicates which transformations can act on a trained model without changing its output. The normalizations used in practice were adopted on heuristic grounds, for training stability or empirical feature decorrelation, rather than from an analysis of which symmetry they remove; the gauge view supplies the missing analysis and lets us rank them by residual symmetry. We characterize the two-sided schemes used in practice, in order of increasing identifiability, and find that only whitening removes the symmetry completely. Cosine normalization is the weakest two-sided scheme (17). Two-sided unit normalization f~=f/‖f‖ f=f/\|f\|, g~=g/‖g‖ g=g/\|g\| with scores cos∠(f(u),g(v)) (f(u),g(v)), as employed by contrastive models such as CLIP (17), leaves a residual gauge exactly equal to the orthogonal group: for every rotation R∈OR , the encodings RfRf and RgRg give pointwise identical cosine scores, and no other transformation does (Theorem 4). Consequently, the individual coordinates of a cosine-normalized embedding carry no intrinsic meaning; only rotation-invariant quantities (norms, pairwise inner products, and the eigenvalues of ΣfΣg _f _g) are well-defined. This provides a theorem-level explanation of the widely reported uninterpretability of contrastive embedding dimensions, and it identifies the whitening scheme of Theorem 2 as the remedy (Corollary 2, Appendix C). Two-sided nonnegativity, f(u)⪰0f(u) 0 and g(v)⪰0g(v) 0 (realized by a softplus or exponential output layer (30)), shrinks the residual group to the monomial group Perm⋉D+Perm _+ of coordinate permutations composed with positive diagonal scalings. Under separability conditions (6; 1), the encoders are identified up to permutation and scaling. The cost is an expressivity limitation: negatively correlated interaction modes are excluded (Theorem 5, Appendix C). Whitening is the unique scheme among those used in practice that removes the continuous gauge symmetry entirely. Cosine and nonnegativity each leave a nontrivial residual group and were adopted largely heuristically; by contrast, we prove that whitening admits a full identifiability guarantee: under a spectral gap it pins the interaction modes up to permutation and sign, recovering both the modes and the spectrum itself. Theorem 2 (Whitening fixes the gauge up to permutation and sign). Let the interaction singular values be distinct, σ1>⋯>σd>σd+1 _1>·s> _d> _d+1, and impose the whitening constraints Σg=Id,Σf=Λ=diag(λ1,…,λd),λ1≥⋯≥λd≥0, _g=I_d, _f= =diag( _1,…, _d), _1≥·s≥ _d≥ 0, (9) with Λ diagonal and nonincreasing. Constraints of this form are standard in self-supervised representation learning, where they are used to decorrelate features (8; 26; 2). Then every global minimizer of the population risk satisfies fk=±σkak,gk=±bk,λk=σk2,k=1,…,d,f_k=±\, _k\,a_k, g_k=±\,b_k, _k= _k^2, k=1,…,d, (10) with signs paired so that the product fkgk=σkakbkf_kg_k= _ka_kb_k is uniquely determined, and with the coordinate labeling fixed by the ordering convention in (9). The k-th coordinate of the whitened encoders recovers the k-th interaction mode of the target, and the diagonal of Λ recovers the interaction spectrum. Equivalently, the encoders are identified up to permutation and sign. Proof idea. The rank-truncation bound of Corollary 1 implies that every minimizer attains the minimal risk. When σd>σd+1 _d> _d+1, this forces the represented function to equal the unique truncation FdF_d. The whitening constraints then require the coefficient matrices to satisfy CC⊤=ICC =I and CS2C⊤=ΛCS^2C = , where S=diag(σ1,…,σd)S=diag( _1,…, _d) has distinct eigenvalues; this forces C to be a signed permutation. The full argument is in Appendix C. ∎ Implementing the whitening gauge. Hard constraints (9) can be enforced by minibatch whitening. Alternatively, a quadratic penalty inherits the identifiability of the hard constraint and improves conditioning monotonically with the penalty weight (Theorem 6, Appendix C). In practice, the second moments are estimated from a finite sample. With probability 1−δ1-δ, the identified modes carry error of order r/Δr/ , where r=O(B2log(d/δ)/n)r=O (B^2 (d/δ)/n ), B bounds the encoder outputs, and Δ=mink(σk2−σk+12) = _k( _k^2- _k+1^2) is the spectral gap. Achieving accuracy ε therefore requires n≳B4log(d/δ)Δ2ε2n B^4 (d/δ) ^2 ^2 (11) samples (Appendix E). The same gap thus governs optimization conditioning, identifiability, and estimation simultaneously (Corollary 3). Section 5 measures the gauge, conditioning, and expressivity consequences of the various schemes on controlled targets; the two real systems, branch-trunk operator learning (14) and independently trained CLIP models, are treated in Appendix F (Theorem 4). Appendix C lists the measurable quantities on which the schemes are compared. 4 Sample Complexity and When Not to Use the Head The interaction spectrum controls both estimation error and fundamental limitations. The excess risk of a head fitted from n samples decomposes into an approximation term that decreases as the rank d grows, and an estimation term that increases with d through the complexity of the encoder classes. Balancing these two terms yields a rank-selection rule. When the spectrum is flat, however, every rank-d head suffers a relative-error floor that no amount of capacity or data can overcome; this yields a practical criterion for abandoning the architecture. Setup and notation. We observe n independent samples (ui,vi,yi)(u_i,v_i,y_i) with yi=F(ui,vi)+εiy_i=F(u_i,v_i)+ _i and fit the head by empirical risk minimization over the class ℋd=(u,v)↦⟨f(u),g(v)⟩:f∈ℱ,g∈H_d= \\,(u,v) f(u),g(v) :\ f ,\ g \, \ (12) of rank-d heads. The encoders are bounded, ‖f(u)‖≤Bf\|f(u)\|≤ B_f and ‖g(v)‖≤Bg\|g(v)\|≤ B_g, so the scores are bounded by B=BfBgB=B_fB_g. Let ℜn R_n denote the Rademacher complexity, extended to vector-valued classes coordinatewise. The class complexity collapses to the sum of the marginal complexities: ℜn(ℋd)≤2d(Bgℜn(ℱ)+Bfℜn()), R_n(H_d)≤ 2d (B_g\, R_n(F)+B_f\, R_n(G) ), (13) for shared per-coordinate classes (Lemma 3, Appendix D). The excess risk of the fitted head therefore decomposes into the approximation terms of Theorem 1 plus an estimation term controlled by this complexity: ‖F^−F‖L2(μU⊗μV)2≲∑k>dσk2+ℰreal+Bd(ℜn(ℱ)+ℜn())+B2log(1/δ)n,\| F-F\|_L^2( _U _V)^2\; \; _k>d _k^2+E_real+B\,d ( R_n(F)+ R_n(G) )+B^2 (1/δ)n, (14) where ℰreal=C∑k≤d(σk2ηg,k2+Bg2ηf,k2)E_real=C _k≤ d( _k^2 _g,k^2+B_g^2 _f,k^2) and ≲ hides absolute constants, with probability 1−δ1-δ (Corollary 4, Appendix D). Linear encoders anchor the rate; rank selection. With linear encoders f(u)=U⊤uf(u)=U u, g(v)=V⊤vg(v)=V v on inputs u∈ℝpu ^p, v∈ℝqv ^q, the head computes u⊤Θvu v with Θ=UV⊤ =UV of rank at most d. Estimation reduces to low-rank matrix sensing of the ground-truth matrix Θ⋆ , whose minimax rate, with σε2 _ ^2 the noise variance of εi _i, is known exactly: ‖Θ^−Θ⋆‖F2≍σε2d(p+q)/n.E\| - \|_F^2 _ ^2\,d(p+q)/n. (15) This rate provides the baseline for the general bound (Theorem 7, Appendix D). The rank d is the primary design choice for the head. It faces two opposing forces: the truncation term ∑k>dσk2 _k>d _k^2 decreases with d, while the estimation term grows linearly with d. The optimal rank balances these two contributions. When the spectrum decays exponentially, σk2=O(e−2ck) _k^2=O(e^-2ck), balancing gives d⋆≍12clognp+q,d 12c np+q, (16) and yields a near-parametric rate ‖F^−F‖L22≲(p+q)log(n/(p+q))n.\| F-F\|_L^2^2 (p+q) (n/(p+q))n. (17) Fast spectral decay thus eliminates the curse of dimensionality (Corollary 5, Appendix D). The three terms in (14) form the complete error ledger: fast decay keeps d⋆d small; slow decay makes the head expensive in every respect. The rest of this section analyzes the regime in which even the optimal d⋆d is inadequate. Identifiability has a sample cost. Identifiability, as characterized in Section 3, is a population-level statement. Realizing it from finite samples incurs an additional cost, detailed in Appendix E (Corollary 6). The total mode error of a fitted and subsequently whitened head (Proposition 6) decomposes into the excess risk of Corollary 4 plus the empirical whitening error. Both terms are divided by the spectral gap Δ and decay as n−1/2n^-1/2. Late versus early interaction. We now compare the multiplicative head, which we refer to as a late-interaction model, with early-interaction models in which the two inputs are combined during encoding. By Corollary 1, every rank-d head pays at least the spectral tail, so the spectrum determines the outcome. Define the effective interaction rank dε(F)=mind:∑k>dσk2∑k≥1σk2≤εd_ (F)= \d: _k>d _k^2 _k≥ 1 _k^2≤ \ (18) as the smallest embedding dimension for which a rank-d head achieves relative error ε . The spectrum is called flat at scale N when dε(F)=Ω((1−ε)N)d_ (F)= ((1- )N). The equality function realizes this worst case exactly. On ==±1mU=V=\± 1\^m with N=2mN=2^m, the interaction operator satisfies TFeq=1NIdT_F_eq= 1NId. The singular values are all equal to 1/N1/ N, the interaction rank is N, and every rank-d head achieves relative error exactly 1−d/N1-d/N. Thus, attaining relative error ε requires d≥(1−ε)2md≥(1- )2^m embedding dimensions, exponential in the input size (Theorem 8, Appendix D). By contrast, an early-interaction predictor with O(m)O(m) parameters represents FeqF_eq exactly. The identity ⟨u,v⟩=m−2Ham(u,v) u,v =m-2\,Ham(u,v) allows a simple threshold on the inner product to recover the equality indicator, whether implemented by a cross-attention layer with identity projections or by a joint MLP with O(m)O(m) ReLUs. Combined with Theorem 8, this yields an exponential separation: O(m)O(m) parameters suffice under early interaction, while any late-interaction head requires Ω(2m) (2^m) dimensions to reach constant relative error (Theorem 9, Appendix D). These observations lead to a concrete usability criterion (Theorem 10, Appendix D). When the effective interaction rank dε(F)d_ (F) is small, a late-interaction head is appropriate: its approximation and estimation errors are controllable, and its computational and statistical advantages are preserved. When the spectrum is flat, the truncation floor forces every rank-d head to relative error at least ε , while early-interaction models can still reach ε with polynomial resources when the target has low-dimensional joint structure. The deciding quantity, the spectral decay rate, can be estimated from data by a weighted singular value decomposition of the empirical kernel matrix, making the criterion actionable before either model is trained. The criterion is a boundary, not a verdict. The early-interaction upper bounds assume low-dimensional joint structure; a target that is both high-rank and unstructured is difficult for both architectures. Moreover, the parameter counts in Theorem 9 are statements about representational existence, not about learnability. Section 5 instantiates both sides of the dichotomy: on gapped spectra the head identifies and estimates well, while flat and narrow-band targets drive it to the exponential-rank floor. 5 Experiments Our experiments are designed for mechanism validation rather than benchmark chasing: each study isolates a quantity that the theory predicts exactly. The controlled synthetic setting (Section 5.1) supplies exact ground truth and is reported in full, with the tables and figures supporting each conclusion placed alongside the corresponding paragraph. Two real architectures instantiate the multiplicative head directly, DeepONet and CLIP, and are reported in full in Appendix F due to page limitation; here we state only the conclusions they establish. On DeepONet, post-hoc whitening recovers the analytic operator eigenbasis from an end-to-end-trained model. On two pretrained CLIP pairs, the interaction spectra are shared across models while per-dimension concept probes are not, and fitting the single residual rotation reconciles them. 5.1 Controlled synthetic study Setup. This study validates the identifiability and usability claims of Sections 3 and 4 in a setting with exact ground truth. We fix a target F(u,v)=∑kσkak(u)bk(v)F(u,v)= _k _ka_k(u)b_k(v) with shifted Legendre modes and a prescribed spectrum, fit linear dual encoders over a fixed Legendre feature map, and record four quantities per run: the fit error, the alignment of the recovered modes with the true ak,bka_k,b_k, the encoder-covariance conditioning, and the cross-seed gauge distance. The encoder-realization error of Assumption 1 is zero, so normalization is the only variable across runs. The four paragraphs below use these quantities to test, in turn, the identifiability ranking of the gauges (Theorem 2), the gap law of estimation (Corollaries 3 and 6), the sample-complexity rate (Theorem 7), and the flat-spectrum floor (Theorem 8). Normalization as gauge-fixing. The seven schemes of Table 1 separate cleanly by residual group. Only whitening attains perfect mode alignment (1.0001.000) with zero cross-seed gauge distance and drives the conditioning diagnostic max(condΣf,condΣg) (cond\, _f,cond\, _g) to its theoretical floor σ12/σd2≈9.5 _1^2/ _d^2≈ 9.5 (Corollary 3); every looser gauge remains stuck at alignment 0.530.53 to 0.650.65, its residual group leaving the modes rotationally entangled (Theorem 4). The gauge-invariant diagnostic cond(ΣfΣg)cond( _f _g) cannot separate the schemes (Lemma 2). The two-sided nonnegative gauge incurs error 4.254.25, the predicted out-of-class cost of excluding negatively correlated modes (Theorem 5). Normalization is therefore gauge fixing, and whitening is the only scheme that removes the symmetry completely, exactly as Theorem 2 predicts. Table 1: The seven normalization schemes, four seeds each. Residual gauge: the subgroup that survives the constraint (Theorem 2 predicts Perm⋉D±Perm _± for whitening, OO for cosine, Perm⋉D+Perm _+ for two-sided nonnegativity). Fit MSE: squared regression error. Alignment: mean mode alignment of the two encoders after optimal permutation and sign matching. Cond: max(condΣf,condΣg) (cond\, _f,cond\, _g), gauge dependent. Cross-seed: mean pairwise gauge distance between seeds. Normalization Residual gauge Fit MSE Align. (f,g) Cond Cross-seed none GLdGL_d 0.0004 0.605, 0.619 13.8 0.326 l2_single ≈GLd _d 0.0092 0.561, 0.420 16.8 0.284 per_coord OO+shear 0.0004 0.607, 0.593 18.7 0.330 clip_cosine OO 0.0568 0.529, 0.550 305.9 0.212 nonneg Perm⋉D+Perm _+ 4.251 0.576, 0.446 121.0 0.280 whiten_soft →Perm⋉D± _± 0.0169 0.651, 0.549 105.5 0.348 whiten_hard Perm⋉D±Perm _± 0.0004 1.000, 1.000 9.5 0.000 The gap as the single unifying quantity. The spectral gap is the single quantity governing identification. With the finite-sample whitening estimator of Appendix E, we sweep the gap at fixed sample size and the sample size at fixed gap, eight configurations in all (Table 2; Figure 3 in Appendix F plots the collapse). The mode-identification error scales as 1/Δ1/ at fixed sample size and as n−1/2n^-1/2 at fixed gap, where Δ=mink(σk2−σk+12) = _k( _k^2- _k+1^2), and the two sweeps collapse onto the single relation err⋅Δ⋅n≈2.48err· · n≈ 2.48 (19) with coefficient of variation 0.120.12: gap and sample size enter the identification error only through their product, the concrete face of Corollaries 3 and 6. Table 2: Gap and sample-size sweeps of the finite-sample whitening estimator (Appendix E). Left group: the spectral gap Δ=mink(σk2−σk+12) = _k( _k^2- _k+1^2) swept by scaling a base spectrum at fixed n=3200n=3200. Right group: sample size swept at fixed Δ=0.900 =0.900. Mode error is the mean mode identification error; the product err⋅Δ⋅nerr· · n is near-constant across both groups, ≈2.48≈ 2.48. Fixed n=3200n=3200 Fixed Δ=0.900 =0.900 Scale Δ Mode err n Mode err 1.00 0.900 0.0431 200 0.1979 0.60 0.648 0.0782 800 0.0956 0.35 0.417 0.1258 3200 0.0470 0.20 0.230 0.2458 12800 0.0236 The sample-complexity rate law. In the linear regime the head is exactly a rank-d matrix sensing problem, and Theorem 7 predicts ‖Θ^−Θ⋆‖F2≍σε2d(p+q)/nE\| - \|_F^2 _ ^2d(p+q)/n. We sweep each of the three variables with the others held fixed and fit log-log slopes, the three sweeps shown in Figure 1. The measured slopes are −1.03-1.03 in n, +0.89+0.89 in d, and +1.24+1.24 in p+qp+q, against the theoretical −1,+1,+1-1,+1,+1; the excess in the last slope is the expected Marchenko-Pastur finite-sample correction, and gradient descent on the factored pair matches the least-squares slope. These sweeps reject the loose general bound of Remark 6, which would predict slope −1/2-1/2: estimation is governed by the sum of the marginal encoder complexities, and the 1/n1/n dependence is genuine. Figure 1: The sample-complexity law of Theorem 7. Log-log sweeps of the Frobenius excess risk against sample size n (left), interaction rank d (center), and ambient dimension p+qp+q (right). Points are the measured risk; the dotted line is the theoretical rate σ2d(2p−d)/nσ^2d(2p-d)/n, with the finite-n corrected form (dashed) in the right panel. The usability criterion. A flat spectrum floors the head regardless of capacity. We test two flat and near-flat regimes, both tabulated in Tables 4 and 4. On the Boolean equality kernel F=[u=v]F=1[u=v] on ±1m\± 1\^m, which has a completely flat spectrum, a rank-d head incurs relative error exactly 1−d/N1-d/N and requires d≈0.9⋅2md≈ 0.9· 2^m dimensions to reach 10%10\% error, while an early-interaction threshold on ⟨u,v⟩ u,v is exact with 2m+12m+1 parameters (Theorems 8 and 9). Smooth narrow-band kernels give the polynomial counterpart dε=Θ(1/h)d_ = (1/h) (Remark 7). A trained-model anchor on a block-boxcar kernel realizes the same floor (Table 10 and Figure 5, Appendix F). Table 3: The boolean equality kernel F=[u=v]F=1[u=v] on ±1m\± 1\^m, N=2mN=2^m. Rank d: embedding dimension at which the head first reaches 10%10\% relative error. Early: parameters 2m+12m+1 and error of the early-interaction comparator. m N Rank d d/Nd/N Early params Early error 3 8 8 1.00 7 0 4 16 15 0.94 9 0 5 32 29 0.91 11 0 6 64 58 0.91 13 0 7 128 116 0.91 15 0 8 256 231 0.90 17 0 Table 4: Narrowband periodic-Gaussian kernel of bandwidth h on the circle: interaction rank dεd_ for ε=10% =10\% error. The rank grows as Θ(1/h) (1/h) (Remark 7). h 1/h1/h dεd_ 0.200 5 3 0.100 10 4 0.050 20 8 0.025 40 15 0.0125 80 30 6 Conclusion This paper has examined a single architectural pattern, the multiplicative dual-encoder head, through a unifying lens: the interaction spectrum of the target. Through theoretical analysis and numerical experiments, we answer four central questions. (i) Approximation error decomposes into a truncation term intrinsic to the target and a realization term intrinsic to the encoders, with target smoothness controlling the decay rate. (i) Sample complexity is governed by the sum of the two marginal encoder complexities, with the linear regime anchoring the minimax rate. (i) Usability is determined by the decay rate itself: a flat spectrum imposes an error floor that no rank-d head can escape, while early-interaction models can, and the deciding quantity is measurable from data before either model is trained. (iv) Identifiability, the problem that the framework brings into focus, is gauge fixing: every normalization used in practice is a section of the gauge orbit; whitening pins the interaction modes up to permutation and sign; and the spectral gap simultaneously controls optimization conditioning, identifiability, and estimation. Ethics Statement This work adheres to the principles outlined in the ICLR Code of Ethics. References Arora et al. (2012) S. Arora, R. Ge, R. Kannan, and A. Moitra Computing a nonnegative matrix factorization–provably. In Proceedings of the forty-fourth annual ACM symposium on Theory of computing, p. 145–162. Cited by: Appendix A, Appendix C, §E.2, §3, Definition 3, Theorem 5. Bardes et al. (2022) A. Bardes, J. Ponce, and Y. LeCun VICReg: variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, Cited by: Appendix A, Theorem 2. Barreto et al. (2017) A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. Van Hasselt, and D. Silver Successor features for transfer in reinforcement learning. Advances in neural information processing systems 30. Cited by: Appendix A, Table 5, §1. Bartlett and Mendelson (2002) P. L. Bartlett and S. Mendelson Rademacher and gaussian complexities: risk bounds and structural results. Journal of machine learning research 3 (Nov), p. 463–482. Cited by: Appendix D. Candes and Plan (2011) E. J. Candes and Y. Plan Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory 57 (4), p. 2342–2359. Cited by: Appendix A, Appendix D. Donoho and Stodden (2003) D. Donoho and V. Stodden When does non-negative matrix factorization give a correct decomposition into parts?. Advances in neural information processing systems 16. Cited by: Appendix A, Appendix C, §3, Definition 3, Theorem 5. Eckart and Young (1936) C. Eckart and G. Young The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: §2.1. Ermolov et al. (2021) A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe Whitening for self-supervised representation learning. In International conference on machine learning, p. 3015–3024. Cited by: Appendix A, Theorem 2. Hong et al. (2021) Z. Hong, G. Yang, and P. Agrawal Bi-linear value networks for multi-goal reinforcement learning. In International Conference on Learning Representations, Cited by: Appendix A, Table 5, §1. Jun et al. (2019) K. Jun, R. Willett, S. Wright, and R. Nowak Bilinear bandits with low-rank structure. In International Conference on Machine Learning, p. 3163–3172. Cited by: Appendix A, Remark 6. Karpukhin et al. (2020) V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), p. 6769–6781. Cited by: Appendix A, Table 5, §1. Katharopoulos et al. (2020) A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret Transformers are rnns: fast autoregressive transformers with linear attention. In International conference on machine learning, p. 5156–5165. Cited by: Appendix A, Table 5, §1, §2.3, Remark 2. Little and Reade (1984) G. Little and J. B. Reade Eigenvalues of analytic kernels. SIAM journal on mathematical analysis 15 (1), p. 133–136. Cited by: Appendix B. Lu et al. (2021) L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence 3 (3), p. 218–229. Cited by: Appendix A, Table 5, §1, §2.3, §3, Remark 2. Maurer (2016) A. Maurer A vector-contraction inequality for rademacher complexities. In International Conference on Algorithmic Learning Theory, p. 3–17. Cited by: Appendix D. Negahban and Wainwright (2009) S. N. Negahban and M. J. Wainwright Estimation of (near) low-rank matrices with noise and high-dimensional scaling. In International Conference on Machine Learning, External Links: Link Cited by: Appendix A, Appendix D. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: Table 5, Table 6, §1, §3, §3, Theorem 4. Reade (1983) J. B. Reade Eigenvalues of positive definite kernels. SIAM Journal on Mathematical Analysis 14 (1), p. 152–157. Cited by: Appendix B. Rendle (2010) S. Rendle Factorization machines. In 2010 IEEE International conference on data mining, p. 995–1000. Cited by: Appendix A, Table 5, §1. Rohde and Tsybakov (2009) A. Rohde and A. Tsybakov Estimation of high-dimensional low-rank matrices. Annals of Statistics 39, p. 887–930. External Links: Link Cited by: Appendix A, Appendix D. Schaul et al. (2015) T. Schaul, D. Horgan, K. Gregor, and D. Silver Universal value function approximators. In International conference on machine learning, p. 1312–1320. Cited by: Appendix A, Table 5, §1. Schmidt (1907) E. Schmidt Zur theorie der linearen und nichtlinearen integralgleichungen: i. teil: entwicklung willkürlicher funktionen nach systemen vorgeschriebener. Mathematische Annalen 63 (4), p. 433–476. Cited by: §2.1. Trouillon et al. (2016) T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard Complex embeddings for simple link prediction. In International conference on machine learning, p. 2071–2080. Cited by: Appendix A, Table 5, §1. Yang et al. (2014) B. Yang, W. Yih, X. He, J. Gao, and L. Deng Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. Cited by: Appendix A, Table 5, §1. Yarotsky (2017) D. Yarotsky Error bounds for approximations with deep relu networks. Neural networks 94, p. 103–114. Cited by: Appendix B, §2.3, Remark 7. Zbontar et al. (2021) J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny Barlow twins: self-supervised learning via redundancy reduction. In International conference on machine learning, p. 12310–12320. Cited by: Appendix A, Theorem 2. Zhao et al. (2025) Z. Zhao, T. Chen, Z. Cai, X. Li, H. Li, Q. Chen, and G. Zhu Crossfi: a cross domain wi-fi sensing framework based on siamese network. IEEE Internet of Things Journal 12 (12), p. 20138–20155. Cited by: §1. Zhao and Li (2025) Z. Zhao and S. Li The impacts of data privacy regulations on food-delivery platforms. Transportation Research Part C: Emerging Technologies 181, p. 105364. Cited by: Appendix A. Zhao and Li (2026a) Z. Zhao and S. Li Discriminatory order assignment and payment-setting of on-demand food-delivery platforms: a multi-action and multi-agent reinforcement learning framework. Transportation Research Part E: Logistics and Transportation Review 208, p. 104653. Cited by: Appendix A, Table 5, §1, §2.3. Zhao and Li (2026b) Z. Zhao and S. Li Triple-bert: do we really need marl for order dispatch on ride-sharing platforms?. In International Conference on Learning Representations, Vol. 2026, p. 153780–153808. Cited by: Appendix A, Table 5, Table 6, §1, §3, §3, Remark 2, Remark 4. Appendix Contents Appendix A Derivations for the Unification Table This appendix spells out, for each of the ten families of Table 5, the identification that makes it an instance of the multiplicative dual-encoder head (1): the two input spaces, the encoder maps, the score, the reference measures that define the interaction spectrum, and the boundary cases where an additional output-level normalization acts after the inner product and therefore leaves the gauge untouched. Table 5 collects the ten families, each computing a real-valued output for a pair of inputs as the inner product of two separately learned encodings; reading across the rows makes visible what is shared, and the paragraphs below identify the interaction operator and its spectrum for each family, including the boundary cases. CLIP. Inputs are an image I and a text T; the encoders are the image tower f and the text tower g, both ℓ2 _2-normalized; the score is sim(I,T)=⟨f(I),g(T)⟩sim(I,T)= f(I),g(T) , rescaled by a temperature and fed into a softmax over the batch. The reference measures are the empirical distributions over images and over texts in the training batch. The temperature and the softmax act on the similarity matrix after the inner product, so they are output-level and do not affect the embedding-level gauge; the unit-norm constraints are embedding-level and leave exactly the residual rotation OO of Theorem 4. DeepONet. Inputs are an input function a (on a sensor grid) and a query point y; the encoders are the branch net b and the trunk net t; the score is (a)(y)≈⟨b(a),t(y)⟩G(a)(y)≈ b(a),t(y) (14). The reference measures are the distribution over input functions and the measure over query points. For a linear operator with kernel κ=∑kσkβk(y)γk(x)κ= _k _k _k(y) _k(x) and a white input measure, the interaction spectrum coincides with the operator singular values and the trunk’s ideal basis with the left singular functions, which is the ground truth used in Appendix F.2. The boundary case is the scale imbalance between branch and trunk outputs, familiar in practice; it is the scale redundancy of the gauge group, diagnosed in Section 3. POD-DeepONet substitutes a precomputed orthogonal POD basis for the learned trunk (14), motivated by the same nonuniqueness; our treatment is complementary, characterizing when the learned basis becomes identifiable instead of replacing it (Appendix F.2). Two-tower retrieval. Inputs are a query q and a document d; the encoders are two towers φ,ψ ,ψ; the score is a relevance measure rel(q,d)=⟨φ(q),ψ(d)⟩rel(q,d)= (q),ψ(d) (11). The reference measures are the query distribution and the document distribution, and the interaction spectrum is the relevance spectrum of the underlying ranker. The top-k nearest-neighbor search is output-level. UVFA (universal value function approximators). Inputs are a state s and a goal g; the encoders are the state encoder ϕ(s)φ(s), a learned map from states to features, and the goal encoder ψ(g)ψ(g), a learned map from goals to features; the score is the goal-conditioned value V(s,g)=⟨ϕ(s),ψ(g)⟩V(s,g)= φ(s),ψ(g) (21). The reference measures are the state-visitation distribution and the goal distribution. Goal-conditioned value functions are naturally low-rank when goals and states share a low-dimensional feature structure, which is exactly the case in which the representation (1) is sample-efficient (Section 4). The interaction spectrum also gives these methods a rank-selection rule they currently lack: the number of features in a successor-feature representation or the width of a value-network embedding is typically a tuned hyperparameter, whereas Corollary 5 ties it to the decay of the spectrum. BVN (bilinear value networks). Inputs are a state-action pair (s,a)(s,a) and a state-goal pair (s,g)(s,g); the encoders are the MLP f(s,a)f(s,a) encoding the state-action pair and the MLP φ(s,g) (s,g) encoding the state-goal pair; the score is Q(s,a,g)=⟨f(s,a),φ(s,g)⟩Q(s,a,g)= f(s,a), (s,g) (9). The reference measures are the joint distributions over (s,a)(s,a) and (s,g)(s,g). Note that the two arguments share the state s: the head’s theory applies to the function Q of the pair of arguments, and the shared state is exactly the coupling that makes Q non-separable in (s,a,g)(s,a,g) directly, which is why the two-input factorization matters. Successor features. Inputs are a state s and a reward weight vector w; the encoder on the state side is the successor-feature map ϕ(s)φ(s) and the encoder on the weight side is the identity w (a singleton encoder class, not learned); the score is Vw(s)=⟨ϕ(s),w⟩V_w(s)= φ(s),w (3). This is the head with =idG=\id\, whose Rademacher complexity is zero, so the sample-complexity bounds of Section 4 specialize to the standard guarantee for successor features: only the state-side class matters. DistMult and ComplEx. Inputs are a head entity h and a tail entity t; the encoders are entity embeddings eh,et∈ℝde_h,e_t ^d; the score is the bilinear form ⟨eh,r∘et⟩ e_h,r e_t of DistMult or the real part of the complex product eh⊤diag(r)et¯e_h \,diag(r)\, e_t of ComplEx, where r is the relation embedding (24; 23). Both are rank-d bilinear forms on the entity embeddings, i.e., heads with linear encoders and a fixed middle matrix; the interaction spectrum is governed by the Gram structure of the entity embeddings under the relation. The reference measure is the empirical distribution over observed triples. Linear attention. Inputs are a query q and a key k in a common space; the encoders are the feature map ϕφ applied to each; the attention weight is ⟨ϕ(q),ϕ(k)⟩ φ(q),φ(k) (12). The aggregation step ∑jϕ(kj)vj⊤ _jφ(k_j)v_j makes the compute separation of Proposition 1 exact: the pairwise inner products are never formed, giving the O(n2)→O(n)O(n^2)→ O(n) complexity. The reference measure is the distribution over keys (or key-value pairs). Finite-dimensional feature maps realize the head exactly; softmax attention with random-feature maps is the limiting case of an infinite-dimensional ϕφ. Factorization machines. Inputs are a feature index i and a feature index j; the encoders are the embedding maps vi,vjv_i,v_j; the pairwise term is xixj⟨vi,vj⟩x_ix_j v_i,v_j (19). The reference measure is the empirical distribution over feature pairs weighted by the data. The interaction spectrum of the pairwise term is the spectrum of the Gram matrix of the embeddings, and the rank selection of Section 4 is the standard FM capacity control viewed through the interaction spectrum. QK-attention matching. Inputs are a worker w and an order o; the encoders are a BERT-style encoder f and an MLP g; the score is the match value ⟨f(w),g(o)⟩ f(w),g(o) forming the Q-matrix of a bipartite assignment problem, with a one-sided softplus and unit-norm constraint on the order encoder (30; 29; 28). The reference measure is the empirical distribution over worker-order pairs. The matching layer that consumes the score matrix is output-level; the one-sided constraint is embedding-level but, as Remark 4 explains, it is a stability fix rather than an identifiability scheme. The two papers differ only in the downstream use of the same head, which is why they share the identifiability properties derived in this paper. Table 5: Ten families of methods as instances of the multiplicative dual-encoder head (1): for each family, the two inputs, the encoder maps, the output, and the downstream use. Instance Inputs Encoders Output Downstream use CLIP (17) image, text ViT, text transformer normalized similarity contrastive softmax over a batch DeepONet (14) function a, point y branch net, trunk net ⟨branch(a),trunk(y)⟩ (a),trunk(y) operator output (a)(y)G(a)(y) Two-tower retrieval (11) query, document two towers relevance top-k ANN search UVFA (21) state s, goal g ϕ(s)φ(s), ψ(g)ψ(g) V(s,g)V(s,g) value function BVN (9) (s,a)(s,a), (s,g)(s,g) two MLPs Q(s,a,g)Q(s,a,g) multi-goal RL Successor features (3) state s, weights w ϕ(s)φ(s), w Vw(s)V_w(s) transfer DistMult / ComplEx (24; 23) head, tail entity entity embeddings bilinear / complex product link prediction Linear attention (12) query q, key k feature map ϕφ ⟨ϕ(q),ϕ(k)⟩ φ(q),φ(k) O(n2)→O(n)O(n^2)→ O(n) aggregation Factorization machines (19) feature i, feature j embedding viv_i ⟨vi,vj⟩xixj v_i,v_j x_ix_j pairwise interaction QK-attention matching (30; 29) worker w, order o BERT, MLP ⟨f(w),g(o)⟩ f(w),g(o) Q-matrix for ILP matching Positioning against prior work. The paragraphs above identify each family and its interaction operator; we now position the paper’s claims against the closest prior work, none of which treats the families as instances of a shared architecture or formulates the identifiability question. On the estimation side, the linear case of the head is rank-d matrix sensing, and the paper relies on the classical minimax theory, upper bounds via restricted-isometry-type analyses (5; 16) and matching lower bounds (20); bilinear bandits (10) study the online counterpart. The contribution relative to this line is a translation rather than a new bound: the estimation problem of a trained head is identified with matrix sensing, and the minimax rate is connected to the rank-selection rule of Section 4 through the decay of the interaction spectrum, which the matrix-sensing literature does not formulate because it treats the rank as given. On the identifiability side, the direct precursor of our remedy is self-supervised whitening: W-MSE applies hard whitening to the embeddings (8), and Barlow Twins and VICReg implement its soft relaxations, pushing the cross-correlation matrix toward the identity and regularizing variance and covariance (26; 2); it is also known that whitening reduces a general linear gauge to an orthogonal ambiguity, a polar-decomposition observation. Against this background, our contribution is not the whitening operation itself, which we do not claim, but three things: the interpretation of normalization as a section of the gauge orbit and the classification of the residual subgroups across the normalizations used in practice (Table 6, Appendix C); the theorem that whitening together with a spectral gap pins the modes down to permutation and sign, strictly beyond the orthogonal ambiguity that decorrelation alone leaves (Theorem 2); and the statement that the interaction spectrum is the unique gauge-invariant content of a trained model, an interpretability claim with a theorem behind it rather than an empirical observation. The nonnegative variant of the bilinear form has its own classical identifiability theory, separability-type conditions for uniqueness up to permutation and scale (6; 1), which the gauge analysis of Section 3 subsumes as a special case (Theorem 5). Appendix B Proofs for Section 2 We prove the results of Section 2 in order. Throughout, ∥⋅∥\|·\| denotes the L2(μU⊗μV)L^2( _U _V) norm, and ∥⋅∥L2(μU)\|·\|_L^2( _U), ∥⋅∥L2(μV)\|·\|_L^2( _V) the marginal norms. Assumption 1 (Encoder realizability). The encoder classes ℱF and G contain functions f^,g f, g such that, for some nonnegative errors ηf,k,ηg,k _f,k, _g,k, ∥f^k−σkak∥L2(μU)≤ηf,k,∥g^k−bk∥L2(μV)≤ηg,k,k=1,…,d.\| f_k- _ka_k\|_L^2( _U)≤ _f,k, \| g_k-b_k\|_L^2( _V)≤ _g,k, k=1,…,d. (20) Remark 1 (What the decomposition buys). The truncation term is intrinsic to the problem, the realization term to the model; when the encoder classes are dense in L2L^2 the bound approaches the lower bound of Corollary 1, so the decomposition is tight up to the constant. Whether training finds the coordinate alignment of (20) is the question of identifiability addressed in Section 3. Proposition 1 (Compute separation). Evaluating a rank-d head on all n×mn× m input pairs costs O((n+m)Cnet+nmd),O ((n+m)\,C_net+nm\,d ), (21) where CnetC_net is the cost of one encoder forward pass, whereas a model that processes each pair jointly costs O(nmCnet)O(nm\,C_net). When the encoder forward pass dominates the inner product, the head is cheaper by a factor Cnet/dC_net/d. Remark 2 (Known complexity results as instances). Proposition 1 recovers, as special cases, the multiplicative-to-additive complexity argument of the QK-attention matching networks (30), the O(n2)→O(n)O(n^2)→ O(n) aggregation of linear attention (12), where the attention weight is written as the inner product of feature maps, and the efficient branch-trunk evaluation of DeepONet (14). Remark 3 (Where low interaction rank comes from). Three structural sources make the truncation term small: exact low rank (F=∑j≤rψj(u)χj(v)F= _j≤ r _j(u) _j(v) implies i-rank(F)≤ri-rank(F)≤ r), tensor-product structure (for F in a tensor-product RKHS the spectrum is governed by the marginal kernels), and smoothness (Theorem 3 gives explicit rates). Conversely, a flat spectrum, developed in Section 4, means the truncation term does not decay and no choice of d helps. Lemma 1 (Rank equivalence). A function S∈L2(μU⊗μV)S∈ L^2( _U _V) can be written as S(u,v)=⟨f(u),g(v)⟩S(u,v)= f(u),g(v) for some encoders f:→ℝdf:U ^d and g:→ℝdg:V ^d if and only if rank(TS)≤d (T_S)≤ d. In particular, the parametric class of eq. (2) coincides with ℳd=F:i-rank(F)≤dM_d=\F:i-rank(F)≤ d\, and we use the two descriptions interchangeably. Proof of Lemma 1 (Rank equivalence). The forward direction is immediate: if S=∑k≤dfk⊗gkS= _k≤ df_k g_k, then the interaction operator is a sum of at most d rank-one operators, TS=∑k≤d⟨⋅,gk⟩L2(μV)fk,T_S= _k≤ d ·,g_k _L^2( _V)\,f_k, (22) so rank(TS)≤d (T_S)≤ d. Conversely, if rank(TS)=r≤d (T_S)=r≤ d, the Schmidt decomposition of S reads S=∑k≤rσkak⊗bkS= _k≤ r _k\,a_k b_k with σk>0 _k>0, and the encoders f=(f1,…,fd)f=(f_1,…,f_d), g=(g1,…,gd)g=(g_1,…,g_d) defined by fk=σkakf_k= _ka_k and gk=bkg_k=b_k for k≤rk≤ r, and fk=gk=0f_k=g_k=0 for k>rk>r, satisfy S=⟨f,g⟩S= f,g . Corollary 1 (Spectral truncation lower bound). For every rank-d head (f,g)(f,g), ‖F−⟨f,g⟩‖L2(μU⊗μV)2≥∑k>dσk2,\|F- f,g \|_L^2( _U _V)^2\;≥\; _k>d _k^2, (23) and the bound is attained in L2L^2 by the truncated Schmidt series Fd=∑k≤dσkak⊗bkF_d= _k≤ d _k\,a_k b_k, which is the unique best rank-d approximation whenever σd>σd+1 _d> _d+1. Proof of Corollary 1 (Spectral truncation lower bound). The map S↦TSS T_S is an isometric isomorphism between L2(μU⊗μV)L^2( _U _V) and the Hilbert-Schmidt class HS(L2(μV),L2(μU))HS(L^2( _V),L^2( _U)) equipped with the inner product ⟨TS,TS′⟩=tr(TS∗TS′) T_S,T_S = (T_S^*T_S ); indeed ‖TS‖HS2=∑k≥1σk2=‖S‖L22\|T_S\|_HS^2= _k≥ 1 _k^2=\|S\|_L^2^2. Under this isometry, the rank constraint of Lemma 1 becomes the operator rank constraint, and the Eckart-Young-Mirsky theorem for compact operators (see e.g. Gohberg and Krein) gives that the best rank-d approximation of TFT_F is the truncation to its leading d singular triples, with squared error ∑k>dσk2 _k>d _k^2. Uniqueness: when σd>σd+1 _d> _d+1, the leading d-dimensional singular subspace of TFT_F is unique, hence Fd=∑k≤dσkak⊗bkF_d= _k≤ d _ka_k b_k is the unique minimizer. Proof of Theorem 1 (Approximation error decomposition). Fix realizations f^∈ℱ f , g^∈ g satisfying Assumption 1, and write fk⋆=σkakf_k = _ka_k, gk⋆=bkg_k =b_k for the ideal modes, so that Fd=∑k≤dfk⋆⊗gk⋆=⟨f⋆,g⋆⟩F_d= _k≤ df_k g_k = f ,g . Set εk=fk⋆−f^k _k=f_k - f_k and δk=gk⋆−g^k _k=g_k - g_k, with ‖εk‖L2(μU)≤ηf,k\| _k\|_L^2( _U)≤ _f,k and ‖δk‖L2(μV)≤ηg,k\| _k\|_L^2( _V)≤ _g,k. Split the error around FdF_d: ‖F−⟨f^,g^⟩‖≤‖F−Fd‖⏟=ℰtrunc+R,R:=‖Fd−⟨f^,g^⟩‖,\|F- f, g \|≤ \|F-F_d\|_= E_trunc+R, R:=\|F_d- f, g \|, (24) where ℰtrunc:=∑k>dσk2E_trunc:= _k>d _k^2 by Corollary 1. For each coordinate, the product rule for tensor products gives fk⋆⊗gk⋆−f^k⊗g^k=εk⊗bk+σkak⊗δk−εk⊗δk.f_k g_k - f_k g_k= _k b_k+ _ka_k _k- _k _k. (25) Sum over k≤dk≤ d and take norms. The first sum is diagonalized by the orthonormality of bk\b_k\: ‖∑k≤dεk⊗bk‖2=∑k≤d‖εk‖L2(μU)2=∑k≤dηf,k2, \| _k≤ d _k b_k \|^2= _k≤ d\| _k\|_L^2( _U)^2= _k≤ d _f,k^2, (26) and likewise the orthonormality of ak\a_k\ diagonalizes the second sum: ‖∑k≤dσkak⊗δk‖2=∑k≤dσk2‖δk‖L2(μV)2≤∑k≤dσk2ηg,k2. \| _k≤ d _ka_k _k \|^2= _k≤ d _k^2\| _k\|_L^2( _V)^2≤ _k≤ d _k^2 _g,k^2. (27) This is the step that avoids a factor of d: bounding the sums coordinatewise and applying Cauchy-Schwarz over the d coordinates would pay a factor d, whereas the orthonormality of the ideal modes makes the two sums orthogonal. The third sum is bounded by Cauchy-Schwarz: ‖∑k≤dεk⊗δk‖≤∑k≤d‖εk‖‖δk‖≤(∑k≤dηf,k2)1/2(∑k≤dηg,k2)1/2. \| _k≤ d _k _k \|≤ _k≤ d\| _k\|\,\| _k\|≤ ( _k≤ d _f,k^2 )^1/2 ( _k≤ d _g,k^2 )^1/2. (28) With X=(∑kηf,k2)1/2X=( _k _f,k^2)^1/2, Y=(∑kσk2ηg,k2)1/2Y=( _k _k^2 _g,k^2)^1/2, Z=(∑kηg,k2)1/2Z=( _k _g,k^2)^1/2, the three bounds give R≤X+Y+XZR≤ X+Y+XZ, hence by (x+y+z)2≤3(x2+y2+z2)(x+y+z)^2≤ 3(x^2+y^2+z^2), R2≤3∑k≤dηf,k2+3∑k≤dσk2ηg,k2+3(∑k≤dηf,k2)(∑k≤dηg,k2).R^2≤ 3 _k≤ d _f,k^2+3 _k≤ d _k^2 _g,k^2+3 ( _k≤ d _f,k^2 ) ( _k≤ d _g,k^2 ). (29) In the informative regime the mode errors are small: assuming ∑k≤dηg,k2≤1 _k≤ d _g,k^2≤ 1 (the modes are realized with bounded total error, the only regime in which the bound is informative) and ηg,k≤1/2 _g,k≤ 1/2 for all k (so that Bg=maxk‖g^k‖L2(μV)≥mink(‖bk‖−‖δk‖)≥1/2B_g= _k\| g_k\|_L^2( _V)≥ _k(\|b_k\|-\| _k\|)≥ 1/2), the cross term absorbs into the first term and ∑kηf,k2≤4Bg2∑kηf,k2 _k _f,k^2≤ 4B_g^2 _k _f,k^2, giving R2≤3∑k≤dσk2ηg,k2+24Bg2∑k≤dηf,k2.R^2≤ 3 _k≤ d _k^2 _g,k^2+24B_g^2 _k≤ d _f,k^2. (30) Finally, since ‖F−⟨f^,g^⟩‖≤ℰtrunc+R\|F- f, g \|≤ E_trunc+R, ‖F−⟨f^,g^⟩‖2≤2ℰtrunc+2R2≤2ℰtrunc+6∑k≤dσk2ηg,k2+48Bg2∑k≤dηf,k2.\|F- f, g \|^2≤ 2E_trunc+2R^2≤ 2E_trunc+6 _k≤ d _k^2 _g,k^2+48B_g^2 _k≤ d _f,k^2. (31) As 2ℰtrunc≤48ℰtrunc2E_trunc≤ 48E_trunc trivially, the right-hand side is at most 4848 times the sum of the truncation term and the realization term of the theorem. Taking the infimum over f∈ℱf , g∈g , the bound holds with C≤48≤144C≤ 48≤ 144; we do not optimize the constant, which is what the statement of the theorem records. Theorem 3 (Smoothness controls the truncation term). Let ⊂ℝmUU ^m_U and ⊂ℝmVV ^m_V be bounded domains, and let μU,μV _U, _V have densities bounded above and below with respect to Lebesgue measure. If F is s-smooth in its second argument, F∈Hs(×)F∈ H^s(U×V), then σk=O(k−s/mV),∑k>dσk2=O(d1−2s/mV)for 2s>mV. _k=O (k^-s/m_V ), _k>d _k^2=O (d^1-2s/m_V ) 2s>m_V. (32) If F is real analytic on ×U×V, then σk=O(e−ck) _k=O(e^-ck) for some c>0c>0 and ∑k>dσk2=O(e−2cd) _k>d _k^2=O(e^-2cd). Proof of Theorem 3 (Smoothness controls the truncation term). The squared singular values of TFT_F are the eigenvalues of the self-adjoint, positive semidefinite integral operator TF∗TFT_F^*T_F on L2(μV)L^2( _V) with kernel K(v,v′)=∫F(u,v)F(u,v′)dμU(u),K(v,v )= _UF(u,v)\,F(u,v )\,d _U(u), (33) which is positive definite and inherits the smoothness of F in its second argument: differentiating under the integral, an s-smooth kernel on a bounded mVm_V-dimensional domain has eigenvalues decaying as O(k−2s/mV)O(k^-2s/m_V); this is the classical eigenvalue decay for positive definite kernels (18), and the analytic case decays exponentially (13). Taking square roots gives σk=O(k−s/mV) _k=O(k^-s/m_V), and summing the tail, ∑k>dσk2=O(∫d∞x−2s/mVdx)=O(d1−2s/mV)for 2s>mV, _k>d _k^2=O ( _d^∞x^-2s/m_V\,dx )=O (d^1-2s/m_V ) 2s>m_V, (34) which is finite exactly when 2s>mV2s>m_V. In the analytic case, σk=O(e−ck) _k=O(e^-ck) gives ∑k>dσk2=O(e−2cd) _k>d _k^2=O(e^-2cd) by the geometric series. Proposition 2 (Approximation separation). Let F∈ℳdF _d with interaction modes ak∈Hs()a_k∈ H^s(U), bk∈Hs()b_k∈ H^s(V). To approximate F within ε in L2(μU⊗μV)L^2( _U _V), a dual-encoder head whose encoders are ReLU networks requires Ndual=O(d(ε−mU/s+ε−mV/s))N_dual=O (d ( ^-m_U/s+ ^-m_V/s ) ) (35) parameters, whereas any single-tower model that approximates F as a general (mU+mV)(m_U+m_V)-dimensional HsH^s function requires Ω(ε−(mU+mV)/s) ( ^-(m_U+m_V)/s) parameters in the worst case. Hence Ndual/Nsingle→0N_dual/N_single→ 0 as ε→0 → 0. Proof of Proposition 2 (Approximation separation). Since F∈ℳdF _d with modes ak∈Hs()a_k∈ H^s(U), bk∈Hs()b_k∈ H^s(V), the truncation term of Theorem 1 vanishes for the rank-d head, and the entire approximation error is the realization term. Approximate each of the 2d2d modes by ReLU networks with the standard HsH^s rates (25): to reach mode error ε on a marginal space of dimension m requires O(ε−m/s)O( ^-m/s) parameters. Setting the mode errors to ε makes the realization term of Theorem 1 of order O(ε2)O( ^2), so a total of Ndual=O(d(ε−mU/s+ε−mV/s))N_dual=O (d ( ^-m_U/s+ ^-m_V/s ) ) (36) parameters suffice. For the lower bound, a single-tower model that treats F as a general (mU+mV)(m_U+m_V)-dimensional HsH^s function must contend with the Kolmogorov width of the Sobolev class, whose L2L^2 approximation is bounded below by Ω(ε−(mU+mV)/s) ( ^-(m_U+m_V)/s) parameters in the worst case (classical Sobolev width results). Since mU+mV>mU,mVm_U+m_V>m_U,m_V, the exponent is additive in the dimensions, so Ndual/Nsingle=O(εmmin/s)→0N_dual/N_single=O ( ^m_ /s )→ 0 as ε→0 → 0, where mmin=minmU,mVm_ = \m_U,m_V\. Proof of Proposition 1 (Compute separation). Counting operations: the head requires n forward passes of f, m forward passes of g, and nmnm inner products of dimension d, i.e., nmdnmd multiply-adds, for a total of O((n+m)Cnet+nmd)O((n+m)C_net+nmd). A joint model processes each of the nmnm pairs separately, costing O(nmCnet)O(nm\,C_net). When Cnet≫dC_net d, the head is cheaper by a factor of order Cnet/dC_net/d in the regime where nm≫n+mnm n+m. Appendix C Proofs for Section 3 We prove the results of Section 3 in order. Throughout, ℒ(f,g)=u,v[(F(u,v)−⟨f(u),g(v)⟩)2]L(f,g)=E_u,v[(F(u,v)- f(u),g(v) )^2] is the population risk. Table 6 collects the normalization taxonomy moved from Section 3; Proposition 3 and Remark 4 complete the picture. Table 6: Embedding-level normalizations as sections of the gauge orbit. Each row shows a constraint on the encoders, the part of the gauge it pins, the residual group that survives, and what the scheme identifies. See Proposition 3 and Theorems 4 to 2 for the statements, and Appendix C for proofs. Normalization Constraint Pins Residual gauge Identifies to None none nothing GLdGL_d nothing One-sided L2 and softplus (30) ‖g(v)‖=1\|g(v)\|=1, g(v)⪰0g(v) 0 pointwise scale of g not controlled no guarantee Per-coordinate standardization diag(Σg)=Idiag( _g)=I coordinate scales d2−d^2-d, shears and rotations no guarantee Cosine, two towers (17) ‖f‖=‖g‖=1\|f\|=\|g\|=1 angles OO up to a rotation Two-sided nonnegative f⪰0f 0, g⪰0g 0 signs, shears, rotations Perm⋉D+Perm _+ permutation and scale Whitening Σg=I _g=I, Σf=Λ _f= all continuous gauge D±D_±, order fixed permutation and sign The gauge group GLdGL_d stratifies into per-coordinate scalings (D+D_+, d dimensions), rotations (OO, d(d−1)/2d(d-1)/2), permutations and signs (discrete), and the remaining shears. The guarantees of Section 3 come from two-sided constraints, which act on both encoders and can pin the joint freedom of the pair; one-sided constraints cannot do this in general, and the schemes that use them are best understood as stability fixes. Normalization schemes are therefore compared on four measurable quantities: the residual gauge subgroup of Table 6, the Hessian condition number on the constrained manifold, related to the gap by Corollary 3, the expressivity cost of the constraint, for example the bounded range [−1,1][-1,1] of cosine normalization, and the output semantics, which determines which decoder and loss can be attached. Remark 4 (One-sided constraints fix scale, not identifiability). The scheme used by the QK-attention matching networks applies a softplus and a pointwise unit-norm constraint to one encoder only (30). It removes the global scaling redundancy of the representation, the parameter α in (αf)(g/α)(α f)\,(g/α), whose growth caused training instabilities in early versions of those networks, and it leaves the other encoder unconstrained. The constraint therefore provides stability without identifiability: the residual freedom of the pair depends on the geometry of the trained encodings and is not controlled by the constraint. The per-coordinate standardization of Proposition 3 has the same character, pinning the d coordinate scales of one encoder and nothing else. Proposition 3 (Per-coordinate standardization pins the scales). Consider the constraint diag(Σg)=Idiag( _g)=I, which standardizes each coordinate of one encoder to unit variance. It pins the positive diagonal subgroup D+D_+ of dimension d; generically the residual gauge has dimension d2−d^2-d and contains the shears and rotations that mix coordinates. Lemma 2 (Gauge invariance). For every A∈GLdA _d and every pair of inputs (u,v)(u,v), ⟨Af(u),A−⊤g(v)⟩=⟨f(u),g(v)⟩ Af(u),A^- g(v) = f(u),g(v) , so ΦA _A leaves the represented function and hence the population risk unchanged. With the second moments Σf=u[f(u)f(u)⊤] _f=E_u[f(u)f(u) ] and Σg=v[g(v)g(v)⊤] _g=E_v[g(v)g(v) ], the action transforms them by ΣAf=AΣfA⊤ _Af=A _fA and ΣA−⊤g=A−⊤ΣgA−1 _A^- g=A^- _gA^-1, so the eigenvalues of the product ΣfΣg _f _g are gauge invariants. Proof of Lemma 2 (Gauge invariance). The identity is the cancellation ⟨Af(u),A−⊤g(v)⟩=f(u)⊤A⊤A−⊤g(v)=f(u)⊤g(v)=⟨f(u),g(v)⟩, Af(u),A^- g(v) =f(u) A A^- g(v)=f(u) g(v)= f(u),g(v) , (37) so the represented function and hence the risk are unchanged. For the second moments, substitution gives ΣAf=u[Af(u)f(u)⊤A⊤]=AΣfA⊤,ΣA−⊤g=A−⊤ΣgA−1, _Af=E_u[Af(u)f(u) A ]=A _fA , _A^- g=A^- _gA^-1, (38) and the product transforms by the similarity ΣAfΣA−⊤g=AΣfΣgA−1 _Af _A^- g=A\, _f _g\,A^-1, whose eigenvalues are invariant. At a global minimizer, the represented function equals the rank-d truncation FdF_d of the target when σd>σd+1 _d> _d+1 (Corollary 1), and the coefficient computation of the proof of Theorem 2 below shows that the eigenvalues of ΣfΣg _f _g are then the squared singular values σk2 _k^2, which is the claim of Remark 5. Remark 5 (The invariant content of a trained head). The gauge-invariant content of a trained head is exactly the function it represents, summarized by its interaction spectrum; Lemma 2 adds a finite-dimensional shadow: at any global minimizer of the population risk, the eigenvalues of ΣfΣg _f _g are the squared interaction singular values. This quantity is well defined under every normalization, and it reappears in Corollary 2. Proposition 4 (Ill-posedness from unfixed gauge). Let (f⋆,g⋆)(f ,g ) be a global minimizer of the risk and let S be its stabilizer, the set of A∈GLdA _d with ΦA(f⋆,g⋆)=(f⋆,g⋆) _A(f ,g )=(f ,g ). Then the Hessian of the risk at (f⋆,g⋆)(f ,g ) has at least dimker∇2ℒ(f⋆,g⋆)≥d2−dimS ∇^2L(f ,g )\;≥\;d^2- S (39) zero directions, coming from the tangent space of the gauge orbit. Without normalization this is generically d2d^2 degenerate directions, so the second-order problem is rank-deficient and the optimization problem is ill-posed. Proof of Proposition 4 (Ill-posedness from unfixed gauge). Let (f⋆,g⋆)(f ,g ) be a global minimizer and S its stabilizer. By Lemma 2, the risk is constant along the orbit =ΦA(f⋆,g⋆):A∈GLdO=\ _A(f ,g ):A _d\, so for every X∈dX∈ gl_d the curve A(t)=exp(tX)A(t)= (tX) gives a path γX(t)=ΦA(t)(f⋆,g⋆) _X(t)= _A(t)(f ,g ) along which ℒ∘γX≡ℒ(f⋆,g⋆)L _X (f ,g ). The velocity is γ˙X(0)=(Xf⋆,−X⊤g⋆)∈T(f⋆,g⋆), γ_X(0)=(Xf ,\,-X g )∈ T_(f ,g ), (40) and since the risk is constant along the path, its second derivative in this direction vanishes. Because (f⋆,g⋆)(f ,g ) is a global minimizer, ∇2ℒ⪰0∇^2L 0, and a semidefinite quadratic form vanishes in a direction if and only if that direction lies in the kernel, so γ˙X(0)∈ker∇2ℒ γ_X(0)∈ ∇^2L for every X. The map X↦γ˙X(0)X γ_X(0) has kernel equal to the Lie algebra of the stabilizer S, hence its image, which lies entirely in the kernel of the Hessian, has dimension d2−dimSd^2- S. This proves the bound. Without normalization the stabilizer at a generic minimizer is trivial, so the Hessian has at least d2d^2 zero directions. Proof of Proposition 3 (Per-coordinate standardization pins the scales). The constraint diag(Σg)=Idiag( _g)=I is d scalar conditions that fix the d coordinate scales of g: for the transformation A−1=diag(s1,…,sd)∈D+A^-1=diag(s_1,…,s_d) _+ acting on g, the condition forces sk2[Σg]kk=1s_k^2[ _g]_k=1, pinning each sks_k. Transformations A that mix coordinates generically change diag(A−⊤ΣgA−1)diag(A^- _gA^-1) unless they also rotate Σg _g to a new matrix with the same diagonal; the residual group is the set of A preserving the diagonal condition, which contains the orthogonal group conjugated by any diagonal scale and the shears that preserve the diagonal of Σg _g. Counting dimensions, the constraint fixes exactly the d-dimensional subgroup D+D_+, and the generic residual dimension is d2−d^2-d. Theorem 4 (Cosine normalization leaves a full rotation). Consider two-sided unit normalization, f~=f/‖f‖ f=f/\|f\| and g~=g/‖g‖ g=g/\|g\|, with scores cos∠(f(u),g(v)) (f(u),g(v)), as in the contrastive models of Table 5 (17); a temperature that rescales the logits changes no gauge. If the encodings are nondegenerate, in that the ranges of f and g contain open subsets of ℝdR^d, the residual gauge group is exactly the orthogonal group, R=A∈GLd:A⊤A=I=O.R= \\,A _d:\;A A=I\, \=O. (41) For every rotation R∈OR , the encodings RfRf and RgRg give pointwise identical cosine scores, and no transformation outside OO does. Corollary 2 (Contrastive dimensions are uninterpretable). Under the residual symmetry of Theorem 4, the individual coordinates of a cosine-normalized embedding carry no intrinsic meaning: replacing (f,g)(f,g) by (Rf,Rg)(Rf,Rg) for any rotation R yields the same model and the same loss, and only rotation-invariant quantities are well defined, among them norms, pairwise inner products, and the eigenvalues of ΣfΣg _f _g, which by Remark 5 are the squared interaction singular values at any optimum. This is a theorem-level explanation of the widely reported observation that contrastive embedding dimensions are uninterpretable; the remedy is the whitening scheme of Theorem 2. Proof of Theorem 4 (Cosine normalization leaves a full rotation). Let R be the residual group of the two-sided unit normalization, i.e., the set of A∈GLdA _d such that, after renormalizing, the cosine scores are unchanged. (⊇)( ) For R∈OR , ‖Rf(u)‖=‖f(u)‖\|Rf(u)\|=\|f(u)\| and ‖Rg(v)‖=‖g(v)‖\|Rg(v)\|=\|g(v)\| by orthogonality, and ⟨Rf(u),Rg(v)⟩=⟨f(u),g(v)⟩ Rf(u),Rg(v) = f(u),g(v) , hence cos∠(Rf(u),Rg(v))=⟨Rf(u),Rg(v)⟩‖Rf(u)‖‖Rg(v)‖=⟨f(u),g(v)⟩‖f(u)‖‖g(v)‖ (Rf(u),Rg(v))= Rf(u),Rg(v) \|Rf(u)\|\,\|Rg(v)\|= f(u),g(v) \|f(u)\|\,\|g(v)\| (42) pointwise, so O⊆RO R. (⊆)( ) Suppose A preserves every cosine between the encodings. Since the ranges of f and g contain open subsets of ℝdR^d, the encodings realize a full-dimensional family of directions, so A preserves all angles between vectors. A linear map that preserves every angle is a similarity: by the classical characterization of conformal linear maps, A=cRA=cR for some c>0c>0 and R∈OR . The scalar c is absorbed by the unit renormalization of both encoders, so within the cosine score A acts identically to the rotation R, giving R⊆OR . Corollary 2 follows immediately. For any rotation R∈OR the pair (Rf,Rg)(Rf,Rg) is a different encoder pair with pointwise identical cosine scores and identical loss, so the individual coordinates carry no intrinsic meaning. The rotation-invariant quantities are the norms, the inner products, and the eigenvalues of ΣfΣg _f _g; by Remark 5 these are the squared interaction singular values at any optimum. Corollary 2 is the statement of these two sentences at the theorem level. Theorem 5 (Two-sided nonnegativity gives NMF identifiability). Suppose both encoders are constrained to nonnegative outputs, f(u)⪰0f(u) 0 and g(v)⪰0g(v) 0, as obtained by applying a softplus or exponential nonlinearity to raw encodings. On a finite set of inputs the model is the nonnegative matrix factorization M=FG⊤M=FG of a nonnegative matrix M. If the encodings are nondegenerate, in that their ranges generate the positive orthant, the residual gauge group shrinks to the monomial group, R=A∈GLd:A⪰0,A−⊤⪰0=Perm⋉D+,R= \\,A _d:\;A 0,\;A^- 0\, \=Perm _+, (43) the group of coordinate permutations composed with positive diagonal scalings. If the factorization is separable in the sense of 6, with an anchor row for each latent factor as in 1, then it is essentially unique and the encoders are identified up to permutation and scaling. Proof of Theorem 5 (Two-sided nonnegativity gives NMF identifiability). On a finite set of inputs, the nonnegative encoders form matrices F∈ℝ≥0n×dF _≥ 0^n× d, G∈ℝ≥0m×dG _≥ 0^m× d, and the model is the nonnegative factorization M=FG⊤M=FG . The residual group consists of A∈GLdA _d such that Af⪰0Af 0 and A−⊤g⪰0A^- g 0 for all nonnegative encodings. Since the ranges generate the positive orthant, Af⪰0Af 0 for all nonnegative f forces A⪰0A 0 entrywise, and likewise A−⊤⪰0A^- 0. A real matrix that is nonnegative and whose inverse transpose is also nonnegative is a generalized permutation matrix: its inverse transpose being nonnegative forces each row and column of A to have exactly one nonzero entry, and nonnegativity makes that entry positive. Hence R⊆Perm⋉D+R _+, and the reverse inclusion is immediate, so R=Perm⋉D+R=Perm _+. Under separability in the sense of 6, each latent factor has an anchor row whose encoding is nonzero in exactly one coordinate, which identifies the extreme rays of the latent cone and hence the factor directions; the constructive algorithm of 1 then yields the decomposition uniquely up to permutation and scaling. The degradation of this guarantee when separability holds only approximately is analyzed in Appendix E. Proof of Theorem 2 (Whitening fixes the gauge up to permutation and sign). The proof has four steps. (1) Every global minimizer represents the rank-d truncation. By Corollary 1 the minimal risk is ∑k>dσk2 _k>d _k^2, attained exactly by the truncated series Fd=∑k≤dσkak⊗bkF_d= _k≤ d _ka_k b_k, which is unique since σd>σd+1 _d> _d+1. Hence any minimizer satisfies ⟨f(u),g(v)⟩=Fd(u,v) f(u),g(v) =F_d(u,v) almost everywhere, i.e., ∑k=1dfk(u)gk(v)=∑k=1dσkak(u)bk(v). _k=1^df_k(u)\,g_k(v)= _k=1^d _k\,a_k(u)\,b_k(v). (44) (2) The span of the encoders is pinned. As rank-d representations of the same tensor, the left factor spaces and right factor spaces coincide: spanf1,…,fd=spana1,…,adspan\f_1,…,f_d\=span\a_1,…,a_d\ and spang1,…,gd=spanb1,…,bdspan\g_1,…,g_d\=span\b_1,…,b_d\. So there exist coefficient matrices B,C∈ℝd×dB,C ^d× d with f=Ba[d]f=B\,a_[d] and g=Cb[d]g=C\,b_[d], where a[d]=(a1,…,ad)⊤a_[d]=(a_1,…,a_d) and likewise for b. Substituting into the identity above and using the orthonormality of the modes gives B⊤C=diag(σ1,…,σd)=:S.B C=diag( _1,…, _d)=:S. (45) (3) The whitening constraints force C to be a signed permutation. By the orthonormality of ak\a_k\, Σf=u[f(u)f(u)⊤]=BB⊤,Σg=CC⊤. _f=E_u[f(u)f(u) ]=B\,B , _g=C\,C . (46) The constraint Σg=I _g=I gives CC⊤=ICC =I, so C∈OC ; the constraint Σf=Λ _f= diagonal gives BB⊤=ΛB = . From B⊤C=SB C=S and C∈OC we get B⊤=SC−1=SC⊤B =SC^-1=SC , so B=CSB=CS, and therefore BB⊤=CS2C⊤=Λ.B =C\,S^2\,C = . (47) Thus the orthogonal matrix C diagonalizes the diagonal matrix S2=diag(σ12,…,σd2)S^2=diag( _1^2,…, _d^2). Since the σk _k are distinct, the eigenspaces of S2S^2 are the coordinate axes, and any orthogonal matrix diagonalizing S2S^2 maps each axis to itself up to sign, so C∈Perm⋉D±C _± and Λ=S2 =S^2, i.e., λk=σk2 _k= _k^2 with the ordering convention matching the decreasing σk _k. With the ordering fixed, C=diag(±1,…,±1)C=diag(± 1,…,± 1) and B=CS=diag(±σ1,…,±σd)B=CS=diag(± _1,…,± _d). (4) Substitute back. With the coordinate order fixed, gk=±bkg_k=± b_k and fk=±σkakf_k=± _ka_k, the signs being paired so that the product fkgk=σkakbkf_kg_k= _ka_kb_k is unique. This is exactly (10), and the encoders are identified up to permutation and sign. Corollary 3 (The spectral gap controls conditioning and convergence). On the manifold defined by the constraints (9), every global minimizer is isolated up to the discrete group of signs and permutations, and the Hessian of the risk is nondegenerate in the tangent directions of the manifold, with smallest nonzero curvature bounded below by a constant multiple of the spectral gap Δ=min1≤k<d(σk2−σk+12) = _1≤ k<d( _k^2- _k+1^2); a projected gradient method that maintains the constraints therefore converges locally at a linear rate 1−μ/L1-μ/L, where μ is bounded below by a constant multiple of Δ and L is the local smoothness constant. Wider gaps make the problem both better conditioned and faster to converge. Proof of Corollary 3 (The spectral gap controls conditioning and convergence). Isolation: by Theorem 2, on the manifold defined by (9) every global minimizer is unique up to the discrete group of signs and the permutation, and the represented function FdF_d is a strict minimum of the risk by the strict inequality σd>σd+1 _d> _d+1; the Hessian of the risk is therefore nondegenerate in the tangent directions of the manifold, and its smallest nonzero curvature is positive. The quantitative bound: the Hessian curvature in the tangent directions is controlled by the second derivative of the risk as the encoders move within the constraint manifold, and by the Davis-Kahan sinΘ theorem the perturbation of a leading singular subspace under a perturbation of size ϵε is at most ϵ/Δε/ , where Δ=mink(σk2−σk+12) = _k( _k^2- _k+1^2). Inverting this stability relation, the curvature of the risk in directions that move the encoders off the ideal modes is bounded below by a constant multiple of Δ : a displacement of size t in function space changes the risk by at least cΔt2c\, \,t^2 for the truncation-optimal function. A projected gradient method that maintains the constraints therefore converges locally at a linear rate 1−μ/L1-μ/L with μ≥cΔμ≥ c\, and L the local smoothness constant, as stated. Theorem 6 (Soft whitening recovers the hard solution). For ρ>0ρ>0, consider the penalized risk ℒρ(f,g)=ℒ(f,g)+ρ(‖Σg−Id‖F2+‖offdiag(Σf)‖F2).L_ρ(f,g)=L(f,g)+ρ ( \| _g-I_d \|_F^2+ \|offdiag( _f) \|_F^2 ). (48) As ρ→∞ρ→∞, any convergent sequence of minimizers of ℒρL_ρ converges to a solution satisfying the constraints of Theorem 2 up to the ordering convention, and hence to the recovered modes (10). For finite ρ, the residual gauge directions that break the constraint Σg=I _g=I carry curvature at least 2ρc2ρ\,c for a constant c>0c>0 that depends on the second moments, so conditioning improves monotonically with ρ. Proof of Theorem 6 (Soft whitening recovers the hard solution). (i) Convergence of minimizers. The zero set of the penalty ρ(‖Σg−I‖F2+‖offdiag(Σf)‖F2)ρ (\| _g-I\|_F^2+\|offdiag( _f)\|_F^2 ) (49) is exactly the whitening constraint (9) up to the ordering convention. Since ℒρ≥ℒL_ρ , any limit point of a convergent sequence of minimizers satisfies the constraint (otherwise the penalty, and with it ℒρL_ρ, would diverge) and is a global minimizer of ℒL subject to the constraint, to which Theorem 2 applies. (i) Curvature of the penalty. Consider a gauge generator X∈dX∈ gl_d acting on the g-side as g(t)=exp(−tX⊤)g(t)= (-tX )g. The second moment evolves as Σg(t)=exp(−tX⊤)Σgexp(−tX) _g(t)= (-tX ) _g (-tX). At a constraint-satisfying point Σg=I _g=I, the second derivative of ‖Σg(t)−I‖F2\| _g(t)-I\|_F^2 at t=0t=0 is 2‖X+X⊤‖F22\|X+X \|_F^2: expanding the exponential gives Σg(t)=I−t(X⊤+X)+t2X⊤X+O(t3) _g(t)=I-t(X +X)+t^2X X+O(t^3), so the squared deviation has leading term t2‖X+X⊤‖F2t^2\|X+X \|_F^2 multiplied by 22. Directions that break Σg=I _g=I have nonzero symmetric part, hence carry penalty curvature at least 2ρc2ρ\,c with c>0c>0 determined by the second moments through the contribution of offdiag(Σf)offdiag( _f); directions with X antisymmetric are rotations, which preserve Σg=I _g=I and are exactly the directions whose residual freedom is fixed by the ordering convention. Conditioning therefore improves monotonically with ρ, as stated. Appendix D Proofs for Section 4 We prove the results of Section 4. Throughout, ℛ(h)=u,v[(F(u,v)−h(u,v))2]R(h)=E_u,v[(F(u,v)-h(u,v))^2] is the population risk and ℛ R its empirical version over the n samples. Corollary 6 and Proposition 6 are proved in Appendix E, as indicated in the main text. Remark 6 (The general bound is conservative in d). Specializing Corollary 4 to the linear class gives ℜn(ℱ)=O(pd/n) R_n(F)=O( pd/n) and ℜn()=O(qd/n) R_n(G)=O( qd/n), hence an estimation error of order Bd3/2(p+q)/nBd^3/2 (p+q)/n, a factor d d worse than the minimax anchor (56). The gap is structural: Lemma 3 sums the d coordinates as independent classes and cannot exploit the low-rank geometry that the matrix analysis of Theorem 7 uses. The general bound is therefore tight in its dependence on n and on the marginal dimensions, and conservative in the power of d; the linear case shows the achievable d-dependence is linear, and local Rademacher arguments close the gap. In the sequential setting, where (ut,vt)(u_t,v_t) are chosen adaptively to maximize an unknown score u⊤Θ⋆vu v, the same scaling appears as regret: bilinear bandits achieve O~(d(p+q)T) O(d (p+q)T) (10), again linear in d and in the marginal dimensions. Remark 7 (The separation is not a discrete artifact). The same separation holds for smooth kernels. On the circle, the narrowband kernel Fh(u,v)=ρh(u−v)F_h(u,v)= _h(u-v) with a periodic Gaussian ρh _h of bandwidth h has a Fourier-diagonal interaction operator with singular values of order e−2π2h2ξ2e^-2π^2h^2ξ^2, so dε(Fh)=Θ(1/h)d_ (F_h)= (1/h): the embedding dimension diverges as the kernel approaches the diagonal. Early interaction computes the difference t=u−vt=u-v exactly and approximates the one-dimensional smooth function ρh _h with O(poly(1/h))O(poly(1/h)) parameters (25), a polynomial separation in the same direction. The strength of the separation is set by how the target depends jointly on its two arguments: through a comparison in the discrete case, through a difference in the smooth case, and not at all when the target is both high-rank and unstructured. Lemma 3 (Rademacher collapse of the inner-product class). For the class (12) with per-coordinate classes ℱ(k),(k)F^(k),G^(k), ℜn(ℋd)≤∑k=1dℜn(ℱ(k)⋅(k))≤ 2∑k=1d(Bgℜn(ℱ(k))+Bfℜn((k))), R_n(H_d)\;≤\; _k=1^d R_n (F^(k)·G^(k) )\;≤\;2 _k=1^d (B_g\, R_n(F^(k))+B_f\, R_n(G^(k)) ), (50) so in particular ℜn(ℋd)≤2d(Bgℜn(ℱ)+Bfℜn()) R_n(H_d)≤ 2d (B_g\, R_n(F)+B_f\, R_n(G) ) for shared per-coordinate classes: the complexity of the head is a sum of marginal complexities, not a complexity of the joint space. Proof of Lemma 3 (Rademacher collapse of the inner-product class). Decompose the score as ⟨f(u),g(v)⟩=∑k=1dfk(u)gk(v) f(u),g(v) = _k=1^df_k(u)g_k(v). Rademacher complexity is subadditive over sums of function classes, so ℜn(ℋd)≤∑k=1dℜn(ℱ(k)⋅(k)), R_n(H_d)≤ _k=1^d R_n (F^(k)·G^(k) ), (51) where ℱ(k)⋅(k)=(u,v)↦fk(u)gk(v)F^(k)·G^(k)=\(u,v) f_k(u)g_k(v)\ is the pointwise product class. For each coordinate, the product map (fk,gk)↦fkgk(f_k,g_k) f_kg_k is Lipschitz in its first argument with constant BgB_g and in its second with constant BfB_f on the bounded range; the vector contraction inequality of 15 then bounds the complexity of the product class by the sum of the two marginal complexities, ℜn(ℱ(k)⋅(k))≤2(Bgℜn(ℱ(k))+Bfℜn((k))), R_n (F^(k)·G^(k) )≤ 2 (B_g\, R_n(F^(k))+B_f\, R_n(G^(k)) ), (52) the factor 22 absorbing the contraction constants. Summing over the d coordinates and using ℜn(ℱ)=maxkℜn(ℱ(k)) R_n(F)= _k R_n(F^(k)) gives the displayed bound. The key structural point is that no complexity of the joint space ×U×V appears: the head is paid for by its two marginal classes. Corollary 4 (Excess risk of empirical risk minimization). Let F F be the empirical risk minimizer over ℋdH_d. With probability at least 1−δ1-δ over the sample, the excess risk decomposes into an approximation part, the truncation and realization terms of Theorem 1, and an estimation part through the class complexity: ‖F^−F‖L2(μU⊗μV)2≲∑k>dσk2+ℰreal+Bd(ℜn(ℱ)+ℜn())+B2log(1/δ)n,\| F-F\|_L^2( _U _V)^2\; \; _k>d _k^2+E_real\;+\;B\,d ( R_n(F)+ R_n(G) )\;+\;B^2 (1/δ)n, (53) where ℰreal=C∑k≤d(σk2ηg,k2+Bg2ηf,k2)E_real=C _k≤ d( _k^2 _g,k^2+B_g^2 _f,k^2) is the realization term and ≲ hides absolute constants. Proof of Corollary 4 (Excess risk of empirical risk minimization). The regression oracle decomposition: for any h∈ℋdh _d, writing F F for the empirical risk minimizer, ‖F−F^‖L22≤‖F−h‖L22+2suph′∈ℋd|ℛ^(h′)−ℛ(h′)|.\|F- F\|_L^2^2≤\|F-h\|_L^2^2+2 _h _d | R(h )-R(h ) |. (54) The uniform deviation is controlled as follows. The squared loss ℓ(y,h)=(y−h)2 (y,h)=(y-h)^2 is 4B4B-Lipschitz in h on |y|,|h|≤B|y|,|h|≤ B, so by the Talagrand contraction inequality the Rademacher complexity of the loss class is at most 4Bℜn(ℋd)4B\, R_n(H_d) (4). Symmetrization bounds the expected uniform deviation by twice the Rademacher complexity of the loss class, and McDiarmid’s inequality, with the bounded loss |y−h|2≤4B2|y-h|^2≤ 4B^2, gives the concentration term: suph′∈ℋd|ℛ^(h′)−ℛ(h′)|≲Bℜn(ℋd)+B2log(1/δ)n _h _d | R(h )-R(h ) | B\, R_n(H_d)+B^2 (1/δ)n (55) with probability at least 1−δ1-δ. Choosing h=h⋆h=h , the population optimum, makes ‖F−h⋆‖2\|F-h \|^2 exactly the approximation term of Theorem 1, and Lemma 3 bounds ℜn(ℋd)≤2d(Bgℜn(ℱ)+Bfℜn()) R_n(H_d)≤ 2d(B_g R_n(F)+B_f R_n(G)); absorbing the Lipschitz and contraction constants into the absolute constant gives (14). For parameterized encoder classes with pfp_f and pgp_g parameters under norm control, the standard Rademacher bound for parameterized classes gives ℜn(ℱ)=O(pf/n) R_n(F)=O( p_f/n) and ℜn()=O(pg/n) R_n(G)=O( p_g/n), and the estimation term becomes O(Bd(pf+pg)/n)O(Bd (p_f+p_g)/n). Theorem 7 (Linear heads are low-rank matrix sensing). Let Θ⋆∈ℝp×q ^p× q have rank d and ‖Θ⋆‖F≤B\| \|_F≤ B, and suppose the sensing matrices uivi⊤u_iv_i are drawn from an isotropic sub-Gaussian family and the noise is sub-Gaussian with variance σε2 _ ^2. The rank-constrained estimator attains ‖Θ^−Θ⋆‖F2≍σε2d(p+q)n,E \| - \|_F^2\; \; _ ^2\, d(p+q)n, (56) and this rate is minimax optimal. Equivalently, ‖F^−F‖L22≍σε2d(p+q)/nE\| F-F\|_L^2^2 _ ^2\,d(p+q)/n. Proof of Theorem 7 (Linear heads are low-rank matrix sensing). The head computes u⊤Θvu v with Θ=UV⊤ =UV , and the observations are yi=⟨Θ⋆,uivi⊤⟩F+εiy_i= ,\,u_iv_i _F+ _i, the standard form of low-rank matrix sensing with sensing matrices Ai=uivi⊤A_i=u_iv_i and rank constraint rank(Θ)≤d ( )≤ d. Upper bound: the rank-d manifold has d(p+q−d)d(p+q-d) free parameters, and under restricted strong convexity of the sensing operator, which holds for isotropic sub-Gaussian sensing matrices, the constrained estimator satisfies ‖Θ^−Θ⋆‖F2≲σε2d(p+q)/nE\| - \|_F^2 _ ^2\,d(p+q)/n; this is the robust error bound of 5 together with the high-dimensional rate of 16. Lower bound: a Fano argument over a packing of the rank-d manifold gives the matching minimax lower bound Ω(σε2d(p+q)/n) ( _ ^2\,d(p+q)/n) (20). The equivalence with the function-space rate follows from the isotropy of u and v: u,v[(u⊤Δv)2]≍‖Δ‖F2E_u,v[(u v)^2] \| \|_F^2 for sub-Gaussian isotropic arguments, so ‖F^−F‖L22≍σε2d(p+q)/nE\| F-F\|_L^2^2 _ ^2\,d(p+q)/n. Corollary 5 (Optimal rank selection). If the interaction spectrum decays exponentially, σk2=O(e−2ck) _k^2=O(e^-2ck), and the encoders are linear, the total error is minimized at d⋆≍12clognp+q,d \; \; 12c np+q, (57) and the achieved error is ‖F^−F‖L22≲(p+q)log(n/(p+q))/n\| F-F\|_L^2^2 (p+q) (n/(p+q))/n, a near-parametric rate in which fast spectral decay eliminates the curse of dimensionality. Proof of Corollary 5 (Optimal rank selection). With exponential decay and linear encoders the total error of Theorem 7 and the truncation term balance as err(d)≲e−2cd+d(p+q)n.err(d)\; \;e^-2cd+ d(p+q)n. (58) Setting the two terms equal, e−2cd⋆≍d⋆(p+q)/ne^-2cd d (p+q)/n, and solving gives d⋆≍(1/2c)log(n/(p+q))d (1/2c) (n/(p+q)), where the logarithmic factor in d is absorbed by the constant. Substituting back, e−2cd⋆≍(p+q)/ne^-2cd (p+q)/n, so the achieved error is ‖F^−F‖L22≲(p+q)n(1+lognp+q)≍(p+q)log(n/(p+q))n,\| F-F\|_L^2^2 (p+q)n (1+ np+q ) (p+q) (n/(p+q))n, (59) the near-parametric rate stated in the corollary. Theorem 8 (Flat spectrum forces an exponential embedding). Let ==±1mU=V=\± 1\^m with uniform measure and N=2mN=2^m, and consider the equality function Feq(u,v)=[u=v]F_eq(u,v)=1[u=v]. Its interaction operator is TFeq=1NIdT_F_eq= 1NId, so all N singular values equal 1/N1/N and the interaction rank is N. Every rank-d head therefore satisfies inff,g‖Feq−⟨f,g⟩‖L22‖Feq‖L22= 1−dN, _f,g \|F_eq- f,g \|_L^2^2 \|F_eq \|_L^2^2\;=\;1- dN, (60) so achieving relative error ε requires d≥(1−ε)2md≥(1- )2^m, an embedding dimension exponential in the input size. Proof of Theorem 8 (Flat spectrum forces an exponential embedding). Compute the interaction operator of FeqF_eq on the uniform measure over ±1m\± 1\^m with N=2mN=2^m: (TFeqh)(u)=∑v∈±1m[u=v]h(v)1N=h(u)N,(T_F_eqh)(u)= _v∈\± 1\^m1[u=v]\,h(v)\, 1N= h(u)N, (61) so TFeq=1NIdT_F_eq= 1NId and all N singular values equal 1/N1/N. Hence ‖Feq‖L22=∑k≥1σk2=N/N2=1/N\|F_eq\|_L^2^2= _k≥ 1 _k^2=N/N^2=1/N and ∑k>dσk2=(N−d)/N2 _k>d _k^2=(N-d)/N^2 for d<Nd<N. By Corollary 1, the relative squared error of the best rank-d head is inff,g‖Feq−⟨f,g⟩‖L22‖Feq‖L22=(N−d)/N21/N=1−dN, _f,g\|F_eq- f,g \|_L^2^2\|F_eq\|_L^2^2= (N-d)/N^21/N=1- dN, (62) which is at most ε only if d≥(1−ε)N=(1−ε)2md≥(1- )N=(1- )2^m. Theorem 9 (Early interaction escapes). For the equality function FeqF_eq on ±1m×±1m\± 1\^m×\± 1\^m, an early-interaction predictor with O(m)O(m) parameters represents FeqF_eq exactly, with zero error. Together with Theorem 8 this is an exponential separation: O(m)O(m) parameters suffice under early interaction, while any late-interaction head needs Ω(2m) (2^m) embedding dimensions to reach constant relative error. Proof of Theorem 9 (Early interaction escapes). The construction uses the identity uivi=1u_iv_i=1 if and only if ui=viu_i=v_i, so that ⟨u,v⟩=∑i=1muivi=m−2Ham(u,v), u,v = _i=1^mu_iv_i=m-2\,Ham(u,v), (63) and ⟨u,v⟩=m u,v =m if and only if u=vu=v, with the next possible value m−2m-2. Hence Feq(u,v)=[⟨u,v⟩≥m−1]F_eq(u,v)=1[ u,v ≥ m-1]. A cross-attention layer with identity projections computes the query-key inner product ⟨u,v⟩ u,v with O(m)O(m) parameters, and a threshold unit ϕ(s)=ReLU(s−(m−1))−ReLU(s−(m+1))φ(s)=ReLU(s-(m-1))-ReLU(s-(m+1)) evaluates to 11 exactly on s=ms=m and to 00 elsewhere on the discrete range, using O(1)O(1) parameters. A joint MLP computes the same quantity bit by bit: each product uivi=((ui+vi)2−2)/2u_iv_i=((u_i+v_i)^2-2)/2 costs O(1)O(1) ReLU units, the sum costs O(m)O(m) additions, and the same threshold completes the network, again O(m)O(m) parameters in total. A cross-encoder, being a strict superset of cross-attention, inherits the construction. Together with Theorem 8, this is the exponential separation O(m)O(m) versus Ω(2m) (2^m). Theorem 10 (Usability criterion). Given a problem (F,μU,μV)(F, _U, _V), a target relative error ε , and a sample budget n: if the effective interaction rank dε(F)d_ (F) is small, a late-interaction head is appropriate, since its approximation and estimation errors are then small and it enjoys the compute and statistical separations of Proposition 1 and Lemma 3. If the spectrum is flat, dε(F)=Ω(N)d_ (F)= (N), the truncation floor of Corollary 1 locks every rank-d head to relative error at least ε regardless of capacity, samples, or training, while early-interaction models reach ε polynomially when the target admits a low-dimensional joint structure. The deciding quantity, the spectral decay rate, is estimable from data by a weighted singular value decomposition of the empirical kernel matrix, so the criterion is actionable before training either model. Proof of Theorem 10 (Usability criterion). First case: if dε(F)d_ (F) is small, take d≳dε(F)d d_ (F). The approximation error of Theorem 1 is at most ε‖F‖2 \|F\|^2 plus the realization term, and the estimation error of Corollary 4 scales as d(ℜn(ℱ)+ℜn())d( R_n(F)+ R_n(G)), small in d; the compute separation of Proposition 1 and the marginal-complexity collapse of Lemma 3 apply without further conditions. Second case: if the spectrum is flat with dε(F)=Ω(N)d_ (F)= (N), then for every d<dε(F)d<d_ (F) the truncation floor of Corollary 1, in the explicit form of Theorem 8, locks the relative error of every rank-d head above ε regardless of capacity, samples, or training, while the constructive bounds of Theorem 9 and Remark 7 show that early-interaction models reach ε with a polynomial parameter budget when the target depends on its arguments through a low-dimensional joint statistic. Actionability: the empirical kernel matrix M^ij=y(ui,vj) M_ij=y(u_i,v_j) on a grid of input pairs, weighted by the reference measures, has a weighted singular value decomposition whose values estimate σk\ _k\; a log-linear fit of the decay of σ^k σ_k against k (exponential) or logk k (polynomial) decides the decay type and reads off dεd_ before either model is trained. Appendix E Finite-Sample Technical Lemmas This appendix collects the finite-sample lemmas used in the main text. Section A makes the whitening identification of Theorem 2 quantitative: concentration of empirical second moments (Lemma 4), stability of the whitening map (Lemma 5), and a Davis-Kahan propagation step (Theorem 11) deliver the sample complexity of Corollary 6. Section B quantifies the degradation of the NMF route when separability holds only approximately (Theorem 12). Section C couples identification with estimation in one ledger (Proposition 5), which is the proof of Proposition 6. Throughout, ∥⋅∥\|·\| on matrices is the spectral norm and ∥⋅∥F\|·\|_F the Frobenius norm. E.1 Empirical whitening and mode identification Lemma 4 (Matrix Bernstein concentration of empirical moments). Let uii=1n\u_i\_i=1^n be drawn i.i.d. from μU _U and let f have bounded outputs, ‖f(u)‖≤Bf\|f(u)\|≤ B_f. The empirical second moment Σ^f=1n∑if(ui)f(ui)⊤ _f= 1n _if(u_i)f(u_i) satisfies, for every t>0t>0, Pr[∥Σ^f−Σf∥≥t]≤2dexp(−nt2/2νf+Bf2t/3),νf:=Bf2∥Σf∥, [\| _f- _f\|≥ t ]≤ 2d\, \! ( -nt^2/2 _f+B_f^2t/3 ), _f:=B_f^2\| _f\|, (64) and hence, with probability at least 1−δ1-δ, ‖Σ^f−Σf‖≤2νflog(2d/δ)n+2Bf2log(2d/δ)3n=:rf(n,δ),\| _f- _f\|≤ 2 _f (2d/δ)n+ 2B_f^2 (2d/δ)3n=:r_f(n,δ), (65) which is O(Bf2log(d/δ)/n)O (B_f^2 (d/δ)/n ) for large n. The symmetric statement holds for g with radius rg(n,δ)r_g(n,δ). Proof. The matrices Xi=f(ui)f(ui)⊤−ΣfX_i=f(u_i)f(u_i) - _f are independent, zero mean, and symmetric, with ‖Xi‖≤‖f(ui)f(ui)⊤‖+‖Σf‖≤2Bf2\|X_i\|≤\|f(u_i)f(u_i) \|+\| _f\|≤ 2B_f^2. For the variance parameter, since [Xi2]⪯‖f(ui)‖2f(ui)f(ui)⊤⪯Bf2ΣfE[X_i^2] \|f(u_i)\|^2f(u_i)f(u_i) B_f^2 _f, we have ‖∑i[Xi2]‖≤nBf2‖Σf‖=nνf \| _iE[X_i^2] \|≤ nB_f^2\| _f\|=n _f. The matrix Bernstein inequality (Tropp, An Introduction to Matrix Concentration Inequalities) applied to ∑iXi _iX_i gives the tail bound, and solving the tail for t at the level δ gives the displayed high-probability bound. ∎ Lemma 5 (Stability of the whitening map). Let Σg⪰γI _g γ I for γ>0γ>0, and suppose ‖Σ^g−Σg‖≤rg≤γ/2\| _g- _g\|≤ r_g≤γ/2. Then ∥Σ^g−1/2−Σg−1/2∥≤C0rgγ3/2, \| _g^-1/2- _g^-1/2 \|≤ C_0\,r_gγ^3/2, (66) for an absolute constant C0C_0. Proof. The map A↦A−1/2A A^-1/2 is Fréchet differentiable on the cone A⪰γIA γ I with derivative bounded by 12γ−3/2 12γ^-3/2: using the integral representation A−1/2=1π∫0∞(A+sI)−1s−1/2dsA^-1/2= 1π _0^∞(A+sI)^-1s^-1/2\,ds, the derivative at A applied to H is −1π∫0∞(A+sI)−1H(A+sI)−1s−1/2ds- 1π _0^∞(A+sI)^-1H(A+sI)^-1s^-1/2\,ds, whose norm is at most ∥H∥⋅1π∫0∞(γ+s)−2s−1/2ds=∥H∥12γ−3/2\|H\|· 1π _0^∞(γ+s)^-2s^-1/2\,ds=\|H\| 12γ^-3/2. The condition rg≤γ/2r_g≤γ/2 keeps Σ^g⪰γ2I _g γ2I by Weyl’s inequality, so the mean value theorem applies along the segment and gives the bound with C0C_0 absorbing the integration constant. ∎ Theorem 11 (Finite-sample mode identification). Under the assumptions of Theorem 2, suppose the encoders represent the population optimum exactly, ⟨f,g⟩=Fd f,g =F_d, and whitening is performed on the empirical moments of a sample of size n, with encoder outputs bounded by B. Let r=max(rf,rg)r= (r_f,r_g) be the moment radii of Lemma 4, let Δ=min1≤k<d(σk2−σk+12) = _1≤ k<d( _k^2- _k+1^2) be the spectral gap, and assume r≤c1min(γ,Δ)r≤ c_1 (γ, ) for a small absolute constant c1c_1. Then with probability at least 1−δ1-δ, the empirically whitened encoders (f^,g^)( f, g) recover the modes up to |sin∠(g^k,bk)|≤CrΔ,‖g^k−(±bk)‖≤C′rΔ,|Λ^kk−σk2|≤C′r, | ( g_k,b_k) |≤ C\,r , \| g_k-(± b_k)\|≤ C \,r , | _k- _k^2|≤ C \,r, (67) where Λ^kk _k is the estimated k-th interaction strength. In particular the identification error is O(B2log(d/δ)/n/Δ)O (B^2 (d/δ)/n/ ), vanishing as n−1/2n^-1/2 and amplified by the inverse spectral gap. Proof. (1) Empirical whitening followed by an eigendecomposition of the whitened second moment is equivalent to a singular value decomposition of the empirical interaction operator. Since the encoders represent Fd=∑k≤dσkak⊗bkF_d= _k≤ d _ka_k b_k exactly, the population singular subspaces are spanned by bk\b_k\ on the g-side and ak\a_k\ on the f-side. (2) The perturbation of the empirical operator relative to its population version is bounded by C3r/γ3/2≤C4rC_3\,r/γ^3/2≤ C_4\,r: the moment error ‖Σ^f−Σf‖≤rf\| _f- _f\|≤ r_f from Lemma 4, the whitening error from Lemma 5, and the remaining factors are absorbed into the constant, which for brevity absorbs the γ dependence as well. (3) The k-th empirical mode is the k-th singular vector of the perturbed operator. By the Davis-Kahan sinΘ theorem, the perturbation of a singular vector is bounded by the operator perturbation divided by the gap between the corresponding singular value squared and its neighbors, which is at least Δ : sin∠(g^k,bk)≤‖E‖min(σk−12−σk2,σk2−σk+12)≤CrΔ. ( g_k,b_k)≤ \|E\| ( _k-1^2- _k^2,\ _k^2- _k+1^2)≤ C\,r . (68) Choosing the sign so that the inner product is nonnegative, ‖g^k−(±bk)‖≤2sin∠≤C′r/Δ\| g_k-(± b_k)\|≤ 2\, ≤ C r/ . (4) For the strengths, Weyl’s inequality gives |Λ^kk−σk2|≤‖E‖≤C′r| _k- _k^2|≤\|E\|≤ C r. ∎ Corollary 6 (Sample complexity of whitening). Under the assumptions of Theorem 11, whitening the empirical second moments of a sample of size n identifies the interaction modes to accuracy ε with probability at least 1−δ1-δ provided n≳B4log(d/δ)Δ2ε2,n\; \; B^4\, (d/δ) ^2\, ^2, (69) where Δ=min1≤k<d(σk2−σk+12) = _1≤ k<d( _k^2- _k+1^2) is the spectral gap. Proof of Corollary 6 (Sample complexity of whitening). Set Cr/Δ≤εCr/ ≤ with r=O(B2log(d/δ)/n)r=O (B^2 (d/δ)/n ) from Lemma 4 and solve for n: n≳B4log(d/δ)Δ2ε2.n\; \; B^4\, (d/δ) ^2\, ^2. (70) The sample budget scales as 1/Δ21/ ^2 in the gap and 1/ε21/ ^2 in the target accuracy, matching (69). Corollary 7 (Block stability under small gaps). When a pair of singular values nearly coincides, Δk=σk2−σk+12→0 _k= _k^2- _k+1^2→ 0, the single-mode bound of Theorem 11 diverges: individual coordinates become unidentifiable. The corresponding eigenspace remains stable: treating k,k+1\k,k+1\ as a block, the projection error of the block is at most O(r/Δ~)O(r/ ), where Δ~ is the gap between the block and the rest of the spectrum. Under repeated singular values the identification therefore degrades to uniqueness up to rotation within the block, the quantitative version of the degenerate-case remark of Section 3. E.2 Approximate separability in the NMF route Definition 3 (α-approximate separability). A nonnegative factor matrix F∈ℝ≥0n×dF _≥ 0^n× d with rows f(ui)f(u_i) is α-separable if for every latent factor k there is a row i(k)i(k) with Fi(k),k≥1,∑ℓ≠kFi(k),ℓ≤α,F_i(k),k≥ 1, _ ≠ kF_i(k), ≤α, (71) i.e., a near-anchor row dominated by coordinate k with residual mass at most α. The case α=0α=0 is exact separability (6; 1). Theorem 12 (Degradation of approximately separable NMF). Let M=FG⊤M=FG be a population nonnegative rank-d factorization in which F is α-separable and the true factor cone is well conditioned, with the smallest angle between its extreme rays at least θ0>0 _0>0. Then every nonnegative optimal factorization (F^,G^)( F, G) satisfies, in a suitable permutation and scaling gauge, minP∈Perm,D∈D+‖F^PD−F‖≤Cαsinθ0‖F‖, _P ,\ D _+\| F\,P\,D-F\|≤ C\,α _0\,\|F\|, (72) so the identification error is linear in the separability deficit α and amplified by the cone condition number 1/sinθ01/ _0. As α→0α→ 0 the bound recovers the exact uniqueness of Theorem 5. Proof sketch. Identification in separable NMF is recovery of the extreme rays of the latent cone from the data points: under exact separability the anchor rows are exactly the extreme rays, and the constructive algorithm of 1 identifies them uniquely up to permutation and scaling. Under α-approximate separability the near-anchor rows deviate from the true extreme rays by at most α, and by the stability of vertex recovery in convex cones, a perturbation of the vertex set by α moves the recovered rays by O(α/sinθ0)O(α/ _0) when the adjacent rays are separated by angle at least θ0 _0; this is the robust version of separable NMF (1) following the noise-robustness analysis of Gillis and Vavasis. The two measurable quantities α and θ0 _0 therefore provide an identifiability budget for the NMF route, which is what Section 5 reports for its nonnegativity condition. ∎ E.3 Coupling identification with estimation Proposition 5 (Identification and estimation in one ledger). Let F F be the empirical risk minimizer of Corollary 4 with encoder capacity fixed and whitening applied on the empirical moments. Then the total identification error of the k-th mode decomposes as ‖g^k−(±bk)‖≤O(‖F^−F‖L22Δ)+O(rΔ),\| g_k-(± b_k)\|≤ O\! ( \| F-F\|_L^2^2 )+O\! ( r ), (73) where the first term converts the excess risk of the estimate into subspace error via Davis-Kahan and the second is the empirical whitening error of Theorem 11. Both terms scale as 1/Δ1/ and decay as n−1/2n^-1/2. Proof. Split the error around the population whitened solution by the triangle inequality. The second piece vanishes: at the population optimum the whitening theorem (Theorem 2) identifies the modes exactly. The first piece is the deviation of the estimate: ‖F^−Fd‖L22≤‖F^−F‖L22\| F-F_d\|_L^2^2≤\| F-F\|_L^2^2 by optimality, and the Davis-Kahan step of Theorem 11 converts an L2L^2 deviation of the represented function into a singular-vector deviation divided by the gap, giving the first term. ∎ Proposition 6 (Mode error ledger). Let F F be the empirical risk minimizer of Corollary 4 and let f^,g f, g be its whitened encoders. The error in recovering the k-th interaction mode satisfies max‖f^k−fk‖,‖g^k−gk‖=O(‖F^−F‖L22+rΔ), \\| f_k-f_k\|,\ \| g_k-g_k\| \\;=\;O\! ( \| F-F\|_L^2^2+r ), (74) with r=O(B2log(d/δ)/n)r=O (B^2 (d/δ)/n ); both terms are inversely proportional to the gap and decay as n−1/2n^-1/2. Proof of Proposition 6 (Mode error ledger). Substitute the excess-risk bound of Corollary 4 for ‖F^−F‖L22\| F-F\|_L^2^2 into the first term of Proposition 5, and the moment radius r=O(B2log(d/δ)/n)r=O (B^2 (d/δ)/n ) of Lemma 4 into the second. The result is exactly (74): max‖f^k−fk‖,‖g^k−gk‖=O(‖F^−F‖L22+rΔ). \\| f_k-f_k\|,\ \| g_k-g_k\| \=O\! ( \| F-F\|_L^2^2+r ). (75) Both terms are inversely proportional to the gap Δ and decay as n−1/2n^-1/2, which is the statement that identification and estimation are governed by the same spectral gap. Appendix F Experimental Details and Additional Results This appendix records the complete setups of the experiments of Section 5 and the additional results that the main text summarizes. All numbers are produced by the scripts in the companion repository, whose data files are grouped per study. We state the metrics alongside the tables, because a recurring theme is that gauge-invariant diagnostics cannot separate normalizations while gauge-dependent ones can. F.1 Controlled synthetic study Targets and training. We fix F(u,v)=∑k≤dσkak(u)bk(v)F(u,v)= _k≤ d _k\,a_k(u)b_k(v) with ak,bka_k,b_k shifted Legendre polynomials, orthonormal on [0,1][0,1], and a prescribed spectrum σk\ _k\ chosen distinct and separated. Encoders are linear maps over a fixed Legendre feature map, f(u)=U⊤φ(u)f(u)=U (u) with U∈ℝp×dU ^p× d, so the encoder realization error of Assumption 1 is zero and the normalization is the only variable across runs. Points u,vu,v are drawn on a common grid of n sites per marginal. Training uses Adam with global-norm gradient clipping; the bare bilinear objective is nonconvex and stiff, and plain gradient descent diverges. Analytic gradients are checked against finite differences to about 10−910^-9. Every scheme runs four seeds. The seven normalization schemes. The main text reports the headline of this study; Table 1 in Section 5.1 gives the full panel, shown graphically in Figure 2. Fit MSE is the squared regression error. Alignment is the mean over modes of |⟨f^k,ak⟩|| f_k,a_k | after optimally matching permutation and sign, reported for the two encoders. Cond is the gauge-dependent diagnostic max(condΣf,condΣg) (cond\, _f,cond\, _g). Cross-seed is the gauge distance between solutions from different seeds, the mean over pairs of the Procrustes-optimal embedding distance. Only whitening reaches alignment 1.0001.000 with cross-seed distance 0.0000.000, matching Theorem 2; every looser gauge stalls at alignment 0.530.53 to 0.650.65 with cross-seed drift 0.280.28 to 0.350.35, the residual OO of cosine and the full GLdGL_d of none leaving the modes rotationally entangled (Theorem 4). Whitening drives Cond to its floor σ12/σd2≈9.5 _1^2/ _d^2≈ 9.5 (Corollary 3). The two-sided nonnegative gauge stays at Fit MSE 4.254.25: positivity forces ⟨f,g⟩≥0 f,g ≥ 0, the sign-changing target is out of class (Theorem 5), and identifiability is bought at the cost of excluding negative interactions. One measurement subtlety, exploited throughout: cond(ΣfΣg)cond( _f _g) is gauge invariant, its eigenvalues equal the σk2 _k^2 by Lemma 2, so it cannot separate normalizations and is not reported as a diagnostic. Figure 2: The seven normalization schemes of Table 1, means over four seeds. Left: mode alignment with the true basis (dashed line: 11). Middle: the conditioning diagnostic max(condΣf,condΣg) (cond\, _f,cond\, _g) on a log scale, whose floor is σ12/σd2≈9.5 _1^2/ _d^2≈ 9.5. Right: cross-seed gauge distance (the short tick marks the exact zero of hard whitening). Abbreviations: l2 = L2 normalization, coord = per-coordinate standardization, cos = cosine normalization, nonneg = two-sided nonnegativity, w-soft = soft whitening, w-hard = hard whitening. The gap collapse. The gap sweeps use the finite-sample whitening estimator of Appendix E directly, empirical second-moment whitening followed by eigendecomposition, rather than a trained optimizer: with Legendre features the target is represented exactly, and the training identification error has a sharp threshold around n≈300n≈ 300 that would hide the 1/Δ1/ law, whereas the perturbation mechanism of the estimator exposes it cleanly. Table 2 in Section 5.1 reports the two sweeps. The eight points collapse onto a single constant, err⋅Δ⋅n≈2.48err· · n≈ 2.48 with coefficient of variation 0.120.12: gap and sample size enter the identification error only through the product, the concrete face of Corollaries 6 and 3. Figure 3 shows the collapse. Figure 3: The gap collapse. Mode identification error of the finite-sample whitening estimator: against the inverse spectral gap 1/Δ1/ at fixed n (left) and against sample size n at fixed Δ (center). Right: the product error⋅Δ⋅n\,· · n per run across both sweeps, near-constant at ≈2.48≈ 2.48. The sample-complexity rate law. The linear setting is rank-d matrix sensing, yi=ui⊤Θ⋆vi+εiy_i=u_i v_i+ _i with rankΘ⋆=d =d, and Theorem 7 predicts ‖Θ^−Θ⋆‖F2≍σε2d(p+q)/nE\| - \|_F^2 _ ^2d(p+q)/n. We sweep each variable with the others fixed and fit log-log slopes: n at p=q=4p=q=4, d=2d=2 gives −1.03-1.03 (theory −1-1); d at p=q=16p=q=16, n=2048n=2048 gives +0.89+0.89 (theory +1+1); p+qp+q at n=104n=10^4, d=4d=4 gives +1.24+1.24 (theory +1+1). The excess of the last slope is accounted for by the Marchenko-Pastur finite-sample correction of the Gaussian design: measured ratios to σε2d(2p−d)/n _ ^2d(2p-d)/n are 1.00,1.05,0.98,1.09,1.131.00,1.05,0.98,1.09,1.13 across p=q=8,…,32p=q=8,…,32, tracking the correction factor pq/(n−pq)pq/(n-pq), which grows from 1.011.01 to 1.111.11 over the same range. Gradient descent on the factored pair (U,V)(U,V), the estimator class of this paper, is warm-started from the truncated least-squares factors and reproduces the least-squares slope, −1.01-1.01, on the same grid. The sweep rejects the loose general bound of Remark 6, which scales as d3/2(p+q)/nd^3/2 (p+q)/n and predicts slope −1/2-1/2 in n: the 1/n1/n dependence on samples is real (Figure 1). The negative results. Tables 4 and 4 in Section 5.1 give the two regimes; Figure 4 shows both graphically. On the boolean equality kernel F=[u=v]F=1[u=v] on ±1m\± 1\^m, the embedding rank needed for 10%10\% error tracks d≈0.9⋅2md≈ 0.9· 2^m, exponential in m, while the early-interaction threshold on ⟨u,v⟩ u,v represents F exactly with 2m+12m+1 parameters (Theorems 8 and 9). On the narrowband periodic-Gaussian kernel Fh(u,v)=ρh(u−v)F_h(u,v)= _h(u-v) of bandwidth h on the circle, the required rank dεd_ grows as Θ(1/h) (1/h) as the kernel approaches the diagonal (Remark 7). Figure 4: The negative results. Left: on the boolean equality kernel, parameters needed to reach 10%10\% relative error against the number of input bits m, for the dual-encoder head (rank d) and the early-interaction threshold. Center: for the narrowband periodic-Gaussian kernel, the dual-encoder rank needed for 10%10\% error against the inverse bandwidth 1/h1/h, with a linear fit. Right: relative squared error of the dual-encoder head against its rank d, one curve per bandwidth h, with the 10%10\% line marked. Figure 5: The flat-spectrum error floor on a trained DeepONet head (Theorems 8 and 9). Left: normalized interaction spectra σk _k for three operators, from exponentially decaying (heat) to polynomially decaying (antiderivative) to exactly flat (boxcar block operator). Right: test relative error of a rank-d dual-encoder head versus d, from an SVD warm start and from random initialization, against the predicted floor 1−d/m1-d/m; the dashed line is an early-interaction MLP baseline. F.2 DeepONet This study carries the identifiability results of Section 3 onto a real dual-encoder architecture. We train branch-trunk DeepONet heads end to end on four operators and ask three questions, one per paragraph below: whether the rank-selection error curve tracks the decay of the operator spectrum (Corollary 5); whether post-hoc whitening, applied after training without touching the loss, lifts the learned trunk basis to the analytic operator eigenmodes and collapses the cross-seed gauge distance (Theorem 2); and whether the whitened gauge improves out-of-distribution error (Corollary 3). Two operators, the heat semigroup and the Volterra antiderivative, have closed-form spectra and provide exact ground truth for the recovered basis; two, viscous Burgers and Darcy flow, have only empirical spectral proxies and are used for the qualitative claims. The main-text Section 5 reports the conclusions; the tables and figures below give the full numbers. Architecture and training. The branch and trunk networks are MLPs of width 128128 and depth 33, outputting embeddings of dimension d=32d=32; the rank-selection sweep of Table 7 varies d from 2 to 64. Training runs 300 epochs with Adam at learning rate 10−310^-3 and cosine annealing, batch size 256, and five seeds per configuration. Inputs are Gaussian random fields. The four operators are the heat semigroup etΔe^t (analytic singular values σk=e−(kπ)2t _k=e^-(kπ)^2t, exponential), the antiderivative or Volterra operator (σk=1/((k−12)π) _k=1/((k- 12)π), polynomial), and viscous Burgers and Darcy flow, whose spectra are empirical proxies, labeled as such. Whitening is realized in two stages: the stable soft-whitening penalty during training, the relaxation of Theorem 6 and closely related to the Barlow Twins objective, and hard whitening post hoc on the active rank-r subspace. The two-stage design is forced by the data: the interaction spectra decay fast, leaving an effective rank of 3 to 8 against an embedding width of 32, and in-forward hard whitening inverts about 25 near-zero covariance eigenvalues and backpropagates through a degenerate eigendecomposition, exactly the small-gap failure predicted by the block-stability analysis of Corollary 7 in Appendix E. Because whitening leaves ⟨b,t⟩ b,t unchanged, applying it after training is exact, and projecting to the active subspace first avoids inverting dead directions. The empirical interaction spectrum is estimated as the square roots of the eigenvalues of Σb1/2ΣtΣb1/2 _b^1/2 _t _b^1/2, computed with eigvalsheigvalsh; this avoids the numerically unstable SVD of the ill-conditioned product Σb1/2Σt1/2 _b^1/2 _t^1/2 and agrees with the direct computation to 10−910^-9 on well-conditioned instances. Spectra and rank selection. The normalized empirical spectra reproduce the dichotomy qualitatively, heat 1,0.22,0.028,…1,0.22,0.028,… and antiderivative 1,0.14,0.042,…1,0.14,0.042,…. Because the network allocates no capacity to near-null modes, the learned spectra decay faster than the analytic ones, so we claim recovery of the dominant modes and of the decay type, not a mode-by-mode match. Table 7 gives the rank sweep. The curves are flat past d≈8d≈ 8 and the argmin is dominated by noise, so we do not report a single optimal rank; the honest quantity is the drop from d=4d=4 to d=8d=8, 5%5\% for heat (early saturation, small optimal rank) versus 31%31\% for antiderivative (more modes needed), consistent with Corollary 5. This is qualitative support, and the figure is labeled accordingly. Table 7: Rank selection, test relative L2L^2 error versus trunk width d. The drop from d=4d=4 to d=8d=8 is the claimed quantity: 5%5\% for heat versus 31%31\% for antiderivative (Corollary 5). Operator d=2d=2 d=4d=4 d=8d=8 d=16d=16 d=32d=32 d=64d=64 Drop 4→84→ 8 heat .070 .056 .053 .052 .055 .056 −5%-5\% antiderivative .106 .081 .056 .055 .070 .060 −31%-31\% burgers .237 .135 .134 .133 .133 .136 flat darcy .167 .150 .149 .153 .158 .169 flat Basis recovery. Table 8 gives the full per-gauge numbers behind Figure 6. Alignment is the mean over modes of |⟨t^k,βk⟩|| t_k, _k | after optimal permutation and sign matching against the analytic left singular functions, before and after the post-hoc whitening gauge fix; cross-seed is the mean pairwise gauge distance between seeds. Whitening lifts the loose gauges from 0.50.5 to 0.60.6 to 0.780.78 to 0.940.94 and collapses the cross-seed distance by roughly an order of magnitude; for heat the recovered modes are the analytic Fourier basis 2sin(kπy) 2 (kπ y), a hard ground truth. On the nonlinear benchmarks, where no analytic basis exists, whitening still tightens the cross-seed distance for Burgers, 0.30→0.080.30→ 0.08; Darcy is the weakest case, 0.56→0.330.56→ 0.33, and its proxy spectrum is the noisiest, which the figure notes. Figure 6: Trunk-encoder basis quality before and after post-hoc whitening (Theorem 2). Each group on the horizontal axis corresponds to one operator (heat, anti = antiderivative, Burg = Burgers, Darcy) trained under one gauge constraint (none, cosine, or whiten = soft whitening). Left: alignment of the learned trunk basis with the analytic operator eigenmodes (11 is exact). Right: distance between bases learned from different random seeds (lower indicates greater stability). Table 8: Basis recovery per gauge, five seeds. Alignment: mean mode alignment against the analytic left singular functions, raw and after post-hoc whitening. Cross-seed: mean pairwise gauge distance, raw and after whitening. The effect is Theorem 2 on a trained model. Operator / gauge Align Whitened align Seed dist Whitened dist heat / none 0.632 0.915 0.442 0.039 heat / cosine 0.529 0.799 0.313 0.007 heat / whiten_soft 0.517 0.780 0.479 0.140 antiderivative / none 0.582 0.940 0.282 0.064 antiderivative / cosine 0.576 0.823 0.245 0.011 antiderivative / whiten_soft 0.587 0.837 0.414 0.018 The payoff of gauge-fixing. Table 9 gives the in-distribution and out-of-distribution relative L2L^2 errors of the none and soft-whitening gauges; the out-of-distribution test set draws inputs from a different smoothness regime of the same operator family. Soft whitening attains the lowest error in all eight cells, with the out-of-distribution gain largest, 2.4×2.4× for heat. Two caveats are recorded with the table. First, soft whitening does not reduce seed-to-seed error variance; if anything it increases it, heat standard deviation 0.0005→0.00360.0005→ 0.0036 and Darcy 0.0017→0.01810.0017→ 0.0181, so the stability it confers is in the identified basis of Table 8, not in the error. Second, the soft-whitening training loss carries the decorrelation penalty and its curve is not directly comparable to the unpenalized gauges, so we make no claim about convergence speed. Table 9: The payoff of gauge-fixing, five seeds: relative L2L^2 error in distribution and out of distribution, none versus soft whitening. Soft whitening is best in all cells; the out-of-distribution gain is the headline (Corollary 3). Operator none, in-dist. whiten, in-dist. none, OOD whiten, OOD OOD gain heat 0.051 0.019 0.038 0.016 2.4×2.4× antiderivative 0.057 0.039 0.046 0.031 1.5×1.5× burgers 0.122 0.112 0.113 0.103 1.1×1.1× darcy 0.151 0.137 0.142 0.120 1.2×1.2× The flat-spectrum anchor. The block-boxcar operator G(a)(y)=cj(y)G(a)(y)=c_j(y), the output is the block mean of the input field at the block containing y, with m=16m=16 blocks, has an exactly flat interaction spectrum: sixteen equal singular values, the seventeenth at 10−1610^-16. Table 10 gives the trained relative error of the dual-encoder head with linear encoders, from both the SVD warm start and random initialization, against the floor 1−d/m1-d/m of Theorem 8; the worst deviation is 0.0140.014, the finite-sample excess, and gradient descent reaches the floor from scratch. The early-interaction comparator, a ReLU MLP on concat(a,onehot(y))concat(a,onehot(y)) with 12.412.4k parameters, four times the budget of the d=16d=16 head, reaches 0.013±0.0010.013± 0.001 on the same test set (Theorem 9). One caveat belongs in the setup: with only 8k training samples the comparator memorizes, training MSE 4×10−54× 10^-5 and test error 1.71.7; at 60k samples, where parameters are far fewer than samples, memorization is impossible and the escape shown in Figure 5 is an honest generalization result. The comparison deliberately pits the floor against a well-trained early model. Table 10: The flat-spectrum anchor: trained relative error of a linear-encoder head on the block-boxcar operator versus the floor 1−d/m1-d/m of Theorem 8,9. The two initializations agree, and the error hugs the floor from either start. d Trained error Floor 1−d/m1-d/m 1 0.936 0.938 2 0.879 0.875 4 0.752 0.750 8 0.502 0.500 12 0.258 0.250 16 0.000 0.000 F.3 CLIP This study tests the gauge view on heterogeneous modalities and on pretrained models that never saw each other. It proceeds in two stages. The white-box synthetic kernel supplies exact ground truth for the mechanism: under linear encoders and white inputs the interaction spectrum equals the singular values of Wf⊤WgW_f W_g, which is gauge invariant, and two independently trained encoders are two random gauges of the same target; on this kernel we verify that the spectrum is invariant and equals the truth, that per-dimension concept probes do not transfer while the canonical subspace does, and that whitening plus gap recovers the text modes exactly. The real CLIP pairs then answer whether the same statements hold for contrastive models trained in the wild: is the interaction spectrum shared across independently pretrained models, are per-dimension probes not, and is the residual relationship a single rotation that whitening removes, exposing interpretable concept axes (Theorem 4)? The main-text Section 5 reports the conclusions; the tables and figures below give the full numbers. The synthetic multimodal kernel. The white-box counterpart is F⋆(u,v)=∑kρk(pk⊤u)(qk⊤v)F (u,v)= _k _k\,(p_k u)(q_k v) with u∼(0,Ip)u (0,I_p), v∼(0,Iq)v (0,I_q), orthogonal canonical directions P,QP,Q, and distinct, gapped canonical correlations ρ=(0.9,0.75,0.6,0.45,0.3,0.15)ρ=(0.9,0.75,0.6,0.45,0.3,0.15). The key identity: under linear encoders and white inputs the interaction spectrum equals the singular values of B=Wf⊤WgB=W_f W_g, which is explicitly gauge invariant, f→Af→ Af, g→A−⊤g→ A^- g leaves Wf⊤WgW_f W_g unchanged, and two independently trained encoders are two random gauges A1,A2A_1,A_2. Table 11 collects the numbers: the spectrum is invariant across gauges to 3×10−163× 10^-16 and equals the true ρ to 4×10−164× 10^-16; per-dimension concept probes transfer at 0.430.43, the best-matched axis reaching only 0.650.65, while the canonical subspace aligns to 1.00001.0000; whitening plus gap recovers the true text modes at |cos|=1.000| |=1.000, against 0.5540.554 raw and 0.5810.581 under cosine, which is stuck at the residual OO; and cross-encoder frame stability rises from 0.6840.684 to 1.0001.000. The distance between the 0.580.58 of cosine and the 1.001.00 of whitening is, numerically, the contribution of whitening beyond the decorrelation that self-supervised methods already perform, precisely the quantity Theorem 2 isolates. Table 11: Synthetic multimodal kernel, ground truth known. The interaction spectrum is the unique gauge-invariant content; whitening plus gap pins the modes, cosine does not. Quantity Value Spectrum invariant across gauges 3×10−163× 10^-16 Spectrum equals true ρ 4×10−164× 10^-16 Per-dimension probe transfer 0.430.43 (best axis 0.650.65) Canonical subspace alignment 1.00001.0000 Text mode recovery, raw 0.5540.554 Text mode recovery, cosine (isotropization) 0.5810.581 Text mode recovery, whitening plus gap 1.0001.000 Cross-encoder frame stability, raw → whitened 0.684→1.0000.684→ 1.000 Real CLIP. Two backbone pairs, each two independent gauges of one similarity function: ViT-B/32 (OpenAI) versus ViT-B/32 (LAION-2B), the same architecture with different pretraining data, and ViT-B/32 (OpenAI) versus RN50 (OpenAI), different architectures. Image features come from CIFAR-100 test images, 49 fine classes spanning animals, vehicles, plants, and household objects, 64 images per class; text features are prompt-ensembled class embeddings plus attribute contrast directions; both towers are projected to a common 32-dimensional working space, the image PCA top-d directions. Whitening is realized via canonical correlation analysis with shrinkage covariance, the finite-sample counterpart of the Theorem 2 gauge fix. Table 12 and Figure 7 give the spectrum comparison across backbone pairs: near-identical spectra, agreement best at the top. Per-dimension probe transfer is 0.280.28 without alignment and 0.920.92 after fitting a single global orthogonal map: the two embeddings are related by one OO rotation, the residual group Theorem 4 predicts for the cosine gauge, and no coordinatewise repair exists short of discovering that rotation. Table 13 gives the group-signal regression of true labels on the top-6 mode activations: the whitened span contains the animate and natural axes, the cosine span contains only the animate axis, and per-mode statistics show why, whitening concentrates the animate signal in a single mode (|t|=3.1|t|=3.1) while cosine spreads it across several. Figure 8 shows the canonical text modes as concept axes: mode 1 separates animals (leopard, bear, wolf, camel) from flowers and fruit, mode 2 is the vehicle axis (bus, streetcar, pickup truck, train), mode 3 separates plants from household objects, and mode 5 the food axis (orange, apple, pear, sweet pepper); the cosine principal coordinates are semantic mixtures by comparison. Cross-backbone frame stability, matching modes by their backbone-independent class-signature profiles, favors whitening in both pairs, 0.770.77 versus 0.720.72 for the same-architecture pair and 0.840.84 versus 0.620.62 across architectures. Table 12: Interaction spectra across independently pretrained CLIP models. Pearson correlation and normalized L1 difference of the two spectra; per-dimension probe transfer raw and after fitting the single residual OO rotation (Theorem 4). Backbone pair Spectrum Pearson Norm. L1 ViT-B/32 (OpenAI) vs ViT-B/32 (LAION) 0.99 0.065 ViT-B/32 (OpenAI) vs RN50 (OpenAI) 0.98 0.096 Figure 7: Shared structure across independently trained CLIP models (Theorem 4). Left: normalized interaction spectra σk _k for three CLIP encoders (a ViT-B/32 reference against a LAION-trained ViT-B/32 and an RN50), with Pearson correlations to the reference in the legend. Right: accuracy of a linear concept probe trained on one model and applied to the other, using raw coordinates and after fitting a single shared rotation, for same-architecture and cross-architecture pairs. Table 13: Group-signal concentration on the reference backbone: R2R^2 of regressing true group labels on the top-6 mode activations, whitened canonical modes versus cosine principal coordinates. Cross-backbone frame stability is reported in the text. Group split Whitened Cosine animate / rest 0.92 0.88 natural / manmade 0.90 0.48 Figure 8: Whitened canonical text modes (rows) over the 49 CIFAR-100 classes (columns) of the reference backbone: modes read as concept axes (animals, vehicles, plants, food), while the cosine principal coordinates are semantic mixtures. What whitening buys for CLIP in practice. None of the steps above requires retraining: whitening is a post-hoc second-moment transform, it leaves the inner-product scores unchanged, and at inference time it costs one fixed linear map. It can therefore be applied as an analysis layer on any deployed CLIP model, and the results above suggest three concrete payoffs. First, model alignment: because independently pretrained pairs are related by a single residual rotation, whitening both models into the shared canonical frame makes their coordinates comparable dimension by dimension, a prerequisite for model merging, distillation, and cross-model probing that raw cosine coordinates do not offer. Second, interpretability with known limits: the whitened axes read as concept axes (animals, vehicles, plants, and food in the CIFAR-100 probe), and the spectral gap quantifies in advance how much interpretability any basis can carry, so an analyst knows where the axes are trustworthy and where the small-gap failure of Appendix E leaves them partially mixed. Third, a spectral summary of the model itself: the interaction spectrum is a gauge-invariant description of what a CLIP pair can discriminate, and its decay predicts the effective interaction rank (Corollary 5) and hence how many concept axes are worth reading out at all. A natural next step is to use the whitened canonical frame as the initialization or constraint for fine-tuning, so that new axes are learned in coordinates that are already interpretable and transferable across models. Summary. The synthetic study establishes the three claims where ground truth is exact: whitening uniquely fixes the gauge, the gap controls identification through the product Δn n (collapse constant 2.482.48), and flat spectra force exponential late-interaction cost. DeepONet carries identifiability onto a real architecture, recovering the analytic eigenbasis with 0.920.92 to 0.940.94 alignment and an out-of-distribution payoff up to 2.4×2.4×; CLIP extends the claim to heterogeneous modalities, the spectrum shared across independently pretrained models while per-dimension probes are not. On a flat spectrum the head is floored at 1−d/m1-d/m while an early-interaction model reaches 0.0130.013: the interaction spectrum decides whether a multiplicative dual-encoder head is the right tool. Appendix G Index of results Table 14 collects the paper’s numbered results with a one-line statement of what each establishes and where it is stated or proved. The results are grouped by the four questions the paper answers, with the finite-sample lemmas of Appendix E last; within each group the numbering follows the order of appearance, and the table doubles as a reading map for the appendices. Remarks and the technical lemmas internal to proofs are not indexed separately; they are cited where they are used. Table 14: Index of the paper’s numbered results. Result What it establishes Where Unification and approximation (Section 2) Definition 1 Interaction spectrum, rank, and modes as the singular data of the target Section 2 Theorem 1 Approximation error decomposes into a truncation term and an encoder-realization term Section 2 Assumption 1 Encoder classes contain approximations of the scaled interaction modes Appendix B Proposition 1 Evaluating a rank-d head on all pairs costs O((n+m)Cnet+nmd)O((n+m)\,C_net+nm\,d) Appendix B Lemma 1 Representability of F by a rank-d head iff rank(TF)≤d (T_F)≤ d Appendix B Corollary 1 The truncation term is the best possible error of any rank-d head Appendix B Theorem 3 Target smoothness controls the decay of the truncation tail Appendix B Proposition 2 The dual head escapes the curse of the joint input dimension Appendix B Identifiability as gauge-fixing (Section 3) Definition 2 Gauge action of GLdGL_d on encoder pairs leaves the represented function unchanged Section 3 Theorem 2 Under a spectral gap, whitening identifies the modes up to permutation and sign Section 3 Proposition 3 Per-coordinate standardization pins the coordinate scales but no more Appendix C Lemma 2 The eigenvalues of ΣfΣg _f _g are gauge invariants, equal to the σk2 _k^2 at any optimum Appendix C Proposition 4 Unfixed gauge renders the risk flat along the orbit and optimization ill-posed Appendix C Theorem 4 Cosine normalization leaves the full orthogonal group as residual gauge Appendix C Corollary 2 Individual coordinates of a cosine-normalized embedding carry no intrinsic meaning Appendix C Theorem 5 Two-sided nonnegativity gives NMF-style identifiability, at the cost of excluding negative modes Appendix C Corollary 3 The spectral gap controls conditioning, identifiability, and convergence simultaneously Appendix C Theorem 6 A quadratic whitening penalty inherits the identifiability of the hard constraint Appendix C Estimation and rank selection (Section 4) Lemma 3 The Rademacher complexity of the inner-product class collapses to the sum of the encoder complexities Appendix D Corollary 4 Excess risk of empirical risk minimization on the head class Appendix D Theorem 7 Linear heads are rank-d matrix sensing with minimax rate σε2d(p+q)/n _ ^2d(p+q)/n Appendix D Corollary 5 Optimal rank grows logarithmically in the sample budget for smooth targets Appendix D The usability criterion (Section 4) Theorem 8 A flat spectrum forces every rank-d head to relative error 1−d/N1-d/N Appendix D Theorem 9 Early-interaction models escape the flat-spectrum floor Appendix D Theorem 10 Usability criterion: decide by the measured spectrum whether the head can succeed Appendix D Finite-sample identification (Appendix E) Lemma 4 Matrix Bernstein concentration for empirical second moments Appendix E Lemma 5 Stability of the empirical whitening map under moment perturbation Appendix E Theorem 11 Finite-sample mode identification with error of order r/Δr/ Appendix E Corollary 6 Sample complexity of the whitening estimator: n≳B4log(d/δ)/(Δ2ε2)n B^4 (d/δ)/( ^2 ^2) Appendix E Corollary 7 Block stability of mode recovery under small gaps Appendix E Definition 3 α-approximate separability as a quantitative relaxation of NMF conditions Appendix E Theorem 12 Approximately separable targets degrade identifiability gracefully Appendix E Proposition 5 Identification and estimation errors combine into one ledger Appendix E Proposition 6 Mode error as the bridge between risk and identified basis Appendix E