Paper deep dive
Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers
Zhen Zhang, Amr Alanwar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/8/2026, 3:11:09 AM
Summary
The paper introduces Matrix Zonotopic Attention (MZAttn), a context-adaptive value projection mechanism for set transformers. It addresses the limitation of standard multi-head attention, which uses a fixed linear value projection, by replacing it with a matrix zonotope family that adapts to the input set. The authors define Transformation Degrees of Freedom (TDOF) to measure the complexity of target operators, proving that standard attention requires depth proportional to TDOF, whereas MZAttn can represent high-TDOF targets in a single layer. Experiments show MZAttn outperforms baselines on high-rank, sparsely combinatorial tasks while remaining competitive on aggregate-statistic tasks.
Entities (8)
Relation Signals (5)
Matrix Zonotopic Attention → replaces → fixed value projection
confidence 95% · MZAttn replaces the fixed value projection with a context-adaptive matrix-zonotope family
Transformation Degrees of Freedom → measures → complexity of target operator
confidence 92% · TDOF is a complexity measure counting the input-dependent directions an exact representation requires
Matrix Zonotopic Attention → represents → high-TDOF targets
confidence 90% · a single layer with a context-adaptive value family can represent the same target
Standard Attention → requires → depth proportional to TDOF
confidence 90% · context-rigid attention needs depth proportional to the target's TDOF
Matrix Zonotopic Attention → outperforms → Set Transformer
confidence 85% · a 1-layer MZAttn (244K params) outperforms standard Set Transformers up to depth 6
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-head attention combines an input-dependent softmax routing with an input-independent linear value projection, so the per-sample operator mapping aggregated values to outputs is the same for every input set. We study the consequences of this asymmetry for permutation-invariant set targets. We introduce the Transformation Degrees of Freedom (TDOF) of a target operator, a complexity measure counting the input-dependent directions an exact representation requires, and present a depth-separation analysis showing that context-rigid attention needs depth proportional to the target's TDOF, whereas a single layer with a context-adaptive value family can represent the same target. Building on this analysis, we propose Matrix Zonotopic Attention (MZAttn), which replaces the fixed value projection with a context-adaptive matrix-zonotope family: a centre matrix plus a sum of generator matrices weighted by input-dependent gates. The construction reduces to standard multi-head attention at initialisation, preserves permutation equivariance, and admits a data-driven reachability interpretation. Experiments on a range of set-prediction tasks are consistent with the TDOF prediction that the architectural advantage is selective: it appears on targets that depend on the input set in a high-rank, sparsely combinatorial way, and is small on aggregate-statistic targets where parameter-matched standard attention is already competitive.
Tags
Links
- Source: https://arxiv.org/abs/2608.05472v1
- Canonical: https://arxiv.org/abs/2608.05472v1
Trouble viewing inline? Open PDF directly →
Full Text
181,175 characters extracted from source content.
Expand or collapse full text
Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers Zhen Zhang Amr Alanwar School of Computation, Information and Technology Technical University of Munich, Germany zhenzhang.zhang, alanwar@tum.de Abstract Multi-head attention combines an input-dependent softmax routing with an input-independent linear value projection, so the per-sample operator mapping aggregated values to outputs is the same for every input set. We study the consequences of this asymmetry for permutation-invariant set targets. We introduce the Transformation Degrees of Freedom (TDOF) of a target operator, a complexity measure counting the input-dependent directions an exact representation requires, and present a depth-separation analysis showing that context-rigid attention needs depth proportional to the target’s TDOF, whereas a single layer with a context-adaptive value family can represent the same target. Building on this analysis, we propose Matrix Zonotopic Attention (MZAttn), which replaces the fixed value projection with a context-adaptive matrix-zonotope family: a centre matrix plus a sum of generator matrices weighted by input-dependent gates. The construction reduces to standard multi-head attention at initialisation, preserves permutation equivariance, and admits a data-driven reachability interpretation. Experiments on a range of set-prediction tasks are consistent with the TDOF prediction that the architectural advantage is selective: it appears on targets that depend on the input set in a high-rank, sparsely combinatorial way, and is small on aggregate-statistic targets where parameter-matched standard attention is already competitive. 1 Introduction Attention-based set architectures span point-cloud analysis [21, 28], molecular modelling [33], in-context learning [17], and recent linear-attention variants [16, 39, 44]. A basic question has gone largely unexamined: how much of a standard multi-head attention [41] layer actually adapts to its input? It combines a softmax routing whose weights depend on the input set with a linear value projection that does not (Figure 1, left). On four textbook computational-geometry targets (minimum enclosing ball, convex hull volume, minimum-spanning-tree weight, rotation-dependent matching), every parameter-matched attention baseline using the standard PMA pooling fails to meaningfully improve over the mean predictor as per-point dimension grows, while MZAttn maintains a clean separation (Figure 1, right). The failure is structural rather than capacity-driven: even pushing a HyperNet update to full key-dimension rank at several times MZAttn’s parameter budget does not close the gap (Figure 1, right; full sweep in Appendix D.1.5). Figure 1: Standard attention uses a fixed value projection; MZAttn replaces it with a context-adaptive matrix-zonotope family. Left: standard attention applies the same WVW_V to every input set, making the per-sample operator independent of S. Middle: MZAttn replaces WVW_V with M(μ(S))=Mc+∑lγl(μ)MlM(μ(S))=M_c+ _l _l(μ)M_l, a context-gated matrix zonotope whose realisation varies per sample. Right: on MEB radius (d=8d=8, 55 seeds), all six parameter-matched PMA-pool baselines—including HyperNet at r=dk=64r=d_k=64 (1.31.3M params, 3.6×3.6× MZAttn’s budget)—stay at R2≤0.025R^2≤ 0.025 while MZAttn reaches R2=0.69R^2=0.69: capacity is not the bottleneck. Tasks requiring a per-sample linear operator (a sample-specific rotation, a change-of-basis map, or a sparse indicator selecting hull boundary points) lie strictly outside the function family a stack of such layers can represent. We formalise this relational capacity bottleneck through the transformation degrees of freedom (TDOF) of the target operator and prove a matching depth-separation: a worst-case task of TDOF k requires standard-attention depth proportional to k (with a constant determined by FFN width), whereas a single layer of our proposed architecture suffices (Theorem 2). The non-linear targets above only approximately inhabit the linearised class; their inheritance of the same bottleneck is an empirical prediction the theory motivates, validated by the synthetic-to-real transfer study below. We propose Matrix Zonotopic Attention (MZAttn), which replaces the fixed value projection with a learnable matrix zonotope (centre plus a few generator matrices, gated by set-level statistics; Section 2). The parameterisation is minimax-optimal in the parameter-versus-TDOF sense (Theorem 11) and yields single-pass uncertainty estimates. The advantage is selective and predicted by TDOF: on low-TDOF benchmarks (point-cloud classification, molecular property prediction) MZAttn is within 11–33 p of the best parameter-matched baseline; positive transfer requires high per-sample rank, sparse combinatorial targets, and absence of imposed symmetry, conditions verified on CIFAR-100 ResNet-feature point clouds with a k-center cost target. 2 Method: Matrix Zonotopic Attention We first introduce the necessary background on zonotopes, matrix zonotopes, and the Set Transformer, then describe the three components of MZAttn: (1) zonotope token representation, (2) the MZ-attention mechanism, and (3) the full MZ-Set Transformer architecture. 2.1 Preliminaries Definition 1 (Zonotope and Matrix Zonotope [19, 3, 1]). A zonotope with center c∈ℝnc ^n and generator columns G1,…,Gng∈ℝnG_1,…,G_n_g ^n (the columns of the generator matrix G∈ℝn×ngG ^n× n_g) is the set =c+∑g=1ngαgGg:αg∈[−1,1]Z=\c+ _g=1^n_g _gG_g: _g∈[-1,1]\. A matrix zonotope with center Mc∈ℝm×nM_c ^m× n and matrix generators M1,…,ML∈ℝm×nM_1,…,M_L ^m× n is the set of matrices ℳ=Mc+∑l=1LξlMl|ξl∈[−1,1].M= \M_c+ _l=1^L _lM_l~ |~ _l∈[-1,1] \. (1) The interval hull of a zonotope is the tightest axis-aligned bounding box, with half-widths δ=∑g=1ng|Gg|δ= _g=1^n_g|G_g| (absolute value taken componentwise). The product of a matrix zonotope with a zonotope is again a zonotope [1]: if ℳM has L generators and Z has h generators, then ℳ≜Mz:M∈ℳ,z∈MZ \Mz:M ,z \ has center MccM_cc and at most h+L+Lh+L+Lh generators. The O(Lh)O(Lh) growth motivates our fixed-budget management scheme (Section 2.4). Set Transformer. The Set Transformer [21] processes unordered sets through MAB / ISAB / PMA blocks built from standard scaled dot-product attention; MZAttn replaces the fixed value projection vj↦WVvjv_j W_Vv_j in each block. 2.2 Zonotope Token Representation Given an input element xi∈ℝdinx_i ^d_in, we project it into a zonotope (ci,Gi)(c_i,G_i): ci=LayerNorm(GELU(Wcxi+bc)),Gi=1ngd(Wgxi+bg)∈ℝng×d,c_i=LayerNorm(GELU(W_cx_i+b_c)), G_i= 1 n_gd(W_gx_i+b_g) ^n_g× d, (2) where Wc∈ℝd×dinW_c ^d× d_in, Wg∈ℝ(ngd)×dinW_g ^(n_gd)× d_in, and ngn_g is the fixed generator budget. The scaling factor 1/ngd1/ n_gd ensures that generators start as small perturbations, so the initial representation is close to a standard vector embedding. 2.3 MZAttn Attention: From Scalar Scores to Relational Operators Given query, key, and value zonotopes with centers ci∈ℝdkc_i ^d_k and generator matrices Gi∈ℝng×dkG_i ^n_g× d_k (the g-th row Gi[g,:]G_i[g,:] is the g-th generator vector), standard attention scores αij=softmax(qi⊤kj/dk) _ij= *softmax(q_i k_j/ d_k) are computed on centers, values are aggregated (c^i,G^i)=∑jαij(vjc,vjG)( c_i, G_i)= _j _ij(v_j^c,v_j^G), and the head-h matrix zonotope ℳ(h)=Mc(h)+∑l=1LξlMl(h)M^(h)=M_c^(h)+ _l=1^L _lM_l^(h) transforms the output: oic=Mc(h)c^i∈ℝdk,oiG=G^i(Mc(h))⊤+Wmixstackl=1L[γl(μ)Ml(h)c^i]∈ℝng×dk,o_i^c=M_c^(h) c_i\;∈\;R^d_k, o_i^G= G_i\,(M_c^(h)) \;+\;W_mix\,stack_l=1^L\! [ _l(μ)\,M_l^(h) c_i ]\;∈\;R^n_g× d_k, (3) where G^i(Mc(h))⊤ G_i(M_c^(h)) applies Mc(h)M_c^(h) row-wise to each generator (preserving the ng×dkn_g× d_k shape), stackl[⋅]stack_l[·] assembles the L vectors γlMl(h)c^i∈ℝdk _l\,M_l^(h) c_i ^d_k into an L×dkL× d_k matrix (rows indexed by l), Wmix∈ℝng×LW_mix ^n_g× L is a learned mixing matrix that preserves the fixed generator budget across layers, and γ=tanh(Wγc¯kv)∈[−1,1]Lγ= (W_γ c^kv)∈[-1,1]^L is the context-dependent gate computed from the set-level mean c¯kv=1nkv∑jcjkv c^kv= 1n_kv _jc_j^kv. Mc(h)=IdkM_c^(h)=I_d_k and Ml(h)≈0M_l^(h)≈ 0 at initialisation, so the block reduces to standard attention. Full step-by-step derivation, generator-budget control, and richer permutation-invariant summaries (attention-pooled statistics, higher-order moments) are in Appendix A.1. 2.4 MZ-Set Transformer Architecture We replace each attention block in a Set Transformer [21] with its MZ counterpart (MZ-MAB, MZ-SAB, MZ-ISAB, and MZ-PMA), yielding InputProj→[MZ-ISAB]Lenc→MZ-PMAk→[MZ-SAB]Ldec→OutputHeadInputProj→[MZ -ISAB]^L_enc -PMA_k→[MZ -SAB]^L_dec . The output head concatenates the pooled center c∈ℝkdc ^kd and the interval-hull half-width δ=∑g|Gg|δ= _g|G_g| before a two-layer MLP, giving a deterministic prediction and a single-pass uncertainty estimate for free. The expressiveness Theorems 2 and 8 below additionally invoke a variant readout that reads the signed pooled generators ∑gGg _gG_g rather than the IHW ∑g|Gg| _g|G_g| (signed access is straightforward to add as a parallel head; the IHW preserves the uncertainty-bound interpretation of Theorem 4, while the signed readout is what the expressiveness proofs exploit). Both readouts share the same MZ-attention layer; only the post-pooling reduction differs. 3 Theoretical Analysis We present three groups of results: an expressiveness bottleneck with minimax-optimal resolution (§3.1), TDOF-based depth separation (§3.2), and uncertainty and generalization guarantees (§3.3). Full proofs are in Appendix B.4; additional results in Appendix B.3. 3.1 Expressiveness: Context-Adaptive vs. Context-Rigid Attention Definition 2 (Context-rigid and context-adaptive relational operators). Consider an attention layer operating on set S=x1,…,xn⊂ℝdS=\x_1,…,x_n\ ^d, and let v^i=∑jαijvj v_i= _j _ijv_j denote the aggregated value for query i. The relational operator T maps v^i v_i into the per-head output. Standard attention is context-rigid: (v^)=Wv^T( v)=W v for a fixed W∈ℝdk×dkW ^d_k× d_k, independent of S. MZAttn is context-adaptive: (v^;S)=M(μ(S))v^T( v;S)=M(μ(S)) v where M(μ)=Mc+∑l=1Lγl(μ)MlM(μ)=M_c+ _l=1^L _l(μ)M_l depends on a permutation-invariant statistic μ(S)μ(S), and the gates γl(μ)∈[−1,1] _l(μ)∈[-1,1] are the trained instantiations of the abstract zonotope coefficients ξl _l of Eq. (1). This distinction has a fundamental information-theoretic consequence. Standard attention routes information through a scalar bottleneck: the score αij _ij controls how much of each value to aggregate, but the subsequent transformation W is frozen. We formalise this via the relational complexity / TDOF of the target function, which counts the intrinsic number of context-dependent directions an exact representation requires. Definition 3 (Relational complexity / TDOF). For a permutation-invariant target f(S)=T(S)μ(S)f(S)=T(S)\,μ(S) with matrix-valued T:≤N→ℝm×dT:\;X^≤ N ^m× d that factors through μ(S)μ(S), the relational complexity (henceforth TDOF) is TDOF(f)=minL:T(S)=M0+∑l=1Lσl(μ(S))Ml,σl 1-Lipschitz,|σl|≤1TDOF(f)= \L:T(S)=M_0+ _l=1^L _l(μ(S))\,M_l,\; _l 1-Lipschitz,\;| _l|≤ 1\. The model-free linear dimension dT=dim(spanT(S)−T(S′):S,S′∈≤N)d_T= (span\T(S)-T(S ):S,S ^≤ N\) used in Appendix B.3 (Definition 4) lower-bounds this constructive count; for f∈ℱLf _L with bounded 1-Lipschitz coordinate functionals (e.g. all natural set-statistical operators in Proposition 12) the two coincide. Functions with TDOF(f)=0TDOF(f)=0 require no input-dependent transformation; those with TDOF(f)>0TDOF(f)>0 fundamentally require the transformation itself to adapt. We write ℱL=S↦(M0+∑l=1Lσl(μ(S))Ml)μ(S):‖Ml‖F≤B,|σl|≤1F_L=\S (M_0+ _l=1^L _l(μ(S))\,M_l)\,μ(S):\|M_l\|_F≤ B,\,| _l|≤ 1\ for the linearised operator family with TDOF at most L. ℱLF_L is the natural linearisation class for any smooth context-adaptive operator: if T(μ)T(μ) has Jacobian J with rank(J)=rrank(J)=r at μ0 _0, then L=rL=r MZ generators capture its first-order behaviour exactly via the SVD of J (Appendix B.3, Proposition 13). The non-linear targets we evaluate (MST, MEB, hull) do not sit exactly in ℱLF_L, so TDOF acts as a structural hypothesis validated empirically. Theorem 1 (Relational capacity bottleneck). (a) Standard attention without FFN has zero relational capacity: Tstd=∑hWo(h)WV(h)T_std= _hW_o^(h)W_V^(h) is constant, independent of S. (b) MZAttn-attention represents any function in ℱLF_L with L generators using (L+1)dk2+Ldk(L+1)d_k^2+Ld_k parameters per head, and this Θ(Ldk2) (Ld_k^2) count is minimax-optimal. Proof sketch. For (a), constant sets StS_t make softmax uniform, so the output is a fixed linear map. The matching Ω(Ldk2log(BR/ϵ)) (Ld_k^2 (BR/ε)) lower bound, the FFN-augmented Yarotsky-style bound on attention with FFN, the Eckart–Young residual bound, and a strict separation construction are in Appendix B.3. ∎ Part (a) is the no-FFN zero-capacity statement (the cleanest qualitative form of the bottleneck); with FFN, zero capacity is replaced by the explicit width/depth tradeoff bound used in part (b) and Theorem 2. Thus MZAttn-attention is minimally parameterised for ℱLF_L: Θ(Ldk2) (Ld_k^2) parameters versus O(dk4)O(d_k^4) for a general hypernetwork. Natural set operators have explicitly computable TDOF: Mahalanobis distance and whitening have TDOF=d(d+1)/2TDOF=d(d+1)/2, top-k PCA projection has TDOF=kd−k(k+1)/2TDOF=kd-k(k+1)/2, and per-coordinate z-score has TDOF=dTDOF=d (Appendix B.3, Proposition 12); soft Chamfer matching has TDOF≥dTDOF≥ d (Proposition 14), accounting for the MZAttn gap on Chamfer set-matching (Task B). 3.2 Transformation Degrees of Freedom and Depth Separation The transformation degrees of freedom TDOF(f)=dim(spanT(S)−T(S′))TDOF(f)= (span\T(S)-T(S )\) is intrinsic to f (see Definition 4). It yields a depth lower bound: Theorem 2 (Depth separation via TDOF, worst-case over ℱkF_k). Fix the input distribution to be sets S whose mean μ(S)∈ℝdμ(S) ^d has i.i.d. standard-Gaussian coordinates. There exists f∈ℱkf _k on this input distribution (the explicit witness of Lemma 15 with M0=0M_0=0) for which any standard attention network of depth D with FFN width dffd_f that represents f exactly requires D≥kdk/dffD≥ k\,d_k/d_f, while a single MZAttn layer with L≥kL≥ k generators followed by the signed-generator readout (Section 2.4) represents the same f exactly. For approximate representation in L2(μ)L^2(μ) with L2L^2-residual error of order O(Var(f)/L)O(Var(f)/L), the same depth lower bound holds up to constants. A compositional-orthogonality argument over the Gaussian input distribution (Appendix B.3, Lemma 15) realises the bound for an explicit Frobenius-orthonormal witness (Ml=eled⊤M_l=e_le_d , M0=0M_0=0) even with FFN and residual connections; extending the lower bound to a generic high-TDOF target family with non-trivial cross-layer matrix products is open. Empirical verification. We verify Theorem 2 on synthetic quadratic targets f(S)=⊤(M0+∑l=1Lμ(S)lMl)μ(S)f(S)=w \! (M_0+ _l=1^Lμ(S)_lM_l )\,μ(S) (4) with Frobenius-orthogonal MlM_l giving TDOF(f)=LTDOF(f)=L. Across L∈4,8,16,24,32L∈\4,8,16,24,32\ the four-layer Set Transformer’s test R2R^2 degrades from 0.970.97 to 0.400.40 while a single MZAttn layer holds R2≥0.90R^2≥ 0.90, with a 66–15×15× residual-MSE gap (Table 1): adding depth alone cannot cure the bottleneck. EGNN [33] hard-codes rotation equivariance and collapses to R2≈0R^2≈ 0 for every L because the target depends on a fixed coordinate frame — the complementary failure mode of symmetry-enforcing architectures. The same picture holds at the architecture level: a 1-layer MZAttn (244K params) outperforms standard Set Transformers up to depth 6 (≥480≥ 480K params; Appendix Table 12), confirming depth separation on real architectures, not only the synthetic-task sweep. Table 1: Depth-separation verification on the quadratic-TDOF target (Eq. 4; mean± , 3 seeds). 1-layer MZAttn holds R2≥0.90R^2≥ 0.90 across L while 4-layer standard ST degrades. L MZAttn-1L R2R^2 ↑ ST-4L R2R^2 ↑ MSE ratio ↑ 44 0.995±0.0020.995± 0.002 0.970±0.0050.970± 0.005 6.6×6.6× 88 0.988±0.0010.988± 0.001 0.809±0.0110.809± 0.011 15.4×15.4× 1616 0.950±0.0010.950± 0.001 0.618±0.0170.618± 0.017 7.6×7.6× 2424 0.944±0.0110.944± 0.011 0.547±0.0200.547± 0.020 8.0×8.0× 3232 0.908±0.0160.908± 0.016 0.396±0.0010.396± 0.001 6.5×6.5× 3.3 Uncertainty, Generalization, and Structural Properties The zonotope representation yields directional perturbation certificates strictly tighter than scalar Lipschitz bounds, with the interval-hull width (IHW) bounding output diameter and variance and thereby serving as a single-pass uncertainty estimate. Appendix B.2 collects the precise statements: a norm-controlled generalisation bound of O~(BMZ/γn) O(B_MZ/γ n) in the spirit of Bartlett et al. [5], universal approximation, and permutation equivariance. At initialisation, MZAttn reduces exactly to standard multi-head attention. Empirically (Appendix D.3.2, Table 21), the IHW estimate matches a 55-model Deep Ensemble on ECE (0.3160.316 vs. 0.3370.337) and is ≈2×≈ 2× better on NLL (5.025.02 vs. 9.179.17), at 1×1× inference cost vs. 5×5×; MC-Dropout achieves the lowest ECE/NLL but at 30×30× inference cost. 4 Relationship to Prior Work Zonotopes have served as external verification tools [6, 24, 15, 7] and as set-valued reachability primitives in data-driven control [1, 2, 18]; MZAttn folds a matrix zonotope into a trainable value-projection family inside an attention layer. The closest architectural relatives are context-adaptive linear maps spanning a capacity–parameters spectrum — FiLM [27], discrete MoE [36], low-rank Hypernetworks [12, 34], dynamic filters [14] — among which MZAttn’s matrix-zonotope parameterisation occupies the minimax-optimal middle at O(Ldk2)O(Ld_k^2) parameters per head (Theorem 1); we compare against them as parameter-matched baselines. Linear-attention and selective state-space variants [39, 44, 30, 11] adopt different value-projection inductive biases (input-independent or, for Mamba, input-dependent only along diagonal channels); a Mamba (S6) encoder with PMA pooling matches MZAttn-Full at d=8d=8 on the MEB target but degrades to R2=0.142±0.134R^2=0.142±0.134 at d=32d=32 vs. MZAttn’s stable 0.378±0.0300.378±0.030 (Table 3, Mamba + PMA row), indicating that input-dependent diagonal selectivity alone is dimension-conditional rather than a structural escape from Theorem 1; MZAttn is orthogonal to and composable with them. Set-input networks [47, 21, 28] represent relationships as scalar scores, which we generalise to matrix-valued context-adaptive operators. Full discussion in Appendix C. 5 Experiments We evaluate MZAttn on eight task families, comparing against seven baselines: Deep Sets [47], a standard Set Transformer [21], ST + FiLM [27], ST-Large (parameter-matched wider variant), HyperNet (low-rank input-dependent value-projection update), Perceiver [13], and Slot Attention [22]. ST-Large, Perceiver, and HyperNet are matched to MZAttn’s parameter count (∼ 360–368K) to control for capacity. Unless noted, experiments are repeated over multiple seeds with mean ± std reported. 5.1 Experimental Setup All attention-based architectures share a base configuration of d=64d=64, H=4H=4 heads, dff=128d_f=128, two ISAB encoder layers, dropout 0.10.1, and identical training (AdamW [23] with learning rate 10−410^-4, weight decay 10−510^-5, cosine schedule, early stopping at patience 2020, all implemented in PyTorch [26]); ST-Large, Perceiver, and HyperNet are configured to match MZAttn’s parameter count of ∼ 365K, and MZAttn uses ng=8n_g=8 zonotope generators and L=4L=4 matrix-zonotope generators. Deep Sets, Slot Attention, the FiLM-conditioned Set Transformer, and the per-baseline configurations of HyperNet rank, Perceiver latent count, and Slot iterations are described in Appendix A.1. We evaluate on eight tasks spanning a range of relational complexity (full descriptions in Appendix A.2): set regression on ℝ16R^16 (A), Chamfer-distance set matching in ℝ3R^3 (B), ModelNet40 [43] (C) and ScanObjectNN [40] (E) point-cloud classification, few-shot set retrieval [37] (D), rotation-dependent matching in ℝ8R^8 (F), minimum-spanning-tree weight in ℝ8R^8 (G), and QM9 molecular property prediction [31] (H). All experiments use 200 epochs with early stopping; results are averaged over 3–5 seeds. 5.2 Results Table 2: Main results across seven synthetic and point-cloud benchmarks (Tasks A–G). QM9 (Task H) and Slot Attention appear separately in Tables 28 and 29. Best in bold among the four parameter-matched attention models (ST-Large, Perceiver, HyperNet, MZAttn, all at ∼ 360–368K); mean ± std over 3–5 seeds. Smaller-budget baselines (ST + FiLM at 225225K, Standard Set Transformer at 208208K) are reported for reference and are not subject to the bold rule. Tasks C and E are point-cloud classification; Task G is MST weight prediction in ℝ8R^8. Task A: Set Regr. Task B: Set Match. Task F: Rot. Match. Task G: MST Wt. Task C Task E Task D: Few-Shot Model MSE ↓ R2R^2 ↑ MSE ↓ R2R^2 ↑ MSE ↓ R2R^2 ↑ MSE ↓ R2R^2 ↑ MN40 ↑ Scan ↑ MSE (×10−2× 10^-2) ↓ R2R^2 ↑ Params Deep Sets – – 490.2± 33.4 0.652± 0.024 – – – – – 40.6± 1.0% 2.05± 0.38 0.722± 0.052 36K Set Transformer 3.43± 0.09 0.811± 0.005 138.0± 10.3 0.901± 0.007 487.5± 31.1 0.904± 0.006 209.8± 1.8 0.712± 0.002 77.5± 0.6% 71.6± 2.2% 2.60± 0.24 0.646± 0.032 208K ST + FiLM 3.77± 0.26 0.792± 0.014 48.0± 7.9 0.966± 0.006 490.5± 17.8 0.904± 0.003 232.7± 12.5 0.681± 0.017 78.4± 0.8% 68.9± 3.2% 2.77± 0.21 0.624± 0.029 225K ST-Large 3.36± 0.10 0.815± 0.005 106.5± 5.3 0.924± 0.004 423.2± 51.8 0.917± 0.010 168.7± 23.0 0.769± 0.032 79.9± 1.0% 72.9± 2.4% – – 362K Perceiver 3.47± 0.05 0.809± 0.003 136.2± 17.5 0.902± 0.013 872.1± 163.8 0.829± 0.032 249.0± 4.7 0.659± 0.006 75.0± 0.3% 67.3± 1.2% – – 341K HyperNet 3.51± 0.01 0.806± 0.001 76.1± 5.7 0.945± 0.004 1373.0± 141.6 0.730± 0.028 291.2± 9.2 0.601± 0.013 75.5± 1.7% 66.1± 0.9% – – 360K MZAttn (ours) 2.92± 0.09 0.839± 0.005 58.4± 11.9 0.958± 0.009 241.2± 3.0 0.953± 0.001 99.4± 1.9 0.864± 0.003 78.1± 0.7% 69.4± 3.7% 2.10± 0.16 0.715± 0.022 365K High-TDOF relational tasks (A, B, F, G). MZAttn achieves the best performance on all four: 13%13\% MSE reduction vs. ST-Large on set regression (A), 45%45\% on set matching (B), %51\% over FiLM on rotation matching (F), and %41\% over ST-Large on MST weight (G). On (B), FiLM matches the low-dimensional (ℝ3R^3) regime (MSE 48.048.0), but HyperNet’s rank-88 update is insufficient (76.176.1) and higher rank degrades further (Appendix D.1.4). On (F), FiLM performs no better than standard attention, consistent with diagonal modulation’s inability to express off-diagonal rotations. On (G), MZAttn’s seed std is ±1.9± 1.9 vs. ±23.0± 23.0 for ST-Large, indicating stable representation of the pairwise-distance geometry. Low-TDOF tasks (C, D, E). On point-cloud classification (ModelNet40 / ScanObjectNN), ST-Large achieves the best accuracy (79.9%79.9\% / 72.9%72.9\%) with MZAttn within 11–33 p (78.1%78.1\% / 69.4%69.4\%); these tasks depend primarily on per-element features, so Theorem 1 predicts no advantage for context-adaptivity. On few-shot retrieval (D), Deep Sets (R2=0.722R^2=0.722) and MZAttn (R2=0.715R^2=0.715) tie and both exceed Standard attention (0.6460.646) and FiLM (0.6240.624) by +0.07+0.07–0.100.10 R2R^2, indicating set-aggregation pooling dominates over per-sample operator structure at this scale. On QM9 (Table 28), MZAttn matches the best parameter-matched attention baseline within 0.050.05 R2R^2 on every target without leading on any individual one, the predicted low-TDOF behaviour. Slot Attention lags all attention-based models on every task tested (Table 29); all models maintain permutation invariance to within 10−510^-5 across 1010 random permutations. 5.3 Computational Geometry and Real-World Transfer The MEB radius depends on at most d+1d+1 extremal points: a sparse combinatorial selector that context-rigid attention cannot express. Table 3 reports R2R^2 across dimensions for parameter-matched baselines. Table 3: MEB radius prediction (R2R^2, mean± , 5 seeds). Point clouds are mixtures of Gaussians with n∈[10,30]n∈[10,30]. LA-ST: linear attention [16]. †Mamba + PMA replaces the ISAB encoder with a Mamba (S6) selective state-space encoder (mambapy 1.2.0). Mean-pool/Mamba dimension-dependence and MZAttn-Large variant (Appendix D.4.1) are discussed in the surrounding paragraph; EGNN comparison: Table 7. Model Params d=8d=8 R2R^2 d=16d=16 R2R^2 d=32d=32 R2R^2 Standard 208K 0.025± 0.009 0.020± 0.005 0.006± 0.012 ST-Large 362K 0.018± 0.001 0.012± 0.015 −-0.006± 0.011 HyperNet (r=8r=8) 360K 0.004± 0.013 0.005± 0.012 −-0.001± 0.001 HyperNet (r=64r=64, full key-rank) 1.3M 0.011± 0.012 0.001± 0.009 0.000± 0.002 Perceiver 341K 0.024± 0.006 0.016± 0.028 0.010± 0.010 LA-ST (Linear Attn.) 359K 0.017± 0.016 0.015± 0.009 0.001± 0.002 Standard (mean-pool head) 140K 0.784± 0.004 0.647± 0.363 0.449± 0.261 Mamba + PMA† 645K 0.713± 0.025 0.617± 0.041 0.142± 0.134 MZAttn-Full 368K 0.691± 0.012 0.629± 0.023 0.378± 0.030 Every parameter-matched baseline using the standard PMA pooling, including LA-ST, fails at R2≤0.025R^2≤ 0.025, whereas MZAttn-Full reaches a 2828–155×155× gap (median ≈40×≈ 40× across 1313 positive-baseline cells, with the 155×155× upper bound at d=8d=8 vs. HyperNet’s 0.00440.0044; Figure 2, left). The collapse is structural: HyperNet at r=dk=64r=d_k=64 (1.31.3M params, 3.6×3.6× budget) leaves R2R^2 within ±0.012± 0.012 of zero (Appendix D.1.5). Replacing PMA with mean-pool partially escapes the ceiling at d=8d=8 (0.784±0.0040.784±0.004, Standard mean-pool row above) but exhibits heavy-tail bimodal failure at higher dimensions (d=16d=16: 0.647±0.3630.647±0.363 with 1/51/5 seeds catastrophically collapsing to R2≈0R^2≈ 0; d=32d=32: 0.449±0.2610.449±0.261 with another 1/51/5 catastrophic). Replacing the ISAB encoder with a Mamba (S6) selective state-space encoder while keeping PMA pooling (Mamba + PMA row, 1.75×1.75× MZAttn’s budget) matches MZAttn-Full at d=8d=8 (0.713±0.0250.713±0.025) but degrades to R2=0.142±0.134R^2=0.142±0.134 at d=32d=32 with all 55 seeds below 0.300.30. MZAttn-Full + PMA maintains structural stability (σ≤0.030σ≤ 0.030) at R2≥0.378R^2≥ 0.378 across all three dimensions, providing the only uniform high-d escape we observed. Convex-hull-volume and a sparse-geometric-set-predicates family confirm the collapse (Appendices D.4.2, D.4.5); rotation matching at d=256d=256 has all baselines at R2∈[−50,−62]R^2∈[-50,-62] while MZAttn stays at −0.005-0.005 (Table 4). On MST and convex hull, the raw-target negative-R2R^2 magnitude is partly an optimisation artefact of unnormalised regression on large-magnitude targets; under target normalisation the architectural advantage on MST shrinks to +0.04+0.04–0.080.08 R2R^2 and the hull d=9d=9 cell ties (Appendix D.2.4). The MEB and quadratic-TDOF separations are scale-free or O(1)O(1)-magnitude, so the gap there is structural rather than scale-driven. Table 4: Rotation matching scaling (R2R^2, mean over 3 seeds). Context-rigid baselines collapse to large negative R2R^2 as d grows; MZAttn tracks the mean predictor up to d=256d=256. At d=512d=512 all architectures diverge but MZAttn’s MSE stays 2.3×2.3× smaller than the next-best. Model d=16d=16 R2R^2 d=32d=32 R2R^2 d=64d=64 R2R^2 d=128d=128 R2R^2 d=256d=256 R2R^2 d=512d=512 R2R^2 Standard 0.084 −-0.002 −-1.20 −-21.54 −-61.69 −-165.4 ST-Large 0.285 −-0.001 −-0.023 −-9.47 −-50.15 −-145.1 Perceiver −-0.001 −-0.006 −-1.17 −-19.18 −-61.52 −-165.2 HyperNet 0.020 −-0.005 −-1.10 −-16.51 −-61.48 −-163.0 MZAttn 0.826 0.519 0.071 −-0.001 −-0.005 −-61.6 Real-world transfer. On a CIFAR-100 ResNet-feature setup that meets the three prerequisites (high-rank per-sample points from layer-3 features; sparse k=5k=5 k-center furthest-cost target; no imposed symmetry), MZAttn-Full ranks first across 7 parameter-matched baselines on both ResNet-18 (d=256d=256) and ResNet-50 (d=1024d=1024) backbones, with Welch’s t≈5.2t≈ 5.2, p≈0.0018p≈ 0.0018 at d=1024d=1024 (n=5n=5 seeds; absolute gap +0.0046+0.0046 R2R^2, uncorrected for the 18-cell TDOF analysis below). The advantage is restricted to the sparse-combinatorial k-center target: LinearAttn matches or beats MZAttn on dense-statistical siblings (diameter, k-N radius, MST) on real ResNet features (Appendix D.5, Tables 33 and 32). Aggregating across eighteen (task, scale) settings, MZAttn’s MSE reduction over the best parameter-matched baseline correlates with the estimated TDOF of each task’s target operator (Figure 2, right): Spearman ρ=0.74ρ=0.74 (n=18n=18, two-sided parametric p<0.005p<0.005). The TDOF estimate is approximate (exact for linear-form tasks, order-of-magnitude for non-linear ones), so we treat the correlation as descriptive. A ±50%± 50\% TDOF-perturbation sensitivity check over 10410^4 trials keeps Spearman in [0.675,0.791][0.675,0.791] at 55%–9595% (Appendix D.2.1). Tasks with TDOF ≤5≤ 5 cluster near zero improvement (range −2-2 to +19%+19\%); TDOF ≥6≥ 6 show median 58%58\% MSE reduction (range 1515 to 98%98\%). Figure 2: The architectural advantage tracks transformation degrees of freedom on tasks where TDOF grows with d. Left: on MEB radius prediction, parameter-matched attention baselines fail at R2≤0.025R^2≤ 0.025 across dimensions, while MZAttn reaches R2=0.69,0.63,0.38R^2=0.69,0.63,0.38 at d=8,16,32d=8,16,32. Right: across 1818 (task, scale) settings, MZAttn’s MSE reduction over the best baseline correlates with target TDOF (Spearman ρ=0.74ρ=0.74, two-sided parametric p<0.005p<0.005, n=18n=18). Low-TDOF tasks cluster near zero; high-TDOF tasks span 1515–98%98\% reduction with median 58%58\%. Architectural ablations, robustness studies, uncertainty calibration, scaling experiments, and the equivariant comparison are reported in Appendix D. 5.4 Analysis: What Does the Matrix Zonotope Learn? We address why the matrix-zonotope parameterisation works while equally-flexible alternatives do not. Capacity is not the driver. HyperNet at r=dk=64r=d_k=64 (1.31.3M params, 3.6×3.6× budget) leaves R2≈0R^2≈ 0 on MEB (Table 3), and the full r∈16,32,64r∈\16,32,64\ sweep on Hull and MST at high d (Table 5) stays within 0.130.13 of r=16r=16 everywhere; conversely MZ-Slim (ng=2n_g=2, L=1L=1, 1.6×1.6× Standard’s parameters) still beats parameter-matched ST-Large and HyperNet variants by 44–2323p R2R^2 on Max regression, MST weight, and rotation matching (Table 6). On the synthetic quadratic-TDOF task (Table 8), even L=1L=1 retains R2=0.978R^2=0.978 and the curve plateaus once L exceeds target TDOF: the gain is the centre-plus-gated-generator structure, not the generator count. Table 5: HyperNet rank extended ablation (R2R^2, mean± , 3 seeds). Low-rank update r∈16,32,64r∈\16,32,64\ on collapse-exhibiting tasks. ‡Hull d=9d=9 MZ-Full (0.0030.003) is within seed std of zero. Target-normalised equivalents: Table 18. Task (d) HyperNet r=16r=16 r=32r=32 r=64r=64 MZAttn-Full Hull d=3d=3 0.191±.008 0.190±.029 0.204±.020 0.804±.014 Hull d=6d=6 −-0.214±.013 −-0.203±.002 −-0.214±.027 0.103±.020 Hull d=9d=9 −-0.095±.001 −-0.098±<.001±<.001 −-0.100±.002 0.003±.006‡ MEB d=8d=8 0.012±.006 0.022±.006 0.011±.012 0.691±.012 MEB d=16d=16 0.007±.003 0.000±.008 0.001±.009 0.629±.023 MEB d=32d=32 −-0.020±.029 −-0.000±<.001±<.001 0.000±.002 0.378±.030 MST d=64d=64 −-1.96±.31 −-2.06±.28 −-1.92±.18 0.58±.01 MST d=128d=128 −-3.09±.29 −-3.00±.17 −-2.80±.17 0.48±.02 MST d=256d=256 −-3.99±.15 −-3.78±.05 −-3.97±.17 0.38±.04 Table 6: MZ parameter efficiency (R2R^2, mean, 3 seeds). Even at ng=2,L=1n_g=2,L=1 MZ retains its high-TDOF advantage: structure dominates parameter count. Model Max R2R^2 ↑ RotMatch R2R^2 ↑ MST R2R^2 ↑ Params Standard 0.812 0.706 0.719 208K MZ-Slim (ng=2,L=1n_g=2,L=1) 0.871 0.919 0.835 333K MZ-Med (ng=4,L=2n_g=4,L=2) 0.863 0.933 0.857 344K MZ-Full (ng=8,L=4n_g=8,L=4) 0.850 0.936 0.852 368K A trained L=8L=8 checkpoint probed with 1,0001,000 contexts shows near-orthogonal generators in operator space, gates active on 78.6%78.6\% of evaluations, and the soft effective rank of M(μi)−M¯i\M( _i)- M\_i saturating the architectural budget when L≤TDOFL and growing sublinearly above (Appendix D.1, Figure 4); the trained model allocates operator-space variation up to but not beyond the intrinsic TDOF, with L a soft upper bound. 6 Discussion and Conclusion MZAttn replaces the fixed value projection of standard attention with a learnable matrix zonotope at the same parameter-budget scaling. The gain is selective and tracks the theory (Figure 2): improvements concentrate on tasks with high-TDOF, sparse-combinatorial targets, and disappear on tasks dominated by per-element features, spatial locality, or known symmetry. Equivariance versus context-adaptivity. EGNN [33] matches or outperforms MZAttn on every rotation-invariant task tested: rotation matching at d=8d=8 (R2=0.99R^2=0.99 vs. 0.950.95), MEB radius (R2=0.78R^2=0.78 vs. 0.380.38 at d=32d=32), and Mahalanobis distance (R2=0.40R^2=0.40 vs. −0.02-0.02). On the quadratic-TDOF target (genuine fixed-frame dependence), EGNN’s invariant-output constraint forces R2≈0R^2≈ 0 while MZAttn reaches R2≥0.90R^2≥ 0.90 (Table 7). The two are complementary: EGNN when symmetry is known, MZAttn when it is unknown or absent. Table 7: MZAttn vs. EGNN. EGNN dominates on rotation/translation-invariant targets; MZAttn dominates on the non-equivariant quadratic-TDOF target where invariance is structurally inadequate. Rot. matching Hull volume MEB radius Statistical (Cov, Mah, PCA) Quadratic TDOF d=8d=8 d=32d=32 d=64d=64 d=3d=3 d=9d=9 d=8d=8 d=32d=32 CN d=8 Mah d=8 PCA d=16 L=4L=4 L=8L=8 L=16L=16 ST-Large 0.92 0.00 −-0.02 0.50 −-0.09 0.02 −-0.01 0.90 −-0.004 0.11 0.83 0.79 0.78 MZAttn (ours) 0.95 0.52 0.07 0.80 0.003 0.69 0.38 0.97 −-0.022 0.58 0.96 0.90 0.90 EGNN 0.99 0.99 0.98 0.95 0.85 0.66 0.78 0.99 0.40 0.93 −-0.002 0.000 0.006 Real-world transfer. On convex-hull-volume across three pretrained backbones (ResNet-50, DINOv2, CLIP) with five parameter-matched context-rigid baselines, 1414 of 1515 cells yield negative R2R^2 (Appendix D.5), while MZAttn-Full maintains mean R2≥0.82R^2≥ 0.82 on all three backbones (MZAttn-Large reaches 0.998±0.0050.998± 0.005 on RN50). On dense-statistical siblings (diameter, k-N radius, MST) all architectures cluster within ≤0.03≤ 0.03 R2R^2, the predicted TDOF behaviour. On SST-2 sentiment [38] (low-TDOF), MZAttn (0.7982±0.01140.7982± 0.0114) matches a parameter-matched Standard Set Transformer (0.7989±0.00840.7989± 0.0084) within 0.070.07p (33 seeds), the strict-generalisation property a value-projection replacement should satisfy. Scope and limitations. The advantage is structurally targeted, not universal: synthetic high-TDOF benchmarks serve as the controlled testbed; the 1818-setting Spearman ρ=0.74ρ=0.74 (Section 5) is the falsifiability check; QM9 and point-cloud rows are the predicted low-TDOF negative controls (within ≤0.04≤ 0.04 R2R^2). MZAttn costs 1.71.7–1.9×1.9× parameters and 2.42.4–2.7×2.7× inference vs. Standard ST (Appendix D.5.5); the Slim variant retains most of the advantage at 1.6×1.6× params. Deep Sets tying on few-shot retrieval (Task D) and ST-Large winning the parameter-fair 7-way meta-regression at L=32L=32 (Table 27; an earlier comparison against an undertuned vanilla baseline overstated this gap, corrected here) instantiate the predicted boundary. Very-large-scale sets and structured-zonotope variants remain open. References [1] A. Alanwar, A. Koch, F. Allgöwer, and K. H. Johansson (2021) Data-driven reachability analysis using matrix zonotopes. In Learning for Dynamics and Control, Vol. 144, p. 163–175. Cited by: Appendix C, §2.1, §4, Definition 1. [2] A. Alanwar, A. Koch, F. Allgöwer, and K. H. Johansson (2023) Data-driven reachability analysis from noisy data. IEEE Transactions on Automatic Control 68 (5), p. 3054–3069. External Links: Document Cited by: Appendix C, §D.5.1, §4. [3] M. Althoff (2010) Reachability Analysis and its Application to the Safety Assessment of Autonomous Cars. Ph.D. Thesis, Technische Universität München. Cited by: §B.1, §B.4.5, §B.4.5, Definition 1. [4] C. B. Barber, D. P. Dobkin, and H. Huhdanpaa (1996) The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software 22 (4), p. 469–483. External Links: Document Cited by: §D.4.2. [5] P. L. Bartlett, D. J. Foster, and M. J. Telgarsky (2017) Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §B.2, §3.3. [6] G. Bonaert, D. I. Dimitrov, M. Baader, and M. Vechev (2021) Fast and precise certification of transformers. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, p. 466–481. External Links: Document Cited by: Appendix C, §4. [7] L. K. Chung and S. Kousik (2025) Provably-safe neural network training using hybrid zonotope reachability analysis. In 2025 IEEE 64th Conference on Decision and Control (CDC), External Links: Link, 2501.13023 Cited by: Appendix C, §4. [8] R. A. DeVore (1998) Nonlinear approximation. Acta Numerica 7, p. 51–150. External Links: Document Cited by: §B.4.2. [9] Y. Gal and Z. Ghahramani (2016) Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In International Conference on Machine Learning, p. 1050–1059. Cited by: Appendix C. [10] J. Gasteiger, J. Groß, and S. Günnemann (2020) Directional message passing for molecular graphs. In International Conference on Learning Representations, Cited by: §D.5.2, Table 28, Table 28. [11] A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. In Conference on Language Modeling, Cited by: §4. [12] D. Ha, A. Dai, and Q. V. Le (2017) HyperNetworks. In International Conference on Learning Representations, Cited by: Appendix C, §4. [13] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira (2021) Perceiver: general perception with iterative attention. In Proceedings of the 38th International Conference on Machine Learning, p. 4651–4664. Cited by: §5. [14] X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool (2016) Dynamic filter networks. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: Appendix C, §4. [15] M. Jordan, J. Hayase, A. G. Dimakis, and S. Oh (2022) Zonotope domains for lagrangian neural network verification. In Advances in Neural Information Processing Systems, Vol. 35. Cited by: Appendix C, §4. [16] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret (2020) Transformers are RNNs: fast autoregressive transformers with linear attention. In International Conference on Machine Learning, Vol. 119, p. 5156–5165. Cited by: Table 23, §1, Table 3, Table 3. [17] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh (2019) Attentive neural processes. In International Conference on Learning Representations, Cited by: §D.5.6, §1. [18] N. Kochdumper and M. Althoff (2023) Constrained polynomial zonotopes. Acta Informatica 60 (3), p. 279–316. External Links: Document Cited by: §B.1, Appendix C, §4. [19] W. Kühn (1998) Rigorously computed orbits of dynamical systems without the wrapping effect. Computing 61 (1), p. 47–67. External Links: Document Cited by: Definition 1. [20] B. Lakshminarayanan, A. Pritzel, and C. Blundell (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: Appendix C. [21] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh (2019) Set transformer: a framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, p. 3744–3753. Cited by: Appendix C, §1, §2.1, §2.4, §4, §5, Corollary 6. [22] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf (2020) Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, Vol. 33, p. 11525–11538. Cited by: §D.5.3, §5. [23] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: §A.1, §5.1. [24] M. Mirman, T. Gehr, and M. Vechev (2018) Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, Vol. 80, p. 3578–3586. Cited by: Appendix C, §4. [25] M. Mohri, A. Rostamizadeh, and A. Talwalkar (2018) Foundations of machine learning. 2 edition, MIT Press, Cambridge, MA. External Links: ISBN 9780262039406 Cited by: §B.2. [26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §5.1. [27] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville (2018) FiLM: visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, Cited by: Appendix C, §4, §5. [28] C. R. Qi, H. Su, K. Mo, and L. J. Guibas (2017) PointNet: deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 652–660. Cited by: Appendix C, §1, §4. [29] C. R. Qi, L. Yi, H. Su, and L. J. Guibas (2017) PointNet++: deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: Appendix C. [30] Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huang, D. Liu, J. Zhou, and J. Lin (2025) Gated attention for large language models: non-linearity, sparsity, and attention-sink-free. In Advances in Neural Information Processing Systems, Note: Best Paper Award Cited by: §4. [31] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld (2014) Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data 1 (1), p. 140022. External Links: Document Cited by: §A.2, §D.5.2, §5.1. [32] RDKit Contributors (2024) RDKit: open-source cheminformatics software. Note: https://w.rdkit.org Cited by: §D.5.2. [33] V. G. Satorras, E. Hoogeboom, and M. Welling (2021) E(n) equivariant graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, p. 9323–9332. Cited by: Appendix C, §D.5.4, §1, §3.2, §6. [34] S. Schug, S. Kobayashi, Y. Akram, J. Sacramento, and R. Pascanu (2025) Attention as a hypernetwork. In International Conference on Learning Representations, Cited by: Appendix C, §4. [35] K. T. Schüt, P. Kindermans, H. E. Sauceda, S. Chmiela, A. Tkatchenko, and K. Müller (2017) SchNet: a continuous-filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: Appendix C, §D.5.2, Table 28, Table 28. [36] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, Cited by: Appendix C, §4. [37] J. Snell, K. Swersky, and R. Zemel (2017) Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §5.1. [38] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, p. 1631–1642. Cited by: §6. [39] Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei (2023) Retentive network: a successor to transformer for large language models. arXiv preprint arXiv:2307.08621. Cited by: Table 23, §1, §4. [40] M. A. Uy, Q. Pham, B. Hua, D. T. Nguyen, and S. Yeung (2019) Revisiting point cloud classification: a new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1588–1597. Cited by: §A.2, §5.1. [41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §1. [42] E. Welzl (1991) Smallest enclosing disks (balls and ellipsoids). In New Results and New Trends in Computer Science (Lecture Notes in Computer Science vol. 555), p. 359–370. External Links: Document Cited by: §D.4.1. [43] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao (2015) 3D ShapeNets: a deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 1912–1920. Cited by: §A.2, §5.1. [44] S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim (2024) Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 56501–56523. Cited by: Table 23, §1, §4. [45] Y. Yang and A. Barron (1999) Information-theoretic determination of minimax rates of convergence. Annals of Statistics 27 (5), p. 1564–1599. External Links: Document Cited by: §B.4.2. [46] D. Yarotsky (2017) Error bounds for approximations with deep ReLU networks. Neural Networks 94, p. 103–114. External Links: Document Cited by: §B.3, §B.4.1, §B.4.4. [47] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola (2017) Deep sets. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: Appendix C, §4, §5. Supplementary Material: Overview The supplementary material is organised as follows. Appendix A reports experimental setup, hyperparameters, and compute resources. Appendix B contains the algebraic preliminaries, uncertainty and generalisation results, and full proofs for all theorems and propositions. Appendix C expands the related-work discussion. Appendix D reports the extended empirical study, organised into five thematic blocks: architectural ablations, robustness and generalisation, uncertainty calibration, computational-geometry scaling, and real-world transfer plus equivariant comparison. Appendix A Experimental Setup, Hyperparameters, and Compute Appendix A.1 gives the model and optimisation choices common to all experiments; Appendix A.2 specifies the per-task data-generation and training settings; Appendix A.3 reports compute resources. A.1 Implementation Details Hyperparameter summary. Headline configuration shared across all attention-based architectures except where overridden per task or by the parameter-matched variants: Optimiser AdamW [23] LR 10−410^-4 LR schedule cosine to 0 LR warmup 55 epochs (linear 0→10−40→ 10^-4) Weight decay 10−510^-5 Gradient clip 1.01.0 (ℓ2 _2 norm) Batch size 128128 (synthetic) / 256256 (geom. tasks) Epochs (cap) 200200 (synthetic), 150150 (real-world) Early stopping val loss patience 2020 Dropout 0.10.1 (constant) dmodeld_model 6464 (default) / 8080 (ST-Large) H heads 44 dffd_f 128128 (default) / 200200 (ST-Large) ISAB inducing points 1616 MZAttn ngn_g 88 (default) / 22 (Slim) / 1616 (Large) MZAttn L 44 (default) / 11 (Slim) / 88 (Large) The ResNet-50 (d=1024d=1024) experiments use early-stopping patience 300300 (Appendix D.4.7 verifies the rank ordering is stable for any patience ≥100≥ 100). MZ parameter initialization. Center matrices Mc(h)M_c^(h) are initialized as IdkI_d_k. Generator matrices Ml(h)M_l^(h) are initialized from (0,0.022)N(0,0.02^2). Mixing weights WmixW_mix are initialized from (0,0.12)N(0,0.1^2). All other linear layers use Xavier uniform initialization. Generator FFN. The generator path uses a lighter FFN with hidden dimension dff/2d_f/2 (compared to dffd_f for centers), since generators represent small perturbation directions and require less modeling capacity. Computational cost comparison. For the default configuration (d=64d=64, H=4H=4, ng=8n_g=8, L=4L=4), the MZ-Set Transformer has approximately 372372K parameters compared to 208208K for the standard Set Transformer. The additional parameters come primarily from the generator projection layer (WgW_g: din×ngd_in× n_gd), generator FFNs, and the MZ parameters (H×(1+L)×dk2=4×5×256=5,120H×(1+L)× d_k^2=4× 5× 256=5,120 matrix-zonotope parameters per attention layer, plus a small O(LHdk)O(LHd_k) for the gating network and the mixing matrix WmixW_mix). A.2 Per-Task Data and Training Settings Task A: Set regression. Given a set S=x1,…,xnS=\x_1,…,x_n\ with xi∈ℝ16x_i ^16 and n∼Uniform(10,100)n (10,100), predict y=∑i[maxdimxi]y= _i[ _dimx_i], i.e., the sum of dimension-wise maxima. 10K train, 2K val, 2K test; 200 epochs; 5 seeds (3 seeds for parameter-matched models). Task B: Set matching. Given two point clouds A,B⊂ℝ3A,B ^3 (each 10–30 points), predict the Chamfer distance CD(A,B)CD(A,B). Set B is either a rotated-noisy version of A (50%) or an independent random cloud (50%). Both sets are packed into a single input with a binary type indicator feature, giving din=4d_in=4. 5K train, 1K val, 1K test; 200 epochs; 3 seeds. Task C: Point cloud classification (ModelNet40). We classify 3D shapes from ModelNet40 [43], which comprises 40 categories with 9,843 training and 2,468 test instances. Each shape is represented by 1,024 randomly sampled surface points in ℝ3R^3. Training uses random y-axis rotation and jittering for augmentation; 200 epochs; 5 seeds. Task D: Few-shot set retrieval. Given a support set S of K=5K=5 examples from a target class and a query set Q of 20 elements from mixed classes, predict the fraction of Q belonging to the target class. Each element has 8 feature dimensions plus a binary type indicator, yielding din=9d_in=9. Class separability varies per sample (noise scale ∈[0.3,1.5]∈[0.3,1.5]), creating natural relational ambiguity. 8K train, 1.5K val, 1.5K test; 200 epochs; 3 seeds. Task E: ScanObjectNN classification (real-world). We evaluate on ScanObjectNN [40], a realistic 3D point cloud classification benchmark constructed from real indoor scans. We use the OBJ_BG variant (15 classes, ∼ 2,300 train, ∼ 600 test). Each shape is represented by 1,024 randomly sampled points in ℝ3R^3 with y-axis rotation and jitter augmentation. 200 epochs; 5 seeds. Task F: Rotation-dependent set matching. Given two point clouds A,B⊂ℝ8A,B ^8 where B=R(θ)A+εB=R(θ)A+ for a sample-dependent rotation R(θ)R(θ) (composed from Givens rotations in all (82)=288 2=28 planes) and i.i.d. noise ε∼(0,0.12I) (0,0.1^2I), predict the post-alignment Chamfer distance. This task requires the model to internally discover and apply a context-dependent rotation to align A and B before computing the residual distance, an inherently off-diagonal operation. FiLM conditioning, which applies γ(μ)⊙z+β(μ)γ(μ) z+β(μ) (diagonal scale and shift), provably cannot express rotation in ℝ8R^8. Both sets are packed with a binary type indicator, giving din=9d_in=9. 5K train, 1K val, 2K test; 200 epochs; 3 seeds. Task G: Minimum spanning tree weight in ℝ8R^8. Given a point set S=x1,…,xn⊂ℝ8S=\x_1,…,x_n\ ^8 with n∼Uniform(10,30)n (10,30), predict the total weight (sum of edge lengths) of the minimum spanning tree of S. This is a classical computational geometry quantity that depends on the global pairwise distance structure of the set, a natural relational task where the output is determined by the collective geometric arrangement rather than any single element. Points are generated with 1–3 random clusters (varying centers and spreads) to create diverse geometric configurations. No explicit context signal is broadcast; the model must infer the relational structure from pairwise distances in high-dimensional space. 5K train, 1K val, 2K test; 200 epochs; 3 seeds. Task H: QM9 molecular property prediction. Given a molecule represented as a set of atoms S=a1,…,anS=\a_1,…,a_n\ with n≤29n≤ 29 and ai∈ℝ9a_i ^9 (5-dim atom type one-hot encoding for H/C/N/O/F, 3D Cartesian coordinates, partial charge), predict quantum chemical properties: dipole moment (μ, Debye) and isotropic polarizability (α, Bohr3). Data is drawn from the QM9 dataset [31], which contains ∼ 134K small organic molecules with DFT-computed properties. We use 50K molecules split 80/10/10; 100 epochs; 3 seeds. A.3 Compute Resources All experiments were run on a mix of consumer and datacentre GPUs. Synthetic benchmarks (Tasks A, B, F, G, MEB, hull, MST, rotation matching at d≤64d≤ 64, sparse-geometric set predicates, gen-/HyperNet-/MZ-Slim ablations, uncertainty calibration, set-size generalisation, ambiguity sweep, quadratic-TDOF / depth separation / sequence MZ) used a single NVIDIA RTX 5090 (32 GB). Higher-dimensional rotation matching (d∈128,256,512d∈\128,256,512\) and MST scaling (d∈64,128,256d∈\64,128,256\) used a single NVIDIA H200 (141 GB). ResNet-50 layer-3 transfer at d=1024d=1024 used a single NVIDIA H200. Per-seed training time on the 5090 ranged from 0.50.5 to 3030 minutes depending on task and dimension. The full empirical sweep (all main-body tables and appendix ablations combined) accumulates approximately 1,2001,200 GPU-hours when re-run from scratch with default seeds. Appendix B Theory: Algebra, Structural Properties, and Proofs This section consolidates the theoretical material that was previously fragmented across four short appendices (zonotope algebra, uncertainty certificates, additional theoretical results, and full proofs). It collects the algebraic preliminaries used by the proofs in Section 3, the structural-property statements summarised in the body, and the proofs of every theorem and proposition cited above. B.1 Zonotope and Matrix-Zonotope Algebra For readers unfamiliar with zonotopes, we provide a self-contained primer. A zonotope =c+Gξ:‖ξ‖∞≤1Z=\c+Gξ:\|ξ\|_∞≤ 1\ can be visualized as a centrally symmetric polytope generated by “sweeping” the center along each generator direction. In 2D, a zonotope with 3 generators is a hexagon; with 4 generators, an octagon. The interval hull IH()=[c−δ,c+δ]IH(Z)=[c-δ,c+δ] with δi=∑k|Gik| _i= _k|G_ik| is the tightest axis-aligned box containing Z. Key operations: • Linear map: A=Ac+AGξ=⟨Ac,AG⟩AZ=\Ac+AGξ\= Ac,AG . Zonotopes are closed under linear maps. • Minkowski sum: 1⊕2=⟨c1+c2,[G1,G2]⟩Z_1 _2= c_1+c_2,[G_1,G_2] . Generator count grows additively. • MZ multiplication: ℳMZ produces a zonotope with center MccM_cc and generators Mcgk∪Mlc∪Mlgk\M_cg_k\∪\M_lc\∪\M_lg_k\. Generator count grows as h+L+Lh+L+Lh (hence the need for budget control). For a thorough treatment including constrained and polynomial variants, see Kochdumper and Althoff [18] and Althoff [3]. B.2 Uncertainty Certificates, Generalization, and Structural Properties This section contains the full theorem statements deferred from Section 3.3. Theorem 3 (Directional vs. scalar sensitivity). Consider a single MZ-attention layer with fixed attention scores. Under value perturbation c^~=c^+G^ξ c= c+ Gξ for ξ∈[−1,1]ngξ∈[-1,1]^n_g: (a) A standard attention layer with output projection WoW_o provides only a scalar bound: ‖Δo‖≤‖Wo‖2⋅‖G^‖1,2\| o\|≤\|W_o\|_2·\| G\|_1,2. (b) MZ-attention provides a directional zonotope certificate: Δo∈Δ=McG^ξ:‖ξ‖∞≤1 o _ =\M_c Gξ:\|ξ\|_∞≤ 1\, contained in the ball of radius ‖Mc‖2‖G^‖1,2\|M_c\|_2\,\| G\|_1,2. (c) For matched output-projection norms (‖Wo‖2=‖Mc‖2\|W_o\|_2=\|M_c\|_2), the MZ zonotope is contained in the scalar ball, Δ⊆B(‖Mc‖2‖G^‖1,2)Z_ B(\|M_c\|_2\,\| G\|_1,2), and is strictly tighter than the ball along every direction orthogonal to the column span of McG^M_c G whenever that column span is not the full ambient space. The qualitative content of part (c) is that MZ replaces a uniform-radius ball with an axis-aligned-in-input-space zonotope; this distinguishes MZAttn from generic context-adaptive methods, which provide only a scalar reachable diameter, by specifying the change per direction. Theorem 4 (IHW as prediction uncertainty bound). Let IHW(Zout)=∑g=1ng‖Gg‖1IHW(Z_out)= _g=1^n_g\|G_g\|_1 denote the interval hull width of the output zonotope. Then: (a) The reachable output diameter satisfies diam((S))≤2‖Wout‖2⋅IHWdiam(Y(S))≤ 2\,\|W_out\|_2·IHW. (b) The output variance under uniform ξ satisfies Varξ[Wout⊤(cout+Goutξ)]≤13‖Wout‖22⋅IHW2Var_ξ[W_out (c_out+G_outξ)]≤ 13\|W_out\|_2^2·IHW^2. (c) Higher IHW implies a strictly wider reachable output range (monotonicity). Theorem 5 (Norm-controlled generalization). For an MZ-Set Transformer with D encoder layers, i.i.d.i.i.d. training data of size n, and margin γ>0γ>0, the generalization gap is bounded by O~(BMZ/γn) O(B_MZ/γ n), where BMZ=∏ℓ=1D(‖Mc(ℓ)‖2+∑l‖Ml(ℓ)‖2)⋅‖WV(ℓ)‖2⋅(∑ℓRℓ2/3)3/2B_MZ= _ =1^D(\|M_c^( )\|_2+ _l\|M_l^( )\|_2)·\|W_V^( )\|_2·( _ R_ ^2/3)^3/2. At initialisation, when Mc=IM_c=I and Ml≈0M_l≈ 0, BMZB_MZ matches standard attention; it grows only as the model learns to depart from context-rigid behaviour. Proof sketch. The result follows from the spectrally-normalised Rademacher complexity bound of Bartlett et al. [5] with one substitution. The post-aggregation map of an MZ-attention layer is v^↦M(μ(S))v v M(μ(S)) v with worst-case spectral norm ‖M(μ(S))‖2≤‖Mc‖2+∑l|γl(μ)|‖Ml‖2≤‖Mc‖2+∑l‖Ml‖2\|M(μ(S))\|_2≤\|M_c\|_2+ _l| _l(μ)|\,\|M_l\|_2≤\|M_c\|_2+ _l\|M_l\|_2, since |γl(μ)|≤1| _l(μ)|≤ 1. This replaces the per-layer spectral factor ‖W(ℓ)‖2\|W^( )\|_2 in the standard bound by the MZ-layer worst-case spectral norm. The aggregation step ∑jαijvj _j _ijv_j is a softmax-weighted convex combination, hence a 11-Lipschitz contraction in v v that does not inflate the Rademacher complexity by the Talagrand contraction principle [25, Lemma 5.7]. Composing across the D MZ-encoder layers and including the value projections WV(ℓ)W_V^( ) yields the stated BMZB_MZ; the residual (∑ℓRℓ2/3)3/2( _ R_ ^2/3)^3/2 factor and the post-hoc-margin transfer are identical to the original argument. ∎ This bound has a natural interpretation: generalisation is controlled by how much context-adaptivity the model learns to use, giving an interpretable complexity measure that connects to the relational-capacity framework. Corollary 6 (Universality is preserved). The MZ-Set Transformer family contains the standard Set Transformer family as the special case Ml(ℓ)=0M_l^( )=0, Wmix(ℓ)=0W_mix^( )=0, Mc(ℓ)=Wo(ℓ)M_c^( )=W_o^( ). Consequently, the universal-approximation property of Lee et al. [21, Theorem 2] carries over: for any compact ⊂ℝdK ^d, continuous permutation-invariant f:≤N→ℝmf:\;K^≤ N ^m, and ϵ>0ε>0, an MZ-Set Transformer ϵε-approximates f uniformly. The MZ family is strictly larger (Theorem 8), so universality is preserved without loss. Proposition 7 (Structural properties). MZ-SAB and MZ-ISAB are permutation-equivariant; the full MZ-Set Transformer is permutation-invariant. At initialisation, with Mc=IM_c=I, Ml≈0M_l≈ 0, and Wmix≈0W_mix≈ 0, MZAttn reduces exactly to standard multi-head attention with an identity output projection. B.3 Additional Theoretical Results This section contains the expressiveness results and supporting lemmas deferred from Section 3. Theorem 8 (Strict expressiveness separation). For any nonzero A∈ℝd×dA ^d× d, define fA(S)=(I+tanh(μ(S)1)⋅A)μ(S)f_A(S)=(I+ (μ(S)_1)· A)\,μ(S). Then: (a) a single MZAttn-attention head with L=1L=1 followed by a fixed linear readout (over the pooled center and the first signed pooled generator) computes fAf_A exactly via the explicit closed-form construction below; (b) no standard multi-head attention layer (any H, without FFN, with any output linear projection) can represent fAf_A on the family of constant inputs Stt∈ℝ\S_t\_t . Proof. (a) We construct each parameter explicitly. Set WQ=0W_Q=0 so all softmax weights are uniform αij=1/n _ij=1/n, giving the aggregated center c^i=μ(S) c_i=μ(S) for every query i. Set WV=IdW_V=I_d so the value zonotope’s center equals the input. Set Mc(h)=IdM_c^(h)=I_d, L=1L=1, M1(h)=AM_1^(h)=A, the gate weight wγ=e1∈ℝdw_γ=e_1 ^d so γ1(μ)=tanh(e1⊤μ)=tanh(μ1) _1(μ)= (e_1 μ)= ( _1), and the mixing weight Wmix=e1∈ℝng×1W_mix=e_1 ^n_g× 1 (only the first generator slot is used). Substituting into Eq. (3) gives center output oic=μ(S)o_i^c=μ(S) and generator output whose first row is oiG[1,:]=tanh(μ1)Aμ(S)o_i^G[1,:]= ( _1)\,A\,μ(S), with rows 2,…,ng2,…,n_g all zero. After PMA pooling with a single seed vector configured analogously (WQ=0W_Q=0, uniform aggregation), the pooled center is μ(S)μ(S) and the pooled signed first-generator is tanh(μ1)Aμ(S) ( _1)\,A\,μ(S). The fixed linear readout Wout:(c,g1)↦c+g1W_out:(c,g_1) c+g_1 then yields Wout(oc,oG[1,:])=μ(S)+tanh(μ1)Aμ(S)=(I+tanh(μ1)A)μ(S)=fA(S),W_out(o^c,o^G[1,:])=μ(S)+ ( _1)\,A\,μ(S)=(I+ ( _1)\,A)\,μ(S)=f_A(S), exactly. No nonlinear approximation is invoked: every operation in the construction is either a fixed linear map or the native MZ gate tanh(wγ⊤c¯kv) (w_γ c^kv), both of which compute their output without finite-width truncation. (b) On constant sets St=te1,…,te1S_t=\te_1,…,te_1\, softmax symmetry gives o(St)=(∑hWo(h)WV(h))te1=tWe1o(S_t)=( _hW_o^(h)W_V^(h))te_1=t\,We_1 (linear in t). But fA(St)=te1+ttanh(t)Ae1f_A(S_t)=te_1+t (t)\,Ae_1 is nonlinear in t whenever Ae1≠0Ae_1≠ 0, contradicting linearity. ∎ Corollary 9 (Depth separation without FFN). There exist continuous permutation-invariant functions computable by a single MZ-attention layer that require at least two standard attention layers with intermediate nonlinearity. Theorem 10 (Width efficiency over standard attention with FFN). For ℱL=S↦(M0+∑l=1Lσl(μ(S))Ml)μ(S)F_L=\S (M_0+ _l=1^L _l(μ(S))M_l)\,μ(S)\: (a) MZ-attention represents any f∈ℱLf _L exactly with (L+1)dk2+O(Ldk)(L+1)d_k^2+O(Ld_k) parameters; (b) standard attention with depth-2 ReLU FFN requires dff=Ω(Ldk/ϵ)d_f= (Ld_k/ ε) for ϵε-approximation. Proof sketch. (a) Direct. (b) Each bilinear term σl⋅(Mlh)j _l·(M_lh)_j requires Ω(1/ϵ) (1/ ε) ReLU neurons [46]; summing over L terms and dkd_k coordinates gives the bound. ∎ Theorem 11 (Minimax parameter efficiency). Let ℱL,BF_L,B denote the class ℱLF_L with bounded generator matrices ‖Ml‖F≤B\|M_l\|_F≤ B and 1-Lipschitz gates. (a) MZ-attention parameterizes ℱL,BF_L,B exactly using PMZ=(L+1)dk2+O(Ldk)=Θ(Ldk2)P_MZ=(L+1)d_k^2+O(Ld_k)= (Ld_k^2) parameters; (b) any model class ϵε-approximating ℱL,BF_L,B in sup-norm over inputs of radius R requires P≥Ω(Ldk2log(BR/ϵ))P≥ (Ld_k^2 (BR/ε)) parameters; (c) hence the MZ parameterization is order-optimal up to logarithmic factors. Definition 4 (Transformation degrees of freedom). For a permutation-invariant function f(S)=T(S)μ(S)f(S)=T(S)\,μ(S) with set-dependent linear operator T(S)∈ℝd×dT(S) ^d× d, the transformation degrees of freedom is TDOF(f)=dim(spanT(S)−T(S′):S,S′∈∗),TDOF(f)= (span\T(S)-T(S ):S,S ^*\ ), i.e. the dimension of the subspace of ℝd×dR^d× d spanned by all possible differences of T across inputs. TDOF of natural set functions. Proposition 12 (TDOF of natural set functions). Let S⊂ℝdS ^d with |S|≥d+1|S|≥ d+1 in general position, sample mean μ(S)μ(S), and regularised covariance Σ^(S)=1|S|∑i(xi−μ)(xi−μ)⊤+λI (S)= 1|S| _i(x_i-μ)(x_i-μ) +λ I for λ>0λ>0. (a) Mahalanobis (T(S)=Σ^(S)−1T(S)= (S)^-1) has TDOF=d(d+1)/2TDOF=d(d+1)/2; (b) Whitening (T(S)=Σ^(S)−1/2T(S)= (S)^-1/2) has TDOF=d(d+1)/2TDOF=d(d+1)/2; (c) PCA projection onto the top k principal components has TDOF=kd−k(k+1)/2TDOF=kd-k(k+1)/2; (d) Per-coordinate z-score (T(S)=diag(σj−1)T(S)=diag( _j^-1)) has TDOF=dTDOF=d. Proof. (a)–(b). As S varies over generic configurations of |S|≥d+1|S|≥ d+1 points in ℝdR^d, the regularised covariance Σ^(S) (S) ranges over all of Symd+Sym^+_d (the cone of d×d× d positive-definite matrices), an open subset of the symmetric matrices SymdSym_d. The maps Σ↦Σ−1 ^-1 and Σ↦Σ−1/2 ^-1/2 are smooth diffeomorphisms on Symd+Sym^+_d with non-singular Jacobian (e.g. ∂Σ−1/∂Σ⋅H=−Σ−1HΣ−1∂ ^-1/∂ · H=- ^-1H ^-1 is full-rank as a linear map Symd→SymdSym_d _d), so T(S)T(S) ranges over an open subset of SymdSym_d. Hence spanT(S)−T(S′)span\T(S)-T(S )\ contains a basis of SymdSym_d, giving TDOF=dimSymd=d(d+1)/2TDOF= _d=d(d+1)/2. (c). The top-k PCA projector is T(S)=Vk(S)Vk(S)⊤T(S)=V_k(S)V_k(S) where Vk∈ℝd×kV_k ^d× k has orthonormal columns. As S varies, VkV_k ranges over the Stiefel manifold St(k,d)St(k,d) of dimension kd−k(k+1)/2kd-k(k+1)/2, so the smooth orbit VkVk⊤\V_kV_k \ is the Grassmannian Gr(k,d)Gr(k,d) of the same dimension, and the linear span of differences fills the tangent space at any point, which is also of dimension kd−k(k+1)/2kd-k(k+1)/2. (d). T(S)=diag(1/σ1,…,1/σd)T(S)=diag(1/ _1,…,1/ _d) where σj(S) _j(S) varies independently over (λ1/2,∞)(λ^1/2,∞) as S ranges over configurations whose marginal variances can be set freely; the differences span the d-dimensional space of diagonal matrices. ∎ ℱLF_L as first-order approximation of smooth operators. The following result shows that ℱLF_L is not an artificial class tailored to MZ-attention, but rather the natural linearization class for any smooth context-adaptive operator. Proposition 13 (ℱLF_L captures linearized context-adaptive operators). Let T:ℝd→ℝm×dT:\;R^d ^m× d be twice continuously differentiable, and define f(S)=T(μ(S))μ(S)f(S)=T(μ(S))\,μ(S) where μ(S)=1|S|∑x∈Sxμ(S)= 1|S| _x∈ Sx. Let J=∂vec(T)∂μ|μ0∈ℝmd×dJ= ∂\,vec(T)∂μ |_ _0 ^md× d be the Jacobian at a reference point μ0 _0, with SVD J=∑l=1rslulvl⊤J= _l=1^rs_l\,u_lv_l where r=rank(J)r=rank(J). Define the linearized operator T~(μ)=T(μ0)+∑l=1rsl(vl⊤(μ−μ0))mat(ul), T(μ)=T( _0)+ _l=1^rs_l\,(v_l (μ- _0))\,mat(u_l), where mat:ℝmd→ℝm×dmat:\;R^md ^m× d reshapes vectors into matrices, and the corresponding function f~(S)=T~(μ(S))μ(S) f(S)= T(μ(S))\,μ(S). Then: (a) f~∈ℱr,B f _r,B with B=s1B=s_1 (the largest singular value of J), center matrix M0=T(μ0)M_0=T( _0), generator matrices Ml=slmat(ul)M_l=s_l\,mat(u_l), and gating functions σl(μ)=vl⊤(μ−μ0) _l(μ)=v_l (μ- _0), which are linear and hence 1-Lipschitz after rescaling by 1/‖vl‖1/\|v_l\|. (b) The approximation error satisfies ‖f(S)−f~(S)‖≤12CT‖μ(S)−μ0‖2⋅‖μ(S)‖,\|f(S)- f(S)\|≤ 12\,C_T\,\|μ(S)- _0\|^2·\|μ(S)\|, where CT=supμ‖∇2vec(T)(μ)‖opC_T= _μ\|∇^2vec(T)(μ)\|_op is bounded on any compact domain. (c) Consequently, L=rank(J)L=rank(J) MZ generators suffice to capture the first-order context-adaptive behavior of T around μ0 _0, and Theorem 11 guarantees this representation is parameter-optimal. Proof. Part (a) is immediate from the SVD construction: each component σl(μ)⋅Ml _l(μ)· M_l is a rank-1 term in the Jacobian decomposition of T, and the gating functions are linear projections of μ. Part (b) follows from Taylor’s theorem applied to the vector-valued map vec(T)vec(T): ‖vec(T(μ))−vec(T(μ0))−J(μ−μ0)‖≤12CT‖μ−μ0‖2.\|vec(T(μ))-vec(T( _0))-J(μ- _0)\|≤ 12C_T\|μ- _0\|^2. Reshaping and multiplying by μ(S)μ(S): ‖f(S)−f~(S)‖=‖(T(μ)−T~(μ))μ‖≤‖T(μ)−T~(μ)‖F⋅‖μ‖≤12CT‖μ−μ0‖2‖μ‖\|f(S)- f(S)\|=\|(T(μ)- T(μ))\,μ\|≤\|T(μ)- T(μ)\|_F·\|μ\|≤ 12C_T\|μ- _0\|^2\|μ\|. Part (c) combines (a) with Theorem 11(a): the Θ(rdk2) (rd_k^2) MZ parameters are both sufficient and necessary (up to log factors) for ϵε-approximation of the linearized operator family. ∎ Remark 1 (Interpretation). Proposition 13 establishes that ℱLF_L is to context-adaptive operators what linear models are to nonlinear regression: the first-order approximation class. The Jacobian rank r determines the intrinsic dimensionality of how the operator varies locally, and MZ-attention with L=rL=r generators is the minimax-optimal parameterization for this class. A standard attention mechanism, which has zero relational capacity by Theorem 1(a), corresponds to the zeroth-order approximation T≡T(μ0)T≡ T( _0), discarding all first-order variation. TDOF of soft Chamfer matching. Proposition 14 (Soft Chamfer matching has nontrivial TDOF). Consider the soft nearest-neighbor operator nβ(a,B)=∑j=1mwj(a,B)bjn_β(a,B)= _j=1^mw_j(a,B)\,b_j with softmax weights wj=softmaxj′(−β‖a−bj′‖2)w_j=softmax_j (-β\|a-b_j \|^2), and define the residual operator Ta(B)=Id−Pa(B)T_a(B)=I_d-P_a(B) where Pa(B)a=nβ(a,B)P_a(B)\,a=n_β(a,B). Then: (a) The operator Ta(B)T_a(B) is context-adaptive: for any m≥2m≥ 2, there exist configurations B,B′B,B such that Ta(B)≠Ta(B′)T_a(B)≠ T_a(B ). (b) TDOF(Ta)≥dTDOF(T_a)≥ d when m≥2m≥ 2 and β>0β>0: as B varies over (ℝd)m(R^d)^m, the operator Ta(B)T_a(B) spans a subspace of ℝd×dR^d× d of dimension at least d. Proof. (a) Immediate: different B configurations change the matching weights wjw_j, hence Pa(B)P_a(B). (b) Fix a∈ℝda ^d and consider B=0,tekB=\0,t\,e_k\ for each standard basis vector eke_k, k=1,…,dk=1,…,d. The soft nearest-neighbor is nβ(a,B)=w2(t,k)tekn_β(a,B)=w_2(t,k)\,t\,e_k where w2=(1+exp(−β(‖a‖2−‖a−tek‖2)))−1w_2=(1+ (-β(\|a\|^2-\|a-te_k\|^2)))^-1. The Jacobian ∂nβ/∂t|t=0 _β/∂ t |_t=0 has a nonzero component in the eke_k direction. Since this holds independently for each k, the operator Pa(B)P_a(B) varies in at least d independent directions as B ranges over (ℝd)m(R^d)^m. By Definition 4, TDOF(Ta)≥dTDOF(T_a)≥ d. ∎ Remark 2. For the full Chamfer distance CD(A,B)=1n∑i‖ai−n(ai,B)‖2CD(A,B)= 1n _i\|a_i-n(a_i,B)\|^2, each query point aia_i contributes d independent relational directions. In the hard-matching limit β→∞β→∞, distinct matchings correspond to combinatorially many regions in configuration space, each defining a different piecewise-linear operator, well beyond the reach of a single context-rigid transformation. This explains the large gap between MZAttn and standard attention on set matching (Task B in Table 2): the Chamfer distance target requires a context-adaptive operator, and the standard Set Transformer’s fixed WOW_O cannot represent the matching-dependent structure. Lemma 15 (Compositional orthogonality under the worst-case construction). Fix k≤d−1k≤ d-1 and the explicit witness f∈ℱkf _k defined by σl(μ):=tanh(μl) _l(μ):= ( _l), Ml:=eled⊤∈ℝd×dM_l:=e_l\,e_d ^d× d for l=1,…,kl=1,…,k, and M0:=0M_0:=0 (the constant-direction term is set to zero so that the network’s linear path provides no free target-direction contribution; this is without loss of generality for the lower-bound argument). Let μ∈ℝdμ ^d have i.i.d. standard-Gaussian coordinates. Then: (a) ⟨Ml,Ml′⟩F=δll′ M_l,M_l _F= _l , so Mll=1k\M_l\_l=1^k is Frobenius-orthonormal. (b) For every r≥2r≥ 2 and every l1,…,lr∈1,…,kl_1,…,l_r∈\1,…,k\, Ml1Ml2⋯Mlr=0M_l_1M_l_2·s M_l_r=0. (c) Consequently, in the tensor-product space L2(μ)⊗ℝd×dL^2(μ) ^d× d with inner product ⟨a⊗A,b⊗B⟩:=μ[ab]⟨A,B⟩F a A,b B :=E_μ[ab]\, A,B _F, every monomial composition term of the form P(σ)⊗(Ml1⋯Mlr)P(σ) (M_l_1·s M_l_r) with composition order r≥2r≥ 2 vanishes identically (the matrix factor is zero), and therefore has zero projection on the target subspace =spanσl⊗Mll=1kV=span\ _l M_l\_l=1^k. Proof. (a) ⟨eled⊤,el′ed⊤⟩F=(el⊤el′)(ed⊤ed)=δll′⋅1=δll′ e_le_d ,e_l e_d _F=(e_l e_l )(e_d e_d)= _l · 1= _l . (b) For any product of length r≥2r≥ 2, Ml1Ml2=(el1ed⊤)(el2ed⊤)=el1(ed⊤el2)ed⊤M_l_1M_l_2=(e_l_1e_d )(e_l_2e_d )=e_l_1\,(e_d e_l_2)\,e_d . Since l2∈1,…,k⊆1,…,d−1l_2∈\1,…,k\ \1,…,d-1\ and d≠l2d≠ l_2, the inner factor ed⊤el2=0e_d e_l_2=0, so Ml1Ml2=0M_l_1M_l_2=0. Any longer product factors through this length-22 block and is therefore also 0. (c) For any monomial P(σ)P(σ) in σl\ _l\ and any composition order r≥2r≥ 2, the matrix factor Ml1⋯Mlr=0M_l_1·s M_l_r=0 by (b). Hence P(σ)⊗(Ml1⋯Mlr)=0P(σ) (M_l_1·s M_l_r)=0 in the tensor-product space, and its projection on V (or on any subspace) is 0. ∎ Remark 3. The construction Ml=eled⊤M_l=e_le_d is the simplest worst-case witness: each MlM_l is a rank-11 outer product whose left factor ele_l varies with l but whose right factor ede_d is a single fixed direction not used as a target index. This makes the matrix family closed under no products of order ≥2≥ 2 (every such product collapses to 0), so a D-layer attention network cannot bootstrap target directions via composition. Other Frobenius-orthonormal Ml\M_l\ admit non-zero matrix products and would require a separate analysis; we use the simplest construction for which the depth lower bound is unconditional. Generator mixing characterization. Proposition 16 (Generator mixing). For mixed generators g~k=gk+∑lwklgl′ g_k=g_k+ _lw_klg _l: (a) Z~⊆Z Z Z if ∑k|wkl|≤1 _k|w_kl|≤ 1 for all l; (b) Z⊆Z~Z Z if ∑k|wkl|≥1 _k|w_kl|≥ 1; (c) equality iff ∑k|wkl|=1 _k|w_kl|=1. Initializing mixing weights with ℓ1 _1 column norms close to 1 yields near-exact generator representation. Score perturbation bound. Proposition 17 (Score perturbation). For input perturbation ‖δj‖≤ρ\| _j\|≤ρ, the total output decomposes as o~−o=McWVG^ξ+ℰ o-o=M_cW_V Gξ+E with ‖ℰ‖≤Cρmaxj‖vj‖\|E\|≤ Cρ _j\|v_j\|, where C=‖Mc‖2‖WQs‖2‖WK‖2/dkC=\|M_c\|_2\|W_Qs\|_2\|W_K\|_2/ d_k. B.4 Full Proofs B.4.1 Proof of Theorem 1 (Relational Capacity Bottleneck) Proof. Part (a). An H-head standard attention layer without FFN computes, for query q (e.g., a PMA seed): o(S)=∑h=1HWo(h)v^(h),v^(h)=∑jαj(h)(S)WV(h)xj.o(S)= _h=1^HW_o^(h) v^(h), v^(h)= _j _j^(h)(S)\,W_V^(h)x_j. The aggregated value v^(h) v^(h) depends on S through the scores αj(h) _j^(h), but the post-aggregation transformation Wo(h)W_o^(h) is a fixed matrix. The effective output transformation is: o(S)=∑hWo(h)WV(h)(∑jαj(h)(S)xj).o(S)= _hW_o^(h)W_V^(h) ( _j _j^(h)(S)\,x_j ). On constant sets St=tei,…,teiS_t=\te_i,…,te_i\, softmax symmetry forces αj(h)=1/n _j^(h)=1/n for all j,hj,h, giving o(St)=(∑hWo(h)WV(h))teio(S_t)=( _hW_o^(h)W_V^(h))\,te_i. The effective transformation Tstd=∑hWo(h)WV(h)T_std= _hW_o^(h)W_V^(h) is independent of StS_t, hence relational capacity is zero. Part (b). After attention, the FFN receives h=x+v^h=x+ v and must produce output ∑l=1kσl(h)⋅Mlh _l=1^k _l(h)· M_lh to represent a function with relational complexity k. Each term σl(h)⋅(Mlh)j _l(h)·(M_lh)_j is the product of a bounded nonlinear scalar and a linear form. By a standard region-counting argument (whose matching upper-bound construction is given by Yarotsky [46], Proposition 3), a depth-2 ReLU network approximating f(a,b)=abf(a,b)=ab on a compact domain to accuracy ϵε requires Ω(1/ϵ) (1/ ε) neurons; the detailed argument is given in the proof of Theorem 2 (Part (b), Appendix). For each relational direction l, the FFN must implement dkd_k such products (one per output coordinate), requiring Ω(dk) (d_k) neurons per direction. With dffd_f total hidden units, at most ⌊dff/dk⌋ d_f/d_k independent relational directions can be represented. Part (c). Direct construction: set Mc=M0M_c=M_0, use L generator matrices M1,…,MLM_1,…,M_L, and gates γl=σl(μ(S)) _l= _l(μ(S)). Then M(μ(S))=M0+∑l=1Lσl(μ(S))Ml=T(S)M(μ(S))=M_0+ _l=1^L _l(μ(S))M_l=T(S), which exactly represents any f∈ℱLf _L. Parameter count: (L+1)(L+1) matrices of size dk×dkd_k× d_k plus L gate weight vectors of size dkd_k. Part (d). Let f(S)=(M0+∑l=1kσl(μ(S))Ml)μ(S)f(S)=(M_0+ _l=1^k _l(μ(S))M_l)\,μ(S) with Mll=1k\M_l\_l=1^k orthogonal in Frobenius norm (⟨Ml,Ml′⟩F=0 M_l,M_l _F=0 for l≠l′l≠ l ). Any mechanism with relational capacity R can represent an output transformation of the form Tθ(S)=M0′+∑l=1Rτl(S)NlT_θ(S)=M_0 + _l=1^R _l(S)N_l for some learned Nll=1R\N_l\_l=1^R and scalar functions τl _l. The optimal R-capacity approximation chooses M0′=M0M_0 =M_0 and Nll=1R\N_l\_l=1^R to be the projection of Mll=1k\M_l\_l=1^k onto the best R-dimensional subspace (in the Frobenius metric, weighted by [σl2‖μ‖2]E[ _l^2\|μ\|^2]). By the Eckart–Young–Mirsky theorem applied to the operator-valued approximation, the residual error is: S[‖f(S)−oθ(S)‖2]≥∑l=R+1kS[σl(μ(S))2⋅‖Mlμ(S)‖2],E_S [\|f(S)-o_θ(S)\|^2 ]≥ _l=R+1^kE_S [ _l(μ(S))^2·\|M_lμ(S)\|^2 ], where we order the terms by [σl2‖Mlμ‖2]E[ _l^2\|M_lμ\|^2] in decreasing order and the mechanism captures the top R. The bound is tight: it is achieved when Nl\N_l\ aligns with the top-R relational directions. ∎ B.4.2 Proof of Theorem 11 (Minimax Parameter Efficiency) Proof. Part (a). The MZ-attention head has parameters Mc∈ℝdk×dkM_c ^d_k× d_k, M1,…,ML∈ℝdk×dkM_1,…,M_L ^d_k× d_k, and gate weights w1,…,wL∈ℝdkw_1,…,w_L ^d_k, totalling (L+1)dk2+Ldk=Θ(Ldk2)(L+1)d_k^2+Ld_k= (Ld_k^2). By Theorem 10(a), this realises every f∈ℱL,Bf _L,B exactly. Part (b). We use a metric entropy argument. The function class ℱL,BF_L,B is parameterized by M0,M1,…,ML∈ℝdk×dkM_0,M_1,…,M_L ^d_k× d_k with ‖Ml‖F≤B\|M_l\|_F≤ B, and Lipschitz gates σl _l. We lower-bound the packing number by considering the subclass with fixed M0=IM_0=I, fixed σl=id _l=id (identity, clipped to [−1,1][-1,1]), and varying M1,…,MLM_1,…,M_L. For two functions f,f′f,f with generator matrices Ml\M_l\ and Ml′\M_l \: sup‖x‖≤R|f(Sx)−f′(Sx)|≥sup‖x‖≤R‖∑lσl(x)(Ml−Ml′)x‖. _\|x\|≤ R|f(S_x)-f (S_x)|≥ _\|x\|≤ R\| _l _l(x)(M_l-M_l )x\|. For the specific input Sx=x,…,xS_x=\x,…,x\ with μ=xμ=x, and choosing x=Rej/‖ej‖x=Re_j/\|e_j\| for the coordinate maximizing ‖(Ml−Ml′)ej‖\|(M_l-M_l )e_j\|: ‖f−f′‖∞≥c⋅R⋅maxl‖Ml−Ml′‖F\|f-f \|_∞≥ c· R· _l\|M_l-M_l \|_F for a universal constant c>0c>0 depending only on the dimension. The δ-packing number of M∈ℝdk×dk:‖M‖F≤B\M ^d_k× d_k:\|M\|_F≤ B\ in Frobenius norm is at least (B/δ)dk2(B/δ)^d_k^2 (volume comparison in ℝdk2R^d_k^2). With L independent generator matrices, the packing number of ℱL,BF_L,B at resolution ϵε is at least: ℳ(ϵ,ℱL,B)≥(cBR/ϵ)Ldk2.M(ε,F_L,B)≥ (cBR/ε )^Ld_k^2. Any model class parameterized by θ∈ℝPθ ^P that ϵε-approximates all of ℱL,BF_L,B must distinguish at least ℳ(ϵ)M(ε) functions. By a standard argument [45, 8], the parameter space ℝPR^P can cover at most exp(CPlog(1/ϵ)) (CP (1/ε)) ϵε-cells (where C depends on the model’s Lipschitz properties in θ). Matching: Plog(1/ϵ)≥Ω(Ldk2log(BR/ϵ))P (1/ε)≥ (Ld_k^2 (BR/ε)), giving P≥Ω(Ldk2log(BR/ϵ)/log(1/ϵ))P≥ (Ld_k^2 (BR/ε)/ (1/ε)). Since log(BR/ϵ)/log(1/ϵ)≥1 (BR/ε)/ (1/ε)≥ 1 for BR≥1BR≥ 1, we obtain P≥Ω(Ldk2)P≥ (Ld_k^2). Including the logarithmic precision factor: P≥Ω(Ldk2log(BR/ϵ))P≥ (Ld_k^2 (BR/ε)). Part (c). MZ-attention uses PMZ=(L+1)dk2+Ldk=Θ(Ldk2)P_MZ=(L+1)d_k^2+Ld_k= (Ld_k^2) parameters and achieves ϵ=0ε=0 (exact representation). The lower bound requires P≥Ω(Ldk2)P≥ (Ld_k^2) for any constant ϵε. Hence MZ-attention is order-optimal up to logarithmic factors. ∎ B.4.3 Proof of Theorem 8 (Expressiveness Separation) We restate and prove both parts in full detail; the construction matches the inline proof in Section B.3 and uses the signed-generator readout variant of Section 2.4. Proof of part (a). We configure a single MZ-attention head as follows. Set the query projection WQ=0∈ℝdk×dW_Q=0 ^d_k× d, so αij=softmaxj(0)=1/n _ij= *softmax_j(0)=1/n for all i,ji,j, yielding uniform aggregation c^i=1n∑jWVcj c_i= 1n _jW_Vc_j. With WV=IdW_V=I_d, c^i=μ(S) c_i=μ(S). Set Mc=IdkM_c=I_d_k, L=1L=1, M1=AM_1=A, gate weight wγ=e1w_γ=e_1, and mixing matrix Wmix=e1∈ℝng×1W_mix=e_1 ^n_g× 1 (use only the first generator slot). The context-dependent gate is γ1(μ)=tanh(e1⊤μ)=tanh(μ1) _1(μ)= (e_1 μ)= ( _1). By Eq. (3), the per-token output has center oc=Mcc^=μ(S)o^c=M_c c=μ(S) and generator stream whose first row is oG[1,:]=tanh(μ1)Aμ(S)o^G[1,:]= ( _1)\,A\,μ(S) with the remaining rows zero. After PMA pooling (uniform aggregation by symmetry), the pooled center is μ(S)μ(S) and the pooled signed first-generator is tanh(μ1)Aμ(S) ( _1)\,A\,μ(S). Applying the signed-generator readout Wout:(c,g1)↦c+g1W_out:(c,g_1) c+g_1 yields μ(S)+tanh(μ1)Aμ(S)=(I+tanh(μ1)A)μ(S)=fA(S)μ(S)+ ( _1)\,A\,μ(S)=(I+ ( _1)\,A)\,μ(S)=f_A(S), exactly. Every operation in the construction is either a fixed linear map or the native MZ gate, both computed without finite-width truncation; no UAT-based ReLU approximation is invoked. ∎ Proof of part (b). An H-head standard attention layer without FFN computes, for a query q (e.g., a PMA seed vector s): o(S)=Woconcat(WV(1)v^(1),…,WV(H)v^(H))o(S)=W_oconcat\! (W_V^(1) v^(1),\,…,\,W_V^(H) v^(H) ) where v^(h)=∑jαj(h)(S)xj v^(h)= _j _j^(h)(S)\,x_j and αj(h)=softmaxj(s(h)⊤WQ(h)⊤WK(h)xj/dk) _j^(h)= *softmax_j(s^(h) W_Q^(h) W_K^(h)x_j/ d_k). Step 1. Consider the subfamily of constant sets St=tei,…,teiS_t=\te_i,…,te_i\ (n copies of teite_i) for a basis vector eie_i and scalar t∈ℝt . By symmetry of the softmax with identical inputs, αj(h)=1/n _j^(h)=1/n for all j and h. Hence v^(h)=WV(h)(tei)=tWV(h)ei v^(h)=W_V^(h)(te_i)=tW_V^(h)e_i, and: o(St)=Woconcat(tWV(1)ei,…,tWV(H)ei)=t⋅Woconcat(WV(1)ei,…,WV(H)ei)=:t⋅w~i.o(S_t)=W_oconcat(tW_V^(1)e_i,\,…,\,tW_V^(H)e_i)=t· W_oconcat(W_V^(1)e_i,\,…,\,W_V^(H)e_i)=:t· w_i. This is a linear function of t. Step 2. On the same inputs, fA(St)=(I+tanh(tδi1)A)(tei)=tei+ttanh(tδi1)Aeif_A(S_t)=(I+ (t _i1)A)(te_i)=te_i+t (t _i1)Ae_i, where δi1=[ei]1 _i1=[e_i]_1. For i=1i=1: fA(St)=te1+ttanh(t)Ae1f_A(S_t)=te_1+t (t)Ae_1. If Ae1≠0Ae_1≠ 0, the map t↦te1+ttanh(t)Ae1t te_1+t (t)Ae_1 is nonlinear (since ttanh(t)t (t) is nonlinear and Ae1≠0Ae_1≠ 0 ensures this nonlinearity survives in at least one coordinate). A linear function t↦tw~1t t w_1 cannot agree with this nonlinear function on any open interval. Step 3. If Ae1=0Ae_1=0, since A≠0A≠ 0 there exists eje_j with Aej≠0Ae_j≠ 0. If j=1j=1, we already have a contradiction. If j≠1j≠ 1, use St=tej+ϵe1,…,tej+ϵe1S_t=\te_j+ε e_1,…,te_j+ε e_1\ for small ϵ>0ε>0: μ(St)=tej+ϵe1μ(S_t)=te_j+ε e_1, so tanh(μ1)=tanh(ϵ)≠0 ( _1)= (ε)≠ 0, and Aμ=tAej+ϵAe1Aμ=tAe_j+ε Ae_1. Since Aej≠0Ae_j≠ 0, the nonlinearity persists. ∎ B.4.4 Proof of Theorem 10 (Width Efficiency) Proof. Part (a) is immediate: the MZ parameters Mc∈ℝdk×dkM_c ^d_k× d_k, M1,…,ML∈ℝdk×dkM_1,…,M_L ^d_k× d_k, and gate weights w1,…,wL∈ℝdkw_1,…,w_L ^d_k total (L+1)dk2+Ldk(L+1)d_k^2+Ld_k. For part (b), consider a single bilinear term σl(h)⋅(Mlh)j _l(h)·(M_lh)_j that the FFN must compute, with a=σl(h)∈[−1,1]a= _l(h)∈[-1,1] and b=(Mlh)j∈[−Bl,Bl]b=(M_lh)_j∈[-B_l,B_l], Bl=‖Ml‖2RB_l=\|M_l\|_2R. The FFN must approximate f(a,b)=abf(a,b)=ab on this rectangle to sup-norm error ϵε. By a standard region-counting argument for shallow ReLU approximation of strictly-convex bilinear functions (the matching upper-bound construction is Yarotsky [46], Proposition 3), any depth-2 ReLU network approximating abab on [−1,1]×[−Bl,Bl][-1,1]×[-B_l,B_l] to sup-norm error ϵε requires n=Ω(Bl/ϵ)n= (B_l/ ε) hidden units. The standard underlying argument bounds a depth-2 ReLU network’s piecewise-linear regions by O(n2)O(n^2) on ℝ2R^2 and applies the strict convexity of abab (∂2f/∂a∂b=1∂^2f/∂ a∂ b=1): on a region of diameter r the best affine approximant errs by Ω(r2) (r^2), and an O(n2)O(n^2)-region partition of a [−Bl,Bl]×[−1,1][-B_l,B_l]×[-1,1] rectangle has at least one region of diameter Ω(Bl/n) (B_l/n), giving worst-case error Ω(Bl2/n2) (B_l^2/n^2) and hence n=Ω(Bl/ϵ)n= (B_l/ ε). Summing over L generators and dkd_k output coordinates: dff≥∑ldk⋅Ω(Bl/ϵ)=Ω(LdkB/ϵ)d_f≥ _ld_k· (B_l/ ε)= (Ld_kB/ ε) where B=maxlBlB= _lB_l. For unit-norm inputs (B=O(1)B=O(1)): dff=Ω(Ldk/ϵ)d_f= (Ld_k/ ε). ∎ B.4.5 Proof of Theorem 3 (Directional Sensitivity) Proof. (a) Standard attention output: o=Wov^o=W_o v with v^=∑jαijvj v= _j _ijv_j. Under perturbation v^~=v^+G^ξ v= v+ Gξ: ‖Δo‖=‖WoG^ξ‖≤‖Wo‖2‖G^ξ‖≤‖Wo‖2∑k‖G^k‖⋅|ξk|≤‖Wo‖2‖G^‖1,2\| o\|=\|W_o Gξ\|≤\|W_o\|_2\| Gξ\|≤\|W_o\|_2 _k\| G_k\|·| _k|≤\|W_o\|_2\| G\|_1,2. This is a scalar (norm) bound: all directions in ℝdkR^d_k are treated equally. (b) MZ-attention output: Δo=McG^ξ o=M_c Gξ. As ξ ranges over [−1,1]ng[-1,1]^n_g, Δo o traces the zonotope Δ=McG^ξ:‖ξ‖∞≤1Z_ =\M_c Gξ:\|ξ\|_∞≤ 1\ with generators McG^kk=1ng\M_c G_k\_k=1^n_g. The vertex count 2(ngdk−1)2 n_gd_k-1 is a standard result from zonotope geometry [3]. (c) Every point in ΔZ_ satisfies ‖z‖≤∑k‖McG^k‖2≤‖Mc‖2‖G^‖1,2\|z\|≤ _k\|M_c G_k\|_2≤\|M_c\|_2\| G\|_1,2, so Δ⊆B(‖Mc‖2‖G^‖1,2)Z_ B(\|M_c\|_2\| G\|_1,2). For directions u∈ℝdku ^d_k orthogonal to the column span spanMcG^kk=1ngspan\M_c G_k\_k=1^n_g, the support function hΔ(u)=∑k|u⊤McG^k|=0<‖Mc‖2‖G^‖1,2=hB(u)h_Z_ (u)= _k|u M_c G_k|=0<\|M_c\|_2\| G\|_1,2=h_B(u), so the MZ certificate is strictly tighter than the ball whenever the column span is a proper subspace. The volume of a zonotope with generators g1,…,gng∈ℝdkg_1,…,g_n_g ^d_k is [3] vol(Δ)=2dk∑|I|=dk|det([gi]i∈I)|,vol(Z_ )=2^d_k _|I|=d_k| ([g_i]_i∈ I)|, which is strictly smaller than vol(B)vol(B) for dk≥2d_k≥ 2 since the zonotope is a proper inscribed subset of the ball; for typical trained generators with unequal norms and non-orthogonal directions, vol(Δ)vol(Z_ ) is empirically several orders of magnitude smaller than vol(B)vol(B), reflecting the anisotropic structure of the model’s sensitivity. ∎ B.4.6 Proof of Proposition 16 (Generator Mixing) Proof. Let Zng+LZ_n_g+L have generators g1,…,gng,g1′,…,gL′\g_1,…,g_n_g,g _1,…,g _L\ in general position. A point in Zng+LZ_n_g+L is p=c+∑k=1nggkζk+∑l=1Lgl′ζl′p=c+ _k=1^n_gg_k _k+ _l=1^Lg _lζ _l with |ζk|,|ζl′|≤1| _k|,|ζ _l|≤ 1. A point in Z~ng Z_n_g is p~=c+∑k=1ng(gk+∑lwklgl′)ζ~k p=c+ _k=1^n_g(g_k+ _lw_klg _l) ζ_k with |ζ~k|≤1| ζ_k|≤ 1. Expanding: p~=c+∑kgkζ~k+∑l(∑kwklζ~k)gl′ p=c+ _kg_k ζ_k+ _l( _kw_kl ζ_k)g _l. (a) Z~ng⊆Zng+L Z_n_g Z_n_g+L: Set ζk=ζ~k _k= ζ_k and ζl′=∑kwklζ~kζ _l= _kw_kl ζ_k. We need |ζl′|=|∑kwklζ~k|≤∑k|wkl|⋅|ζ~k|≤∑k|wkl||ζ _l|=| _kw_kl ζ_k|≤ _k|w_kl|·| ζ_k|≤ _k|w_kl|. If ∑k|wkl|≤1 _k|w_kl|≤ 1 for all l, then |ζl′|≤1|ζ _l|≤ 1 and p~∈Zng+L p∈ Z_n_g+L. (b) Zng+L⊆Z~ngZ_n_g+L Z_n_g: Given a point p∈Zng+Lp∈ Z_n_g+L with parameters (ζ,ζ′)(ζ,ζ ), we seek ζ~ ζ with |ζ~k|≤1| ζ_k|≤ 1 matching both gkg_k and gl′g _l coefficients. By general position, coefficient matching requires ζ~k=ζk ζ_k= _k (from gkg_k) and ∑kwklζk=ζl′ _kw_kl _k=ζ _l (from gl′g _l). For the latter to hold for all ζl′∈[−1,1]ζ _l∈[-1,1], the range of ∑kwklζk _kw_kl _k (over |ζk|≤1| _k|≤ 1) must contain [−1,1][-1,1]. This range is [−∑k|wkl|,∑k|wkl|][- _k|w_kl|, _k|w_kl|], so containment requires ∑k|wkl|≥1 _k|w_kl|≥ 1. (c) Combining (a) and (b): equality holds when ∑k|wkl|=1 _k|w_kl|=1 for all l. ∎ B.4.7 Proof of Theorem 4 (IHW Uncertainty Bound) Proof. Part (a). The output zonotope is Zout=cout+Goutξ:‖ξ‖∞≤1Z_out=\c_out+G_outξ:\|ξ\|_∞≤ 1\. Under a linear readout Wout∈ℝdkW_out ^d_k, the output is: y(ξ)=Wout⊤(cout+Goutξ)=Wout⊤cout+∑g=1ng(Wout⊤Gg)ξg.y(ξ)=W_out (c_out+G_outξ)=W_out c_out+ _g=1^n_g(W_out G_g) _g. This is a scalar zonotope (interval) with center Wout⊤coutW_out c_out and half-width ∑g|Wout⊤Gg| _g|W_out G_g|. Hence: diam()=2∑g=1ng|Wout⊤Gg|≤2∑g‖Wout‖2‖Gg‖2≤2‖Wout‖2∑g‖Gg‖1=2‖Wout‖2⋅IHW.diam(Y)=2 _g=1^n_g|W_out G_g|≤ 2 _g\|W_out\|_2\|G_g\|_2≤ 2\|W_out\|_2 _g\|G_g\|_1=2\|W_out\|_2·IHW. Part (b). Since ξ1,…,ξng _1,…, _n_g are i.i.d. uniform on [−1,1][-1,1]: [ξg]=0E[ _g]=0, [ξg2]=1/3E[ _g^2]=1/3, and [ξgξg′]=0E[ _g _g ]=0 for g≠g′g≠ g . The variance of y(ξ)=Wout⊤cout+∑g(Wout⊤Gg)ξgy(ξ)=W_out c_out+ _g(W_out G_g) _g is: Var[y]=∑g=1ng(Wout⊤Gg)2⋅Var[ξg]=13∑g(Wout⊤Gg)2.Var[y]= _g=1^n_g(W_out G_g)^2·Var[ _g]= 13 _g(W_out G_g)^2. By the Cauchy–Schwarz inequality applied coordinate-wise: (Wout⊤Gg)2≤‖Wout‖22‖Gg‖22≤‖Wout‖22‖Gg‖12(W_out G_g)^2≤\|W_out\|_2^2\|G_g\|_2^2≤\|W_out\|_2^2\|G_g\|_1^2. Hence ∑g(Wout⊤Gg)2≤‖Wout‖22∑g‖Gg‖12≤‖Wout‖22(∑g‖Gg‖1)2=‖Wout‖22⋅IHW2 _g(W_out G_g)^2≤\|W_out\|_2^2 _g\|G_g\|_1^2≤\|W_out\|_2^2( _g\|G_g\|_1)^2=\|W_out\|_2^2·IHW^2, where the last inequality uses ∑ai2≤(∑ai)2Σ a_i^2≤(Σ a_i)^2 for ai≥0a_i≥ 0. Part (c). From part (a), diam((S))=2‖Wout‖2⋅IHW(Zout(S))diam(Y(S))=2\|W_out\|_2·IHW(Z_out(S)). Since ‖Wout‖2\|W_out\|_2 is shared between inputs (it is a model parameter, not input-dependent), the monotonicity in IHW directly implies monotonicity in the reachable diameter. ∎ B.4.8 Proof of Theorem 2 (Depth Separation) Proof. We prove the depth lower bound on the input distribution where μ(S)∈ℝdμ(S) ^d has i.i.d. standard-Gaussian coordinates, using a compositional-orthogonality argument in L2(μ)L^2(μ) together with the explicit worst-case construction of Lemma 15. Setup. Let f(S)=(∑l=1kσl(μ(S))Ml)μ(S)f(S)=( _l=1^k _l(μ(S))\,M_l)\,μ(S) be the witness of Lemma 15: σl(μ):=tanh(μl) _l(μ):= ( _l), Ml:=eled⊤M_l:=e_le_d for l=1,…,kl=1,…,k with k≤d−1k≤ d-1, and M0=0M_0=0. Each MlM_l is rank-1 with Frobenius-orthonormal Mll=1k\M_l\_l=1^k (Lemma 15(a)). The choice M0=0M_0=0 ensures the network’s linear path μ↦M0μ M_0μ contributes no free target-direction component on V, so the depth lower bound is determined by the FFN multiplicative budget alone. We view f as an element of the tensor-product space L2(μ)⊗ℝd×dL^2(μ) ^d× d with inner product ⟨a⊗A,b⊗B⟩:=μ[ab]⟨A,B⟩F a A,b B :=E_μ[a\,b]\, A,B _F. The target directions σl⊗Mll=1k\ _l M_l\_l=1^k are mutually orthogonal in this inner product (independent μl _l’s make [σlσl′]=0E[ _l _l ]=0 for l≠l′l≠ l , and Frobenius orthonormality handles the matrix factor), so they span a k-dimensional subspace V. Layer-by-layer FFN bilinear count. A D-layer standard attention network with residual connections and depth-22 ReLU FFN of width dffd_f computes z(ℓ+1)=z(ℓ)+FFN(ℓ)(z(ℓ)+v^(ℓ)),z(0)=μ.z^( +1)=z^( )+FFN^( ) (z^( )+ v^( ) ), z^(0)=μ. A bilinear term in V has the form σl(μ)⊗Ml _l(μ) M_l, requiring the FFN to multiply a gate σl _l by the linear coordinate map μ↦Mlμ M_lμ. By Theorem 1(b), each multiplicative bilinear coordinate of this kind requires Ω(dk) (d_k) ReLU neurons (Yarotsky bilinear lower bound), so a single layer’s FFN of width dffd_f contributes at most ⌊dff/dk⌋ d_f/d_k new directions that lie in V. Cross-layer compositions vanish on the matrix side. After ℓ layers, the residual representation can additionally contain monomial composition terms of the form P(σ)⊗(Ml1⋯Mlr)P(σ) (M_l_1·s M_l_r) where r≥2r≥ 2 comes from successive FFNs reading earlier-layer bilinear outputs and re-multiplying them through linear maps. By Lemma 15(b)–(c), under the construction Ml=eled⊤M_l=e_le_d every such matrix product with r≥2r≥ 2 is identically zero, so the entire term vanishes in the tensor-product space and therefore has zero projection on V. Hence cross-layer composition cannot bootstrap any new target direction; only the layer-wise FFN bilinear count contributes. Counting. After D layers the network has captured at most ΛD≤D⋅⌊dff/dk⌋ _D≤ D· d_f/d_k of the k target directions in V. The remaining k−ΛDk- _D directions contribute an irreducible L2(μ)L^2(μ) error (by the Eckart–Young residual of Theorem 1(d)) bounded below by Ω((k−ΛD)/k)⋅Var(f) ((k- _D)/k)·Var(f). For exact representation we need ΛD=k _D=k, giving D≥kdk/dffD≥ k\,d_k/d_f. For approximate representation in L2(μ)L^2(μ) with residual at most O(Var(f)/L)O(Var(f)/L), achieving this residual requires k−ΛD=O(k/L)k- _D=O(k/L) uncaptured directions, hence ΛD≥k(1−O(1/L)) _D≥ k(1-O(1/L)), giving D≥Ω(kdk/dff)D≥ (k\,d_k/d_f) up to constants since each missing direction contributes Θ(Var(f)/k) (Var(f)/k). Single MZ-attention layer with signed-generator readout. Setting Mc=M0=0M_c=M_0=0 and using the k generators M1,…,MkM_1,…,M_k of f with gates γl(μ)=σl(μ) _l(μ)= _l(μ), Eq. (3) produces a per-token output whose center is 0 and whose generator stream contains γl(μ)Mlμ _l(μ)\,M_l\,μ as the l-th signed generator slot. After PMA pooling (uniform aggregation, as in the construction of Theorem 8), the signed-generator readout Wout:(c,g1,…,gk)↦∑l=1kglW_out:(c,g_1,…,g_k) _l=1^kg_l recovers f(S)=∑lσl(μ)Mlμf(S)= _l _l(μ)M_lμ exactly. The IHW readout (Section 2.4) gives only ∑l|σl(μ)|⋅‖Mlμ‖ _l| _l(μ)|·\|M_lμ\|-bounded scalar information rather than the signed combination, so it does not extract f(S)f(S) exactly; the expressiveness statement is for the signed-readout variant. ∎ B.4.9 Proof of Proposition 7 (Permutation Equivariance) Proof. Let π be a permutation of 1,…,n\1,…,n\ and denote the permuted set as π(S)=xπ(1),…,xπ(n)π(S)=\x_π(1),…,x_π(n)\. MZ-MAB equivariance: The attention scores satisfy αi,π(j)(π(S))=αij(S) _i,π(j)(π(S))= _ij(S) because they depend on pairwise dot products qi⊤kjq_i k_j which are permutation-equivariant. The aggregation c^i=∑jαijvj c_i= _j _ijv_j is thus equivariant: permuting the set permutes the query outputs. The MZ transform (Eq. 3) operates on each query independently. The gate γ=tanh(Wγc¯)γ= (W_γ c) depends on the set mean c¯=1n∑jcj c= 1n _jc_j, which is permutation-invariant. Hence MZ-MAB is equivariant: MZ-MAB(π(Q),π(KV))=π(MZ-MAB(Q,KV))MZ-MAB(π(Q),π(KV))=π(MZ-MAB(Q,KV)). MZ-SAB inherits equivariance since Q=KV=SQ=KV=S. Permutation invariance of the full model: MZ-PMA uses learnable seed vectors s1,…,sks_1,…,s_k as queries, which are independent of the input set. Thus MZ-PMA(π(S))=MZ-PMA(S)MZ-PMA(π(S))=MZ-PMA(S), and the output is permutation-invariant. ∎ Appendix C Extended Related Work Zonotopes as external verification tools vs. as internal representations. Zonotopes have a long history in formal verification and abstract interpretation of neural networks [6, 24, 15, 7], where they are computed post-hoc around a trained model’s activations to certify robustness, and in control for data-driven reachability [1, 2, 18]. To our knowledge MZAttn is the first architecture to make the matrix-zonotope an internal, trainable representation of the value-projection family: center and generator matrices are jointly optimised by gradient descent, and the zonotope structure participates in every forward pass rather than being wrapped around a frozen model. Concurrent and prior work has explored related ideas in adjacent spaces (e.g. structured low-rank reparameterisations and gated value updates), but we are not aware of an internal trainable matrix-zonotope value family with the gating, mixing, and budget-control structure of Eq. (3). Set-input networks. Deep Sets [47], PointNet [28], PointNet++ [29], and Set Transformer [21] established permutation invariance and attention-based pooling as standard tools for set inputs; PointNet++ in particular adds hierarchical local-region feature aggregation but still composes fixed per-point MLPs with set pooling, so its value transformation remains context-rigid in our sense. All of these architectures express relationships between elements via scalar scores (softmax weights or learned aggregations) that multiply fixed linear projections of inputs; MZAttn generalizes this to matrix-valued context-adaptive operators, which is precisely what high-TDOF tasks require. Context-adaptive linear maps. Hypernetworks [12, 34] and dynamic filter networks [14] generate full dk2d_k^2-parameter weight updates per sample, giving unlimited expressiveness but paying a quadratic cost that makes the effective rank hard to control. FiLM [27] restricts to diagonal (scale-and-shift) modulation, which is provably insufficient for off-diagonal structure such as rotations or covariance-dependent maps. Mixture-of-Experts [36] routes between a discrete set of fixed experts and is therefore piecewise-constant in context. The matrix-zonotope parameterization of MZAttn occupies a middle ground: O(Ldk2)O(Ld_k^2) parameters with L continuous gates realize a convex, rank-controlled family of linear operators that is minimax-optimal for context-adaptive linear maps (Theorem 11) while retaining the geometric and uncertainty structure of the zonotope family. Equivariant architectures. EGNN [33], SchNet [35], and similar E(n)- or SO(n)-equivariant architectures achieve very strong performance on rotation- and translation-invariant targets by hard-coding the symmetry into the forward pass. On such tasks MZAttn is at a structural disadvantage (it does not bake in the symmetry) and our experiments confirm this. The complementary regime is tasks whose target depends on a specific coordinate frame, e.g. the quadratic TDOF task of Section 3.2, where equivariance is actively harmful: invariant outputs cannot distinguish the target from its rotations. MZAttn is the right tool when the task’s symmetry is unknown a priori and cannot be built in by construction. Uncertainty quantification. MC-Dropout [9] and Deep Ensembles [20] obtain predictive uncertainty by multiple stochastic forward passes; MZAttn’s zonotope geometry (Theorem 4) yields a single-pass uncertainty estimate via the interval-hull width of the output zonotope, at 3.7×3.7× lower inference cost than the baselines (Appendix B.2, Table 20). Appendix D Additional Experiments This section reports the full battery of additional experiments referenced from the main body. It is organised into five thematic blocks: D.1 architectural ablations isolating which design choice drives the gain; D.2 robustness and generalisation studies; D.3 uncertainty calibration; D.4 high-dimensional and large-scale scaling on computational-geometry targets; and D.5 real-world transfer studies plus the equivariant baseline comparison. D.1 Architecture and ablation studies Studies that isolate which architectural choice in MZAttn (generator count, generator-info routing, low-rank update form, depth) drives the empirical advantage. They support the conclusion of Section 5.4: the centre-plus-gated-generator structure, not raw capacity, is what drives the gain. Figure 3: Effective value-transformation operator (head 0) on the quadratic-TDOF task. Standard attention uses one fixed WVW_V regardless of context (left); MZAttn produces a context-specific M(μ)=Mc+∑lγl(μ)MlM(μ)=M_c+ _l _l(μ)\,M_l for each input set (centre two panels). The context-dependent gates γl(μ) _l(μ) (right) drive the adaptation; the resulting Frobenius-norm change is ‖M(μA)−M(μB)‖F/‖Mc‖F=8.0%\|M( _A)-M( _B)\|_F/\|M_c\|_F=8.0\%. Per-sample operator adaptation is non-trivial. Visualising M(μA),M(μB)M( _A),M( _B) on two contexts at head 0 (Figure 3), the relative operator change is ‖M(μA)−M(μB)‖F/‖Mc‖F=8.0%\|M( _A)-M( _B)\|_F/\|M_c\|_F=8.0\% and the eight gates γl(μ) _l(μ) flip sign on five of eight generators between contexts, ruling out gate collapse to constants. Figure 4: Mechanistic anatomy of a trained MZAttn layer (L=8L=8, head 0, 1,0001,000 probe contexts), from left to right. Cosine similarity of vec(Mc)vec(M_c) and vec(Ml)vec(M_l): generator directions are near-orthogonal (mean off-diagonal |ρ|<0.01|ρ|<0.01). Per-generator Frobenius norms cluster in [0.146,0.169][0.146,0.169] (CV 5.1%5.1\%), each ∼1/22 1/22 of ‖Mc‖F\|M_c\|_F. Effective rank of M(μi)−M¯i\M( _i)- M\_i as architectural budget L varies on a fixed TDOF-88 task: saturates the budget when L≤TDOFL , grows sublinearly above. Gate magnitudes |γl(μ)|| _l(μ)| over L×1000L× 1000 evaluations: 78.6%78.6\% exceed 0.050.05, mean 0.130.13, p95 0.290.29. Trained generator basis + TDOF utilisation. Probing a trained L=8L=8 checkpoint with 1,0001,000 test contexts (Figure 4, cosine, Frobenius-norm, and gate panels): generators are near-orthogonal in operator space, comparably sized, and gates active on 78.6%78.6\% of evaluations — no dead generators or constant-bias collapse. Sweeping L∈4,8,16,32L∈\4,8,16,32\ at target TDOF =8=8, the soft effective rank of 200200 extracted operators saturates the budget when L≤TDOFL and grows sublinearly above (effective-rank panel). The trained model allocates operator-space variation up to but not beyond the intrinsic TDOF, with L a soft upper bound. D.1.1 MZ Generator Budget Table 8: Ablation on MZ generator matrices nmzn_mz (mean± , 3 seeds). Per-seed JSONs for all nmz∈1,2,4,8,16,32n_mz∈\1,2,4,8,16,32\ are in gen_ablation.json. L=16L=16 L=32L=32 nmzn_mz R2R^2 Params R2R^2 Params 1 0.978±0.0040.978± 0.004 247K 0.937±0.0140.937± 0.014 247K 2 0.976±0.0120.976± 0.012 252K 0.959±0.0190.959± 0.019 252K 4 0.956±0.0110.956± 0.011 260K 0.961±0.0070.961± 0.007 260K 8 0.960±0.0110.960± 0.011 278K 0.949±0.0210.949± 0.021 278K 16 0.948±0.0160.948± 0.016 313K 0.923±0.0110.923± 0.011 313K 32 0.943±0.0020.943± 0.002 383K 0.907±0.0100.907± 0.010 383K Even nmz=1n_mz=1 exceeds deep standard Set Transformers. Practical guidelines for L and ngn_g. The two main hyperparameters of MZAttn are the number of MZ generator matrices L (controlling relational capacity) and the zonotope generator budget ngn_g (controlling per-token uncertainty resolution). Based on the ablation studies in Tables 9 and 8, we recommend: • MZ generators L: Start with L=2L=2–44. Even L=1L=1 substantially outperforms standard attention on high-TDOF tasks (Table 8). Increasing L beyond 4 adds parameters (Ldk2Ld_k^2 per head per layer) with diminishing returns unless the task’s TDOF is known to be high. For tasks where the TDOF can be estimated (e.g., d for Chamfer matching, d(d+1)/2d(d+1)/2 for Mahalanobis-type operators), setting L to match the estimated TDOF is theoretically motivated by Theorem 11. • Zonotope generators ngn_g: Start with ng=4n_g=4–88. Table 9 shows that ng=4n_g=4 peaks for set matching; larger budgets overfit. Since ngn_g primarily controls the resolution of the uncertainty envelope rather than the relational capacity, moderate values suffice; the output head aggregates generators via interval hull width regardless of ngn_g. • Scaling regime: When computational budget is constrained, prioritize L over ngn_g: the MZ generators directly determine the model’s relational capacity (Theorem 1(c)), whereas ngn_g affects uncertainty granularity, which is less critical for prediction quality. D.1.2 Generator Ablation Table 9: Ablation on zonotope generators ngn_g (set matching, seed 42). ngn_g MSE ↓ MAE ↓ R2R^2 ↑ 2 45.80 3.50 0.967 4 26.74 3.33 0.981 8 32.46 3.39 0.977 16 55.05 4.47 0.961 Performance peaks at ng=4n_g=4 and degrades for larger budgets due to overfitting. D.1.3 Generator Info in Q/K Scoring Table 10: Ablation: incorporating generator information (IHW) into Q/K scoring. MZ-GenQK adds projections Wqgen,WkgenW_q^gen,W_k^gen of per-token interval hull widths into the attention score computation. Task A: Set Regr. Task B: Set Match. Model MSE ↓ R2R^2 ↑ MSE ↓ R2R^2 ↑ Params MZAttn (centers-only Q/K) 2.19± 0.09 0.879± 0.005 55.0± 10.5 0.961± 0.008 365K MZ-GenQK (gen info in Q/K) 1.64± 0.13 0.909± 0.007 68.8± 5.6 0.951± 0.004 415K Incorporating generator information into Q/K scoring yields task-dependent results: on Task A, MZ-GenQK improves R2R^2 from 0.8790.879 to 0.9090.909 (+0.030+0.030 absolute, +3.4%+3.4\% relative), but on Task B it degrades MSE from 55.055.0 to 68.868.8 (+25%+25\%). The mixed outcome, combined with the additional ∼50 50K parameters, indicates that centers-only scoring is a reasonable default. The benefit may arise when the prediction depends on generator structure (as in regression), while for matching tasks the center already encodes sufficient information for routing attention. D.1.4 HyperNet Rank Ablation Table 11: HyperNet rank ablation (mean ± std over 3 seeds). All models are parameter-matched at ∼ 360–370K. Increasing the low-rank update dimension does not close the gap to MZAttn; on Task B, performance degrades monotonically. Task A: Set Regr. Task B: Set Match. Model MSE ↓ R2R^2 ↑ MSE ↓ R2R^2 ↑ Params HyperNet (r=8r=8) 3.51± 0.01 0.806± 0.001 76.1± 5.7 0.945± 0.004 360K HyperNet (r=16r=16) 3.50± 0.02 0.807± 0.001 95.9± 8.0 0.931± 0.006 370K HyperNet (r=32r=32) 3.54± 0.02 0.805± 0.001 133.7± 8.0 0.904± 0.006 359K MZAttn 2.19± 0.09 0.879± 0.005 55.0± 10.5 0.961± 0.008 365K Increasing the HyperNet rank from 8 to 32 does not improve performance: Task A R2R^2 stays flat at ∼0.806 0.806, while Task B MSE increases from 76.1 to 133.7, a 76%76\% degradation. This indicates that the advantage of MZAttn does not stem from having a higher-rank dynamic update, but from the structured zonotope parameterization: the matrix zonotope’s center-plus-generators decomposition, context-dependent gating, and generator mixing provide an inductive bias that generic low-rank hypernetworks cannot replicate regardless of rank. D.1.5 HyperNet Rank Extended (Collapse Tasks) The main HyperNet rank ablation (Table 11) sweeps r∈8,16,32r∈\8,16,32\ on the set-regression and set-matching tasks. We extend it to the collapse-exhibiting tasks, MEB, Hull, and MST at high d, with r∈16,32,64r∈\16,32,64\ (where r=64r=64 matches the key dimension dk=64d_k=64, giving full-rank capacity to the low-rank update U(μ)VhyperU(μ)V_hyper). The full numerical content is reported in main-body Table 5. The trend is strikingly flat: across all ranks r∈16,32,64r∈\16,32,64\ on the hull and MEB cells, HyperNet’s R2R^2 changes by at most 0.0200.020 (MEB d=32d=32), with most cells changing by under 0.0150.015. The MST cells show somewhat larger inter-rank variation (≤0.29≤ 0.29 at d=128d=128), but every cell’s three rank values fall well below their corresponding seed-pool envelope and well below MZAttn-Full’s R2R^2 on the same task. Ranking up the low-rank update U(μ)VhyperU(μ)V_hyper from r=8r=8 (main table) to r=64r=64 (full key-dimension rank) does not close the gap to MZAttn: the largest HyperNet R² on each task remains 0.60.6+ R² behind MZAttn-Full (Hull d=3d=3: best HyperNet 0.2040.204 vs. MZAttn-Full 0.8040.804; MEB d=8d=8: best HyperNet 0.0220.022 vs. MZAttn-Full 0.6910.691). Combined with the main Table 11, this provides evidence that the advantage of MZAttn is architectural (the zonotope structure Mc+∑lγlMlM_c+ _l _lM_l) and not a capacity argument about generic input-dependent weight generation. D.1.6 Parameter Efficiency Ablation To address the concern that MZAttn’s advantage might stem from its larger parameter count (∼ 2× Standard), we ablate the two parameters controlling MZ capacity: ngn_g (generator slots per zonotope token) and L (matrix zonotope generator count). We test three variants: MZ-Full (ng=8,L=4n_g=8,L=4, used in main experiments), MZ-Med (ng=4,L=2n_g=4,L=2), and MZ-Slim (ng=2,L=1n_g=2,L=1). The full numerical content is reported in main-body Table 6. Table 6 shows that MZ’s advantage is largely insensitive to the generator count. MZ-Slim (333K parameters, 1.6×1.6× Standard) achieves the best Max result (R2=0.871R^2=0.871 vs. Standard’s 0.8120.812), a 41%41\% MSE reduction on MST (0.8350.835 vs. 0.7190.719), and a 72%72\% MSE reduction on rotation matching (0.9190.919 vs. 0.7060.706). Performance is nearly flat across the three MZ variants: the largest gap between MZ-Slim and MZ-Full on any task is 0.0170.017 R2R^2 (MST), within the seed variance for high-TDOF problems. On Max, MZ-Slim in fact outperforms MZ-Full by 0.0210.021 R2R^2, which suggests smaller generator budgets can act as regularization on simpler tasks. Together these results indicate the advantage comes from the matrix-zonotope structure rather than the number of generators; even L=1L=1 generator is enough to produce the key inductive bias for high-TDOF tasks. D.1.7 Depth Separation Validation Table 12: Depth separation: 1-layer MZAttn vs. D-layer standard Set Transformer (mean ± std over 3 seeds). 1-layer MZ outperforms standard depths up to 6 layers at 2×2× more parameters. Model Layers MSE ↓ R2R^2 ↑ Params MZAttn 1 42.3± 1.6 0.969± 0.001 244K Set Transformer 1 53.3± 0.7 0.961± 0.001 140K Set Transformer 2 57.3± 4.6 0.958± 0.003 208K Set Transformer 3 68.6± 16.4 0.950± 0.012 276K Set Transformer 4 59.6± 3.7 0.956± 0.003 344K Set Transformer 5 63.4± 8.3 0.953± 0.006 412K Set Transformer 6 57.1± 5.9 0.958± 0.004 480K 1-layer MZAttn outperforms all standard depths including 6-layer (2×2× more parameters). D.1.8 MZ in Standard Transformers Table 13: MZ-attention in a standard Transformer with positional encoding. L MZ-Transformer Std-Transformer Params (MZ / Std) 4 .9998±<.001±<.001 .9982±<.001±<.001 83K / 44K 8 .9981±<.001±<.001 .9864±.001 87K / 44K 16 .9964±.001 .9619±.002 96K / 44K 24 .9963±<.001±<.001 .9526±.007 105K / 44K 32 .9957±.002 .9313±.003 114K / 44K MZ-attention also achieves R2>0.99R^2>0.99 in a sequence Transformer with positional encoding (Table 13), suggesting compatibility beyond Set Transformers; broader sequence-domain evaluation is left to future work. D.2 Robustness and generalization Sensitivity of the architectural advantage to nuisance factors: input noise, set-size shift at test time, target-scale preprocessing, and TDOF-estimate uncertainty. MZAttn retains its lead across all four. D.2.1 TDOF Estimate Sensitivity (Spearman robustness) The Spearman correlation ρ=0.74ρ=0.74 between estimated TDOF and MZAttn’s MSE reduction (Section 5, Figure 2, right) relies on the per-task TDOF assignments documented in our analysis. To verify this correlation is not an artefact of any specific TDOF estimate, we perturb each of the 18 task TDOFs independently by a multiplicative factor in [0.5,1.5][0.5,1.5] (uniform), recompute the Spearman, and repeat 10,00010,000 times. We also run a log-symmetric variant (×[1/1.5,1.5]×[1/1.5,1.5] in log scale) to be invariant to the wide TDOF dynamic range (11 to 20162016). Table 14: Spearman robustness to per-task TDOF perturbation. Both ±50%± 50\% uniform and ×[1/1.5,1.5]×[1/1.5,1.5] log-symmetric perturbations preserve the strong correlation: in 99.98%99.98\% (9,998/10,0009,998/10,000) of uniform trials and in at least 95%95\% of log-symmetric trials, the resulting Spearman remains ≥0.6≥ 0.6. The headline correlation is therefore not driven by any single TDOF estimate. Perturbation scheme 55th pctile ρ median ρ 9595th pctile ρ frac. trials with ρ≥0.6ρ≥ 0.6 Baseline (no perturbation) ρ=0.7402ρ=0.7402 — ±50%± 50\% uniform multiplicative 0.6750.675 0.7330.733 0.7910.791 99.98%99.98\% ×[1/1.5,1.5]×[1/1.5,1.5] log-symmetric 0.6900.690 0.7350.735 0.7810.781 ≥95%≥ 95\% The Spearman correlation is robust: 9,9989,998 of 10,00010,000 uniform-perturbation trials yield ρ≥0.6ρ≥ 0.6 (min ρ=0.59ρ=0.59), and the 55th-percentile ρ stays above 0.670.67 in both perturbation schemes, indicating that the headline relationship between TDOF and MSE reduction does not depend on any specific assignment within a ±50%± 50\% window. Verification script: Base/tdof_sensitivity_analysis.py writes the full statistics to results/tdof_sensitivity.json. D.2.2 Ambiguity Sensitivity Table 15: Ambiguity sweep on set matching: MSE as a function of noise σ. σ ST MSE ST R2R^2 MZ MSE MZ R2R^2 Δ 0.05 62.06 0.957 41.12 0.971 −-33.7% 0.1 62.58 0.956 44.69 0.969 −-28.6% 0.3 59.87 0.957 38.92 0.972 −-35.0% 0.5 57.25 0.958 39.21 0.971 −-31.5% 1.0 58.28 0.955 44.80 0.965 −-23.1% MZAttn consistently reduces MSE by 23–35%, with the largest advantage at moderate noise (σ=0.3σ=0.3). D.2.3 Set Size Generalization Table 16: Size generalization: trained on sets of size 10–30, tested on larger sets. Test size ST MSE ST R2R^2 MZ MSE MZ R2R^2 Δ [10,30][10,30] 59.87 0.957 38.92 0.972 −-35.0% [30,50][30,50] 24.87 0.978 19.85 0.982 −-20.2% [50,80][50,80] 33.86 0.968 24.07 0.977 −-28.9% [80,120][80,120] 34.56 0.969 30.40 0.972 −-12.0% MZAttn maintains its advantage across out-of-distribution sizes (12–29% MSE reduction). D.2.4 Optimization Confound Ablation An important question is whether the catastrophic negative R2R^2 of baselines at high d reflects a representational limit or an optimization artifact from training on raw large-scale regression targets. We directly test this by repeating ST-Large on MST d=128d=128 and d=256d=256 under target normalization (subtract train mean, divide by train std) and compare to the same runs without normalization. We therefore re-ran the full main-table MST and convex hull scaling sweeps with target normalization and 5 seeds per setting, to give the cleanest architectural gap estimate (Tables 17 and 18). Table 17: MST weight, normalized targets (R2R^2, mean over 5 seeds). Under target normalization, the dramatic negative-R2R^2 collapse of baselines (Table 23) disappears: all models reach R2∈[0.37,0.74]R^2∈[0.37,0.74]. MZAttn-Full nonetheless retains a monotonically shrinking architectural advantage of +0.04+0.04 to +0.08+0.08 R2R^2 over the best context-rigid baseline, tying only at d=256d=256. The dramatic collapse in the raw-target table is thus partially an optimization artifact of scale-sensitive regression on large-magnitude targets; the persistent (if smaller) normalized gap is the architectural effect. d Standard ST-Large HyperNet Perceiver MZAttn-Full MZAttn-Large 16 0.660 0.658 0.495 0.467 0.738 0.688 32 0.565 0.567 0.435 0.417 0.640 0.617 64 0.539 0.524 0.447 0.435 0.594 0.560 128 0.510 0.493 0.443 0.401 0.554 0.497 256 0.461 0.462 0.373 0.363 0.469 0.447 Table 18: Convex hull volume, normalised targets (R2R^2, mean over 5 seeds). Under target normalisation the hull gap is non-monotonic: MZAttn-Full wins at d=6d=6 (+0.14+0.14), ties at d=3d=3, and is within seed variance of the leader at d=9d=9. Compare to Table 22 (raw targets), where the mid-d baseline R2R^2 plunges to −0.18-0.18 from training-scale instability. d Standard ST-Large HyperNet Perceiver LA-ST MZAttn-Full MZAttn-Large 3 0.822 0.866 0.727 0.725 0.824 0.863 0.795 6 0.292 0.412 0.195 0.209 0.408 0.555 0.497 9 0.138 0.160 0.108 0.119 0.172 0.123 0.143 Interpretation. These normalized tables, not the raw-target collapse tables, are the cleanest reading of the MST and convex hull scaling experiments. Raw-target-claim caveats: the negative-R2R^2 plunge in Table 23 is substantially the behavior of Standard and ST-Large on unnormalized regression targets of magnitude 10410^4–10510^5 (MST weights) or 10110^1–10310^3 (hull volumes); under standard target normalization the effect largely vanishes. MZAttn is less sensitive to this choice, which is in itself a useful practical property (one fewer hyperparameter to tune), but it does not establish a representational gap by itself. The MEB (Table 3) and quadratic TDOF (Table 7) results do establish such a gap: MEB has O(1)O(1)-magnitude targets for which normalization changes nothing and yet MZAttn reaches R2=0.38R^2=0.38–0.690.69 while all baselines are stuck at R2≤0.025R^2≤ 0.025; the quadratic TDOF task is scale-free by construction and yet every non-MZ architecture (including EGNN) collapses to R2≤0.01R^2≤ 0.01 for L≥4L≥ 4. These are the cleanest architectural evidence. On the SGSP family (Table 24), MZAttn wins uniformly at d=8d=8, wins on 3 of 4 tasks at d=16d=16, and crosses over with baselines at d=32d=32. We thus position MZAttn’s empirical contribution as: (i) a clean architectural win on a subset of tasks (MEB, quadratic-TDOF, low-d SGSP); (i) a consistent smaller advantage on MST scaling under normalization; (i) robustness to target scale decisions that affect context-rigid baselines more severely. Item (i) is a real-world usability advantage but is not itself architectural. D.3 Uncertainty calibration Empirical evaluation of the single-pass interval-hull-width (IHW) uncertainty estimator inherited from the matrix-zonotope output, both qualitatively (Spearman / Pearson correlation with error) and via standard calibration metrics (ECE, NLL) against MC-Dropout and Deep Ensembles. D.3.1 Uncertainty Calibration Table 19: Uncertainty calibration: samples binned by IHW. Higher IHW → higher error (Spearman r=0.51r=0.51). Bin IHW (mean) Mean ||error|| Count 1 (lowest) 5.33 1.13 400 2 5.52 2.01 400 3 5.72 3.15 400 4 6.03 4.20 400 5 (highest) 6.64 6.81 400 Table 20: Uncertainty quality comparison on set matching. Method Spearman ↑ Pearson ↑ Inference Passes MZAttn (IHW) 0.513 0.442 0.10s 1 MC-Dropout 0.553 0.547 0.37s 10 Deep Ensemble 0.470 0.542 0.40s 5 IHW achieves competitive uncertainty quality with a single forward pass at 3.7×3.7× lower cost. D.3.2 Quantitative Uncertainty Calibration (ECE and NLL) Table 21: Uncertainty calibration on set matching (mean ± std over 3 seeds). ECE: expected calibration error (lower better); NLL: Gaussian negative log-likelihood (lower better). MZAttn’s single-pass IHW uncertainty is on par with the 55-model Deep Ensemble on ECE and lower than it on NLL; MC-Dropout achieves the best ECE/NLL but at 30×30× inference cost. Method MSE ↓ ECE ↓ NLL ↓ Passes Models MZAttn (IHW) 55.0± 10.5 0.316± 0.001 5.02± 0.03 1 1 MC-Dropout 59.8± 6.1 0.197± 0.030 3.52± 0.43 30 1 Deep Ensemble 47.8± 1.1 0.337± 0.019 9.17± 1.96 1 5 MC-Dropout achieves the lowest ECE and NLL but requires 30×30× inference cost. MZAttn’s IHW yields NLL 5.025.02, nearly 2×2× lower than Deep Ensemble (9.179.17), while matching Ensemble-level ECE (0.3160.316 vs. 0.3370.337) in a single forward pass at 1×1× storage. MZAttn is therefore the only method here that requires neither extra forward passes nor extra stored model copies; the practical use case is latency- and storage-constrained deployment where 30×30× MC-Dropout passes or 5×5× ensemble storage are prohibitive. The remaining ECE gap relative to MC-Dropout may be reducible via post-hoc temperature scaling. D.4 Computational geometry and scaling High-dimensional and large-set scaling behaviour on classical computational-geometry targets. Together they establish that the architectural advantage is preserved (and often grows) at scale on tasks satisfying the structural prerequisites of Theorem 1. D.4.1 Minimum Enclosing Ball: A Computational-Geometry Benchmark The MEB benchmark and parameter-matched baseline collapse are reported in the main body (Table 3, Section 5). We expand here on the experimental design and the qualitative pattern across dimensions, and report MZAttn-Large as a generator-budget ablation that complements the headline MZAttn-Full configuration. The full benchmark uses Welzl’s algorithm [42] for ground-truth radii; point clouds are mixtures of Gaussians with n∈[10,30]n∈[10,30]. MZAttn-Large is reported as a generator-budget ablation. At d=8d=8, MZAttn-Large reaches R2=0.621±0.013R^2=0.621± 0.013 under the headline 5-seed / 5-epoch-warmup protocol (meb_results_combined.json). At d=16,32d=16,32, the higher-dimensional MZ-Large training is sensitive to LR warmup, so we report 10 seeds with extended 10-epoch warmup (meb_results_warmup10*.json): R2=0.488±0.099R^2=0.488± 0.099 at d=16d=16 and R2=0.221±0.074R^2=0.221± 0.074 at d=32d=32. All three are consistently below the MZAttn-Full numbers in the main body, supporting the choice of ng=8,L=4n_g=8,L=4 as the recommended configuration. Three observations: (i) Baseline collapse is uniform and clean. None of the five baselines, spanning the full 2017–2024 range (Set Transformer, Perceiver, HyperNet, a parameter-matched Set Transformer variant, and a linear-attention baseline representing the 2020s efficient-attention family), achieves R2>0.025R^2>0.025 on any dimension. The task is structurally hostile to context-rigid attention: to predict the radius, the model must identify a small subset of convex-hull extrema (at most d+1d+1 points) from among the input, and context-rigid WVW_V cannot express the required indicator-like selector. (i) MZAttn dominates by 2828–155×155×. MZ-Full achieves R2=0.691R^2=0.691 at d=8d=8 (≈155×≈ 155× HyperNet’s full-precision 0.00440.0044, 28×28× Standard’s 0.0250.025), R2=0.629R^2=0.629 at d=16d=16 (40×40× Perceiver’s 0.0160.016), and R2=0.378R^2=0.378 at d=32d=32 (37×37× Perceiver’s 0.0100.010). (i) The Linear Attention baseline (a 2023–2024-era design family) performs indistinguishably from Set Transformer. LA-ST reaches only R2=0.017,0.015,0.001R^2=0.017,0.015,0.001 at d=8,16,32d=8,16,32, within 1σ1σ of Set Transformer. This rules out the hypothesis that “modern efficient attention solves this”: the gap is not a softmax-vs-linear issue but an architectural capacity issue for context-adaptive operators, consistent with Theorem 8. D.4.2 Convex Hull Volume: Another Computational-Geometry Benchmark To verify the MEB collapse pattern is not specific to one task, we evaluate on a second classical computational-geometry quantity: the volume of the convex hull of a point set in ℝdR^d, computed exactly via the QHull algorithm [4]. The convex hull is determined by O(n⌈d/2⌉)O(n d/2 ) extremal points and its volume is smooth in their positions, giving a regression target with sparse combinatorial support similar to MEB but with substantially richer d-scaling. Table 22: Convex hull volume prediction (R2R^2, mean ± std over 5 seeds; n∈[15,35]n∈[15,35] points in ℝdR^d). At d=3d=3 MZAttn-Large beats the best baseline by +0.32+0.32 R2R^2. From d=6d=6 onwards every parameter-matched baseline collapses to negative R2R^2 on raw targets; MZAttn-Large is the only architecture significantly above R2=0R^2=0 at all three tested dimensions. The d=9d=9 MZ-Full result (0.003±0.0060.003± 0.006) is statistically indistinguishable from zero and is reported for completeness; Appendix D.2.4 reports normalised-target equivalents where the d=9d=9 gap structure differs. Model Params d=3d=3 R2R^2 d=6d=6 R2R^2 d=9d=9 R2R^2 Standard 208K 0.273± 0.081 −-0.209± 0.016 −-0.099± 0.002 ST-Large 362K 0.504± 0.058 −-0.178± 0.012 −-0.088± 0.004 HyperNet 360K 0.136± 0.082 −-0.211± 0.013 −-0.097± 0.003 Perceiver 341K 0.164± 0.058 −-0.221± 0.007 −-0.096± 0.003 LA-ST (Linear Attn.) 359K 0.473± 0.036 −-0.126± 0.022 −-0.077± 0.005 MZAttn-Full 368K 0.804± 0.014 0.103± 0.020 0.003± 0.006 MZAttn-Large 457K 0.826± 0.016 0.191± 0.053 0.056± 0.003 The pattern exactly mirrors MEB: at low d, MZAttn substantially outperforms baselines; at moderate d, baselines cross zero while MZAttn remains positive; even at d=9d=9, where the advantage narrows, MZAttn is the only architecture producing meaningful predictions. Together with MEB (Section D.4.1), this gives two independent computational-geometry benchmarks (MEB: boundary of minimum-radius ball; convex hull: boundary of convex hull) on which every context-rigid attention variant, including 2020s linear-attention, cannot exceed baselines, while MZAttn remains on the right side of zero. D.4.3 Rotation Matching Scaling at High Dimension The paper’s Task F (rotation matching) in ℝ8R^8 shows MZAttn outperforming baselines by 4141–57%57\% MSE. Our theory predicts this advantage should grow with dimension d, since off-diagonal rotation components (which FiLM-style diagonal modulation provably cannot express) become more numerous: a rotation in ℝdR^d requires d(d−1)/2d(d-1)/2 independent Givens angles, while FiLM only has d scalar parameters. We test this prediction at d∈16,32,64,128,256,512d∈\16,32,64,128,256,512\. The full numerical content is reported in main-body Table 4. The gap between MZAttn and the baselines grows monotonically with d: at d=16d=16, MZAttn reaches R2=0.826R^2=0.826 while the best baseline (ST-Large) reaches 0.2850.285, a 76%76\% MSE reduction; at d=32d=32, all four baselines drop to R2≈0R^2≈ 0 (no better than predicting the training mean), while MZAttn still attains R2=0.519R^2=0.519; at d=64d=64, the baselines have negative R2R^2 (actively worse than the mean predictor), and MZAttn is the only architecture producing useful predictions (R2=0.071R^2=0.071); at d=128d=128, the baselines collapse catastrophically to R2=−9R^2=-9 to −22-22 (MSE 1010–23×23× worse than the mean predictor, with large seed variance), while MZAttn remains at R2≈0R^2≈ 0 (MSE within 1%1\% of the mean predictor’s); at d=256d=256, all baselines reach R2≈−50R^2≈-50 to −62-62 (∼50× 50× MSE ratio vs. MZAttn’s R2=−0.005R^2=-0.005); at d=512d=512, MZAttn also begins to diverge (R2=−62R^2=-62) but remains 2.3×2.3× better in raw MSE than the next-best baseline (ST-Large at −145-145). The divergence at d=512d=512 reflects that a d=512d=512 rotation has d(d−1)/2=130,816d(d-1)/2=130,816 Givens parameters, exceeding our generator budget L=4L=4 by four orders of magnitude; the architecture has the capacity to express the operator family (dk2=4,096d_k^2=4,096 entries per generator, L⋅dk2≈16L· d_k^2≈ 16K free parameters in the headline configuration) but training stability becomes the bottleneck before representational capacity does, a separate scaling regime from the d≤256d≤ 256 rows where the headline gap holds. This matches the theoretical analysis: context-rigid attention (fixed WVW_V) cannot represent the context-adaptive rotation operator R(μ)R(μ), and the gap widens monotonically with d because the space of rotations grows as d(d−1)/2d(d-1)/2. Standard attention’s R2R^2 also becomes highly variable at d≥64d≥ 64 (std ±0.47± 0.47 at d=64d=64; ±3.5± 3.5 at d=128d=128), consistent with its failure being due to an expressibility limit rather than a tractable optimization problem. D.4.4 MST Weight Scaling at High Dimension The paper’s Task G (MST weight in ℝ8R^8) already showed MZAttn reducing MSE by 41%41\% over ST-Large. We extend this sweep to d∈16,32,64,128,256d∈\16,32,64,128,256\ to test whether the advantage on this combinatorial-geometry quantity also grows with dimension (as theory predicts: pairwise-distance structure has O(d2)O(d^2) degrees of freedom). Table 23: MST weight scaling (R2R^2, mean ± std over 5 seeds). As d increases past 3232, every non-MZ baseline collapses to strongly negative R2R^2 (worse than predicting the mean). MZAttn is the only architecture that remains above R2=0R^2=0 for all d≤256d≤ 256, with MSE 44–8×8× lower than the best baseline at d≥64d≥ 64. LA-ST is the linear-attention baseline [16], whose feature-map kernel underlies 2023–2024 architectures including RetNet [39] and Gated Linear Attention [44]. Model d=16d=16 d=32d=32 d=64d=64 d=128d=128 d=256d=256 Standard 0.295± 0.072 −-0.202± 0.367 −-1.326± 0.245 −-2.789± 0.511 −-3.221± 0.305 ST-Large 0.526± 0.044 0.274± 0.140 −-0.252± 0.046 −-1.230± 0.430 −-2.425± 0.247 HyperNet 0.250± 0.068 −-0.211± 0.244 −-1.352± 0.116 −-1.985± 0.251 −-3.423± 0.222 Perceiver 0.400± 0.097 −-0.082± 0.180 −-1.049± 0.242 −-2.361± 0.248 −-3.320± 0.299 MZAttn-Full 0.724± 0.010 0.625± 0.027 0.581± 0.008 0.484± 0.019 0.384± 0.042 MZAttn-Large 0.612± 0.009 0.531± 0.018 0.538± 0.038 0.497± 0.021 0.435± 0.012 Three features of Table 23 are notable: (i) Both MZAttn variants dominate all baselines across every dimension tested. The gap grows monotonically with d: MZ-Full beats ST-Large by +0.20+0.20 R2R^2 at d=16d=16 and +0.35+0.35 at d=32d=32, and from d=64d=64 onwards every single baseline has negative R2R^2 while MZAttn remains above 0.380.38. At d=64d=64, MZ-Full’s MSE is 3×3× lower than the best baseline; at d=256d=256 the ratio is over 5×5×. (i) Baseline failure at d≥64d≥ 64 is not just a performance drop but a qualitative one: Standard, HyperNet, and Perceiver all reach R2<−1R^2<-1, indicating MSEs larger than a constant mean predictor’s, and ST-Large, which remained competitive at d≤32d≤ 32, also falls to R2=−2.43R^2=-2.43 at d=256d=256. MZAttn’s MSEs stay within 22–3×3× of the mean predictor’s, with seed std below 0.040.04, whereas baselines have seed std of 0.20.2–0.40.4. (i) At smaller dimensions (d≤64d≤ 64), MZ-Full outperforms MZ-Large (consistent with Appendix D.1.6: extra generators act as overparameterization on moderate-TDOF tasks); at d≥128d≥ 128, MZ-Large overtakes MZ-Full (0.4350.435 vs. 0.3840.384 at d=256d=256), suggesting that as the task’s effective TDOF grows with d, the larger generator budget begins to pay off. This crossover aligns with the theoretical prediction that generator count should match the target function’s TDOF. D.4.5 Sparse Geometric Set Predicates (SGSP) Motivated by the MEB, convex-hull, and MST successes, we identify a broader task family we call Sparse Geometric Set Predicates: targets that are (i) computed from a continuous d-dimensional point cloud (no symbolic or one-hot encoding), (i) a smooth function of a combinatorially sparse subset of the input (extreme pairs, nearest neighbors, boundary points, or small distance-subset cuts). This family is where MZAttn’s matrix-zonotope machinery is predicted to provide the largest advantage because the target depends on a per-sample “selector operator” that extracts the relevant sparse subset. We test four members of this family beyond the MEB, hull, and MST results reported above: Table 24: Sparse Geometric Set Predicates benchmark (R2R^2, mean over 5 seeds; 7 models × 5 seeds × 3 dims × 4 tasks == 420 runs total). MZAttn-Full wins every d=8d=8 task with margin +0.04+0.04 to +0.13+0.13, maintains an advantage at d=16d=16 on 3 of 4 tasks, and is overtaken by parameter-matched Set Transformer variants at d=32d=32. This crossover is consistent with concentration-of-measure: at high d, distances between random points become tightly concentrated and sparse-combinatorial targets (max pairwise, k-th-N, k-center) reduce to aggregate statistics readily captured by context-rigid attention. Task d Standard ST-Large HyperNet Perceiver LA-ST MZAttn-Full MZAttn-Large diameter 8 0.723 0.768 0.702 0.674 0.787 0.876 0.814 16 0.732 0.748 0.706 0.682 0.755 0.806 0.727 32 0.774 0.775 0.687 0.683 0.740 0.710 0.663 knn-radius 8 0.721 0.786 0.697 0.611 0.815 0.869 0.810 16 0.703 0.699 0.655 0.593 0.710 0.784 0.721 32 0.740 0.728 0.651 0.629 0.697 0.685 0.653 k-smallest sum 8 0.116 0.109 0.034 0.083 0.108 0.152 0.097 16 0.009 0.009 −-0.005 0.005 0.007 0.010 −-0.039 32 −-0.010 −-0.024 −-0.003 −-0.011 −-0.017 0.014 0.001 furthest-first 8 0.422 0.478 0.275 0.298 0.456 0.604 0.531 16 0.396 0.391 0.163 0.249 0.365 0.438 0.205 32 0.315 0.276 0.100 0.209 0.252 0.225 0.206 These tasks are classical combinatorial-geometry quantities computed by standard algorithms (pairwise-distance matrix, k-nearest-neighbor sort, greedy farthest-first traversal, k-smallest pair enumeration). None has an exploitable group symmetry; targets depend on the magnitudes of specific pairwise distances, not on the global coordinate frame, so equivariant baselines gain no free advantage. The pattern in Table 24 has three regimes: (i) Low dimension (d=8d=8): MZAttn-Full wins on all four tasks with margins +0.036+0.036 (min-sum) to +0.126+0.126 (furthest-first) over the best context-rigid baseline. This is the regime where the sparse-combinatorial structure is most difficult for context-rigid attention, and MZAttn’s per-sample operator adapts to identify the relevant subset (extremal pair for diameter, k-th-N for knn-radius, farthest cluster center for k-center). (i) Moderate dimension (d=16d=16): MZAttn-Full retains an advantage on 3 of 4 tasks (+0.04+0.04 to +0.08+0.08), though the gap narrows. (i) High dimension (d=32d=32): MZAttn is overtaken by Standard or ST-Large on 3 of 4 tasks. We attribute this to concentration of measure: in high dimension, pairwise distances between isotropic points concentrate tightly around their mean, so sparse-combinatorial targets (diameter, k-th-N, k-center) become near-deterministic functions of simple aggregate statistics (mean pairwise distance, set variance) that context-rigid attention readily extracts. The per-sample-operator advantage of MZAttn no longer applies because all samples are, statistically, nearly identical. This d-dependent crossover is the mirror image of the MST and rotation-matching scaling pattern (Tables 23, 4), where MZAttn’s advantage grows with d. The two regimes correspond to distinct mechanisms: MST and rotation involve O(d2)O(d^2)-parameter operators (pairwise-distance tree, off-diagonal rotation) whose complexity scales with d; SGSP involves O(1)O(1)-parameter selectors (pick 2 extreme points, pick k boundary points) whose intrinsic complexity is dimension-independent and whose d-scaling concentrates rather than diversifies. The unified theoretical prediction is that MZAttn helps when TDOF grows with d relative to the information content of the aggregate statistics: a prediction confirmed by both scaling directions. D.4.6 Large-Scale Set Task Generalization To verify that MZAttn’s advantage persists at larger set sizes n and element dimensions d than those used in the main experiments, we evaluate on three synthetic tasks (max, sum, range) across a sweep of scales: d∈16,64,128d∈\16,64,128\ and n ranging from 1010–5050 (small) up to 500500–10001000 (large). Each configuration is trained for 150 epochs with 3 seeds. Table 25: Large-scale generalization (R2R^2, mean over 3 seeds). MZAttn’s advantage on the sum task grows with scale: at d=64,n=200–500d=64,n=200--500, MZ reaches R2=0.998R^2=0.998 while ST-Large reaches only 0.9460.946, a 30×30× MSE reduction. On the simpler max task, architectural differences diminish as capacity becomes the bottleneck; on range, all models struggle regardless of architecture. Max Sum Range Model d=16d=16, n=10-50n=10-50 d=64d=64, n=200-500n=200-500 d=128d=128, n=500-1000n=500-1000 d=64d=64, n=200-500n=200-500 d=64d=64, n=200-500n=200-500 d=128d=128, n=500-1000n=500-1000 Standard 0.748 0.744 0.632 0.939 0.083 0.038 ST-Large 0.822 0.744 0.631 0.946 0.083 0.038 Perceiver 0.728 0.745 0.628 0.910 0.077 0.036 MZAttn 0.843 0.745 0.627 0.998 0.082 0.039 Table 25 reveals three scaling regimes: (i) Sum at scale: MZAttn retains its large-margin advantage (R2=0.998R^2=0.998 vs. 0.9460.946 for ST-Large at d=64d=64, n=200-500n=200-500), showing that the zonotope structure’s inductive bias for aggregation tasks holds at scale. (i) Max at scale: while MZAttn leads at small scale (d=16d=16: R2=0.843R^2=0.843 vs. 0.7480.748 for Standard), at larger scales (n≥200n≥ 200) the max task becomes capacity-bottlenecked and all models converge to similar performance (R2≈0.74R^2≈ 0.74 at medium, 0.630.63 at large). (i) Range at scale: a known-hard task where all models struggle uniformly (R2<0.1R^2<0.1 at n≥200n≥ 200). Range over n≥200n≥ 200 i.i.d. samples concentrates tightly around its expectation (the empirical max minus min has variance scaling as O(logn/n)O( n/n)), so the regression target is nearly constant with respect to typical input variations; the low R2R^2 across architectures reflects this concentration ceiling rather than a relational-capacity gap, and is a property of the task at this scale. These findings support the theoretical predictions of Theorems 1 and 2: MZ’s advantage is selective, tied to the relational complexity (TDOF) of the target function, and persists at scale when the task demands high-TDOF operators (as in sum, which requires a full linear combination of elements). D.4.7 Early-Stopping Patience Ablation (ResNet-50 d=1024d=1024) The main-text Table 33 reports ResNet-50 (d=1024d=1024) results with early-stopping patience set to 300300, a departure from our default patience of 2020 used elsewhere. To verify that this is not a setting where the patience choice selectively favours MZAttn, we sweep the patience value ∈20,100,300∈\20,100,300\ for the four top contenders (MZ-Full, ST-Large, HyperNet, LinearAttn) on ResNet-50 k-center furthest (k=5k=5, n=16n=16, d=1024d=1024) with 5 seeds each; all other hyperparameters are held fixed. Results are in Table 26. Table 26: Patience sweep on ResNet-50 k-center furthest (R2R^2 mean ± std over 5 seeds, 8080 epochs). Across all three patience values 20,100,300\20,100,300\ every model converges to within 0.0020.002 R2R^2 of its patience-300300 plateau, and the ranking stabilises to MZAttn-Full >> ST-Large ∼ HyperNet >> LinearAttn. Patience choice does not selectively favour any architecture: MZAttn-Full’s lead over the next-best architecture is 0.0030.003–0.0050.005 R2R^2 across the full patience range. Per-seed JSONs in real_geom_cifar_resnet50_furthest_diag_p20_*.json, _sweep_p100_*.json, and _diag_p300_*.json. Model patience =20=20 patience =100=100 patience =300=300 MZAttn-Full 0.795±0.0010.795 0.001 (1st) 0.794±0.0020.794 0.002 (1st) 0.796±0.0010.796 0.001 (1st) ST-Large 0.790±0.0010.790±0.001 (3rd) 0.791±0.0010.791±0.001 (2nd) 0.791±0.0020.791±0.002 (2nd) HyperNet 0.790±0.0030.790±0.003 (2nd) 0.790±0.0030.790±0.003 (3rd) 0.789±0.0030.789±0.003 (3rd) LinearAttn 0.788±0.0020.788±0.002 (4th) 0.788±0.0030.788±0.003 (4th) 0.789±0.0030.789±0.003 (4th) Interpretation. In a fresh 5-seed sweep at the three patience values 20,100,300\20,100,300\ (epochs 8080, batch size 128128, learning rate 10−410^-4, identical across runs), every model converges to within ±0.002± 0.002 R2R^2 of its patience-300300 plateau. The MZ-Full >> ST-Large ∼ HyperNet >> LinearAttn ranking is therefore stable across the full patience range, with MZAttn-Full’s lead over the next-best architecture remaining at 0.0030.003–0.0050.005 R2R^2 throughout. The main-text claim that MZAttn-Full ranks first on ResNet-50 k-center furthest does not depend on the specific patience value chosen for early stopping. D.5 Real-world experiments and equivariant comparison Real-data and meta-learning evaluations beyond the synthetic high-TDOF regime, plus a side-by-side comparison with E(n)-equivariant graph networks on tasks where strict symmetry is or is not present. These delineate where MZAttn’s architectural advantage transfers and where it does not. D.5.1 Few-shot Meta-Regression: a 7-way Parameter-Fair Comparison The main-body experiments test whether MZAttn can fit a fixed target whose intrinsic complexity requires a high-TDOF operator. A natural follow-up question is whether MZAttn also excels when the operator must be inferred from a support set at inference time. We design a few-shot meta-regression task that is the meta-learning analogue of the quadratic TDOF benchmark in Section 3.2 and report a parameter-fair 7-way comparison against every architecture used in the main results table, addressing the concern that a 2-method (Standard vs. MZAttn) baseline is not parameter-fair. Setup. We fix a shared basis M1,…,ML\M_1,…,M_L\ of random matrices Ml∈ℝdout×dinM_l ^d_out× d_in. For each task τ, we draw coefficients cτ∼(0,IL)c_τ (0,I_L) and define the task-specific operator Aτ=∑l=1Lcτ[l]MlA_τ= _l=1^Lc_τ[l]\,M_l. The model observes K=30K=30 support pairs (xi,yi)(x_i,y_i) with yi=Aτxiy_i=A_τx_i and must predict yq=Aτxqy_q=A_τx_q for a new query xqx_q. AτA_τ is never given; it must be inferred from the support context. We set din=dout=8d_in=d_out=8 and vary L∈4,8,16,32L∈\4,8,16,32\ to control the intrinsic dimension of the operator family. All seven architectures are instantiated with dmodel=64d_model=64 and 22 or 44 encoder layers; 3 seeds; 50 epochs with patience 10. Table 27: Few-shot meta-regression at L=32L=32 (highest-TDOF setting), parameter-fair 7-way comparison (MSE ↓ , mean± over 3 seeds). At shallow depth (n=2n=2), most non-MZ baselines fail to learn the task (MSE ≈30≈ 30); Standard narrowly wins against MZAttn. At matched depth (n=4n=4), ST-Large is the clear winner and MZAttn-Large places second, beating Standard, HyperNet, Perceiver, and LinearAttn. Perceiver fails to converge at either depth. Depth Standard ST-Large HyperNet Perceiver LinearAttn MZAttn-Full MZAttn-Large n=2n=2 0.790.79 0.830.83 30.7530.75 30.7630.76 30.7730.77 1.011.01 1.391.39 n=4n=4 0.450.45 0.170.17 0.580.58 30.7630.76 1.241.24 0.470.47 0.36¯ 0.36 Bold: best in each row. Underline: 2nd best. Param counts shown for the L=32L=32 (n=4n=4) configuration: Standard 146146K, ST-Large 255255K, HyperNet 297297K, Perceiver 314314K, LinearAttn 380380K, MZAttn-Full 334334K, MZAttn-Large 485485K; the L=32L=32 (n=2n=2) row uses the same architectures with about 5050–55%55\% of these counts. Result interpretation. The 7-way comparison shows that generic meta-regression, where the target is a random linear operator in an L-dimensional subspace, is not a setting where MZAttn dominates. ST-Large, a wider Set Transformer, wins at L=32L=32 (MSE 0.170.17 vs. ≥0.36≥ 0.36 for every other method), and at L=4L=4–88 it wins or ties with LinearAttn. MZAttn-Large ranks second at L=32L=32 with 44 layers (MSE 0.360.36), beating Standard, HyperNet, Perceiver, and LinearAttn. Previous versions of this paper reported a 3838–51%51\% MSE reduction of MZ over Standard; that comparison was against a single 7777K-parameter vanilla Standard Transformer and did not include parameter-matched ST-Large (255255K) or LinearAttn (380380K), both of which in fact outperform MZ here. What the corrected result implies. The mismatch between this meta-regression outcome and the main-body wins on MEB, SGSP, MST, rotation matching, and convex hull is informative. Those main-body tasks share a feature that meta-regression does not: their target operators are algorithmic (boundary-point selection for MEB and hull, minimum-spanning-tree edge selection for MST, rotation for rotation matching), and therefore have sharp combinatorial structure that an expressive but unstructured operator class (such as wide attention’s effective WVW_V) struggles to approximate with smooth gradient updates. Meta-regression by contrast has dense, smooth AτA_τ operators drawn from a Gaussian, which a high-capacity standard attention layer can fit directly. We therefore scope the empirical claim precisely: MZAttn is the architecture of choice when the target operator has sharp algorithmic and combinatorial structure and a matching TDOF (MEB, SGSP, MST, rotation, convex hull), not as a universal improvement to attention on any high-TDOF-seeming task. Connection to data-driven reachability. The algebraic machinery of MZAttn has a natural interpretation through the lens of data-driven reachability analysis [2]. In that framework, a matrix zonotope represents the set of system matrices consistent with noisy observations; in ours, the learned MZ represents the family of “plausible relational mappings” consistent with the observed context. MZ–zonotope multiplication propagates this set-valued information through the network, maintaining a structured envelope of possible transformations at each layer, an inductive bias that is absent from pointwise architectures. D.5.2 QM9 Molecular Property Prediction To evaluate on a real-world scientific benchmark, we test all models on QM9 molecular property prediction [31]. We parse the original gdb9.sdf with RDKit [32] to obtain true 3D atomic coordinates; each molecule is represented as a set of atoms with 9-dimensional features (5-dim atom type one-hot for H/C/N/O/F, 3D Cartesian coordinates in Å, Gasteiger partial charge), with up to 29 atoms per molecule. We use 50K molecules split 80/10/10 and predict three properties spanning distinct physical regimes: dipole moment (μ, depends on charge distribution), isotropic polarizability (α, depends on spatial extent), and HOMO–LUMO gap (electronic structure). All models use the same architecture as the main experiments (d=64d=64, H=4H=4, dff=128d_f=128); results are averaged over 3 seeds with 100 training epochs. Table 28: QM9 molecular property prediction across all 12 targets (R2R^2, mean over 3 seeds; best in bold; the Gap column is unbolded because ST-Large 0.8790.879 and MZ-Large 0.8740.874 tie within 0.0050.005 R2R^2). Parameter-matched attention models (ST-Large, Perceiver, HyperNet, MZAttn) cluster within 0.040.04 R2R^2 on the 8 thermodynamic targets (α, R2R^2, ZPVE, U0U_0, U298U_298, H298H_298, G298G_298, CvC_v); on the 4 electronic-structure / frontier-orbital targets (μ, HOMO, LUMO, Gap) the spread is wider (HyperNet trailing by 0.080.08–0.150.15 R2R^2), reflecting their higher relational complexity. MZ-Large (ng=16n_g=16, L=8L=8) is evaluated under batch size 6464 on all 1212 targets to keep the protocol consistent across the row (the same per-target budget MZ-Large was originally validated under for HOMO/LUMO); MZ-Large tops the original ST-Large on HOMO (+0.033+0.033 R2R^2) and LUMO (+0.074+0.074 R2R^2), and matches it on Gap (within 0.0050.005 R2R^2, where ST-Large 0.8790.879 ≥ MZ-Large 0.8740.874); a parameter-matched ST-XLarge subsequently closes the HOMO/LUMO gap (Appendix D.5.2); on the remaining nine targets MZ-Large fails to improve over the default MZAttn and degrades by 0.110.11–0.260.26 R2R^2, consistent with the TDOF prediction that scaling generator capacity helps only on targets that admit a non-trivial context-adaptive solution. Specialised equivariant models (SchNet [35], DimeNet [10]) achieve higher absolute accuracy with hand-designed features. Model Params μ α HOMO LUMO Gap R2R^2 ZPVE U0U_0 U298U_298 H298H_298 G298G_298 CvC_v Deep Sets 35K 0.437 0.865 – – 0.653 – – – – – – – Slot Attention 59K 0.607 0.917 – – 0.753 – – – – – – – Set Transf. 208K 0.752 0.905 0.803 0.838 0.858 0.959 0.954 0.856 0.856 0.856 0.859 0.940 ST-Large 361K 0.795 0.914 0.842 0.852 0.879 0.967 0.958 0.864 0.862 0.863 0.858 0.950 ST-XLarge 443K – – 0.883 0.937 – – – – – – – – Perceiver 341K 0.766 0.920 0.774 0.819 0.824 0.967 0.955 0.891 0.886 0.883 0.888 0.948 HyperNet 360K 0.641 0.902 0.722 0.773 0.786 0.955 0.934 0.871 0.863 0.869 0.876 0.922 MZAttn 368K 0.761 0.911 0.802 0.830 0.855 0.947 0.952 0.877 0.875 0.871 0.867 0.940 MZAttn-Large 417K 0.500 0.698 0.875 0.926 0.874 0.823 0.702 0.759 0.760 0.722 0.757 0.676 Table 28 reports all 12 QM9 targets. With the default generator budget (ng=8n_g=8, L=4L=4), MZAttn is competitive with the best parameter-matched attention baseline on every target (within 0.050.05 R2R^2), confirming the TDOF prediction that adding context-adaptive capacity does not regress on low-to-moderate-TDOF molecular targets. MZAttn matches or marginally exceeds Standard attention on 10 of 12 targets (the exceptions are LUMO, where Standard leads by 0.0080.008 R2R^2, and the spatial-extent target R2R^2 where Standard leads by 0.0130.013); on μ and α MZAttn edges Standard by 0.0090.009 and 0.0070.007 R2R^2 respectively. MZAttn is not the best on any single target. On the hardest electronic-structure targets (HOMO, LUMO), where baselines vary most, we also test a larger variant MZAttn-Large with ng=16n_g=16 zonotope generators and L=8L=8 matrix-zonotope generators (417417K parameters, 1.16×1.16× ST-Large). MZAttn-Large outperforms the original ST-Large on HOMO (0.8750.875 vs. 0.8420.842) and LUMO (0.9260.926 vs. 0.8520.852); however, when we additionally test ST-XLarge, a Set Transformer scaled to 443443K parameters by widening d from 8080 to 9292, ST-XLarge attains R2=0.883±0.005R^2=0.883±0.005 on HOMO and 0.937±<0.0010.937±<0.001 on LUMO (n=3n=3 seeds; qm9_xlarge_results_homo.json, _lumo.json), exceeding MZAttn-Large on both. The HOMO–LUMO gap of MZAttn-Large over ST-Large therefore reflects a parameter-count advantage rather than a structural one, and a parameter-matched context-rigid baseline closes the gap. The complementary observation is that MZAttn-Large improves over the default MZAttn on three frontier-orbital / electronic-structure targets (HOMO +0.073+0.073, LUMO +0.096+0.096, Gap +0.019+0.019 R2R^2) and underperforms it on the remaining nine targets by 0.110.11–0.260.26 R2R^2 (Table 28, last row), with the largest degradation on the heat capacity (CvC_v: 0.6760.676 vs. MZAttn’s 0.9400.940). The TDOF-based capacity bound is again confirmed: scaling generator budget pays off only on the small subset of targets (HOMO, LUMO, Gap) that admit a non-trivial context-adaptive solution, and adds variance without bias correction on aggregate-statistic targets where capacity is already saturated. This is consistent with the broader scope claim of the paper: on low-to-moderate TDOF real-data targets like QM9, where the per-sample operator does not vary in a high-rank subspace, scaling context-rigid attention is a competitive alternative to structural augmentation. The complementary regime is the high-TDOF synthetic suite (MEB, MST, rotation matching, SGSP), on which an MZ-Slim variant with L=1L=1 and only 1.6×1.6× Standard’s parameter count already outperforms parameter-matched ST-Large (Appendix D.1.6); structural inductive bias dominates there, and capacity scaling does not catch up. The QM9 ST-XLarge equivalence and the synthetic MZ-Slim parameter-efficiency win together demarcate the boundary precisely. Deep Sets lags every attention-based model on dipole moment by ≥0.20≥ 0.20 R2R^2, consistent with the higher relational complexity of that target; Slot Attention is competitive with the weakest attention baselines (within 0.040.04 R2R^2 on μ and α). Specialized equivariant models (SchNet [35], DimeNet [10]) achieve higher absolute accuracy, but require hand-designed radial basis functions and spherical harmonics, whereas all models compared here use the same generic set-input interface. D.5.3 Slot Attention Baseline Comparison To compare against the modern Slot Attention baseline [22] on tasks where our existing models have established results, we evaluate on four synthetic set tasks: max, sum, median, and range. These tasks span a range of relational complexity from low-TDOF (max, median) to moderate-TDOF (sum, range). Table 29: Slot Attention baseline comparison on synthetic set tasks (R2R^2, mean ± std over 3 seeds). Models are reported at canonical sizes (Slot Attention 5959K, Deep Sets 3636K, Set Transformer 208208K, MZAttn ∼365 365K) rather than parameter-matched; the parameter-matched comparison against ST-Large (362362K) and HyperNet (360360K) is in Table 2. Slot Attention’s iterative routing trails the standard and matrix-zonotope attention variants on all four tasks; MZAttn attains the highest R2R^2 on the sum task (0.9990.999). Model Max R2R^2 ↑ Sum R2R^2 ↑ Median R2R^2 ↑ Range R2R^2 ↑ Slot Attention 0.749± 0.004 0.892± 0.004 0.652± 0.002 0.324± 0.002 Deep Sets 0.823± 0.002 0.995±<0.001±<0.001 0.646± 0.003 0.348± 0.003 Set Transformer 0.817± 0.001 0.960± 0.002 0.659± 0.008 0.344±<0.001±<0.001 MZAttn (ours) 0.847± 0.008 0.999±<0.001±<0.001 0.654± 0.001 0.343± 0.001 Table 29 shows Slot Attention underperforming every other architecture, including Deep Sets. Iterative slot competition was designed for object-centric image decomposition and does not appear to transfer to set-to-scalar regression. MZAttn is best on max (R2=0.847R^2=0.847 vs. 0.8170.817 for Standard) and reaches R2=0.999R^2=0.999 on sum compared to 0.9600.960 for Set Transformer, a 40×40× residual-MSE reduction relative to the Set Transformer baseline. On the low-TDOF median task, all models perform similarly (R2≈0.65R^2≈ 0.65), matching the theory’s prediction. D.5.4 Comparison with E(n)-Equivariant GNNs We compare MZAttn to E(n)-equivariant graph neural networks (EGNN) [33], a symmetry-enforcing baseline that uses pairwise distances ‖xi−xj‖\|x_i-x_j\| as invariant features. All runs use 33 seeds; EGNN is parameter-matched at ∼ 303K (close to MZAttn’s 368368K) with hdim=140h_dim=140 and 44 message-passing layers. The full numerical content is reported in main-body Table 7. The pattern is architecturally interpretable: (i) Where EGNN wins. On every task where the label is invariant under rotations and translations of the input (rotation matching, convex hull volume, MEB radius, covariance Frobenius norm, Mahalanobis distance, PCA reconstruction error), EGNN outperforms MZAttn. This confirms the expected pattern: hard-coding the correct symmetry beats learning it. (i) Where MZAttn wins. The quadratic TDOF target (Eq. 4) depends on specific Frobenius-orthogonal matrices Ml\M_l\ in a fixed basis; rotating the input changes the target. EGNN, whose output is rotation-invariant by construction, cannot express this target and collapses to R2≈0R^2≈ 0 across L∈4,8,16L∈\4,8,16\. MZAttn handles this regime because its matrix-zonotope operator is not tied to any group action. (i) Target scale sensitivity. Our default baselines train on raw regression targets (see Appendix D.2.4); the collapse of Standard and ST-Large at high d on MST is partly an optimization artifact that target normalization can mitigate. The MZAttn vs. ST-Large gap narrows under normalization on MST and Hull, but persists on MEB (where target magnitudes are small) and on the quadratic TDOF task where the issue is architectural rather than numerical. EGNN, being a different architecture family, is unaffected by this confound. The comparison delineates MZAttn’s scope: MZAttn is the right tool for context-adaptive linear operators that are not reducible to a known group action; EGNN is the right tool when the target’s symmetry is known and can be enforced by construction. D.5.5 Computational Cost Table 30: Computational overhead (ratio MZ/Standard). RTX 5090, batch 64. Config Params Inference Training Memory Small (d=64d=64, n=30n=30) 1.9×1.9× 2.4×2.4× 2.2×2.2× 2.5×2.5× Medium (d=128d=128, n=50n=50) 1.9×1.9× 2.5×2.5× 3.8×3.8× 2.0×2.0× Large (d=256d=256, n=100n=100) 1.7×1.7× 2.7×2.7× 5.5×5.5× 2.5×2.5× D.5.6 Negative Transfer: Neural-Process Image Completion and Tabular Meta-Regression To test whether MZAttn’s advantage generalizes beyond synthetic high-TDOF relational tasks, we evaluated it on two families of real-world benchmarks where the structural prerequisites (per-sample operator variability in a high-dimensional subspace) are weak or absent. Both yielded ties or slight losses rather than wins; we report these honestly as they sharpen the scope of the claim. Neural-Process image completion (MNIST, CIFAR-100). Following the Attentive Neural Process setup [17], each image defines a 2D function f:(x,y)↦pixel valuef:(x,y) value; the model receives K context pixels with (xy,value)(xy,value) tokens and predicts values at T target coordinates. We compared Standard, ST-Large, HyperNet, Perceiver, LinearAttn, MZAttn-Full, and MZAttn-Large at matched depth (4 layers) using a Gaussian likelihood head. On MNIST (K=50K=50, T=200T=200, 5 seeds), ST-Large achieved the best test NLL of −1.657-1.657; MZAttn-Large reached −1.599-1.599 at 3.4×3.4× the parameters of Standard (−1.617-1.617). On CIFAR-100 (K=100K=100, T=300T=300, 3–5 seeds), Standard attained −1.147-1.147 NLL at 144144K parameters while MZAttn-Full achieved only −1.140-1.140 NLL at 332332K parameters; MZ lost by 0.0070.007 NLL while using 2.3×2.3× the parameters. This pattern aligns with the theory: image pixel prediction is dominated by spatial-locality inductive bias, not per-image operator variance; the high-TDOF prerequisite for MZAttn’s gain is absent, so extra capacity in the form of matrix-zonotope value projections yields no architectural advantage. Few-shot meta-regression on real tabular data (OpenML-CTR23 subset). We built a meta-learning benchmark from six real OpenML regression datasets (California Housing, Diabetes, Energy Efficiency, Abalone, Airfoil, Boston Housing), each restricted or padded to 8 features. Three were used for meta-training and three were held out for test. Each episode sampled K=50K=50 context rows and T=50T=50 query rows from a single dataset; models were trained to predict the standardized target under a Gaussian likelihood. Under the held-out split, both Standard and MZAttn-Full exhibit severe distribution shift (test NLL jumps from −0.43-0.43 at validation to +18+18–2020 at test, with comparable cross-seed standard deviations of ±6.06± 6.06 for Standard and ±5.45± 5.45 for MZAttn-Full at 1010 seeds with patience 1212), indicating that small-K in-context operators do not identify the true per-dataset regression operator when the test datasets differ structurally from training. The two architectures are within seed variance on NLL (MZAttn-Full +20.02+20.02 vs. Standard +18.66+18.66; gap 1.361.36, pooled std 5.765.76) and MZAttn-Full has slightly lower test MSE (1.5151.515 vs. 1.5971.597). Under the easier same-distribution split (test held-out rows from training datasets), MZAttn-Full wins 5.5%5.5\% on MSE but loses 1.4%1.4\% on NLL, a mixed signal. Neither split supports a dominance claim, and the apparent five-fold seed-variance ratio reported in earlier 5-seed pilots is an artefact of small-sample standard-deviation estimation; the 10-seed run reports comparable spreads. Systematic real-world transfer study. To empirically characterise which real-data configurations inherit the synthetic gain of MZAttn and which do not, we ran the geometric-target family (MEB, convex hull, MST weight, k-center furthest cost) on a systematic grid of point-cloud constructions across MNIST, Fashion-MNIST, and CIFAR-100, organised into three tiers of per-sample rank. Tier 1: raw 2D pixel coordinates of on-pixels, with optional per-sample 2×22×2 affine perturbations (TDOF-4) or 3-digit multi-cluster mixtures. Tier 2: 8D feature-lifted pixels (position + intensity + gradient + polynomial features) and CIFAR-100 4×44×4 image patches encoded as 8D patch-feature vectors (mean and std RGB, intensity, colorfulness), with variable-size and class-mixture variants. Tier 3: 256D ImageNet-pretrained ResNet-18 layer-3 features of CIFAR-100 (adaptively average-pooled to n=16n=16 points per image). Together the tiers span a 128×128× increase in per-sample dimension while holding the target family fixed, yielding a controlled ablation of the structural prerequisites the theory predicts. Table 31 reports MZAttn-Full’s rank among 7 parameter-comparable baselines in each (construction, target) cell; the full numerical R2R^2 values and per-method seed statistics are in the supplementary JSON dumps. Table 31: Systematic transfer study: MZAttn-Full rank (out of 7 parameter-matched methods) per (construction, target) cell, grouped by per-sample rank tier. 3 seeds per cell; dashes mark cells not in the grid. Mean ranks across populated cells: 3.93.9 on Tier-1 (low-rank 2D pixel, n=11n=11 cells), 3.93.9 on Tier-2 (medium-rank 8D lift, n=5n=5), 3.03.0 on Tier-3 (high-rank deep feature, n=2n=2); the Tier-3 mean is descriptive only given the small cell count, and the rank=1rank=1 Furthest cell drives it. Expected rank under the no-advantage null is 44. All printed ranks reproduce from the per-cell JSONs via Base/collect_transfer_matrix.py. Construction d MEB Hull MST Furthest Tier 1: low-rank 2D pixel coordinates MNIST raw pixel 2 4 5 – – MNIST + affine (TDOF-4) 2 5 – – – MNIST 3-digit mixture 2 4 4 4 – Fashion-MNIST pixel 2 – 4 – 2 Fashion-MNIST + affine 2 – 3 – – Fashion-MNIST 3-digit mix 2 – – 4 – CIFAR-100 raw pixel 2 – 3 – 5 Tier 2: medium-rank 8D feature-lift MNIST 8D polynomial lift 8 3 3 – – CIFAR-100 4×44×4 patch features 8 5 2 – – CIFAR-100 variable-size patches 8 5 – 4 – CIFAR-100 class-mixture patches 8 5 – – – Tier 3: high-rank deep feature CIFAR-100 ResNet-18 layer-3 256 (unstable) – 5 1 The rank improvement of MZAttn-Full at Tier-3 (deep-feature space), relative to Tier-1 and Tier-2 where ranks are tied within seed noise, is the empirical signature the theory predicts: MZAttn’s advantage requires data to have jointly (i) genuinely high-rank per-sample structure (not a 2D manifold embedded via a low-rank projection), (i) sparse combinatorial targets (such as the 3–9 extreme points that determine MEB or the k=5k=5 centers determining k-center cost), and (i) the absence of a stronger spatial or equivariance prior that standard attention can exploit. On every cell where any one of these is absent, MZAttn-Full ranks 3rd–5th, not catastrophically worse, but also not ahead. The rank-1 cell appears only when all three conditions are constructively met. The synthetic reference point (mixture-of-Gaussians MEB at d=8d=8, n=10n=10–3030, where MZAttn-Full reaches R2=0.684R^2=0.684 against baselines stuck at R2≤0.023R^2≤0.023; Table 3) reproduces exactly in the same code. The systematic grid therefore serves as the empirical scaffold on which the theoretical scope of the claim rests: it is not a set of failed transfer attempts but a controlled verification that the architectural advantage tracks the structural prerequisites of Theorem 1. Positive transfer: CIFAR-100 ResNet-18 features. Guided by the diagnostic above, we construct a real-data setting satisfying the three prerequisites: (1) ResNet-18 (ImageNet-pretrained) layer-3 feature maps of CIFAR-100 images, adaptively average-pooled to spatial 4×44× 4, giving n=16n=16 genuinely-high-rank points in ℝ256R^256 per image (not a low-rank projection of pixel coordinates); (2) the sparse combinatorial SGSP-family targets (MEB radius, hull volume, MST weight, diameter, k=5k=5 k-center furthest cost, k=5k=5 median k-N radius); (3) no imposed symmetry. Table 32 summarises the 7-way comparison at 5 seeds per method per task. Table 32: Real-world SGSP on CIFAR-100 ResNet-18 layer-3 features (n=16n=16, d=256d=256, R2R^2 mean ± std, bold = best). Seed counts differ by row: k-center furthest uses 5 seeds (file: real_geom_cifar_resnet_furthest_5seed.json); Diameter, k-N radius, MST weight, MEB radius use 3 seeds (default seed pool 42,123,456\42,123,456\). The MEB cell at n=16≤d+1=257n=16≤ d+1=257 is structurally degenerate (no sparse-combinatorial selector available); we mark it “unstable” and exclude it from the ranking claim. On the remaining four tasks, methods tie within ≤0.03≤ 0.03 R2R^2, and MZAttn-Full ranks 1st on k-center furthest cost. Task Standard ST-Large HyperNet Perceiver LinearAttn MZAttn-Full Furthest (k=5k=5) 0.7890.789 0.7930.793 0.7920.792 0.7460.746 0.7970.797 0.8010.801 Diameter 0.8560.856 0.8720.872 0.8400.840 0.8150.815 0.8820.882 0.8760.876 k-N radius 0.8920.892 0.8990.899 0.8980.898 0.8480.848 0.9040.904 0.8980.898 MST weight 0.9120.912 0.9190.919 0.9160.916 0.9080.908 0.9220.922 0.9140.914 MEB radius 0.8000.800 0.8150.815 0.7830.783 0.7280.728 0.8170.817 0.5420.542 (unstable) Across the full SGSP family on real ResNet-18 features, LinearAttn ranks first on three of the four valid cells (diameter, k-N radius, MST weight; the MEB cell is structurally degenerate at n=16≤d+1n=16≤ d+1 and is excluded from the ranking claim), and the gap between the top and bottom attention-based method is narrow (≤0.03≤ 0.03 R2R^2) on these dense-statistical targets. Aggregating many pairwise distances reduces these targets to dense statistics that context-rigid attention readily extracts, so the absence of an MZ advantage there is consistent with the TDOF theory rather than evidence against it. Only the sparsest combinatorial target, k-center furthest cost, yields a ranking on which MZAttn-Full is first among the seven parameter-matched baselines in both a 5-seed and an independent 10-seed verification run. We are explicit that the ranking, not the magnitude, is the evidence at the smaller backbone: 5-seed 0.8010.801 vs. LinearAttn 0.7970.797 and 10-seed 0.7970.797 vs. 0.7960.796 are both within pooled seed standard deviation. The cleaner statistical separation arrives at the larger backbone (Section 5, ResNet-50 d=1024d=1024, Welch’s t≈5.4t≈ 5.4, p<0.005p<0.005); the ResNet-18 result establishes that the rank ordering is stable across two independent seed pools, a necessary but weaker form of evidence. Dimension scaling: ResNet-50 features (d=1024d=1024). Table 33 reports the seven-baseline dimension-scaling comparison. The reported ResNet-50 numbers use patience 300300 to give every model ample budget to converge at d=1024d=1024, and the patience sweep in Appendix D.4.7 confirms the MZAttn-Full >> ST-Large ∼ HyperNet >> LinearAttn ranking is stable across patience 20,100,300\20,100,300\ at 8080 training epochs. Per-method standard deviations are: MZAttn-Full 0.7955±0.00090.7955± 0.0009, MZAttn-Large 0.7930±0.00240.7930± 0.0024, ST-Large 0.7909±0.00170.7909± 0.0017, with MZAttn-Full also the most stable method in the comparison. The absolute magnitude of the d=1024d=1024 gap is small (+0.0046+0.0046 R2R^2, equivalent to a 5.5%5.5\% reduction in 1−R21-R^2 residual error) but reproduces across 55 seeds at Welch p≈0.0018p≈ 0.0018. Table 33: Dimension scaling on the k-center furthest target (k=5k=5, n=16n=16, R2R^2 mean over 5 seeds). ResNet-50 numbers use early-stopping patience 300300 for full convergence at d=1024d=1024; a patience sweep 20,100,300\20,100,300\ confirms ranking stability for any patience ≥100≥ 100 (Appendix D.4.7). Feature Standard ST-Large HyperNet Perceiver LinearAttn MZAttn-Large MZAttn-Full ResNet-18 (d=256d=256) 0.7890.789 0.7930.793 0.7920.792 0.7460.746 0.7970.797 0.7890.789 0.8010.801 ResNet-50 (d=1024d=1024) 0.7890.789 0.7910.791 0.7890.789 0.7840.784 0.7890.789 0.7930.793 0.7960.796 What these results show. MZAttn’s gain is structural: it bites when the target operator genuinely varies per sample in a high-rank subspace and depends on a sparse combinatorial subset of the input. When the target is governed by spatial locality (image pixels), smooth interpolation (Neural-Process regression), or a known symmetry (equivariant tasks), MZAttn’s per-sample operator prior is not the right tool. Real high-rank deep-feature point clouds with a sparse combinatorial target (CIFAR-100 ResNet features, k-center furthest cost) do satisfy the prerequisites, and MZAttn-Full wins there. The real-world transfer is therefore architecturally precise rather than universal, in line with the TDOF theory.