Paper deep dive

The Polytopal Neural Network

A. Emilie J. Wedenborg, Anders V. Nørskov, Teresa Dorszewski, Kristoffer Wickstrøm, Morten Mørup

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 88

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/10/2026, 2:50:10 AM

Summary

The paper introduces Polytopal Neural Networks (PNNs), a framework that enforces polytope-based structures on neural network representations to enhance interpretability. By constraining latent representations to lie on a polytope defined by learned archetypes (convex combinations of corpus points), PNNs provide intrinsic, layer-wise explanations. The method scales via batched streaming updates and amortized simplex inference, and offers a direct optimization route for Vector Quantization (VQ) training without requiring heuristics like the straight-through estimator or commitment losses.

Entities (7)

Polytopal Neural Networks · model-architecture · 98%Technical University of Denmark · organization · 95%Amortized Simplex Inference · technique · 93%Archetypal Analysis · method · 92%Vector Quantized Variational Autoencoders · model-architecture · 90%Straight-Through Estimator · technique · 88%PathMNIST · dataset · 85%

Relation Signals (6)

A. Emilie J. Wedenborg → affiliatedwith → Technical University of Denmark

confidence 95% · A. Emilie J. Wedenborg Technical University of Denmark

Polytopal Neural Networks → uses → Archetypal Analysis

confidence 95% · Integrating AA constraints into deep architectures, PNNs bridge the gap between interpretable, prototype-based modeling and high-capacity representation learning

Polytopal Neural Networks → employs → Amortized Simplex Inference

confidence 93% · We scale our approach using learned corpus representations and an amortized simplex inference procedure

Polytopal Neural Networks → avoids → Straight-Through Estimator

confidence 90% · PNNs employ continuous convex combinations of archetypes, avoiding discrete assignment operations... without EMA updates, a commitment loss, or a straight-through estimator

Polytopal Neural Networks → reducesto → Vector Quantized Variational Autoencoders

confidence 90% · Replacing the simplex with one-hot coordinates reduces the PNN to a VQ model whose codebook is grounded in the corpus

Vector Quantized Variational Autoencoders → requires → Straight-Through Estimator

confidence 88% · VQ-VAE relies on... auxiliary optimization heuristics such as the straight-through estimator

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Understanding how deep neural networks process information remains a central challenge. Existing interpretability methods often compromise structural fidelity, rely on prespecified corpora, or explain models post-hoc. We propose Polytopal Neural Networks (PNNs), a framework that extracts distinct layer-wise aspects by enforcing a polytope-based structure that is used directly in subsequent information processing. We scale our approach using learned corpus representations and an amortized simplex inference procedure and highlight how the framework also gives a direct route to vector quantized (VQ) training. In PNNs, observations are explicitly described by their alignment with layer-specific aspects. Empirical results show that imposing polytopal constraints on neural network representations preserves meaningful structures in the latent space with minimal degradation in performance, favorable compressed representations when compared to VQ representations in unsupervised learning, while also providing a performant new approach to VQ deep learning training. Our findings suggest that deep networks can enforce interpretable polytope-based representations, offering a principled path toward more transparent AI systems with minimal performance compromise.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

85,012 characters extracted from source content.

Expand or collapse full text

Abstract Understanding how deep neural networks process information remains a central challenge. Existing interpretability methods often compromise structural fidelity, rely on prespecified corpora, or explain models post-hoc. We propose Polytopal Neural Networks (PNNs), a framework that extracts distinct layer-wise aspects by enforcing a polytope-based structure that is used directly in subsequent information processing. We scale our approach using learned corpus representations and an amortized simplex inference procedure and highlight how the framework also gives a direct route to vector quantized (VQ) training. In PNNs, observations are explicitly described by their alignment with layer-specific aspects. Empirical results show that imposing polytopal constraints on neural network representations preserves meaningful structures in the latent space with minimal degradation in performance, favorable compressed representations when compared to VQ representations in unsupervised learning, while also providing a performant new approach to VQ deep learning training. Our findings suggest that deep networks can enforce interpretable polytope-based representations, offering a principled path toward more transparent AI systems with minimal performance compromise. The Polytopal Neural Network A. Emilie J. Wedenborg Technical University of Denmark Anders V. Nørskov Technical University of Denmark Teresa Dorszewski Technical University of Denmark Kristoffer Wickstrøm UiT The Arctic University of Norway Morten Mørup Technical University of Denmark Figure 1: Two PathMNIST test images traced through two PNN layers (K=10K=10). Each layer passes a convex combination of archetypes to the next layer. Numbers denote mixture weights; each archetype is illustrated by its two nearest training images. 1 Introduction The increasing deployment of deep learning systems in scientific, industrial, and societal contexts has intensified the demand for methods that make their internal computations transparent and trustworthy (Longo et al., 2024). Yet, many existing approaches fail to explain how neural networks structure and transform information across layers. A substantial body of explainable AI (XAI) research has focused on post-hoc analyses of input-output behavior, including saliency-based attribution methods and activation visualizations (Bach et al., 2015; Selvaraju et al., 2017; Uendes et al., 2025). Although such techniques can identify influential input regions, they offer limited insight into the internal mechanisms directing the representation learning within the network. This limitation is especially noticeable given consistent empirical evidence that deep networks develop structured, hierarchical representations, where early layers encode low-level attributes such as color and texture, while deeper layers assemble these primitives into higher-order concepts and object-level abstractions (Bau et al., 2017; Dorszewski et al., 2025; Vielhaben et al., 2024). Despite their central role in learning and inference, these latent representations remain largely unconstrained and difficult to interpret, restricting their usefulness for mechanistic understanding, model validation, and scientific discovery. We introduce Polytopal Neural Networks (PNNs), a class of architectures in which latent representations are explicitly constrained to lie on a polytope by use of simplex constraints (i.e., imposing non-negativity and sum-to-one constraints). This induces a structured, interpretable latent geometry in which each representation is expressed as a convex combination of learned archetypes defining the corners of the polytope (Figure 1). Crucially, this constraint is enforced during training and shapes the network’s internal computations, rather than serving as a post-hoc explanation mechanism (Wedenborg et al., 2026). PNNs are related to corpus-based explanation methods that express predictions as mixtures of exemplars (Crabbe et al., 2021). However, these approaches rely on a fixed, user-defined corpus and aim to explain individual predictions after training. In contrast, PNNs learn the archetypal corpus end-to-end and enforce simplex constraints directly on the latent space, ensuring that predictions are intrinsically consistent with the interpretable polytope representations imposed at every layer. We use interpretability to mean the extent to which a model’s internal representations and computations can be understood in terms of identifiable components and their contributions. An explanation is intrinsic when those components are part of the computation producing the output, rather than estimated afterward. It is local when it describes an individual input and global when it describes structure shared across inputs. PNNs provide intrinsic local explanations through each input’s mixture weights over archetypes, and global structure through the shared archetypes that define each layer’s polytope. This guarantees a faithful description of the constrained representations. PNN’s use of archetypes as reference profiles is motivated by geometric accounts of cognition based on similarity and betweenness (Gärdenfors and Williams, 2001), and by evidence that contrastive categorization can favor extreme, idealized representatives along distinguishing dimensions (Davis and Love, 2010). This framing connects the PNN to classical Archetypal Analysis (A) (Cutler and Breiman, 1994; Mørup and Hansen, 2012), which models data as mixtures of extremal points lying on the convex hull. A based models are well known for their interpretability, as both the archetypes and the mixture coefficients allow intuitive explanations. Integrating A constraints into deep architectures, PNNs bridge the gap between interpretable, prototype-based modeling and high-capacity representation learning, ensuring the results remain strictly grounded in the original data. The proposed framework also relates to vector quantization (VQ) methods, most notably vector quantized variational autoencoders (VQ-VAEs) (van den Oord et al., 2017), which replaces continuous latent manifolds with finite representations to promote compression, disentanglement, and efficient generative modeling. Both VQ-VAE and PNNs reduce latent complexity by constraining representations to a limited set of learned components. However, their mechanisms differ: VQ-VAE relies on hard nearest-neighbor assignments, exponential moving average (EMA) updates, and auxiliary optimization heuristics such as the straight-through estimator and commitment losses (van den Oord et al., 2017), whereas PNNs employ continuous convex combinations of archetypes, avoiding discrete assignment operations while preserving a compact and interpretable latent structure. In summary, PNNs combine the strengths of Archetypal Analysis, Vector Quantization, and deep representation learning. Each layer decomposes activations into convex mixtures of learned archetypes, forming a sequence of sequential polytopes that jointly explain and reconstruct the model’s internal representations. This polytopal structure enforces geometric consistency and provides an intrinsic explanation of each prediction through archetypal coefficients. Figure 2: The Polytopal Neural Network: Points in the input space (left) are used to learn data driven corpus points (×) used to span the polytopes in the layers of the PNN. Each layer maps the observations into a low-dimensional simplex-structured latent space (polytope), where convex combinations of the encoded corpus points define the layer specific archetypal vertices. A subsequent latent transformation refines the geometric organization forming a refined polytope. Finally, points are decoded into the output space (right), where the resulting representation exhibits improved separation and structure. Colored points indicate different data groups, and black vertices denote learned archetypes. Specifically, our main contributions are: (i) Polytopal representation learning. A layer-wise framework that constrains latent features to lie within learned polytopes, with A constrained representations enforced exactly at every constrained layer. (i) Integrated interpretability. The coordinates used to explain a representation are also the coordinates that the next layer receives. (i) Scalable inference. A batched learning framework with (a) a compact corpus learned jointly with the network and updated from mini-batches and (b) amortized simplex inference. (iv) A direct optimization route for VQ. Replacing the simplex with one-hot coordinates reduces the PNN to a VQ model whose codebook is grounded in the corpus and is trained without EMA updates, a commitment loss, or a straight-through estimator. 2 Related Work Archetypal Analysis (A), introduced by Cutler and Breiman (1994), has been incorporated into deep learning frameworks, including sparse autoencoder–based models (Fel et al., 2025a) and other deep architectures (Keller et al., 2019; Keller et al., 2021; Milite et al., 2025; Wieser et al., 2025). Most existing non-linear A approaches place archetypes only at a single latent bottleneck of an autoencoder, and several enforce the archetypal structure only through a penalty (Keller et al., 2019; Keller et al., 2021; Wieser et al., 2025), so the archetypes act as a geometric prior rather than a strict constraint, limiting interpretability and consistency across layers. Early work such as AAnet proposed learning nonlinear archetypal spaces using deep autoencoders that map data into a latent simplex, demonstrating improved recovery of extremal structure in nonlinear domains (van Dijk et al., 2019). Recent research highlights A as a tool for explainable representation learning, from stabilizing sparse autoencoders (Fel et al., 2025a) to identifying task-relevant concepts in vision transformers (Fel et al., 2025b). Post hoc probing confirms that stable archetypes exist within pretrained networks (Wedenborg et al., 2026) but leaves the network unchanged. Deep A architectures, in turn, place archetypes at a single bottleneck. There remains a need for architectures that enforce simplex-constrained representations throughout the network depth with explanations fully consistent with the networks’ information processing. An alternative paradigm learns prototypical summaries jointly with the model. Prototype- and clustering-based approaches encourage representation space to organize around learned centroids or parts, facilitating concept-level explanations (Chen et al., 2024; Liang et al., 2023; Wang et al., 2023; Zhou and Wang, 2024). ProtoPNet (Chen et al., 2019) assigns prototypes to classes, while PIP-Net (Nauta et al., 2023) learns prototype–class associations through supervised classification after self-supervised pretraining. The PNN instead constrains intermediate representations to convex combinations of K archetypes without class assignments, using only the task loss. This constraint applies at multiple classifier depths, in unlabeled autoencoders, and to tokens from pretrained backbones. We therefore compare against the same unconstrained architecture and other representation bottlenecks: projection layers, vector quantization, and the Dirichlet VAE. Centroid-oriented representations emphasize typical instances and are effective for class- or part-centered interpretations. Notably, centroid-based representations are also explored in VQ-VAEs (van den Oord et al., 2017). By contrast, A targets extremal data geometry, decomposing space into vertices of a convex polytope and expressing each point as a convex mixture of these vertices (Cutler and Breiman, 1994; Hastie et al., 2009; Mørup and Hansen, 2012). This shift from central tendency (centroids) to extremal representation (archetypes) yields complementary interpretive advantages where archetypes capture boundary phenomena and diverse modes of variation that centroids can obscure. PNNs more closely resemble the Dirichlet VAE (Joo et al., 2020) in which the latent representation is projected onto the standard simplex with an associated Dirichlet prior imposed to characterize the continuum within the standard simplex. The PNN notably differs here by imposing a learned polytope as opposed to a fixed projection onto the standard simplex. The use of subspace projections of latent representations has also been widely adopted in deep learning using linear projection layers, see also (Hawkins-Hooker et al., 2018) and the recent generalization to arbitrary higher-order feature spaces (Morimoto and Huang, 2025). Whereas these procedures can produce lower-dimensional subspaces they do not promote explainable polytopal latent representations. 3 Methods Notation. Data and representations are observations as rows throughout. ∈ℝN×MX ^N× M holds N observations. At layer ℓ , ℓ∈ℝN×dℓZ ^N× d_ holds the representations before projection, ℓ∈ℝK×NC ^K× N is row-stochastic, and ℓ∈ℝN×KS ^N× K is row-stochastic. A row-stochastic matrix has non-negative entries and rows that sum to one. The archetypes are ℓ=ℓℓ∈ℝK×dℓA =C Z ^K× d_ and the projected representations are ℓ=ℓℓR =S A . 3.1 Archetypal Analysis and the Polytopal Neural Network A approximates the convex hull of the data by a polytope with K vertices, the archetypes, which are convex combinations of the data (Mørup and Hansen, 2012). It solves min,⁡‖−‖F2 _C,S\|X-SCX\|_F^2 (1) s.t. k≥0,k=1∀k,n≥0,n=1∀n, .t. c_k≥ 0,\ c_k1=1\ ∀ k,\ s_n≥ 0,\ s_n1=1\ ∀ n, where kc_k and ns_n are the kkth row of C and the nnth row of S. The rows of CX are the archetypes and S projects each observation onto the polytope they span. Archetypes are therefore idealized, data-grounded extremes, and each observation has a transparent decomposition in terms of these extremes. A deep network applies L layers fθℓf _θ and is trained by minimizing ℒ(,fθ())L(Y,f_θ(X)) with fθ=fLθL∘⋯∘f1θ1f_θ=f^L_θ^L ·s f^1_θ^1. Supervised learning uses labels as Y, and an autoencoder uses the inputs themselves. The PNN inserts a projection operator Δℓ _C after the chosen layers, fθ()=fθL f_θ(X)=f^L_θ^L ∘[ΔL−1L−1∘fθL−1L−1] [ ^L-1_C^L-1 f^L-1_θ^L-1 ] (2) ∘⋯∘[Δ11∘fθ11](), ·s [ ^1_C^1 f^1_θ^1 ](X), and learns θℓ,ℓ\θ ,C \ subject to ℓC being row-stochastic. With ℓZ the output of layer ℓ before projection, Δℓ(ℓ)=ℓℓℓ _C (Z )=S C Z , where ℓ=argmin⁡‖ℓ−ℓℓ‖F2 s.t. n≥0,n=1∀n.S = _S\|Z -SC Z \|_F^2\ s.t. \ s_n≥ 0,\ s_n1=1\ ∀ n. (3) This is a convex quadratic program (QP) per sample. We parameterize ℓ=softmax⁡(~ℓ)C = ( C ) row-wise with unconstrained logits ~ℓ C and train all parameters by back-propagation, treating ℓS as fixed when differentiating with respect to ℓC (Appendix B.1). An exact solve by an active-set method costs up to ⁡(K3)O(K^3) per sample. Sequential minimal optimization (SMO) (Wedenborg and Mørup, 2025) reduces a full pass to ⁡(K2)O(K^2) by updating two coordinates at a time in closed form. Storing X, every ℓZ , and ℓC still costs ⁡(NM+N∑ldℓ+NLK)O(NM+N _ld_ +NLK) in memory. 3.2 Scaling the PNN Scaling Memory Usage by Batched Streaming Updates of Data-Driven Corpus: We decouple the archetypes from the full dataset by learning a corpus of Nc≪N^c N points, c=cX^c=C^cX with row-stochastic c∈ℝNc×NC^c ^N^c× N. The corpus is encoded by the same layers as the data, c,ℓZ^c, , and defines the archetypes ℓ=ℓc,ℓA =C Z^c, with ℓ∈ℝK×NcC ^K× N^c. Let c~ki c_ki be the logits of corpus row k. For numerical stability we subtract the row maximum of all N observations, wki=exp⁡(c~ki−maxj⁡c~kj)w_ki= ( c_ki- _j c_kj), which leaves the softmax unchanged. For a mini-batch B and its complement ¬B B, corpus row k is kc=∑i∈Bwkii+∑i∈¬Bwkii∑i∈Bwki+∑i∈¬Bwki.x^c_k= _i∈ Bw_kix_i+ _i∈ Bw_kix_i _i∈ Bw_ki+ _i∈ Bw_ki. (4) The batch sums carry gradients. The non-batch sums are computed from the current logits without gradients, in chunks. A row outside the batch therefore receives no gradient on that step, but any update it received while in an earlier batch is reflected the next time the statistics are recomputed. Eq. (4) is the exact weighted average at every step, and gradient memory scales with |B||B| and NcN^c rather than with N resulting in a memory usage of ⁡(Nc|B|+NcM+|B|M+Nc∑ldℓ+NcLK)O(N^c|B|+N^cM+|B|M+N^c _ld_ +N^cLK). Amortized simplex inference. Although exact inference solves Eq. (3) for every sample in every forward pass at cost of ⁡(K3)O(K^3) reduced by SMO to ⁡(K2)O(K^2) this can still be prohibitive. We therefore instead learn a map gϕℓg_φ on the archetype correlations n=n(ℓ)⊤∈ℝKh_n=z_n(A ) ^K, ^n=softmax⁡(gϕℓ(n)),^n=^nℓ. s_n= (g_φ (h_n) ), r_n= s_nA . (5) The input of gϕℓg _φ is a layer normalization of nh_n together with log⁡std⁡(n) (h_n), because the projection depends on the scale of nh_n, which layer normalization removes. The task loss is computed through ^n r_n, but gϕℓg _φ is never trained on it. It is trained only on the projection residual ‖n−^nℓ‖2\|z_n- s_nA \|^2 with nz_n and ℓA detached, so it cannot deform the encoder, the head, the corpus, or the archetypes. It is fitted before the archetypes are trained and refit every ten epochs as the polytope moves. At test time the amortized coordinates can be refined by K rounds of batched SMO warm-started at ^n s_n (Appendix B.1). 3.3 Theoretical Properties For a fixed polytope, let A∈ℝK×dA ^K× d hold the archetypes as columns, G=AA⊤G=A , h=Azh=Az, and ΔK−1=s∈ℝK∣s≥0, 1⊤s=1 ^K-1=\s ^K s≥ 0,\ 1 s=1\. Expanding 12‖z−A⊤s‖2 12\|z-A s\|^2 and dropping the term independent of s gives Φ⁡(h)=argmins∈ΔK−1⁡12s⊤Gs−h⊤s. (h)= _s∈ ^K-1 12s Gs-h s. (6) Lemma 1. If G is positive definite, the minimizer is unique and Φ is a continuous piecewise affine map from ℝKR^K to ΔK−1 ^K-1. G is positive definite if and only if A has full row rank, which requires K≤dK≤ d. Theorem 1. If G is positive definite, there is a finite ReLU network fθf_θ such that Φ⁡(h)=sparsemax⁡(fθ(h)) (h)= (f_θ(h)) for all h∈ℝKh ^K. Proofs are in Appendix A. The theorem establishes exact representability for a fixed polytope with sparsemax; we use softmax (Table 11). For CNNs and AEs, we refit the amortizer every ten epochs and apply K SMO rounds per forward pass. Appendix B.2 evaluates projection fidelity. 3.4 Token-level PNN. For pretrained vision backbones we place the polytopal layer on the spatial tokens of the last feature map. Each token n,tz_n,t, t=1,…,Tt=1,…,T, of image n is layer-normalized and linearly projected to 256256 dimensions, and all tokens share one polytope A. The amortizer assigns each token its own coordinates n,t∈ΔK−1s_n,t∈ ^K-1, and a linear classifier acts on the token mean of the projections, n=(1T∑tn,t)⊤+.y_n= ( 1T _ts_n,tA )W +b. (7) The pooled representation therefore stays in the polytope, with coordinates ¯n=1T∑tn,t s_n= 1T _ts_n,t. Because the head is affine, the logits equal the mean of the per-token logits n,t⊤+s_n,tAW +b, so every token makes an exact, additive contribution to the prediction. Correspondingly, the corpus consists of individual (image, token) feature rows. 3.5 Generalizations of PNN Interestingly, the developed PNN methodology naturally reduces to alternative well-known modeling procedures by changing the constraints used in (3). Figure 3: Performance as a function of the number of components across datasets. Shaded regions indicate ± one standard deviation over runs. Top row: VQ reconstruction MSE, with improved reconstruction quality reflected by lower values. Middle row: Classification accuracy. Bottom row: Autoencoder reconstruction MSE. Vector Quantization (VQ): Whereas PNNs approximate latent representations via convex combinations of learned archetypes, VQ-VAEs use a finite set of learnable embedding vectors. Unlike PNNs, where archetypes are learned via gradient-based optimization, the VQ-VAE codebook vectors do not receive gradients directly from the loss. To enable backpropagation through the non-differentiable quantization step, the Straight-Through Estimator (STE) (Bengio et al., 2013) is employed, passing gradients from the codebook reconstruction ℓR to ℓZ unchanged. The resulting training objective consists of the task loss and a commitment loss, ℒ=ℒtask+β∑ℓ‖ℓ−sg⁡[ℓ]‖F2,L=L_task+β _ \|Z -sg[R ]\|_F^2, (8) where sg⁡[⋅]sg[·] is the stop gradient and the commitment term encourages encoder outputs to remain close to their assigned codebook entries and limits latent volume (van den Oord et al., 2017), see also Appendix G.3.1 for further details. Interestingly, the PNN framework provides a direct gradient-based optimization procedure for VQ deep learning by defining ℓR as the solution to minimizeℓ S minimize ‖ℓ−ℓℓc,ℓ‖F2 \|Z -S C Z^c, \|_F^2 (9) subject to to ℓn∈0,1K,ℓn=1,∀n. _n∈\0,1\^K, _n1=1, ∀ n. ns_n is a one-hot vector indicating centroid assignment. Notably, the optimal C used to define the cluster centroids by minimizing the least squares error is accordingly given by =(ℓ⊤ℓ)−1ℓ⊤C=(S S )^-1S , provided every centroid is assigned at least one corpus point. Empirically we found this to always be true, but standard VQ-VAE measures to prevent codebook collapse are also applicable to the PNN-VQ. The equation corresponds to defining the centroids as the average of the observations of the learned corpus c,ℓZ^c, assigned to the centroid thus satisfying the non-negativity and sum to one constraints making this optimization consistent with the PNN formulation. Consequently, this approach can be directly implemented using the existing constraints on ℓC based on the softmax reparameterization, whereas ns_n, instead of solved iteratively, can be estimated directly with one non-zero element p by (nℓ)p=1 where p=argmink⁡‖nℓ−kc,ℓ‖22. (s _n)_p=1 where p=argmin_k\|z _n-c_k\,Z^c, \|_2^2. (10) Thereby, the PNN provides a simple gradient-based training objective without the need to impose a commitment loss. The encoder in the PNN-VQ learns through the corpus features c,ℓZ^c, , and the layers after the projection learn through ℓR . Projection Layers (PL): The use of Projection Layers (PL) in which each layer’s activation is projected onto a subspace before further processing is well established within deep learning to perform dimensionality reduction and compression, see also Hawkins-Hooker et al. (2018) and the recent generalization to arbitrary higher order feature spaces Morimoto and Huang (2025). Such projection layers can naturally be implemented within the PNN framework by imposing unconstrained representations minimizeℓ‖ℓ−ℓℓc,ℓ‖F2subject tonℓ∈ℝK∀n. array[]rl S minimize&\|Z -S C Z^c, \|_F^2\\ subject to& _n ^K ∀ n. array (11) Consequently, ns_n is trivially given as the closed-form solution to the normal equation nℓ=nℓc,ℓ⊤ℓ⊤(ℓc,ℓc,ℓ⊤ℓ⊤)−1. _n=z_n Z^c, C (C Z^c, Z^c, C )^-1. (12) whereas ℓC is optimized unconstrained without the softmax mapping. We will denote the above generalizations PNN-VQ and PNN-PL, respectively. 4 Results and Discussion Experimental setup. We evaluate PNNs on image classification and reconstruction, with additional experiments on tabular classification. The main image benchmarks are MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), and EuroSAT (Helber et al., 2019). These experiments use a CNN classifier with polytopal layers after its two fully connected hidden layers and a convolutional autoencoder (AE) with a polytopal bottleneck. We further evaluate the PNN-CNN on PathMNIST, DermaMNIST, and BloodMNIST from MedMNIST v2 (Yang et al., 2023). For larger backbones and token representations, we evaluate ImageNet-100 (Tian et al., 2020) and ImageNet-1k (Deng et al., 2009) using ConvNeXt-T (Liu et al., 2022), initialized from ImageNet-1k weights and fine-tuned on each benchmark. The PNN projects each spatial token onto a shared polytope before averaging the resulting representations and applying a linear classifier. We also evaluate a PNN head on the pooled features of a frozen compact vision transformer (Hassani et al., 2022) on CIFAR-10. Additional MLP results on Digits (Alpaydin and Kaynak, 1998), Iris (Fisher, 1936), Wine (Aeberhard and Forina, 1992), and Breast Cancer (Wolberg et al., 1993) are reported in Appendix G.1. The main CNN and AE experiments compare PNN with the corresponding unconstrained networks and the PNN-VQ and PNN-PL variants, enabling comparison between polytopal, subspace-based, and codebook-based representations. The AE comparisons additionally include VQ-VAE (van den Oord et al., 2017) and a variational autoencoder with a Dirichlet bottleneck (Joo et al., 2020). The MedMNIST and larger-backbone experiments compare PNNs with their corresponding unconstrained models. Dataset splits, architectures, training procedures, and experiment-specific settings are provided in Appendix E. We first evaluate the proposed scalable inference framework by comparing mini-batched training with full-data (non-batched) updates. Mini-batching with batch-wise corpus updates (Section 3.2) achieves accuracy comparable to full-data training (see Appendix Table 7). We also evaluate the proposed amortized simplex inference procedure and find that with periodic realignment this serves as a fast and accurate replacement for the full QP projection (Appendix B.2). To validate our direct training strategy for PNN-VQ, we compare it to standard VQ training with a commitment loss (van den Oord et al., 2017). As shown in Figure 3, PNN-VQ achieves comparable, and in several cases lower test mean-squared-error (MSE), with detailed results reported in Table 12. We next evaluate classification with a CNN backbone. In Figure 3, the PNN generally approaches the accuracy of the unconstrained CNN as the number of archetypes increases. The encoder dimension remains fixed, while larger values of K allow the constrained representation to use more archetypes. The largest improvements occur when moving away from the smallest archetype counts, although the remaining gap to the unconstrained model varies across datasets. This comparison extends to the three MedMNIST datasets. With K=25K=25, the PNN achieves test accuracies of 89.1% on PathMNIST, 77.4% on DermaMNIST, and 98.1% on BloodMNIST, compared with 90.4%, 78.6%, and 98.3% for the corresponding unconstrained CNNs (Table 9). Increasing K from 10 to 25 improves the mean accuracy on PathMNIST and DermaMNIST, while BloodMNIST remains at 98.1%. We evaluate the proposed method in an unsupervised setting using an AE. Reconstruction quality is measured using MSE, with a single PNN layer structuring the bottleneck under the PNN constraints. As shown in Figure 3, the PNN-AE shows similar tendencies as the PNN-CNN and generally performs competitively to the less constrained variants. PNN-AE generally achieves lower reconstruction MSE than DirVAE. The unconstrained AE and PNN-PL generally achieve lower MSE than PNN-AE, indicating a reconstruction cost associated with the convex constraint that generally shrinks with K. We also compare softmax and sparsemax as the amortizer output map (Table 11). Across MNIST, FashionMNIST, and CIFAR-10, the two variants achieve similar reconstruction MSE under the same refinement protocol. Token-level PNNs on pretrained ConvNeXt-T retain most of the unconstrained model’s accuracy (Table 1). On ImageNet-100, accuracy varies little for K≥100K≥ 100, with overlapping error bars. Full results are in Appendix F. Next, the PNN module is added to a frozen compact transformer where PNN acts on the pooled representation. On CIFAR-10, its PNN head achieves 94.46% test accuracy with K=10K=10, compared to 94.47% for the unconstrained classifier (Table 1). Table 1: Larger backbones and label spaces. Accuracy (%) of the unconstrained model and of the PNN, and the share of unconstrained accuracy the PNN retains. IN-100 and IN-1k are ImageNet-100 and ImageNet-1k, whose backbones are fine-tuned. Full results in Appendix F. Setting Classes Unconstr. K PNN Kept (%) IN-100, ConvNeXt-T 100 94.6± 0.1 25 91.7± 0.5 96.9 IN-1k, ConvNeXt-T 1000 81.2± 0.0 128 78.6± 0.2 96.9 CIFAR-10, compact ViT 10 94.47± 0.21 10 94.46± 0.23 100.0 Figure 4shows the latent space of a PNN-AE with K=3K=3 on FashionMNIST using multidimensional scaling, together with the four training images with the largest weight on each archetype. The displayed examples form recognizable groups of lower-body clothing, upper-body clothing, and footwear. This gives a qualitative illustration of how the learned polytope organizes the representations in this model. Figure 4: PNN-AE (K=3K=3) on FashionMNIST. Left: latent space projected via MDS; crosses indicate archetypes. Right: the four training images with the largest weight on each archetype, showing coherent groups (upper body, lower body, footwear). The token-based ImageNet models allow the same archetypal representation to be inspected spatially. Figure 5 illustrates how individual tokens are represented using a shared polytope before their reconstructed features are averaged for classification. Figure 6 shows token predictions and the spatial weights of selected archetypes for individual images. The associated training examples provide visual references for interpreting these archetypes. Each token is represented by its coordinates inside the polytope. The image-level margin can therefore be described as a sum of archetype scores, each weighted by the archetype’s share of the image (Figure 6). The snake example is described by two archetypes. Most of the tokens fall near an archetype whose training images show a small snake within a scene. The head tokens are primarily described by an archetype whose training images shows the hognose snake scale pattern. From the token votes we see that the head tokens have higher weight which influences the predictions. Unlike prototype networks (Chen et al., 2019; Nauta et al., 2023), which score patches by similarity to learned prototypes, the PNN represents each token as a point in the polytope spanned by the archetypes, so the decomposition uses the token’s own coordinates and requires no prototype-specific losses. Figure 5: Token-level PNN on ImageNet-100 with ConvNeXt-T and 200 archetypes. Each spatial token is represented as a convex combination of shared archetypes. A linear classifier reads the mean of the reconstructed token features. Two archetypes associated with the predicted and competing classes are illustrated using training images. The outlined tokens have the largest weights on the displayed archetypes. Figure 6: Token predictions and archetype weights for two ImageNet-100 images. Each row shows the input, token predictions, and spatial weights of selected archetypes associated with the predicted and competing classes. Training examples are used to illustrate the archetypes. Blue denotes the predicted class, orange the competing class, and gray other classes. The counts below the token-prediction panels show how many tokens favor each displayed class. The image prediction depends on the averaged class scores. From a computational perspective, PNNs add per-iteration cost through corpus-based archetype construction and simplex inference. This cost depends on the batch size, representation dimension, corpus size, number of archetypes, and number of tokens. The inference procedure differs across experiments. The CNN and AE use the amortized simplex infernece procedure followed by K SMO refinement rounds during training and evaluation. The ImageNet models use the amortizer without this refinement in the task path. The larger-scale experiments demonstrate that the construction can be used with spatial token representations, while its computational overhead depends on the chosen inference procedure. Costs are further examined in Appendix E.7. From an explainability perspective, PNNs expose the archetypal decomposition of a representation used in the forward computation. The FashionMNIST and ImageNet examples show how this decomposition can be inspected at the level of an entire input or individual spatial tokens. The archetypes are learned without predefined concept annotations, and representative training examples provide a way to examine their visual meaning. For the ImageNet model’s linear classifier, the archetype contributions can also be traced directly to the class scores. With nonlinear downstream networks, the coordinates describe the intermediate representation, and their effect on the output must be assessed through that subsequent computation. We test this by removing the archetype with the largest simplex coordinate, renormalizing the remaining coordinates, and compared this with removing a random archetype. For the CNN, this changes more predictions than removing a random archetype (Figures 8 and 9). 5 Conclusion We have introduced the Polytopal Neural Network, a novel architecture that prioritizes interpretability and trustworthiness in deep learning by rigorously enforcing A constrained latent representations . Through extensive evaluation across multiple network architectures and diverse datasets, we have demonstrated that PNNs produce inherently interpretable predictions with semantically meaningful explanations. Unlike post-hoc explanation methods, PNNs provide a principled guarantee that the explanations reflect the actual decision-making process learned during training, addressing a fundamental challenge in XAI. Our analysis reveals that PNNs can be understood as a continuous quantization method, establishing a natural connection to vector quantization approaches. In this framework, the learned archetypes form a semantically meaningful codebook that enables both effective classification and transparent reasoning. References Aeberhard and Forina (1992) S. Aeberhard and M. Forina Wine. Note: UCI Machine Learning RepositoryDOI: 10.24432/C5PC7J Cited by: §4. Alpaydin and Kaynak (1998) E. Alpaydin and C. Kaynak Optical recognition of handwritten digits. Note: UCI Machine Learning RepositoryDOI: 10.24432/C50P49 Cited by: §4. Arora et al. (2016) R. Arora, A. Basu, P. Mianjy, and A. Mukherjee Understanding deep neural networks with rectified linear units. CoRR abs/1611.01491. External Links: Link, 1611.01491 Cited by: §A.2. Bach et al. (2015) S. Bach, A. Binder, G. Montavon, F. Klauschen, K. Müller, and W. Samek On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10 (7), p. e0130140. Cited by: §1. Bau et al. (2017) D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba Network dissection: quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6541–6549. Cited by: §1. Bengio et al. (2013) Y. Bengio, N. Léonard, and A. Courville Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. Cited by: §3.5. Chen et al. (2019) C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems 32. Cited by: §2, §4. Chen et al. (2024) G. Chen, X. Li, Y. Yang, and W. Wang Neural clustering based visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5714–5725. Cited by: §2. Crabbe et al. (2021) J. Crabbe, Z. Qian, F. Imrie, and M. van der Schaar Explaining Latent Representations with a Corpus of Examples. Advances in Neural Information Processing Systems 34, p. 12154–12166. Cited by: §1. Cutler and Breiman (1994) A. Cutler and L. Breiman Archetypal Analysis. Technometrics 36 (4), p. 338–347. External Links: Document, ISSN 0040-1706 Cited by: §1, §2, §2. Davis and Love (2010) T. Davis and B. C. Love Memory for category information is idealized through contrast with competing options. Psychological Science 21 (2), p. 234–242. Note: PMID: 20424052 External Links: Document, Link, https://doi.org/10.1177/0956797609357712 Cited by: §1. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , p. 248–255. External Links: Document Cited by: §E.1, §4. Dorszewski et al. (2025) T. Dorszewski, L. Tětková, R. Jenssen, L. K. Hansen, and K. K. Wickstrøm From colors to classes: emergence of concepts in vision transformers. arXiv preprint arXiv:2503.24071. Cited by: §1. Fel et al. (2025a) T. Fel, E. S. Lubana, J. S. Prince, M. Kowal, V. Boutin, I. Papadimitriou, B. Wang, M. Wattenberg, D. E. Ba, and T. Konkle Archetypal SAE: adaptive and stable dictionary learning for concept extraction in large vision models. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 16543–16572. Cited by: §2, §2. Fel et al. (2025b) T. Fel, B. Wang, M. Lepori, M. Kowal, A. Lee, R. Balestriero, S. Joseph, E. Lubana, T. Konkle, D. Ba, and M. Wattenberg Into the rabbit hull: from task-relevant concepts in dino to minkowski geometry. External Links: Document Cited by: §2. Fisher (1936) R. A. Fisher Iris. Note: UCI Machine Learning RepositoryDOI: 10.24432/C56C76 Cited by: §4. Gärdenfors and Williams (2001) P. Gärdenfors and M. Williams Reasoning about categories in conceptual spaces. In Proceedings of the 17th International Joint Conference on Artificial Intelligence - Volume 1, IJCAI’01, San Francisco, CA, USA, p. 385–392. External Links: ISBN 1558608125 Cited by: §1. Hassani et al. (2022) A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, and H. Shi Escaping the big data paradigm with compact transformers. External Links: 2104.05704, Link Cited by: §E.2, §4. Hastie et al. (2009) T. Hastie, R. Tibshirani, and J. Friedman The elements of statistical learning: data mining, inference, and prediction. 2nd edition, Springer Series in Statistics, Springer New York, NY. External Links: ISBN 978-0-387-84857-0, Document Cited by: §2. Hawkins-Hooker et al. (2018) A. Hawkins-Hooker, H. Kenlay, and J. Reid Projection layers improve deep learning models of regulatory dna function. BioRxiv, p. 412734. Cited by: §2, §3.5. Helber et al. (2019) P. Helber, B. Bischke, A. Dengel, and D. Borth EuroSAT: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7), p. 2217–2226. External Links: Document Cited by: §4. Joo et al. (2020) W. Joo, W. Lee, S. Park, and I. Moon Dirichlet variational autoencoder. Pattern Recognition 107, p. 107514. Cited by: §E.3, §2, §4. Keller et al. (2019) S. M. Keller, M. Samarin, M. Wieser, and V. Roth Deep archetypal analysis. In Pattern Recognition, G. A. Fink, S. Frintrop, and X. Jiang (Eds.), Cham, p. 171–185. External Links: ISBN 978-3-030-33676-9 Cited by: §2. Keller et al. (2021) S. Keller, M. Samarin, F. Arend Torres, M. Wieser, and V. Roth Learning extremal representations with deep archetypal analysis. International Journal of Computer Vision 129, p. 1–16. External Links: Document Cited by: §2. Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical report University of Toronto. Note: CIFAR‑10 dataset Cited by: §4. LeCun et al. (2010) Y. LeCun, C. Cortes, and C. J. C. Burges MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist/. Cited by: §4. Liang et al. (2023) J. Liang, Y. Cui, Q. Wang, T. Geng, W. Wang, and D. Liu Clusterfomer: clustering as a universal visual learner. Advances in neural information processing systems 36, p. 64029–64042. Cited by: §2. Liu et al. (2022) Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie A ConvNet for the 2020s . In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , Los Alamitos, CA, USA, p. 11966–11976. External Links: ISSN , Document, Link Cited by: §4. Longo et al. (2024) L. Longo, M. Brcic, F. Cabitza, J. Choi, R. Confalonieri, J. D. Ser, R. Guidotti, Y. Hayashi, F. Herrera, A. Holzinger, R. Jiang, H. Khosravi, F. Lecue, G. Malgieri, A. Páez, W. Samek, J. Schneider, T. Speith, and S. Stumpf Explainable artificial intelligence (xai) 2.0: a manifesto of open challenges and interdisciplinary research directions. Information Fusion 106, p. 102301. External Links: ISSN 1566-2535, Document Cited by: §1. Martins and Astudillo (2016) A. F. T. Martins and R. F. Astudillo From softmax to sparsemax: a sparse model of attention and multi-label classification. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, p. 1614–1623. Cited by: §A.2. Milite et al. (2025) S. Milite, G. Caravagna, and A. Sottoriva MIDAA: deep archetypal analysis for interpretable multi-omic data integration based on biological principles. Genome Biology 26 (1), p. 90. Cited by: §2. Morimoto and Huang (2025) T. Morimoto and S. Huang TensorProjection layer: a tensor-based dimension reduction method in deep neural networks. Neurocomputing, p. 131695. Cited by: §2, §3.5. Mørup and Hansen (2012) M. Mørup and L. K. Hansen Archetypal analysis for machine learning and data mining. Neurocomputing 80, p. 54–63. External Links: Document, ISSN 0925-2312 Cited by: §1, §2, §3.1. Nauta et al. (2023) M. Nauta, J. Schlötterer, M. van Keulen, and C. Seifert PIP-net: patch-based intuitive prototypes for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2744–2753. Cited by: §2, §4. Netzer et al. (2011) Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng SVHN: the street view house numbers dataset. Note: http://ufldl.stanford.edu/housenumbers/ Cited by: §4. Selvaraju et al. (2017) R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra Grad-cam: visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, p. 618–626. Cited by: §1. Tian et al. (2020) Y. Tian, D. Krishnan, and P. Isola Contrastive multiview coding. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI, Berlin, Heidelberg, p. 776–794. External Links: ISBN 978-3-030-58620-1, Link, Document Cited by: §E.1, §4. Uendes et al. (2025) B. Uendes, S. Yu, and M. Hoogendoorn Start smart: leveraging gradients for enhancing mask-based XAI methods. In The Thirteenth International Conference on Learning Representations, Cited by: §1. van den Oord et al. (2017) A. van den Oord, O. Vinyals, and K. Kavukcuoglu Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA, p. 6309–6318. External Links: ISBN 9781510860964 Cited by: §E.3, §1, §2, §3.5, §4, §4. van Dijk et al. (2019) D. van Dijk, D. Burkhardt, M. Amodio, A. Tong, G. Wolf, and S. Krishnaswamy Finding archetypal spaces using neural networks. arXiv preprint arXiv:1901.09078. Cited by: §2. Vielhaben et al. (2024) J. Vielhaben, D. Bareeva, J. Berend, W. Samek, and N. Strodthoff Beyond scalars: concept-based alignment analysis in vision transformers. arXiv preprint arXiv:2412.06639. Cited by: §1. Wang et al. (2023) W. Wang, C. Han, T. Zhou, and D. Liu Visual recognition with deep nearest centroids. In International Conference on Learning Representations (ICLR), Cited by: §2. Wedenborg and Mørup (2025) A. E. J. Wedenborg and M. Mørup Archetypal analysis for binary data. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1–5. External Links: Document Cited by: §B.1, §B.1, §3.1. Wedenborg et al. (2026) A. E. J. Wedenborg, T. Dorszewski, L. K. Hansen, K. K. Wickstrøm, and M. Mørup Explaining latent representations of neural networks with archetypal analysis. In Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL), H. Kim, A. Ramírez Rivera, and B. Ricaud (Eds.), Proceedings of Machine Learning Research, Vol. 307, p. 448–468. External Links: Link Cited by: §1, §2. Wieser et al. (2025) M. Wieser, D. Siegismund, and S. Steigele Revisiting deep archetypal analysis for phenotype discovery in high content imaging. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 3802–3811. Cited by: §2. Wolberg et al. (1993) W. Wolberg, O. Mangasarian, N. Street, and W. Street Breast cancer wisconsin (diagnostic). Note: UCI Machine Learning RepositoryDOI: 10.24432/C5DW2B Cited by: §4. Xiao et al. (2017) H. Xiao, K. Rasul, and R. Vollgraf Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. External Links: 1708.07747 Cited by: §4. Yang et al. (2023) J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni MedMNIST v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10 (1), p. 1–10. External Links: Document, ISSN 2052-4463 Cited by: §E.1, §G.2, §4. Zhou and Wang (2024) T. Zhou and W. Wang Prototype-based semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (10), p. 6858–6872. Cited by: §2. The Polytopal Neural Network Supplementary Materials Appendix A Proofs We use the notation of Section 3.3. The archetype matrix A∈ℝK×dA ^K× d is fixed, G=AA⊤G=A , and ΔK−1=s∈ℝK∣s≥0, 1⊤s=1 ^K-1=\s ^K s≥ 0,\ 1 s=1\. Write Fh(s)=12s⊤Gs−h⊤s.F_h(s)= 12s Gs-h s. (13) A.1 Proof of Lemma 1 Since G is positive definite, FhF_h is strictly convex. The simplex is compact and convex, so a minimizer exists and is unique for every h∈ℝKh ^K. Fix a nonempty index set ⊆1,…,KJ \1,…,K\ and set sc=0s_J^c=0. The stationarity and sum-to-one conditions on these coordinates are (G⊤0)(sλ)=(h1). pmatrixG_JJ&1\\ 1 &0 pmatrix pmatrixs_J\\ λ pmatrix= pmatrixh_J\\ 1 pmatrix. (14) Here λ is the multiplier for ⊤s=11 s=1. Because GG_JJ is positive definite, this system is invertible. Its solution s(h)s^J(h) and λ(h)λ^J(h) is therefore affine in h. This candidate satisfies the remaining KKT conditions exactly when s(h)≥0,(Gs(h)−h+λ(h))c≥0.s^J_J(h)≥ 0, (Gs^J(h)-h+λ^J(h)1 )_J^c≥ 0. (15) These inequalities are affine in h and define a closed polyhedral region. On this region the candidate is the unique minimizer, so Φ is affine there. Every minimizer has a nonempty support and satisfies the KKT conditions for that support. The regions therefore cover ℝKR^K. There are finitely many index sets, and uniqueness ensures that the affine formulas agree wherever their regions overlap. Finally, let hm→h_m→ h. Compactness of the simplex ensures that every subsequence of Φ⁡(hm) (h_m) has a convergent further subsequence. Passing to the limit in Fhm(Φ⁡(hm))≤Fhm(s)for every s∈ΔK−1F_h_m( (h_m))≤ F_h_m(s) every s∈ ^K-1 (16) shows that every such limit minimizes FhF_h. Uniqueness forces the limit to equal Φ⁡(h) (h). Thus Φ⁡(hm)→Φ⁡(h) (h_m)→ (h), establishing continuity. Hence Φ is continuous piecewise affine. ∎ A.2 Proof of Theorem 1 By Lemma 1, each coordinate of Φ is a continuous piecewise affine function with finitely many pieces. Every such scalar function admits an exact finite ReLU representation [Arora et al., 2016]. Representing each coordinate and stacking the resulting networks gives a finite ReLU network fθf_θ satisfying fθ(h)=Φ⁡(h)for all h∈ℝK.f_θ(h)= (h) all h ^K. (17) Sparsemax is the Euclidean projection onto the probability simplex [Martins and Astudillo, 2016], sparsemax⁡(v)=argmins∈ΔK−1⁡12‖s−v‖2. (v)= _s∈ ^K-1 12\|s-v\|^2. (18) It therefore leaves every point of ΔK−1 ^K-1 unchanged. Since Φ⁡(h)∈ΔK−1 (h)∈ ^K-1, sparsemax⁡(fθ(h))=sparsemax⁡(Φ⁡(h))=Φ⁡(h)for all h∈ℝK. (f_θ(h))= ( (h))= (h) all h ^K. (19) This proves the theorem. ∎ Appendix B On the simplex projection B.1 Quadratic Scaling of Projections via. SMO-Updates Instead of the QP based estimation producing the full gradient of ℓC accounting also for the ℓS dependencies on ℓC as given by (20): n⊤=(ℓnℓ)⊤(ℓℓnℓ⊤+λ)⊤(ℓℓℓ⊤ℓ⊤+λ),−1,r_n =(C z_n )_P (C Z z_n +λ 1)_P (C Z Z C + )_P,P^-1, (20) we can reduce the gradient computation of ℓC to only account for the direct influence that ℓC exerts on nr_n keeping ns_n fixed. Consequently, we propagate gradients only through the expression nℓ=nℓℓℓr_n =s _nC Z , treating nℓs _n as constant. This avoids having gradient computations rely on any cubic in scaling matrix inversion. To further reduce the cubic complexity of estimating ns_n we exploit the recently proposed sequential minimal optimization (SMO) procedure for A Wedenborg and Mørup [2025]. The SMO procedure only considers two coefficients (p,q)(p,q) at a time. Defining σiℓ=si,pℓ+si,qℓ,si,pℓ=σiℓαiℓ,si,qℓ=σiℓ(1−αiℓ), _i =s _i,p+s _i,q, s _i,p=σ _iα _i, s _i,q=σ _i(1- _i ), (21) the problem reduces to a one-dimensional quadratic function in αiℓ∈[0,1] _i ∈[0,1] redistributing the mass σiℓ _i between the two elements. Importantly, this has a simple closed-form solution given by αi⋆ℓ=clip[0,1](−σi(Hqpℓ−Hqqℓ)+di,pℓ−di,qℓσi(Hppℓ−2Hpqℓ+Hqqℓ)), _i = [0,1]clip (- _i(H _qp-H _q)+d _i,p-d _i,q _i(H _p-2H _pq+H _q) ), (22) where ℓ=(ℓℓ)(ℓℓ)⊤H =(C Z )(C Z ) , iℓ=−iℓ(ℓℓ)⊤+iℓℓd _i=-z _i(C Z ) +s _iH . Pairs with σi=0 _i=0 are skipped, as they contain no weight. Pairs (p,q)(p,q) are iterated in random order, optionally multiple times per batch in which it was found in Wedenborg and Mørup [2025] that convergence could be achieved with complexity ⁡(K2)O(K^2). All updates are performed outside the computational graph such that gradients are propagated only through archetype parameters ℓC , not through S and can be implemented trivially in parallel across the considered observations. This yields substantial speedups while preserving high-quality simplex-constrained projections. B.2 Fidelity of the Amortizer To test the fidelity of the Amortizer we replace the deployed amortizer with the exact SMO projection at test time. Figure 7 shows the resulting test MSE against the amortized test MSE. All 150 configurations lie on the diagonal. The amortizer therefore does not materially change what the trained network computes. Figure 7: Test MSE of the PNN-AE with the exact SMO projections vs. the Amortized version. Appendix C Interpretability of the Polytope To test the interpretability of the PNN, we test how the output depends on the archetypes, if the archetypes are useful extreme configurations, removing the top archetype should hurt performance more than removing archetypes at random. We therefore take every trained PNN-AE and remove one archetype from the deployed coordinates of 1000 test images. The remaining coordinates are renormalized and decoded by the same decoder. Figures 8 and 9 shows both increases against K. Figure 8: Share of PNN-CNN test predictions that change when the highest-weight archetype is removed and when a single archetype is removed at random, against the number of archetypes K. Mean ± standard deviation for five seeds. Figure 9: Increase in test MSE of the PNN-AE when the highest-weight archetype is removed and when a single archetype is removed at random, against the number of archetypes K. Mean ± standard deviation for five seeds. Appendix D Token-wise Representations To make the method applicable to larger scale models with larger backbones we implement a token based version of the PNN. It works by applying the projection along the feature dimension and applying it to each token of a sequence. A patch embedding produces =[1,…,T]∈ℝT×dZ=[z_1,…,z_T] ^T× d, and each token is projected onto the same polytope, t=tr_t=s_tA with =cA=CZ^c. Stacking gives =R=SA with ∈ℝT×KS ^T× K. In ImageNet, each corpus element is the cached feature vector of one training image and token. These vectors are recomputed from the current backbone every epoch, and column generation updates corpus membership. The same corpus is shared across positions. Appendix E Experimental Details E.1 Data MNIST, FashionMNIST, CIFAR-10 and SVHN use their official test splits. We hold out 10% of each training split for validation. EuroSAT uses a 70/10/20 training, validation and test split. The CNN splits are stratified by class, while the AE splits are unstratified. Each run uses its training seed for the split. MNIST and FashionMNIST images are 28×2828× 28, CIFAR-10 and SVHN images are 32×3232× 32, and EuroSAT images are 64×6464× 64. Classifier inputs use fixed channel normalization. CIFAR-10, SVHN and EuroSAT training batches use reflected random crops with padding 4 and horizontal flips with probability 0.5. MNIST and FashionMNIST use reflected random crops with padding 2 and no flips. Other experiments. PathMNIST, DermaMNIST and BloodMNIST from MedMNIST v2 [Yang et al., 2023] use their official splits at 64×6464× 64 and the classifier augmentation above. ImageNet-100 [Tian et al., 2020] is the class subset of ImageNet-1k [Deng et al., 2009]. Both train on random resized crops with flips and evaluate on a centre crop. The frozen transformer uses CIFAR-10 with 5000 training images held out for validation. E.2 Architectures The image encoder has three strided convolutions, Conv3×3,s=2(C,32)Conv_3× 3,s=2(C,32), Conv3×3,s=2(32,64)Conv_3× 3,s=2(32,64) and Conv3×3,s=2(64,128)Conv_3× 3,s=2(64,128), each followed by group normalization with 8, 16 and 32 groups and a ReLU. The feature map is flattened and linearly projected to ∈ℝ64z ^64. The classifier applies 1=ReLU⁡(W1)h_1=ReLU(W_1z), a PNN projection ~1 h_1, 2=ReLU⁡(W2~1)h_2=ReLU(W_2 h_1), a second PNN projection ~2 h_2, and a linear output layer. The first site forms archetypes from their encoded features. Before the second site, the corpus passes through the first PNN projection and the second fully connected layer, as the input images do. Each site forms its archetypes from the corpus features entering that site. The autoencoders use the same encoder, a PNN bottleneck, and a mirrored transposed-convolution decoder with group normalization, ReLU and a sigmoid output. Other experiments. The MedMNIST classifier is the CNN above with a 128-dimensional latent space. On ImageNet, ConvNeXt-T start from ImageNet-1k weights. Their spatial tokens are projected to 256 dimensions by a PCA-initialized linear layer, and the PNN projects every token (Appendix D). The reconstructions are averaged over tokens before a linear classifier. The unconstrained model applies the same projection to the pooled feature, and the width control uses 128 dimensions. The frozen-head experiment places the PNN after the attention pooling of a compact transformer [Hassani et al., 2022] and initializes its output layer from the backbone classifier. E.3 Model Variants PNN. The amortizer of Eq. (5) has two hidden layers of width max⁡(8K,128) (8K,128) and ReLU activations. The CNN and AE refine its detached output with K SMO rounds during both training and evaluation. PNN-VQ. One-hot coordinates by nearest archetype, with the reconstruction or task loss only. In the autoencoder, C is initialized from k-means clusters of the encoded corpus, using a softened form of (⊤)−1⊤(S S)^-1S from Section 3.5 with 95% of each row of C spread uniformly over its cluster and the remaining 5% spread uniformly over the other corpus points, and is then learned by the task gradient. PNN-PL. Unconstrained C and least squares coordinates. VQ-VAE. A codebook updated by an exponential moving average, the straight-through estimator, and the commitment loss β‖−sg⁡[]‖F2β\|Z-sg[R]\|_F^2 with β=0.25β=0.25 [van den Oord et al., 2017]. The codebook is initialized by k-means++ and unused entries are reinitialized. DirVAE. The encoder outputs positive concentrations. Training uses the inverse Gamma CDF approximation and normalizes the samples onto the simplex [Joo et al., 2020]. Evaluation uses the posterior mean. The loss adds βKLD−1∑kKL(Gamma(αk,1)∥Gamma(1,1)) _KLD^-1 _kKL(Gamma( _k,1)\,\|\,Gamma(1,1)) to mean pixel MSE, where D counts pixel values and βKL=0.01 _KL=0.01. E.4 Training Protocol The main CNN and AE grids use K∈3,5,10,15,20,25K∈\3,5,10,15,20,25\ and seeds 42,123,456,789,1024\42,123,456,789,1024\. Runs use AdamW, cosine learning-rate schedules and gradient-norm clipping. Table 2 lists epoch limits and initial learning rates. Early stopping can shorten each phase. Phase 1 loads the unconstrained network trained with the same seed. Before Phase 2, A on up to 20,000 encoded training images selects one anchor image per corpus point. The anchor starts with weight 0.25 and the remaining weight is spread uniformly over the training set. The archetypes are initialized on this corpus. Phase 2 trains the archetype weights and output layer, with the encoder frozen. The CNN also freezes its two inner fully connected layers. Phase 3 updates the task network after an initial period with the encoder frozen. The corpus logits are trained by the task loss through the mini-batch rows of Eq. (4). Amortizer fitting. The AE fits the amortizer by minimizing ‖z−s^A‖2\|z- sA\|^2, normalized by the mean squared feature magnitude. CNN warm starts and refits minimize coordinate MSE against targets computed with 2K2K batched SMO rounds. Features, archetypes and targets are detached during fitting. Between refits, each training step also takes one amortizer step on the normalized projection residual |−^|2/||2¯|z- sA|^2/ |z|^2, with z and A detached. This step updates only the amortizer, and the task loss never updates it. Refits run every ten epochs and after Phase 2. The AE also refits after Phase 3. Its final refit is retained only if the measured projection residual on training data does not increase. The CNN has no post-Phase-3 refit. Table 2: Hyperparameters of the main CNN and AE grids. Epoch counts are maxima and learning rates are initial values. CNN classifier Autoencoder Latent dimension 64 64 Corpus size 2048 2048 Baseline batch size 256 2048 Phase 2 and Phase 3 batch size 512 512 Unconstrained baseline, epochs 200 200 Phase 2 epochs 60 100 Phase 3 epochs 300 200 Backbone frozen at the start of Phase 3 30 epochs 20 epochs Learning rate, unconstrained baseline 10−310^-3 10−310^-3 Learning rate, archetypes in Phase 2 3⋅10−33· 10^-3 5⋅10−35· 10^-3 Learning rate, output layer in Phase 2 5⋅10−45· 10^-4 2⋅10−32· 10^-3 Learning rate in Phase 3, encoder / output / archetypes 10−410^-4 / 2⋅10−42· 10^-4 / 10−310^-3 10−410^-4 / 2⋅10−42· 10^-4 / 10−310^-3 Learning rate, CNN inner layers in Phase 3 10−410^-4 – Gradient clipping 2.0 1.0 Amortizer epochs, warm start / refit after Phase 2 / after Phase 3 200 / 100 / 0 200 / 100 / 100 Amortizer refit interval / maximum epochs per refit 10 / 30 10 / 30 Initial amortizer temperature τ 1 1 Learning rate, corpus logits, Phase 2 / Phase 3 2⋅10−22· 10^-2 / 5⋅10−35· 10^-3 10−210^-2 / 10−310^-3 Task objective and adaptation. The task loss is pixel MSE for the AE and cross-entropy for the CNN, with no latent A penalty. After periodic amortizer refits, two epochs adapt the output layer at learning rate 10−310^-3. The AE also allows up to 50 decoder-only adaptation epochs before Phase 3. The layer-count and simplex-map ablations use seeds 42,123,456\42,123,456\ and K∈3,10,25K∈\3,10,25\. Their comparison arms use the same seeds. E.5 Training beyond the main grids Table 3 summarizes the MedMNIST, ImageNet and frozen-transformer experiments. Appendices E.1 and E.2 describe their data and architectures. They share the PNN layer, A corpus selection, column generation and an amortizer that the task loss does not update. They differ in backbone, optimizer, coordinate map and reported metric. Table 3: Settings of the experiments beyond the main CNN and AE grids. Unconstrained models use the same backbone, data and optimization as the PNN of their column. The ImageNet-100 PNN has three seeds and its unconstrained model five. MedMNIST ImageNet-100 ImageNet-1k Frozen head Backbone two-site CNN ConvNeXt-T ConvNeXt-T compact ViT Backbone initialization baseline, same seed ImageNet-1k ImageNet-1k CIFAR-10, same seed Input resolution 64×6464× 64 224×224224× 224 224×224224× 224 32×3232× 32 PNN site two FC sites every token every token pooled feature PNN input dimension 128 256 256 256 K 10, 25 25, 50, 100, 150, 200 128, 256 3, 5, 10, 20, 40 Corpus 2048 images 3200 token rows 32,768 token rows 2048 feature rows Seeds 5 3 2 5 Optimizer AdamW SGD SGD AdamW Batch size 256 512 512 512 Epochs 80 + 160 2 + 30 2 + 30 80 Amortizer hidden width max⁡(8K,128) (8K,128) 1024 1024 max⁡(8K,128) (8K,128) Coordinate map K refinement rounds amortizer only amortizer only K refinement rounds Reported split test validation validation test Checkpoint best validation Final epoch Final epoch best validation MedMNIST. Training follows the main CNN protocol with the changes in Table 3. Phase 2 and Phase 3 run for up to 80 and 160 epochs, the encoder stays frozen for the first 20 Phase 3 epochs, and the post-Phase-2 amortizer refit is limited to 50 epochs. ImageNet. Two epochs with the backbone frozen are followed by 30 epochs of fine-tuning with SGD, learning rate 0.02, a backbone multiplier of 0.1, momentum 0.9, weight decay 10−410^-4, label smoothing 0.1 and cosine decay. As in the main grids, A selects the corpus from training tokens and column generation updates it. The amortizer is fitted to projected-gradient targets and refitted when its arg-max agreement falls below 0.85. It is deployed without SMO refinement. Frozen transformer head. Each backbone is trained for 160 epochs with mixup, CutMix, RandAugment and random erasing. The head trains for 80 epochs with AdamW, learning rate 10−310^-3 and cosine decay, using K refinement rounds. Its amortizer is refitted to 2K2K-round SMO targets every ten epochs, when column generation also updates the corpus. The released code records every remaining setting. E.6 Choice of K We define K as the number of archetypes that span the convex polytope approximating the latent representation manifold. This hyperparameter controls the geometric expressivity of the model, meaning that smaller values of K impose a stronger convex constraint, leading to a more compact and interpretable representation, whereas larger values increase the capacity to capture fine-grained variations in the data. Thus, K governs a trade-off between compression and fidelity. From a geometric perspective, increasing K refines the polytope approximation of the underlying data manifold, enabling the network to capture higher-order structures while maintaining interpretability through the archetypal basis. We recommend systematically assessing the impact of the choice of K on performance as assessed in Figure 3 and for ease of interpretation select the smallest K with sufficient performance. We presently used the same number of archetypes for each PNN layer. Whereas this can be defined for each layer separately, tuning separate values of K for each layer requires the fitting of many models and future work should investigate efficient search strategies for identifying suitable layer specific values of K. E.7 Training costs Table 4 shows that training time increases with K, while memory changes only slightly. As expected PNN-PL and PNN-VQ are cheaper than PNN at the same K. Figure 10 shows that a smaller corpus reduces training time and memory. Table 4: Measured training cost of the CNN models on one NVIDIA RTX A5000. Each model trains for 100 epochs from its seed-42 checkpoint with its own training step. Unconstrained PNN K=3K=3 PNN K=10K=10 PNN K=25K=25 PNN-PL K=25K=25 PNN-VQ K=25K=25 Training time (minutes per 100 epochs) MNIST 5.3 10.3 14.9 24.8 7.7 6.7 FMNIST 5.3 10.3 14.9 25.5 7.8 6.6 CIFAR-10 4.5 9.0 12.8 21.1 6.6 5.9 SVHN 6.5 14.2 18.8 31.4 9.7 8.5 EuroSAT 2.1 5.4 7.0 10.5 4.2 3.7 Peak GPU memory in training (GB) MNIST 0.24 0.60 0.61 0.64 0.52 0.52 FMNIST 0.24 0.60 0.61 0.64 0.52 0.52 CIFAR-10 0.70 1.18 1.19 1.22 1.08 1.08 SVHN 1.12 1.60 1.61 1.65 1.51 1.51 EuroSAT 1.31 3.22 3.22 3.26 2.84 2.84 Figure 10: Cost of the PNN-CNN (CIFAR-10, K=10K=10) against the corpus size NcN^c, on one NVIDIA RTX A5000. Left: time per batch of 512 for a Phase-3 training step and for inference with the archetypes cached. Right: peak GPU memory of the same. Using the full training set as the corpus (Nc=45,000N^c=45,000) costs 4.7× the training time and 9.3× the memory of the learned corpus (Nc=2048N^c=2048); inference time does not depend on NcN^c. Appendix F Larger Backbones Table 5: ImageNet-100 at 224×224224×224 with a fine-tuned ConvNeXt-T. Top-1 accuracy (%) on the 5,000 validation images at the final epoch, mean± . Unconstrained K=25K=25 K=50K=50 K=100K=100 K=150K=150 K=200K=200 ConvNeXt-T 94.6± 0.1 91.7± 0.5 92.0± 0.3 91.9± 0.7 92.5± 0.2 92.0± 0.7 Table 5 reports ImageNet-100 with ConvNeXt-T, and Table 6 reports ImageNet-1k with ConvNeXt-T. Experimental details for these runs can be found in Appendix E. Table 6: ImageNet-1k with a fine-tuned ConvNeXt-T. Top-1 accuracy (%) on the 50,000 validation images at the final epoch, mean± over two seeds. Top-1 Seeds Unconstrained, 256-d linear projection 81.2± 0.0 2 Unconstrained, 128-d linear projection 81.1± 0.1 2 PNN, K=128K=128 78.6± 0.2 2 PNN, K=256K=256 78.9± 0.1 2 Appendix G Additional Experimental Results This appendix provides comprehensive experimental results and implementation details to support reproducibility. We organize the material as follows: Section G.1 presents additional MLP classifier results; Section G.2 provides CNN classifier analysis and visualizations; Section G.3 reports autoencoder reconstruction experiments. G.1 Multi-Layer Perceptron Classifiers G.1.1 Tabular Dataset Performance We evaluate PNN-constrained MLPs on four tabular benchmark datasets: Iris, Wine, Breast Cancer, and Digits. These datasets vary in dimensionality (4-64 features), sample size (150-5,620 instances), and task complexity (2-10 classes), enabling systematic assessment of polytopal constraints under diverse conditions. Table 8 presents the mean test accuracy as a function of the number of archetypes K for each dataset. Across all benchmarks, PNN-MLP approaches the performance of the unconstrained baseline as K increases, with minimal degradation even at low archetype counts. Notably, PNN-MLP consistently outperforms both PNN-VQ-MLP and PNN-PL-MLP variants, demonstrating that soft convex combinations better preserve discriminative information compared to hard assignments (VQ) or unconstrained projections (PL). The Wine and Breast Cancer datasets exhibit particularly strong performance, achieving near-baseline accuracy with as few as 5 archetypes, suggesting that the decision boundaries in these tasks align well with polytopal geometry. G.1.2 Validation of Scalable Training Procedures To validate our scalable training framework (3.2), we compare mini-batched updates against full-batch optimization on three small datasets where exact computation is tractable. Table 7 reports test accuracy for three projection solvers: CVX (generic convex solver), QP (quadratic programming with linear constraints), and SMO (our Sequential Minimal Optimization approach). Table 7: Test accuracy (%) comparison between batched and non-batched training across optimization methods. Results show mean ± std. Batched training with SMO achieves statistically equivalent performance to exact methods while enabling scalability. K CVX QP SMO Amortized Batch ¬ Batch Batch ¬ Batch Batch ¬ Batch Batch ¬ Batch Breast 5 96.1± 2.3 97.1± 1.3 96.4± 1.5 96.4± 1.8 95.4± 4.1 96.8± 1.5 96.3± 2.2 96.3± 1.8 10 97.3± 1.2 97.1± 1.2 96.4± 1.2 97.2± 1.4 96.7± 1.1 97.0± 1.2 96.7± 1.7 96.3± 1.7 25 97.4± 1.0 96.9± 1.5 96.5± 0.0 97.4± 2.0 96.9± 2.5 96.9± 1.5 96.0± 0.7 96.3± 1.4 Iris 5 93.7± 4.0 94.7± 2.8 94.3± 3.5 95.0± 3.6 93.7± 4.0 93.0± 5.1 94.0± 5.3 84.7± 15.1 10 95.3± 2.8 95.3± 2.8 95.3± 2.8 95.3± 2.8 95.7± 2.7 96.0± 3.1 94.0± 3.9 94.0± 3.9 25 95.0± 3.2 95.0± 3.2 95.3± 3.2 95.3± 2.8 94.7± 2.8 95.3± 2.8 93.3± 4.7 94.0± 3.9 Wine 5 98.3± 2.3 97.5± 3.3 98.3± 2.3 97.2± 3.2 98.1± 2.3 97.2± 3.2 97.2± 1.8 97.8± 2.1 10 98.1± 2.3 98.3± 2.3 97.5± 2.4 97.5± 2.4 97.2± 2.3 98.1± 2.3 98.9± 1.4 98.3± 2.2 25 100.0± 0.0 98.6± 2.0 98.6± 2.0 98.6± 2.0 97.2± 0.0 98.6± 2.0 98.3± 2.2 97.8± 2.1 The results demonstrate that: (1) mini-batched training achieves equivalent accuracy to full-batch methods, validating our corpus streaming approach; (2) SMO-based updates match the performance of exact QP solvers while avoiding cubic-complexity matrix inversions; and (3) the simplified gradient computation (treating S as constant) introduces negligible bias. These findings confirm that our scalability optimizations preserve model quality while enabling application to large-scale datasets. G.1.3 Training Phase Ablation Table 8 decomposes the performance across the three training phases described in E.4: Pretrained (Phase 1, backbone only), Polytope (Phase 2, polytope only), and Full (Phase 3, end-to-end). Table 8: MLP test classification accuracy (%) across training phases for PNN variants K Pretrained PNN-PL-MLP PNN-VQ-MLP PNN-MLP Polytope Full Polytope Full Polytope Full Breast 3 96.3± 2.3 96.3± 2.3 96.0± 2.9 86.0± 13.9 85.6± 13.7 85.5± 15.6 92.0± 10.5 5 96.3± 2.3 96.5± 2.2 96.1± 2.6 93.3± 4.7 93.0± 4.6 97.1± 2.0 96.8± 1.9 7 96.3± 2.3 96.3± 2.3 96.3± 2.3 89.5± 6.4 89.3± 6.4 96.2± 2.0 95.7± 2.6 10 96.3± 2.3 96.3± 2.3 96.3± 2.3 93.2± 3.4 92.6± 3.1 96.9± 2.2 96.8± 1.9 15 96.3± 2.3 96.3± 2.3 96.3± 2.3 94.4± 1.8 92.6± 5.1 97.0± 1.9 96.9± 2.1 20 96.3± 2.3 96.3± 2.3 96.1± 2.6 94.2± 3.0 93.9± 2.5 97.4± 1.8 96.2± 1.7 25 96.3± 2.3 96.3± 2.3 96.1± 2.6 95.8± 1.7 94.0± 2.7 96.7± 1.7 96.3± 1.4 Digits 3 93.5± 1.5 74.4± 5.3 85.6± 3.0 18.2± 8.3 19.6± 9.5 19.2± 7.8 23.8± 11.5 5 93.5± 1.5 91.9± 1.9 92.9± 2.0 15.7± 6.0 18.7± 7.8 35.7± 14.1 45.0± 15.9 7 93.5± 1.5 93.3± 1.6 93.6± 1.8 22.1± 10.2 23.7± 8.7 48.9± 14.9 59.3± 12.2 10 93.5± 1.5 93.5± 1.5 93.7± 2.2 20.6± 7.6 25.8± 4.0 54.0± 17.9 64.0± 18.6 15 93.5± 1.5 93.5± 1.5 93.8± 2.1 30.2± 8.4 30.7± 9.0 77.1± 4.0 80.7± 3.5 20 93.5± 1.5 93.5± 1.5 93.7± 2.2 26.9± 13.7 31.9± 10.4 74.4± 14.9 81.4± 8.0 25 93.5± 1.5 93.4± 1.5 93.8± 2.1 40.7± 16.3 44.1± 13.4 83.3± 6.1 86.7± 4.9 Iris 3 94.0± 4.3 94.7± 3.8 94.7± 3.8 85.3± 11.4 85.3± 11.4 83.0± 12.1 89.0± 9.2 5 94.0± 4.3 94.7± 3.8 94.7± 3.8 85.3± 7.7 85.3± 7.7 90.0± 13.0 93.3± 5.2 7 94.0± 4.3 94.0± 4.3 94.7± 3.8 92.7± 4.3 92.7± 4.3 90.3± 9.2 93.7± 5.1 10 94.0± 4.3 94.0± 4.3 94.7± 3.8 88.7± 1.8 88.7± 1.8 94.3± 4.2 94.3± 4.2 15 94.0± 4.3 94.0± 4.3 94.7± 3.8 92.0± 4.5 92.0± 4.5 93.3± 5.4 93.0± 6.0 20 94.0± 4.3 94.0± 4.3 94.0± 4.3 93.3± 4.1 93.3± 4.1 94.0± 4.7 93.0± 4.0 25 94.0± 4.3 94.0± 4.3 94.0± 4.3 92.7± 4.3 93.3± 4.7 93.3± 5.2 94.0± 5.4 Wine 3 97.8± 2.3 97.2± 2.0 97.8± 2.3 71.1± 19.1 71.1± 19.1 85.8± 16.1 88.6± 14.7 5 97.8± 2.3 97.8± 2.3 97.8± 2.3 91.7± 5.2 91.1± 4.6 97.2± 2.6 97.5± 1.6 7 97.8± 2.3 97.8± 2.3 97.8± 2.3 97.8± 3.6 97.8± 3.6 97.8± 3.2 97.5± 2.4 10 97.8± 2.3 97.8± 2.3 97.8± 2.3 96.1± 2.5 95.0± 2.3 98.1± 1.9 98.3± 1.4 15 97.8± 2.3 97.8± 2.3 98.3± 2.5 96.7± 3.0 97.2± 2.8 97.2± 2.6 96.9± 3.1 20 97.8± 2.3 97.8± 2.3 98.3± 2.5 96.7± 3.6 97.2± 3.9 96.9± 2.8 97.2± 2.6 25 97.8± 2.3 97.8± 2.3 98.3± 2.5 97.8± 3.0 97.8± 3.0 98.1± 2.3 98.1± 2.3 Several patterns emerge from this ablation. First, for standard PNN-MLP, the Polytope phase stays within 4 points of the Pretrained accuracy on Breast Cancer, Iris and Wine for K≥5K≥ 5, but reaches only 35.7 to 83.3% on Digits. Second, PNN-PL consistently maintains pretrained performance in Phase 2, confirming that unconstrained subspace projections impose negligible information loss. The near-perfect preservation (e.g., 97.8% on Wine in the Polytope phase for every K≥5K≥ 5) demonstrates that the learned subspaces capture the essential structure of the pretrained features. Third, PNN-VQ exhibits substantial performance degradation in Phase 2, particularly at low K values. For the Breast Cancer dataset with K=3K=3, PNN-VQ drops to 86.0% compared to the 96.3% pretrained baseline, indicating that hard discrete assignments cannot faithfully represent the continuous structure of pretrained features without co-adaptation. However, the performance recovers somewhat as K increases, suggesting that a sufficiently large codebook can approximate the required representational capacity. PNN-VQ-MLP stays competitive on Breast Cancer, Iris and Wine at larger K. Finally, Phase 3 end-to-end training (Full) provides modest but consistent improvements for standard PNN. This validates our three-phase protocol: pretraining establishes a strong initialization, polytope-only optimization verifies representational capacity, and joint fine-tuning enables full co-adaptation when necessary. G.2 CNN Experiments Table 9: CNN test accuracy (%) on three MedMNIST datasets at 64×6464×64, mean± over five seeds. Each PNN run starts from the unconstrained network of the same seed and uses the deployed coordinate map. Classes Unconstrained PNN K=10K=10 PNN K=25K=25 PathMNIST 9 90.4± 1.2 88.5± 1.5 89.1± 1.5 DermaMNIST 7 78.6± 0.7 76.4± 0.6 77.4± 0.7 BloodMNIST 8 98.3± 0.0 98.1± 0.2 98.1± 0.1 Table 9 applies the PNN-CNN to PathMNIST, DermaMNIST and BloodMNIST from MedMNIST v2 [Yang et al., 2023], with K∈10,25K∈\10,25\ and five seeds, more information in Appendix E. Table 10: CNN test accuracy (%) with one constrained site or two, deployed coordinates (the amortizer followed by K rounds of warm-started batched SMO), mean± over the same three seeds for every column. The configuration is otherwise identical. K Unconstrained One site Two sites MNIST 3 99.6± 0.0 98.4± 1.3 91.7± 12.8 10 99.4± 0.0 99.4± 0.1 25 99.5± 0.0 99.5± 0.0 FMNIST 3 93.2± 0.1 92.1± 0.1 91.6± 0.6 10 92.8± 0.3 92.8± 0.2 25 93.0± 0.1 92.7± 0.1 CIFAR-10 3 82.3± 0.3 76.9± 0.7 75.7± 0.6 10 81.6± 0.4 81.5± 0.5 25 81.8± 0.4 82.2± 0.4 Table 10 compares one constrained site with two under an otherwise identical configuration on MNIST, FashionMNIST and CIFAR-10. G.3 AE Experiments From Figure 11 we see how the size of the polytope affects the quality of the reconstruction in the datasets. A larger polytope means less compression and therefore more expressive power which results in better reconstruction. (a) SVHN (b) MNIST (c) CIFAR-10 (d) EuroSAT Figure 11: PNN-AE reconstruction by K for SVHN, MNIST, CIFAR-10, EuroSAT From Figure 12 shows how each of the models reconstruct across datasets. We see that the PNN visually offers competitive reconstruction quality. (a) SVHN (b) MNIST (c) CIFAR-10 (d) EuroSAT Figure 12: Reconstruction by model for K=25K=25 for SVHN, MNIST, CIFAR-10 and EuroSAT. In table 11 we further compared using a softmax instead of a sparsemax in the amortizer. Both methods have similar mean MSE. Table 11: PNN-AE reconstruction MSE with a softmax or sparsemax amortizer output. Values are mean± over the same three seeds. K Softmax Sparsemax MNIST 3 0.0399± 0.0003 0.0400± 0.0005 10 0.0147± 0.0005 0.0148± 0.0001 25 0.0073± 0.0001 0.0075± 0.0003 FMNIST 3 0.0266± 0.0002 0.0268± 0.0003 10 0.0127± 0.0001 0.0127± 0.0001 25 0.0099± 0.0002 0.0100± 0.0004 CIFAR-10 3 0.0363± 0.0001 0.0363± 0.0001 10 0.0218± 0.0000 0.0218± 0.0000 25 0.0152± 0.0004 0.0157± 0.0007 G.3.1 Vector Quantization To benchmark polytopal soft-clustering against discrete hard-clustering, we compare to the VQ-VAE. VQ-VAE discretizes the latent space using a finite codebook of embeddings ℓE . The codebook is updated using an exponential moving average (EMA) scheme rather than direct gradient descent. Gradients are propagated through the quantization operation using the Straight-Through Estimator (STE), passing gradients from quantized latents ℓR to encoder outputs ℓZ unchanged. The training objective is ℒ=ℒtask+β∑ℓ‖ℓ−sg⁡[ℓ]‖F2,L=L_task+β _ ||Z -sg[R ]||_F^2, (23) where the commitment term encourages encoder outputs to remain close to their assigned codebook vectors. To prevent codebook collapse, ℓE is initialized via greedy k-means++ on the full training set, and unused embeddings are periodically reinitialized. A direct comparison between the PNN-VQ and the VQ-VAE can be seen in table 12. Table 12: Reconstruction MSE of a standard VQ-VAE (EMA codebook, commitment loss, straight-through estimator) and of PNN-VQ (direct gradient updates of a corpus-grounded codebook, reconstruction loss only), mean± over five seeds. K VQ-VAE PNN-VQ MNIST 3 0.0613± 0.0005 0.0614± 0.0006 5 0.0568± 0.0006 0.0580± 0.0007 10 0.0530± 0.0006 0.0527± 0.0009 15 0.0502± 0.0005 0.0493± 0.0004 20 0.0478± 0.0001 0.0474± 0.0003 25 0.0464± 0.0004 0.0458± 0.0007 FMNIST 3 0.0635± 0.0049 0.0611± 0.0013 5 0.0540± 0.0008 0.0542± 0.0010 10 0.0459± 0.0014 0.0431± 0.0005 15 0.0408± 0.0008 0.0394± 0.0005 20 0.0381± 0.0008 0.0370± 0.0004 25 0.0362± 0.0004 0.0361± 0.0003 CIFAR-10 3 0.0552± 0.0041 0.0477± 0.0004 5 0.0452± 0.0004 0.0444± 0.0003 10 0.0412± 0.0005 0.0414± 0.0005 15 0.0393± 0.0001 0.0395± 0.0004 20 0.0382± 0.0002 0.0383± 0.0001 25 0.0374± 0.0002 0.0376± 0.0002 SVHN 3 0.0339± 0.0025 0.0262± 0.0004 5 0.0278± 0.0032 0.0224± 0.0004 10 0.0279± 0.0058 0.0207± 0.0008 15 0.0221± 0.0037 0.0197± 0.0002 20 0.0186± 0.0011 0.0190± 0.0003 25 0.0172± 0.0002 0.0181± 0.0003 EuroSAT 3 0.0145± 0.0007 0.0120± 0.0005 5 0.0106± 0.0007 0.0107± 0.0006 10 0.0095± 0.0003 0.0096± 0.0005 15 0.0089± 0.0001 0.0092± 0.0002 20 0.0088± 0.0002 0.0090± 0.0001 25 0.0085± 0.0000 0.0091± 0.0003

Instant research discovery