Paper deep dive
Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning
Soutrik Sarangi, Yonatan Sverdlov, Adir Dayan, Haggai Maron, Nadav Dym
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning properties of neural networks has recently attracted growing interest, with existing approaches operating either directly on network parameters or through probe-based representations of network behavior. While probing methods have shown strong empirical performance, their theoretical foundations remain limited. In this work, we study when finite probe-based representations are sufficient for learning neural functionals. We establish general identification and universality results for probing, and show that using intermediate hidden representations can provide significantly more informative representations than relying only on final outputs. Motivated by these results, we introduce HIDDENPROBE, a simple architecture for learning from hidden probe responses. Across a range of neural functional benchmarks, including both MLPs and Transformers, HIDDENPROBE consistently improves over existing probing methods and achieves state-of-the-art performance. Our code is publicly available on GitHub.
Tags
Links
- Source: https://arxiv.org/abs/2609.33901v1
- Canonical: https://arxiv.org/abs/2609.33901v1
Trouble viewing inline? Open PDF directly →
Full Text
209,338 characters extracted from source content.
Expand or collapse full text
Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning Soutrik Sarangi Yonatan Sverdlov11 1 Our focus is on neural functionals which depend only on the function fθf_θ and not on the specific parameterization θ. In the terminology of Dayan et al. (2026) these are called function-space functionals Adir Dayan Haggai Maron Nadav Dym †thanks: Equal contribution. Affiliation: Microsoft Affiliation: Faculty of Mathematics, Technion – Israel Institute of Technology Affiliation: Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology Affiliation: NVIDIA Email: soutriksarangi14@gmail.com Email: yonatans@campus.technion.ac.il Email: adir.dayan@campus.technion.ac.il Email: haggaimaron@technion.ac.il Email: nadavdym@technion.ac.il Abstract Learning properties of neural networks has recently attracted growing interest, with existing approaches operating either directly on network parameters or through probe-based representations of network behavior. While probing methods have shown strong empirical performance, their theoretical foundations remain limited. In this work, we study when finite probe-based representations are sufficient for learning neural functionals. We establish general identification and universality results for probing, and show that using intermediate hidden representations can provide significantly more informative representations than relying only on final outputs. Motivated by these results, we introduce HiddenProbe, a simple architecture for learning from hidden probe responses. Across a range of neural functional benchmarks, including both MLPs and Transformers, HiddenProbe consistently improves over existing probing methods and achieves state-of-the-art performance. Our code is publicly available on GitHub. 1 Introduction In recent years, there has been increasing interest in the topic of learning neural functionals (also known as weight space learning Han et al. (2026)). A neural functional is a mapping ψ taking a neural network fθf_θ to a vector ψ(θ)∈ℝmψ(θ) ^m. For example, one could consider a neural functional predicting the test accuracy of a given neural network. Another example is where fθf_θ is an INR representing an image, and then standard image tasks like image classification become neural functionals. A popular approach to learning neural functionals ψ(fθ)ψ(f_θ) is via weight space methods, which think of the functionals as a function of the parameters ψ~(θ)=ψ(fθ) ψ(θ)=ψ(f_θ), and then apply neural architectures directly to the parameter weights to learn ψ~ ψ. Several recent works have adopted this approach with some success Navon et al. (2023); Zhou et al. (2023b); Kofinas et al. (2024); Zhou et al. (2023a). However, weight-space learning must contend with the fact that many distinct parameter vectors can represent the same function. Accounting for these parameter symmetries is often important for performance, but typically requires symmetry-aware architectures that are more complex to design and implement. Probing provides an alternative way to process neural models that avoids the parameterization ambiguities of weight-space methods. Rather than representing fθf_θ through its parameters θ, probing represents it through its evaluations on a set of inputs, called probes, p1,…,pNp_1,…,p_N. The learning problem is then to find a function ψ^(fθ(p1),…,fθ(pN)) ψ (f_θ(p_1),…,f_θ(p_N) ) that approximates the desired neural functional ψ(fθ)ψ(f_θ). Probe-based methods, such as Kahana et al. (2025), have achieved strong empirical performance on several benchmarks, but currently lack a theoretical justification. In particular, while weight space methods like Navon et al. (2023) can provably approximate all neural functionals Dayan et al. (2026), analogous universality guarantees are not known for probe-based methods. Moreover, whereas the parameters θ fully specify fθf_θ, a finite probe set reveals only the values of fθf_θ at finitely many inputs and therefore appears, a priori, to provide only partial information about the realized function. This raises two closely related questions: Main Questions: 1. Can a function fθf_θ be uniquely reconstructed from a finite number of probes? If so, how many probes are needed? 2. Can any neural functional Λ(fθ) (f_θ) be approximated by a probe-based model? Main Results: As we shall see, our answer to both questions will be positive. Under the assumption that fθf_θ is analytic, we show that the function fθf_θ can be uniquely recovered from 2q+12q+1 probes, where q denotes the dimension of the parameter space. As a result, we will show that any neural functions Λ(fθ) (f_θ) can be approximated by a probe-based model with 2q+12q+1 probes. In addition, we will discuss probe based methods which represent fθf_θ not only through the final probe value fθ(p)f_θ(p), but also through the value of all intermediate layers on the probes. For both MLP and self-attention, we will show that this representation requires fewer probes to obtain the same guarantees. For fully connected neural networks, this brings down the sufficient probe count from O(q)O(q) to O(dmax)O(d_ ), where dmaxd_ is the maximal width, and for self-attention blocks this brings down the count to O(dinh)O(d_inh), with dind_in and h being the input dimension and number of heads, respectively. Motivated by this we suggest a specialized model to learn properties of neural networks with such probes, which we name HiddenProbe (see Figure 1). Empirically, we will show that HiddenProbe attains state of the art results on several standard neural functional datasets. Figure 1: Overview of HiddenProbe. A set of learnable probes p1,…,pNp_1,…,p_N is evaluated on a target neural network fθf_θ, producing intermediate activations xi(pj,θ)x_i(p_j,θ) and final outputs xL(pj,θ)x_L(p_j,θ), which are processed by an invariant model and an MLP to predict a property of the target network. 1.1 Previous Work Weight-space learning. Weight-space learning studies models that process the parameters of other neural networks directly; we refer to Han et al. (2026) for a recent survey. A central challenge in this setting is that different parameterizations can represent the same function, motivating architectures that explicitly respect weight-space symmetries. DWSNets (Navon et al., 2023) and Neural Functional Networks (Zhou et al., 2023a) construct linear layers equivariant to hidden-neuron permutations, while Neural Functional Transformers (Zhou et al., 2023b) incorporate the same symmetry into attention-based architectures. More recent graph-based approaches, including Graph Metanetworks (Lim et al., 2024) and Neural Graphs (Kofinas et al., 2024), represent networks as computational graphs, enabling weight-space processing across a broader range of architectures. Beyond permutations, subsequent methods incorporate additional symmetries, such as ScaleGMN (Kalogeropoulos et al., 2024) and Monomial-NFN (Tran et al., 2024), which account for scaling and sign symmetries induced by particular activation functions. Identification and learning from probes Prior work in weight-space learning has explored combining network weights with responses to input probes to predict network properties (Herrmann et al., 2024; Kofinas et al., 2024; Horwitz et al., 2025). Complementary approaches rely solely on final outputs evaluated at learned probes (Kahana et al., 2025). Beyond weight-space learning, interpolation and identification of neural-network from input–output observations has also been studied in learning theory (Vardi et al., 2022). More recently, Bhattamishra et al. (2026); Kim and Kim (2026) have investigated recovering the weights of frozen transformers from query outputs. Beyond final outputs, hidden activations provide additional information about a network’s internal representations. Prior work has used these activations to study representation properties (Alain and Bengio, 2017; Cotler et al., 2023) and, more recently, to investigate model behavior through mechanistic interpretability (Gurnee et al., 2023). In natural language processing, probing classifiers similarly predict linguistic properties from a model’s hidden representations to study the information they encode (Belinkov, 2022). Within weight-space learning, Heo et al. (2026) similarly probe selected hidden layers to predict network properties. To the best of our knowledge, prior work has not systematically studied property prediction from complete activation traces across all hidden layers, collected by evaluating a network on input probes. We address this gap by studying when neural functions can be identified from the activations induced by input queries, and how universal approximators can be designed to predict network properties from final outputs and hidden activation traces. 1.2 Preliminaries Consider a parametric family f:×Θ→ℝmf:X× ^m, where ⊆ℝdX ^d is the input domain and Θ⊆ℝq ^q is the parameter domain. Every fixed parameter θ∈Θθ∈ defines a function fθ(x)=f(x,θ)f_θ(x)=f(x;θ). We use f(⋅,θ)f(·;θ) and fθf_θ interchangeably. We denote the collection of all such functions by ℱΘF_ . Our motivating application is the problem of learning neural functionals11 1 Our focus is on neural functionals which depend only on the function fθf_θ and not on the specific parameterization θ. In the terminology of Dayan et al. (2026) these are called function-space functionalsΨ:ℱΘ→ℝm :F_ ^m, and our focus is on analyzing methods which rely on probing. We will now define these methods, and focus on two variants: final-layer probing and hidden-layer probing. Final-layer probing Final-layer probing represents a function fθf_θ by considering its output on a number of input points, called probes. Formally, given an N tuple of points ≔(p1,⋯,pN)∈NP (p_1,·s,p_N) ^N, the model fθf_θ is mapped to an m×Nm× N dimensional matrix Ψfinal(,θ)≔(fθ(p1),⋯,fθ(pN))∈ℝm×N. _final(P;θ) (f_θ(p_1),·s,f_θ(p_N)) ^m× N. This matrix is processed to learn properties of the neural network, e.g., by applying an MLP ϕMLP∘Ψfinal(,θ) _MLP _final(P;θ) to the output, to obtain a neural model for learning the functional Λ(fθ) (f_θ). A representative example of this approach is the ProbeGen method of Kahana et al. (2025), which uses this approach precisely with a carefully designed method for learning the probes. Hidden-Layer Probing Heo et al. (2026); Horwitz et al. (2025), In applications, the parametric functions fθf_θ are typically constructed by a composition of simpler functions (e.g. layers), that is fθ=fL,θ∘…∘f2,θ∘f1,θf_θ=f_L,θ … f_2,θ f_1,θ. When this compositional structure is present, hidden layer probing relates to the practice of retaining the output of all intermediate computations on the probes, and not only the final output. Formally, for every input p∈p and parameter θ∈Θθ∈ we define x0(p,θ)=p,x1(p,θ)=f1,θ(x0(p,θ)),…,xL(p,θ)=fL,θ(xL−1(p,θ)).x_0(p,θ)=p,\,x_1(p,θ)=f_1,θ (x_0(p,θ) ),…,x_L(p,θ)=f_L,θ (x_L-1(p,θ) ). The hidden-probe representation of fθf_θ is defined via the mapping Ψall(θ,):=(xi(pj,θ),0≤i≤L, 1≤j≤N) _all(θ,P):= (x_i(p_j,θ),0≤ i≤ L,\,1≤ j≤ N ) 1.3 Overview In the remainder of the paper, we will prove that analytic neural networks of any kind can be identified by a number of final-layer probes proportional to the network’s parameters (Section 2), and will show, for both MLPs and transformers, that the required number of probes can be greatly reduced when allowing Hidden-Layer probing (Sections 3-4). We then describe HiddenProbe, a method for learning using hidden probes (Section 5), prove the universality of probe-based methods (Section 6), and finally demonstrate the effectiveness of HiddenProbe on neural functional learning (Section 7). 2 Identifying Analytic Neural Networks from Probe evaluations Our first result is that final-layer probe evaluations are enough to identify analytic parametric functions, and the number of probes is proportional to the dimension of the parameter space. This theorem applies to all analytic parametric families including MLPs and Transformers, provided that the activations used in these models are analytic. Examples of such activations are tanh, sigmoid, SiLU and sine activations. Theorem 1 (Unique reconstruction from finite samples). Let f:ℝd×ℝq→ℝmf:R^d×R^q ^m be an analytic function. Then, for almost any choice of probes p1,…,p2q+1∈ℝdp_1,…,p_2q+1 ^d, we have that for all θ1,θ2∈Θ _1, _2∈ , f(pj,θ1)=f(pj,θ2),∀j=1,…,2q+1⟺f(⋅,θ1)≡f(⋅,θ2)on ℝdf(p_j, _1)=f(p_j, _2),∀ j=1,…,2q+1 \ f(·, _1)≡ f(·, _2)\ on R^d Proof Idea. Intuitively we can interpret this result as follows: although the family of functions fθ|θ∈ℝq\f_θ| θ ^q\ contains infinitely many functions, it has a finite dimensional parameterization in ℝqR^q, and so it is reasonable that a finite number of ∼q q evaluations can separate the realized functions. This is similar to the well-known result that a degree n polynomial can be identified by n+1n+1 samples. However, this rule of thumb is not always true. For example, for ReLU networks, no finite number of probes suffices to uniquely identify the function (see Subsection 3.2). The proof that such a result does hold for analytic functions is based on the Finite Witness Theorem by Amir et al. (2023). ∎ In Section 6 we will show that this result implies universal approximation: probe based methods with 2q+12q+1 probes can approximate all neural functionals. This result is substantial, since probe based methods typically use a small number of probes, while apriori it would seem that universal approximation would require an exhaustive sampling of the input domain. At the same time, the number of probes suggested by the theorem is still much larger than what is used in practice. To alleviate this disadvantage, we will next focus on hidden probes, where function identification is possible with a much lower probe cardinality. 3 Probing MLPs In this section we specialize to the case where f(x,θ)f(x;θ) is an MLP. The main message of this section is that hidden probes substantially reduce the number of probes necessary to identify an MLP. We begin our discussion by setting our notation for the MLP model class. An MLP f(p,θ)f(p;θ) is determined by a choice of activation σ and a width vector ≔(d0,d1,⋯,dL−1,dL)D (d_0,d_1,·s,d_L-1,d_L). An MLP f(p,θ)f(p;θ) is then defined via x0≔p,xi≔σ(Wixi−1+bi),∀i∈[L],x_0 p,\,x_i σ (W_ix_i-1+b_i ),\,∀ i∈[L],\, (1) Here, Wi∈ℝdi×di−1W_i ^d_i× d_i-1 and bi∈ℝdib_i ^d_i, for i∈[L]i∈[L], are the weights and biases, which together define the parameters θ≔[Wi,bi]i∈[L]θ [W_i,b_i]_i∈[L]. The intermediate features xix_i depend on p,θp,θ and so we denote them by xi(p,θ)x_i(p,θ). The final function is defined by the last feature f(p,θ)=xL(p,θ)f(p,θ)=x_L(p,θ). We denote the total dimension of the parameter space by q_D. 3.1 Hidden probes provide useful information: For MLPs with analytic activations, Theorem 1 implies that a finite number of probes, proportional to q_D, is sufficient. For hidden probes, the number of necessary probes depends only on the maximal width: Theorem 2 (Reconstruction from hidden layer activations). Let σ:ℝ→ℝσ:R be an analytic, injective, non-polynomial activation. Let f(p,θ)f(p,θ) denote an MLP with activation σ and widths ≔(d0,d1,⋯,dL−1,dL)D (d_0,d_1,·s,d_L-1,d_L). Assume that N>maxidiN> _id_i. Then for almost every ∈NP ^N, we have that for almost every θ1,θ2 _1, _2, Ψall(θ1,)=Ψall(θ2,)⟹θ1=θ2,⟹f(⋅,θ1)=f(⋅,θ2). _all( _1,P)= _all( _2,P) \, _1= _2,\, f(·, _1)=f(·, _2). In this result we add the assumption that the activation is non-polynomial and injective (this is the case for e.g., tanh and sigmoid, but not for sine activation). Moreover, identifiability is not on the whole domain, but rather it holds for parameters θ1,θ2 _1, _2 outside an exceptional Lebesgue-null set ℰE. The proof in fact shows a stronger statement: the set ℰE is a zero set of an analytic function, and hence is of strictly lower dimension than the parameter space, and has empty interior. In general, whenever we talk about ”almost everywhere” in this paper, this stronger notion applies. Proof Idea. The idea of the proof is based on a straightforward algorithm described in detail in in Algorithm 1 of Appendix B.1. Intuitively, the parameters Wi,biW_i,b_i of the i-th layer satisfy the linear equations σ−1(xi(pj,θ))=Wixi−1(pj,θ)+bi,j=1,…,Nσ^-1 (x_i(p_j,θ) )=W_ix_i-1(p_j,θ)+b_i, j=1,…,N The proof shows that for generic probes and parameters, these linear equations have a unique solution. ∎ Tightness of results in Appendix B.2 we give examples of analytic MLPs that cannot be identified with dmaxd_ hidden probes (exactly matching the bound of Theorem 2) For final-layer probes, we construct, for every N≤2qD−2N≤ 2q_D-2, a nonempty open set of probe tuples that fail to separate all realized functions. (Theorem 1 gives a sufficient bound of 2q+12q_D+1 generic probes). 3.2 Identifying ReLU Networks from Probe evaluations Figure 2: For any finite set of probes, a nonzero ReLU ‘hat’ function can be constructed whose support avoids all probes. Our results on identifiability of an MLP from a finite number of probes require analytic activations, and thus do not apply to the common ReLU activation. In fact, such a result is not possible for ReLU activations: Proposition 3 (Finite probes do not identify ReLU networks). For any finite set of probes p1,…,pN∈[0,1]p_1,…,p_N∈[0,1], there exist two distinct ReLU networks, each with three hidden neurons, that agree on all the probes. Consequently, no finite set of probes can uniquely identify all ReLU networks. Proof Idea. For given probes p1,…,pNp_1,…,p_N, it is possible to construct two different hat functions which are zero on the probes: for example, the functions plotted in blue and orange in Figure 2. Full proof is in Appendix B.3. ∎ Since exact identification of ReLU networks from finitely many probes is impossible, we instead study approximate identification in L2(μ)L^2(μ). We show that, with probability at least 1−δ1-δ over the sampled probes, any network that agrees with the target network on those probes is within ϵε of the target in L2(μ)L^2(μ). This is similar in spirit to results from PAC learning. The following theorem gives a semi-formal description of these results, which are described and proved in more detail in Theorem 11 of Appendix B.3. Theorem 4 (semi-formal-PAC identification of ReLUReLU networks). Consider the family of ReLUReLU MLPs with widths ≔(d0,d1,⋯,dL−1,dL)D (d_0,d_1,·s,d_L-1,d_L) and q parameters. Let μ be a compactly supported probability distribution on ℝd0R^d_0, and assume that the parameter set is compact. Fix ϵ,δ∈(0,1)ε,δ∈(0,1) and a parameter vector θ0 _0. Then there exist constants C1(ϵ,δ),C2(ϵ,δ)>0C_1(ε,δ),\,C_2(ε,δ)>0 such that • For N1=C1⋅Lqln(q)N_1=C_1· Lq (q) final layer samples, with probability of at least 1−δ1-δ on the samples, any θ which agrees with θ0 _0 on the samples will satisfy ‖f(⋅,θ)−f(⋅,θ0)‖L2(μ)≤ϵ.\|f(·;θ)-f(·; _0)\|_L^2(μ)≤ε. • When hidden layer samples are used, only N2=C2⋅qln(q)N_2=C_2· q (q) samples are sufficient. As in the analytic activation case, this result suggests that hidden activations can reduce the number of probes necessary to (almost) identify a ReLU network, especially for deeper ReLU nets. An experimental verification of this is given in the Appendix D.1. 4 Identifying Multi-Head Attention from Probes Multi-head attention (MHA) forms the backbone of modern transformer architectures. In this section we discuss the identification of multiple-layer multihead attention (MHA) blocks from probes. Additional components typically used in transformers, like FFNs and LayerNorm, are omitted for mathematical tractability. We begin with some basic definitions (details in Appendix B.4). We consider d dimensional inputs and h attention heads. An MHA layer has parameters θ≔(a,a,a,a)a=1h,θ \(Q_a,K_a,V_a,O_a)\_a=1^h, For an input probe P∈ℝn×dP ^n× d whose rows are the token vectors p1⊤,…,pn⊤p_1 ,…,p_n , the layer computes Fn(P;θ)=∑a=1hsoftmax(PaP⊤)Pa,a≔aa⊤k,a≔a. F_n(P;θ)= _a=1^hsoftmax(PM_aP )\,PW_a,\,M_a Q_aK_a k,\,W_a _aO_a. (2) The softmax acts rowwise:[softmax(PaP⊤)]ij=exp(pi⊤apj)∑s=1nexp(pi⊤aps). [softmax(PM_aP ) ]_ij= (p_i M_ap_j) _s=1^n (p_i M_ap_s). A probe specifies P and returns the complete matrix Fn(P,θ)∈ℝn×dF_n(P;θ) ^n× d. We denote by =(θ1,…,θL) θ=( _1,…, _L) the parameters of an L-layer MHA stack and let Θ=∏ℓ=1LΘℓ = _ =1^L _ denote the parameter space of the L-layer MHA architecture, with Θℓ _ being the parameter space of the ℓ -th layer, let q≔dim(Θ)q ( ).Writing Fn(ℓ)F_n^( ) for the ℓ -th layer map, we define x0,n(P;)=P,xℓ,n(P;)=Fn(ℓ)(xℓ−1,n(P;);θℓ),ℓ=1,…,L. x_0,n(P; θ)=P,\,x_ ,n(P; θ)=F_n^( ) (x_ -1,n(P; θ); _ ),\, =1,…,L. (3) Thus xL,nx_L,n is the stack’s final output. For a collection of probes =(P(i))i=1N P=(P^(i))_i=1^N, where P(i)∈ℝni×dP^(i) ^n_i× d. As usual, we define the output-only and hidden-access probe maps for transformers by Ψfinal(,) _final( θ,P) and Ψall(,) _all( θ,P), respectively. Since the softmax attention is analytic, we can apply a variant of Theorem 1, by using the Finite Witness Theorem for the effective manifold dimension of Θ to give function-level identification from probes: Corollary 1 (Generic probes). Fix a token length n≥1n≥ 1. For almost every tuple of 1+2q1+2q probes in ℝn×dR^n× d and all θ,θ′∈Θθ,θ ∈ , Ψfinal(θ)=Ψfinal(θ′),⇔xL,n(⋅,θ)≡xL,n(⋅,θ′) _final(θ)= _final(θ ),\, \,x_L,n(·;θ)≡ x_L,n(·;θ ). Note that all of these parameter counts are O(d2)O(d^2) per layer for the standard multi-head attention blocks. But softmax attention has more structure than analyticity, and we can query probes of varying token lengths to an MHA block, which lets us identify the function using only O(dh)O(dh) probes: Theorem 5 (Structured probing for softmax attention). Let hmax≔maxℓhℓh_ _ h_ and N≔d(2hmax+1)+1N d(2h_ +1)+1. There exists an open set of probe tuples ∈NP ^N of token length at most max3,1+d/hmax \3,1+d/h_ \, such that for almost every target θ0∈Θ _0∈ we have: Ψall(θ,)=Ψall(θ0,)⟹xℓ,n(⋅,θ)≡xℓ,n(⋅,θ0),∀θ∈Θ,∀ℓ∈[L],∀n∈ℕ, _all(θ;P)= _all( _0;P)\, \,x_ ,n(·;θ)≡ x_ ,n(·; _0),\,∀\,θ∈ ,∀\, ∈[L],∀\,n , Proof Idea. Theorem 5 gives a specific way of choosing the probes for identifying Multilayer MHA blocks with hidden access, which we call RecoverMHA. We elaborate on the single layer single head case of RecoverMHA (see also Algorithm 2 in the appendix). In this case, the goal is to reconstruct two matrices: the matrices 0M_0 and 0W_0 from (2). The key trick is to do this in first stages: we first use single-token input: For such input the output of the softmax is always the scalar 1∈ℝ1×11 ^1× 1, and so 0M_0 is eliminated, and one-token probes provide linear equations that can be used to determine 0W_0. Once W is known, we query a single probe consisting of affinely independent tokens, which uniquely recovers the attention weights for a full rank target value matrix. We then recover 0M_0 from log-ratios by utilizing the softmax structure. Algorithm 2 implements these steps using d+1d+1 probes. The extension of these results to multiple heads and layers is discussed in Appendix B.4. ∎ 5 HiddenProbe: Learning using Hidden Neural Activations Our analysis shows that using hidden probes can greatly enhance the identifiability of neural networks from probes. Accordingly, we design a method for learning from hidden probes which we name HiddenProbe. We have two different instantiations of this model, one for MLPs and one for transformers. The exact details are discussed in Appendix A. However, the basic idea is simple and identical. We use learned probes, and utilize all hidden layers. The probes of each hidden layer are processed a permutation-invariant aggregator, and then the results are aggregated using a global MLP to predict the target property. We design HiddenProbe for MLPs so that it respects the permutation invariance of the MLP weight space. Namely, one can apply an independent permutation to the neurons of each hidden layer, and obtain a functionally equivalent MLP with permuted weight matrices. We attain permutation invariance by using permutation invariant Set Transformer Lee et al. (2019). In general, an interesting advantage of (hidden) probe methods over weight space methods is that weight space symmetries become much simpler. Indeed, while in the hidden layer representation the permutations to each hidden layer are independent, in the weight space representation these symmetries are entangled, and designing permutation invariant models for this representation can be cumbersome Navon et al. (2023); Sverdlov et al. (2025). For transformers, the weight space approach needs to deal with GL(n)GL(n) symmetries Tran et al. (2025) which probe based methods avoid since they depend on the functional behavior of MHA and not on the exact parameterization. 6 Universality of probe-based learning In this section, we use our identification results from previous sections to prove universality of probe based models with a fixed number of probes. These results are based on the standard observation that separation implies universality Chen et al. (2019); Dym and Gortler (2025). For our theorem, we will consider analytic functions f:ℝd×ℝq→ℝmf:R^d×R^q ^m as in Theorem 1, but we will assume that we are only interested in a compact input domain ⊆ℝdX ^d with non-empty interior, and a compact parameter domain Θ⊆ℝq ^q. Accordingly, we are interested in the function space ℱΘ(,ℝm):=fθ:→ℝmF_ (X,R^m):= \f_θ:X ^m \ which is a subset of the set of all continuous functions from X to ℝmR^m, endowed with the infinity norm ‖fθ‖∞=maxx∈‖f(x,θ)‖2\|f_θ\|_∞= _x \|f(x,θ)\|_2. A functional Λ:ℱΘ(,ℝm)→ℝ :F_ (X,R^m) is continuous, if it is continuous with respect to the distance induced by this norm. Our first result uses Theorem 1 to show that for general analytic models with q parameters, 2q+12q+1 final-layer probes are sufficient for universality: Theorem 6 (Universality of probe-based learning). With the assumptions listed above, let Λ:ℱΘ(,ℝm)→ℝk :F_ (X,R^m) ^k be a continuous functional. Then there exists p1,…,p2q+1∈p_1,…,p_2q+1 , such that ∀ϵ>0∀\,ε>0, there exists an MLP ϕMLP _MLP such that supθ∈Θ‖Λ(fθ)−ϕMLP(fθ(p1),…,fθ(p2q+1))‖<ϵ _θ∈ || (f_θ)- _MLP (f_θ(p_1),…,f_θ(p_2q+1) )||<ε When using hidden probes, we can obtain universality using our model HiddenProbe with a significantly smaller number of probes, due to our identifiability results: Theorem 7 (Universality of HiddenProbe). With our previous notation, assume that 1. f(x,θ)f(x;θ) is an MLP architecture with analytic, injective, non-polynomial activation, and maximal width dmaxd_max, and denote N=dmax+1N=d_max+1. Or 2. f(x,θ)f(x;θ) is a pure softmax MHA stack with d dimensional inputs and at most H heads per layer. Set N=d(2H+1)+1N=d(2H+1)+1. Then, if K⊆ΘK is a compact subspace of the parameters that does not intersect an appropriate Lebesgue-null set E, there exist N probes, and a hidden probe model ℋHiddenProbeH_ HiddenProbe, such that supθ∈K‖Λ(fθ)−HHiddenProbe(θ)‖2<ϵ. _θ∈ K \| (f_θ)- H_ HiddenProbe(θ) \|_2<ε. We restate formal versions of the above theorems and prove them in Appendix B.5, together with guarantees on operator learning from probes. 7 Experiments We evaluate HiddenProbe for weight space lerning tasks and compare with other prominent baselines. To understand the influence of multi-layer probes, we put a special emphasis on comparison with ProbeGen, which is a successful method relying only on final layer probes. We will show that HiddenProbe typically outperforms ProbeGen and other weight space and probe-based models on weight space learning baselines. INR classification In this setting, each input network is an MLP trained as an implicit neural representation (INR) of a single image, and the goal is to predict the class of the represented image directly from the network. We evaluate on the MNIST, Fashion-MNIST, CIFAR-10 (singke and augmented views). The INR networks across datasets contain two hidden layers. In Table 1, we compare HiddenProbe against the baselines: ScaleGMN (Kalogeropoulos et al., 2024), NG-T and Neural Graphs (Kofinas et al., 2024), NFT (Zhou et al., 2023b), MAGEP-NFN (Vo et al., 2025), and ProbeGen (Kahana et al., 2025). A complete comparison with all available methods is provided in Table 4 in the Appendix. Method MNIST FMNIST CIFAR-10 CIFAR-10 Aug ScaleGMN 0.966±0.0020.966± 0.002 0.808±0.0010.808± 0.001 0.388±0.0010.388± 0.001 0.570±0.0050.570± 0.005 Neural Graphs (128 probes) 0.976±0.0010.976± 0.001 0.745±0.0080.745± 0.008 – – NG-T 0.924±0.0030.924± 0.003 0.727±0.0060.727± 0.006 – – MAGEP-NFN – – 0.3718±0.0030.3718± 0.003 – NFT – – – 0.634±0.0000.634± 0.000 ProbeGen 0.984±0.0010.984± 0.001 0.877±0.0030.877± 0.003 0.573±0.0070.573± 0.007 0.563±0.0030.563± 0.003 HiddenProbe 0.986±0.0010.986± 0.001 0.886±0.0050.886± 0.005 0.580±0.0020.580± 0.002 0.606±0.0020.606± 0.002 Table 1: Classification accuracy (↑ ) on INR weight-space benchmarks. Uncertainties denote the standard error over five seeds. Baseline numbers are copied from the respective papers As observed, (1) HiddenProbe achieves the best performance on MNIST, Fashion-MNIST, and CIFAR-10, and the second-best performance on augmented CIFAR-10. (2) It outperforms ProbeGen everywhere, highlighting the importance of having hidden layer information. (3) Probegen comes second in 3/43/4 datasets, proving the efficacy of probe-based learning in general Test accuracy prediction We next evaluate HiddenProbe on the model accuracy prediction task. In this setting, the input is a CNN trained for image classification, and the goal is to predict its test accuracy directly from the network. We evaluate on the MNIST, Fashion-MNIST, SVHN, CIFAR-10-GS, and CIFAR-10-WP benchmarks, using Kendall’s correltion(τ) as the evaluation metric. In Table 2, we compare against a representative set of strong baselines: NFN (Zhou et al., 2023a), ScaleGMN (Kalogeropoulos et al., 2024), NG-T and Neural Graphs (Kofinas et al., 2024), DNG-Encoder (Wu et al., 2026), and ProbeGen (Kahana et al., 2025). A complete comparison with all available methods is provided in Table 5 in the Appendix. The networks in the MNIST, Fashion-MNIST, SVHN, and CIFAR-10-GS benchmarks contain three hidden layers, whereas CIFAR-10-WP contains heterogeneous architectures with up to four hidden layers. HiddenProbe achieves the best performance on all five benchmarks and outperforms ProbeGen in every case. Method MNIST FMNIST SVHN CIFAR-WP CIFAR-GS NFN 0.942±0.0010.942± 0.001 0.935±0.0000.935± 0.000 0.931±0.0050.931± 0.005 – 0.934±0.0010.934± 0.001 ScaleGMN – – – – 0.941±0.0000.941± 0.000 NG-T – – 0.872±0.0010.872± 0.001 0.817±0.0070.817± 0.007 0.935±0.0000.935± 0.000 DNG-Encoder – – 0.867±0.0020.867± 0.002 0.874±0.0020.874± 0.002 0.936±0.0000.936± 0.000 Neural Graphs – – – 0.885±0.0050.885± 0.005 0.938±0.0000.938± 0.000 ProbeGen 0.953±0.0020.953± 0.002 0.948±0.0050.948± 0.005 0.876±0.0050.876± 0.005 0.932±0.0060.932± 0.006 0.957±0.0010.957± 0.001 HiddenProbe 0.966±0.0040.966± 0.004 0.957±0.0020.957± 0.002 0.957±0.0030.957± 0.003 0.940±0.0060.940± 0.006 0.965±0.0000.965± 0.000 Table 2: Test-accuracy prediction performance measured by Kendall’s τ (↑ ). Uncertainties denote standard error over five seeds. Baseline numbers are copid from the respective papers. As observed: (1) HiddenProbe is the best across all datasets, and probegen comes second in 4/54/5 datasets. (2) In Cifar-WP where the model architecture varies, probe-based models significantly outperform weight space models since they are more robust to architectural changes. Transformer Accuracy Prediction. We evaluate HiddenProbe on test-accuracy prediction for Transformers trained on MNIST and AGNews. Performance for the fuill dataset is shown in Table 3.Following the evaluation protocol of Tran et al. (2026), we additionally report results across different accuracy thresholds in Appendix D.4 in tables 6,7. Method MNIST-Transformers AGNews-Transformers MLP 0.866±0.0020.866± 0.002 0.879±0.0060.879± 0.006 STATNN (Unterthiner et al., 2020) 0.881±0.0010.881± 0.001 0.841±0.0020.841± 0.002 XGBoost (Chen and Guestrin, 2016) 0.860±0.0020.860± 0.002 0.859±0.0010.859± 0.001 LightGBM (Ke et al., 2017) 0.858±0.0020.858± 0.002 0.835±0.0010.835± 0.001 Random Forest (Breiman, 2001) 0.772±0.0020.772± 0.002 0.774±0.0030.774± 0.003 Transformer-NFN (Tran et al., 2025) 0.905±0.0020.905± 0.002 0.910±0.0010.910± 0.001 Transformer-NFN large (Tran et al., 2026) 0.907±0.0010.907± 0.001 0.913±0.0010.913± 0.001 Transformer-NFN Quasi (Tran et al., 2026) 0.911±0.0010.911± 0.001 0.914±0.0010.914± 0.001 ProbeGen 0.898±0.0010.898± 0.001 0.885±0.0010.885± 0.001 HiddenProbe 0.920±0.0030.920± 0.003 0.916±0.0020.916± 0.002 Table 3: Test accuracy prediction on the MNIST-Transformers and AGNews-Transformers benchmarks, evaluated on the full datasets. Performance is measured by Kendall’s τ, reported as mean ± standard error over five seeds. The best and second-best results are highlighted. As observed, (1)HiddenProbe achieves state-of-the-art performance on both benchmarks, beating Transformer-NFN models with lesser parameter count (2) Even output-only ProbeGen is competitive with the benchmarks. Varying performance with the number of probes In the first part of the experiments, we showed that using the intermediate layers adds significant empirical improvement. Now we want to see how the performance of the probe-based models, namely ProbeGen and HiddenProbe, changes as we vary the number of probes. We compare HiddenProbe with ProbeGen using N∈16,32,64,128,256N∈\16,32,64,128,256\ probes. We take seed 0 results for all plots. Results are in Figure 3. Figure 3: Performance of HiddenProbe and ProbeGen as the number of probes N varies. FMNIST INR classification is evaluated by test accuracy, while regression tasks are evaluated by test Kendall’s τ. Higher is better in all panels. We make the following observations: (1) HiddenProbe generally improves as the probe count increases. Such a trend is also observed for ProbeGen but to a weaker extent (2) HiddenProbe is comparable to, or better than, ProbeGen at a smaller ( 0.5×0.5× or less) probe count. This supports the efficiency of including hidden layer information in probe-based learning. Additional Information In all our experiments, HiddenProbe uses the same number of parameters as ProbeGen (see D.5 for a complete parameter count comparison). A runtime comparison of HiddenProbe, ProbeGen, and other DWS models is provided in Section D.6. Synthetic experiments evaluating model reconstruction from finite probes are presented in Appendix D.1. 8 Conclusion In this work, we have done a rigorous theoretical analysis of function-level identification of neural networks from probes. We have shown that one can use probe-based methods to identify a neural network from its outputs and also design universal approximators to learn properties of the neural net from probes. Across architectures like MLPs and Multi head attention blocks, we have shown that this identification can be done with much less number of probes if hidden layer information is incorporated, and we have accordingly proposed HiddenProbe, which learns properties of neural nets from full hidden traces. Our experiments demonstrate consistent improvements of HiddenProbe over ProbeGen and strong weight-space baselines across diverse tasks, achieving state-of-the-art performance in several. We also demonstrated that incorporating hidden-layer information enables our models to match or surpass ProbeGen using substantially fewer probes, often requiring only half as many. Together, these results establish hidden-layer probing as an effective approach to learning neural functionals across different architectures and tasks. An interesting direction of future work could be designing such methods for larger architectures like language models or diffusion kernels. 9 Acknowledgment N.D. and Y.S. are funded by the Israeli Science Foundation grant no.272/23 and DFG-ISF grant no. 3597/26. H.M. is supported by the Israel Science Foundation through a personal grant (ISF 264/23) and an equipment grant (ISF 532/23), and by the Career Advancement Chairs in Artificial Intelligence – Schmidt Futures. AI use statement In this work, we used ChatGPT to assist in refining the exposition of mathematical proofs and to check mathematical arguments. We also used ChatGPT for language editing and improving the readability of the manuscript. All AI-assisted mathematical arguments, claims, and edits were manually reviewed and verified by the authors. We have also used ChatGPT and Claude for AI-assisted coding but all the data and codes were manually checked and verified. Generative AI tools were not used to generate experimental results, numerical results, or data reported in the paper. We take responsibility for the final content of this work, including any text, claims, and mathematical exposition and code produced with the aid of generative AI. References Alain and Bengio (2017) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR), Workshop Track, External Links: Link Cited by: §1.1. Amir et al. (2023) T. Amir, S. J. Gortler, I. Avni, R. Ravina, and N. Dym Neural injective functions for multisets, measures and graphs via a finite witness theorem. In Advances in Neural Information Processing Systems, Vol. 36, p. 42516–42551. External Links: Link Cited by: §B.1, §B.4, Appendix B, §2. Andreis et al. (2024) B. Andreis, S. Bedionita, P. H. S. Torr, and S. J. Hwang Set-based neural network encoding without weight tying. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §D.3. Bartlett et al. (2019) P. L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research 20 (63), p. 1–17. External Links: Link Cited by: §B.3, §B.3. Belinkov (2022) Y. Belinkov Probing classifiers: promises, shortcomings, and advances. Computational Linguistics 48 (1), p. 207–219. External Links: Document, Link Cited by: §1.1. Bhattamishra et al. (2026) S. Bhattamishra, K. Shah, M. Hahn, and V. Kanade Provably learning attention with queries. In Proceedings of the 43rd International Conference on Machine Learning, External Links: Link Cited by: §B.4, §1.1. Blumer et al. (1989) A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth Learnability and the vapnik-chervonenkis dimension. Journal of the ACM 36 (4), p. 929–965. External Links: Document Cited by: §B.3. Breiman (2001) L. Breiman Random forests. Machine Learning 45 (1), p. 5–32. External Links: Document Cited by: Table 6, Table 7, Table 3. Chen and Guestrin (2016) T. Chen and C. Guestrin XGBoost: a scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 785–794. External Links: Document Cited by: Table 6, Table 7, Table 3. Chen et al. (2019) Z. Chen, S. Villar, L. Chen, and J. Bruna On the equivalence between graph isomorphism testing and function approximation with gnns. Advances in neural information processing systems 32. Cited by: §6. Cotler et al. (2023) J. Cotler, K. S. Tai, F. Hernández, B. Elias, and D. Sussillo Analyzing populations of neural networks via dynamical model embedding. External Links: 2302.14078, Link Cited by: §1.1. Dayan et al. (2026) A. Dayan, Y. Eitan, and H. Maron On the expressive power of permutation-equivariant weight-space networks. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1, footnote 1, footnote 1. Dym and Gortler (2025) N. Dym and S. J. Gortler Low-dimensional invariant embeddings for universal geometric learning. Foundations of Computational Mathematics 25 (2), p. 375–415. External Links: Document Cited by: §6. Gurnee et al. (2023) W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas Finding neurons in a haystack: case studies with sparse probing. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §1.1. Han et al. (2026) X. Han, Z. Wang, B. Zhao, B. Zhang, J. Li, D. Borth, R. Yu, H. Maron, Y. Ye, L. Yin, and F. Neri A survey of weight space learning: understanding, representation, and generation. External Links: 2603.10090, Link Cited by: §1.1, §1. Hanin and Rolnick (2019) B. Hanin and D. Rolnick Complexity of linear regions in deep networks. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 2596–2604. External Links: Link Cited by: §D.1. Haussler and Welzl (1987) D. Haussler and E. Welzl ϵε-nets and simplex range queries. Discrete & Computational Geometry 2, p. 127–151. External Links: Document Cited by: §B.3. Heo et al. (2026) E. Heo, K. Seo, and J. Yoo What linear probes miss: multi-view probing for weight-space learning. External Links: 2605.23410, Link Cited by: §1.1, §1.2. Herrmann et al. (2024) V. Herrmann, F. Faccio, and J. Schmidhuber Learning useful representations of recurrent neural network weight matrices. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 18205–18227. External Links: Link Cited by: §1.1. Horwitz et al. (2025) E. Horwitz, B. Cavia, J. Kahana, and Y. Hoshen Learning on model weights using tree experts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 20468–20478. Cited by: §1.1, §1.2. Kahana et al. (2025) J. Kahana, E. Horwitz, I. Shuval, and Y. Hoshen Deep linear probe generators for weight space learning. In The Thirteenth International Conference on Learning Representations, p. 96360–96377. External Links: Link Cited by: §C.1.2, §D.2, §D.3, §1.1, §1.2, §1, §7, §7. Kalogeropoulos et al. (2024) I. Kalogeropoulos, G. Bouritsas, and Y. Panagakis Scale equivariant graph metanetworks. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §D.2, §D.3, Table 8, §1.1, §7, §7. Ke et al. (2017) G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T. Liu LightGBM: a highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, Vol. 30, p. 3146–3154. Cited by: Table 6, Table 7, Table 3. Kim and Kim (2026) S. Kim and I. Kim Provably learning multi-head attention with queries. External Links: 2608.03294, Link Cited by: §B.4, §B.4, §B.4, §1.1. Kofinas et al. (2024) M. Kofinas, B. Knyazev, Y. Zhang, Y. Chen, G. J. Burghouts, E. Gavves, C. G. M. Snoek, and D. W. Zhang Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §D.2, §D.3, §1.1, §1.1, §1, §7, §7. Lee et al. (2019) J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh Set transformer: a framework for attention-based permutation-invariant neural networks. In International conference on machine learning, p. 3744–3753. Cited by: §A.1, §B.5, §5. Leshno et al. (1993) M. Leshno, V. Ya. Lin, A. Pinkus, and S. Schocken Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks 6, p. 861–867. External Links: Link Cited by: §B.5. Lim et al. (2024) D. Lim, H. Maron, M. T. Law, J. Lorraine, and J. Lucas Graph metanetworks for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1.1. Navon et al. (2023) A. Navon, A. Shamsian, I. Achituve, E. Fetaya, G. Chechik, and H. Maron Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, p. 25790–25816. Cited by: §A.1, §D.2, §1.1, §1, §1, §5. Rónyai et al. (2001) L. Rónyai, L. Babai, and M. K. Ganapathy On the number of zero-patterns of a sequence of polynomials. Journal of the American Mathematical Society 14 (3), p. 717–735. External Links: Document Cited by: §B.3. Sverdlov et al. (2025) Y. Sverdlov, I. Springer, and N. Dym Revisiting multi-permutation equivariance through the lens of irreducible representations. In The Thirteenth International Conference on Learning Representations, p. 8505–8527. External Links: Link Cited by: §5. Tran et al. (2026) V. Tran, A. Nguyen, B. Guérand, T. N. Vo, and T. M. Nguyen Quasi-equivariant metanetworks. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §D.2, §D.4, Table 6, Table 6, Table 7, Table 7, Table 8, §7, Table 3, Table 3. Tran et al. (2025) V. Tran, T. N. Vo, A. Nguyen The, T. Tran Huu, M. Nguyen-Nhat, T. Tran, D. Pham, and T. M. Nguyen Equivariant neural functional networks for transformers. In International Conference on Learning Representations, Cited by: §A.2, Table 6, Table 7, Table 8, §5, Table 3. Tran et al. (2024) V. Tran, T. N. Vo, T. Tran-Huu, A. T. Nguyen, and T. M. Nguyen Monomial matrix group equivariant neural functional networks. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §D.2, Table 8, §1.1. Unterthiner et al. (2020) T. Unterthiner, D. Keysers, S. Gelly, O. Bousquet, and I. Tolstikhin Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448. Cited by: §D.3, Table 6, Table 7, Table 3. Vardi et al. (2022) G. Vardi, G. Yehudai, and O. Shamir On the optimal memorization power of relu neural networks. In International Conference on Learning Representations, External Links: Link Cited by: §1.1. Vo et al. (2025) T. Vo, H. V. Tran, T. T. Huu, A. N. The, T. Tran, M. Nguyen-Nhat, D. Pham, and T. M. Nguyen Equivariant polynomial functional networks. In Proceedings of the 42nd International Conference on Machine Learning, Vol. 267, p. 61689–61744. Cited by: §D.2, §7. Wilkie (2009) A. J. Wilkie O-minimal structures. In Séminaire Bourbaki, Volume 2007/2008, Astérisque, p. 131–142. External Links: Document, Link Cited by: §B.4. Wu et al. (2026) D. Wu, H. Liu, Z. Chi, Y. Yu, K. N. Plataniotis, and Y. Wang Dynamic neural graph encoding of inference processes in deep weight space. Transactions on Machine Learning Research. Cited by: §D.3, §7. Zhou et al. (2023a) A. Zhou, K. Yang, K. Burns, A. Cardace, Y. Jiang, S. Sokota, J. Z. Kolter, and C. Finn Permutation equivariant neural functionals. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §D.3, §1.1, §1, §7. Zhou et al. (2023b) A. Zhou, K. Yang, Y. Jiang, K. Burns, W. Xu, S. Sokota, J. Z. Kolter, and C. Finn Neural functional transformers. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §D.2, §D.3, §1.1, §1, §7. Appendix A HiddenProbe: Probe Based learning using Hidden Neural Activations In this appendix we describe in full detail HiddenProbe, our probe-based architecture for learning from hidden-layer activations. A.1 HiddenProbe: A probe-based model for learning on weight spaces We begin by discussing the permutation symmetries of hidden activations induced by weight-space symmetries, which motivate the design of HiddenProbe. Symmetries in weight spaces and hidden Layers Consider a depth-L MLP. Let P0=PL=IP_0=P_L=I, and let P1,…,PL−1P_1,…,P_L-1 be arbitrary permutation matrices. The transformation Wi↦Pi+1WiPi⊤,bi↦Pi+1bi, 0≤i<L, W_i P_i+1W_iP_i ,\,b_i P_i+1b_i,\,0≤ i<L, permutes the neurons within the hidden layers while leaving the realized function unchanged. As shown in prior work Navon et al. (2023), architectures that explicitly respect these symmetries can substantially outperform models that ignore them. We therefore design HiddenProbe to be invariant to hidden-neuron permutations. Under the above transformation, the representation at hidden layer i transforms as xi(p,θ)↦Pixi(p,θ),i∈[L−1]x_i(p;θ) P_ix_i(p;θ),\,i∈[L-1] Thus, each hidden layer is equivariant to its corresponding permutation PiP_i. In contrast, the input and final output are unchanged, since P0=PL=IP_0=P_L=I. We now describe how this symmetry acts on hidden probe responses. Consider a tuple of probes =(p1,…,pN).P=(p_1,…,p_N). For hidden layer i, define the response vector of neuron k across all probes as vi,k≔(xi,k(p1,θ),…,xi,k(pN,θ))∈ℝN.v^i,k (x_i,k(p_1;θ),…,x_i,k(p_N;θ) ) ^N. Since xix_i is equivariant to PiP_i, applying a hidden-neuron permutation simply permutes the collection vi,k:k∈[di]. \v^i,k:k∈[d_i] \. Therefore, each hidden layer is naturally represented as an unordered set of did_i vectors in ℝNR^N. Overview of HiddenProbe Motivated by this observation, HiddenProbe processes each hidden layer using a permutation-invariant set encoder. Specifically, we encode every hidden layer i as rϕi≔Tϕi(vi,k:k∈[di]),r_φ^i T_φ^i ( \v^i,k:k∈[d_i] \ ), where TϕiT_φ^i is a layer-wise set encoder. In our implementation, TϕiT_φ^i is an invariant Set Transformer Lee et al. (2019), making rϕir_φ^i invariant to permutations of the neurons in layer i. The resulting hidden-layer representations are concatenated in layer order, together with the final-layer probe outputs: Φ,ϕhidden(θ)=[rϕ1;…;rϕL−1;f(p1,θ);…;f(pN,θ)]. _P,φ^hidden(θ)= [r_φ^1;…;r_φ^L-1;f(p_1;θ);…;f(p_N;θ) ]. A final predictor ρϕ _φ maps this representation to the target: y^(θ)=ρϕ(Φ,ϕhidden(θ)). y(θ)= _φ( _P,φ^hidden(θ)). The probes p1,…,pNp_1,…,p_N, the layer-wise encoders TϕiT_φ^i (allowed to be different across layers but can have shared parameters to reduce overall parameter count), and the predictor ρϕ _φ are trained jointly. The learned probes are shared across all target networks. A.2 HiddenProbe for transformers Learning a mixed-length probe bank. Here we introduce HiddenProbe, a probe-based model for learning on softmax-attention based transformers. Our reconstruction arguments from Theorem 5 and Algorithm 2 use different sequence lengths: singleton probes recover the value matrix, while multi-token probes recover the attention weights. This suggests that probes of different lengths can provide complementary information about a target. We retain this design principle, but learn a shared set of probes of varying sequence length rather than imposing the exact configurations. Design of HiddenProbe predictor For each probe Pj∈P_j , we record the outputs of the transformer blocks, the final normalized token states, and the decoder output: θ=(X1(Pj,θ),⋯,XL(Pj,θ),Xnorm(Pj,θ),f(Pj,θ))Pj∈.T_θ= (X_1(P_j;θ),·s,X_L(P_j;θ),\,X_norm(P_j;θ),\,f(P_j;θ) )_P_j . For each observed layer and channel, we concatenate its responses across all probes and token positions, using a fixed order. A shared MLP encodes each resulting channel trajectory into one feature vector. These channel features and final logit features are passed to a transformer followed by an MLP to give the output: y^θ=Rϕ(θ) y_θ=R_φ(T_θ). Incorporating weight space symmetries Head permutations and the transformations Qa↦QaRaQ_a Q_aR_a, Ka↦KaRa−⊤K_a K_aR_a^- and Va↦VaSaV_a V_aS_a, Oa↦Sa−1OaO_a S_a^-1O_a, for invertible Ra,SaR_a,S_a, leave the attention output unchanged. Consequently, they leave every recorded block response and classifier output unchanged, so HiddenProbe is invariant to these symmetries by design. Handling such symmetries remains a key challenge in works like Tran et al. (2025), which we get for free. Appendix B Proof of Technical results See 1 Proof. The proof is based on the Finite Witness Theorem (Theorem A.2 in Amir et al. (2023)), which states that, for an analytic G:ℝa×ℝb→ℝG:R^a×R^b , and Lebesgue almost every w1,…,wa+1∈ℝbw_1,…,w_a+1 ^b, y∈ℝa|G(y;wi)=0,∀i=1,…,a+1=y∈ℝa|G(y;w)=0,∀w∈ℝb\y ^a|\,G(y;w_i)=0,∀ i=1,…,a+1\=\y ^a|\,G(y;w)=0,∀ w ^b\ To prove our theorem, we define the real analytic function G(θ1,θ2,p):=‖f(p,θ1)−f(p,θ2)‖22G( _1, _2;p):=\|f(p; _1)-f(p; _2)\|_2^2. Since the pair (θ1,θ2)( _1, _2) resides in ℝ2qR^2q, the Finite Witness Theorem implies that for 2q+12q+1 random samples p1,…,p2q+1p_1,…,p_2q+1, we will have that for all θ1,θ2 _1, _2, f(pi,θ1)−f(pi,θ2)=0f(p_i; _1)-f(p_i; _2)=0 for all i=1,…,2q+1i=1,…,2q+1, if and only if f(p,θ1)−f(p,θ2)=0f(p; _1)-f(p; _2)=0 for all p∈ℝdp ^d. ∎ B.1 Identification of Analytic MLPs from Hidden Probes We borrow notations from Section 3.1 and Algorithm 1, but occasionally them slightly. Let L∈ℕL , L≥1L≥ 1, and let d0,…,dLd_0,…,d_L be positive integers. Let ⊆ℝd0X ^d_0 be nonempty, open, and connected. For each i∈1,…,Li∈\1,…,L\, let Θi⊆ℝpi _i ^p_i be nonempty, open, and connected, and let fi:ℝdi−1×Θi→ℝdif_i:R^d_i-1× _i ^d_i be jointly real analytic. Define the following notations: Θ≔∏i=1LΘi,θ≔(θ1,…,θL)∈Θ,q≔∑i=1Lqi,qmax _i=1^L _i,\,θ ( _1,…, _L)∈ ,\,q _i=1^Lq_i,\,q_ ≔max1≤i≤Lqi. _1≤ i≤ Lq_i. The full model is the composition of these layer maps: x0(u,θ)≔u,xi(u,θ)≔fi(xi−1(u,θ),θi),∀i∈[L],f(u,θ)≔xL(u,θ).x_0(u;θ) u,\,x_i(u;θ) f_i (x_i-1(u;θ), _i ),\,∀ i∈[L],\,f(u;θ) x_L(u;θ). (4) Given a tuple of probes =(p1,…,pN)∈NP=(p_1,…,p_N) ^N, define iθ()≔(xi(p1;θ)⋯xi(pN;θ))∈ℝdi×N,i=0,…,L.¯iθ=(iθN⊤) split&X_i^θ(P) pmatrixx_i(p_1;θ)·s x_i(p_N;θ) pmatrix ^d_i× N,\,i=0,…,L.\\ & X_i^θ= pmatrixX_i^θ\\ 1_N pmatrix split (5) Thus, full intermediate observation records the sampled states at every layer, including the final output. Identification of Arbitrary Analytic Nets from Hidden Layers: Theorem 1 shows that 2q+12q+1 final-layer probes suffice to distinguish any two functionally distinct models in an analytic family. Here, we show that for analytic sequential models of the form in (4), access to intermediate-layer observations reduces the sufficient probe count to q+qmax+1q+q_ +1. Theorem 8 (Identification from intermediate observations). Under the preceding assumptions, suppose N≥q+qmax+1N≥ q+q_ +1. Then, for Lebesgue-almost every ∈NP ^N, simultaneously for every θ,θ′∈Θθ,θ ∈ , one has iθ()=iθ′()∀i∈0,⋯,L⟹f(u,θ)=f(u,θ′),∀u∈ _i^θ(P)=X_i^θ (P)\,∀\,i∈\0,·s,L\\, \,f(u;θ)=f(u;θ ),\,∀\,u Proof. For i∈[L]i∈[L], we use the following notations: q<i≔∑k=1i−1qk,Θ<i≔∏k=1i−1Θk,Qi≔q<i+2qi. q_<i _k=1^i-1q_k,\, _<i _k=1^i-1 _k,\,Q_i q_<i+2q_i. For i=1i=1, set q<1=0q_<1=0 and Θ<1=ℝ0 _<1=R^0. The state xi−1(u,θ)x_i-1(u;θ) depends only on (θ1,⋯,θi−1)( _1,·s, _i-1). For η=(η1,⋯,ηi−1)∈Θ<iη=( _1,·s, _i-1)∈ _<i, we therefore write xi−1(u,η)x_i-1(u;η) for the state generated by these blocks and for i=1i=1, this means x0(u,η)=ux_0(u;η)=u. We define Ωi≔Θ<i×Θi×Θi, _i _<i× _i× _i, and Hi(u,η,α,β)≔‖fi(xi−1(u,η),α)−fi(xi−1(u,η),β)‖22.H_i(u;η,α,β) \|f_i (x_i-1(u;η),α )-f_i (x_i-1(u;η),β ) \|_2^2. By composition, HiH_i is jointly real analytic on ×ΩiX× _i, and Ωi _i is an open connected subset of ℝQiR^Q_i. Moreover, we have Qi+1=q<i+2qi+1≤q+qi+1≤q+qmax+1≤N. Q_i+1=q_<i+2q_i+1≤ q+q_i+1≤ q+q_ +1≤ N. The Finite Witness Theorem (Amir et al., 2023, Theorem A.2) provides a Lebesgue-null set Ei⊆Qi+1E_i ^Q_i+1 such that, for every (p1,⋯,pQi+1)∉Ei(p_1,·s,p_Q_i+1)∉ E_i and every (η,α,β)∈Ωi(η,α,β)∈ _i, Hi(pn,η,α,β)=0,∀n∈[Qi+1]⟹Hi(u,η,α,β)=0,∀u∈. H_i(p_n;η,α,β)=0,\ ∀\,n∈[Q_i+1] H_i(u;η,α,β)=0,\ ∀\,u . We define Bi≔Ei×N−Qi−1,B≔⋃i=1LBi,B_i E_i×X^N-Q_i-1,\,B _i=1^LB_i, where 0X^0 denotes a singleton. Each BiB_i, and hence B, is Lebesgue-null. For every =(p1,⋯,pN)∉BP=(p_1,·s,p_N)∉ B, the preceding implication holds at every layer, simultaneously for all (η,α,β)∈Ωi(η,α,β)∈ _i. In particular, B does not depend on the pair of parameters being compared. Fix ∉BP∉ B and θ=(θ1,⋯,θL)θ=( _1,·s, _L), θ′=(θ1′,⋯,θL′)θ =( _1 ,·s, _L ) satisfying iθ()=iθ′(),i=0,⋯,L.X_i^θ(P)=X_i^θ (P),\,i=0,·s,L. We prove by induction that xi(⋅,θ)≡xi(⋅,θ′)x_i(·;θ)≡ x_i(·;θ ) on X. For i=0i=0, both maps are the identity. Suppose the assertion holds for i−1i-1, and set η=(θ1,⋯,θi−1)η=( _1,·s, _i-1). Then, for every u∈u , we have xi−1(u,η)=xi−1(u,θ)=xi−1(u,θ′),andHi(u,η,θi,θi′)=‖xi(u,θ)−xi(u,θ′)‖22. x_i-1(u;η)=x_i-1(u;θ)=x_i-1(u;θ ),\,and\,H_i(u;η, _i, _i )= \|x_i(u;θ)-x_i(u;θ ) \|_2^2. Equality at the probes P therefore gives Hi(pn;η,θi,θi′)=0,n=1,⋯,N.H_i(p_n;η, _i, _i )=0,\,n=1,·s,N. Since ∉BiP∉ B_i, the finite witness theorem implies Hi(u,η,θi,θi′)=0H_i(u;η, _i, _i )=0 for every u∈u . Thus xi(⋅,θ)≡xi(⋅,θ′)x_i(·;θ)≡ x_i(·;θ ), completing the induction. ∎ When the activation is also injective and non polynomial: We now show that if the activation is analytic, injective, and non-polynomial, the sufficient number of probes can be substantially reduced. We prove a stronger version of Theorem 2 for that. We first prove a couple of lemmas that will be useful in proving the result. Lemma 1. Let y1,…,yK∈ℝqy_1,…,y_K ^q be pairwise distinct. Let σ:ℝ→ℝσ:R be real analytic and non-polynomial. Then span(σ(w⊤y1+b),…,σ(w⊤yK+b)):(w,b)∈ℝq×ℝ=ℝK. \ (σ(w y_1+b),…,σ(w y_K+b) ):(w,b) ^q×R \=R^K. Consequently, for Lebesgue-a.e. (w1,b1),…,(wK−1,bK−1)∈(ℝq×ℝ)K−1,(w_1,b_1),…,(w_K-1,b_K-1)∈(R^q×R)^K-1, the matrix M=(1⋯1σ(w1⊤y1+b1)⋯σ(w1⊤yK+b1)σ(wK−1⊤y1+bK−1)⋯σ(wK−1⊤yK+bK−1)) M= pmatrix1&·s&1\\ σ(w_1 y_1+b_1)&·s&σ(w_1 y_K+b_1)\\ && \\ σ(w_K-1 y_1+b_K-1)&·s&σ(w_K-1 y_K+b_K-1) pmatrix is invertible. Proof. We consider the non-trivial case K≥2K≥ 2. Choose v∈ℝqv ^q such that the numbers tn≔v⊤ynt_n v y_n, n∈[K]n∈[K], are pairwise distinct. Such a choice is possible because the bad set is the finite union of proper hyperplanes ⋃n<n′v∈ℝq:v⊤(yn−yn′)=0 _n<n \v ^q:v (y_n-y_n )=0\, each of which is proper since yn≠yn′y_n≠ y_n . We first prove the span claim. Suppose, for contradiction, that the span is proper. Then there exists a nonzero vector c=(c1,…,cK)∈ℝKc=(c_1,…,c_K) ^K such that ∑n=1Kcnσ(w⊤yn+b)=0∀(w,b)∈ℝq×ℝ⟹∑n=1Kcnσ(b+atn)=0∀a,b∈ℝ,by taking w=av. split& _n=1^Kc_nσ(w y_n+b)=0\,∀\,(w,b) ^q×R\\ & _n=1^Kc_nσ(b+at_n)=0\,∀\,a,b ,\,by taking w=av. split (6) Since σ is real analytic and non-polynomial, none of the derivatives σ(k)σ^(k), k=0,…,K−1k=0,…,K-1, is identically zero. Indeed, if σ(k)≡0σ^(k)≡ 0 for some k, then σ would be a polynomial of degree at most k−1k-1. Therefore, for each k=0,…,K−1k=0,…,K-1, the zero set of σ(k)σ^(k) is a proper discrete subset of ℝR. Hence the finite union ⋃k=0K−1b∈ℝ:σ(k)(b)=0 _k=0^K-1\b :σ^(k)(b)=0\ does not cover ℝR. Choose b0∈ℝb_0 outside this union. Then we have: σ(k)(b0)≠0,k=0,…,K−1. σ^(k)(b_0)≠ 0,\,k=0,…,K-1. Now set b=b0b=b_0 in equation 6. Since σ is analytic at b0b_0, for all sufficiently small a we have: 0=∑n=1Kcnσ(b0+atn)=∑n=1Kcn∑k=0∞σ(k)(b0)k!aktnk 0= _n=1^Kc_nσ(b_0+at_n)= _n=1^Kc_n _k=0^∞ σ^(k)(b_0)k!a^kt_n^k ⟹ 0=∑k=0∞σ(k)(b0)k!(∑n=1Kcntnk)ak. 0= _k=0^∞ σ^(k)(b_0)k! ( _n=1^Kc_nt_n^k )a^k. Since the convergent power series is identically 00 in a neighbourhood around the origin, every coefficient vanishes, and since σ(k)(b0)≠0σ^(k)(b_0)≠ 0 for k≤K−1k≤ K-1 we get ∑n=1Kcntnk=0 _n=1^Kc_nt_n^k=0 for k=0,…,K−1k=0,…,K-1. Hence c lies in the kernel of the Vandermonde matrix V=(1⋯1t1⋯tKt1K−1⋯tKK−1). V= pmatrix1&·s&1\\ t_1&·s&t_K\\ && \\ t_1^K-1&·s&t_K^K-1 pmatrix. But the tnt_n are pairwise distinct, so we have: detV=∏1≤n<n′≤K(tn′−tn)≠0. V= _1≤ n<n ≤ K(t_n -t_n)≠ 0. Therefore c=0c=0, contradicting the choice of c. This proves the span claim. Now we define v(w,b)≔(σ(w⊤y1+b),…,σ(w⊤yK+b))∈ℝK. v(w,b) (σ(w y_1+b),…,σ(w y_K+b) ) ^K. We first show that there exists some (K−1)(K-1)-tuple of parameters such that the matrix M is full rank. We start with the one-dimensional subspace V0≔spanKV_0 \1_K\. If every vector v(w,b)v(w,b) belonged to V0V_0, then the span of all ridge vectors would be contained in V0V_0, but these vectors span ℝKR^K. Hence there exists (w1,b1)(w_1,b_1) such that v(w1,b1)∉V0v(w_1,b_1)∉ V_0. Set V1≔spanK,v(w1,b1)V_1 \1_K,v(w_1,b_1)\. Continuing inductively (as long as dimVj<K V_j<K some ridge vector lies outside VjV_j), after K−1K-1 steps we obtain parameters (w1,b1),…,(wK−1,bK−1)(w_1,b_1),…,(w_K-1,b_K-1) such that the K vectors K,v(w1,b1),…,v(wK−1,bK−1) 1_K,\,v(w_1,b_1),…,v(w_K-1,b_K-1) are linearly independent in ℝKR^K. Since there are K such vectors, they form a basis of ℝKR^K. Equivalently, the corresponding matrix M has nonzero determinant for at least one choice of parameters. Now we define Δ((w1,b1),…,(wK−1,bK−1))≔detM. ((w_1,b_1),…,(w_K-1,b_K-1)) M. Since σ is real analytic, every entry of M is a real-analytic function of the parameters (w1,b1)(w_1,b_1), …, (wK−1,bK−1)(w_K-1,b_K-1), and therefore Δ is real analytic on the connected open set (ℝq×ℝ)K−1(R^q×R)^K-1. The preceding construction shows that Δ is not identically zero. Hence its zero set is the zero set of a nonzero real-analytic function on a connected open set, so it has Hausdorff dimension strictly smaller than (K−1)(q+1)(K-1)(q+1), implying zero Lebesgue measure. Therefore M is invertible for Lebesgue-a.e. choice of the ridge parameters. ∎ Now, for the following lemma, we define another notation for MLPs with width sequence ≔(d=d0,d1,⋯,dL−1,dL=m)D (d=d_0,d_1,·s,d_L-1,d_L=m). For probes ∈NP ^N and θ∈Θθ∈ _D, we recall the definitions ¯iθ()≔(iθ()N⊤)∈ℝ(di+1)×N X_i^θ(P) pmatrixX_i^θ(P)\\ 1_N pmatrix ^(d_i+1)× N, where iθ()X_i^θ(P) is as defined in equation 5. We show that, for sufficiently large N≔||N |P|, the matrix ¯iθ() X_i^θ(P) is almost always full rank. Lemma 2. Assume that σ:ℝ→ℝσ:R is real analytic, injective, and non-polynomial. Let N≥max0≤i≤L−1(di+1)N≥ _0≤ i≤ L-1(d_i+1), and define E≔E1∪E2, where: E E_1∪ E_2,\ where: E1≔=(p1,…,pN)∈N:rank(p1⋯pN1⋯1)<d+1, E_1 \P=(p_1,…,p_N) ^N:rank pmatrixp_1&·s&p_N\\ 1&·s&1 pmatrix<d+1 \, E2≔⋃1≤n<n′≤N∈N:pn=pn′. E_2 _1≤ n<n ≤ N\P ^N:\,p_n=p_n \. Then dimH(E)<Nd _H(E)<Nd, and for every probe tuple ∈N∖EP ^N E there exists a Lebesgue-null set ℳ⊆ΘM_P _D such that rank¯iθ0()=di+1,∀i∈0,⋯,L−1, X_i _0(P)=d_i+1,\,∀\,i∈\0,·s,L-1\, for every θ0∈Θ∖ℳ _0∈ _D _P. Proof. Since X has nonempty interior, choose x0∈x_0 and ϵ>0ε>0 such that B(x0,ϵ)⊆B(x_0,ε) . Let e1,…,ede_1,…,e_d be the standard basis of ℝdR^d. Then the determinant P(p1,…,pd+1)≔det(p1⋯pd+11⋯1) P(p_1,…,p_d+1) pmatrixp_1&·s&p_d+1\\ 1&·s&1 pmatrix is nonzero for the choice of points x0,x0+ϵ2e1,⋯,x0+ϵ2ed\x_0,x_0+ ε2e_1,·s,x_0+ ε2e_d\: subtracting the first column from each of the others gives the value ±(ϵ/2)d±(ε/2)^d. Hence P is not the zero polynomial. If a probe tuple belongs to E1E_1, then every (d+1)×(d+1)(d+1)×(d+1) minor of the matrix in the definition of E1E_1 vanishes, so P(p1,…,pd+1)=0P(p_1,…,p_d+1)=0. Thus E1⊆P=0×N−d−1E_1 \P=0\×X^N-d-1. Since P is a nonzero polynomial, P=0\P=0\ has Hausdorff dimension strictly smaller than (d+1)d(d+1)d. Therefore dimH(E1)<Nd _H(E_1)<Nd. Moreover, each diagonal set pn=pn′\p_n=p_n \ has Hausdorff dimension (N−1)d<Nd(N-1)d<Nd. Since E is a finite union of such sets, we have dimH(E)<Nd _H(E)<Nd. Fix a probe tuple =(p1,…,pN)∈N∖EP=(p_1,…,p_N) ^N E. Then the probes are pairwise distinct and rank¯0()=d+1rank X_0(P)=d+1. Note that ¯iθ() X_i^θ(P) and the sampled states xi(pn,θ)x_i(p_n;θ) depend on θ=(B0,…,BL−1)θ=(B_0,…,B_L-1) only through the blocks B0,…,Bi−1B_0,…,B_i-1. We shall prove inductively that, for Lebesgue-a.e. choice of the blocks B0,…,Bi−1B_0,…,B_i-1, the following two properties hold: rank¯iθ()=di+1, X_i^θ(P)=d_i+1, (RiR_i) xi(p1,θ),…,xi(pN,θ) are pairwise distinct. x_i(p_1;θ),…,x_i(p_N;θ) are pairwise distinct. (DiD_i) For i=0i=0, property (R0)(R_0) holds by the definition of E1E_1, and (D0)(D_0) holds because the probes p1,…,pNp_1,…,p_N are pairwise distinct; neither involves any block. Now fix 0≤i≤L−20≤ i≤ L-2 and blocks B0,…,Bi−1B_0,…,B_i-1 for which (Di)(D_i) holds. Set yn≔xi(pn;θ)∈ℝdi,n=1,…,N. y_n x_i(p_n;θ) ^d_i,\,n=1,…,N. By (Di)(D_i), the vectors y1,…,yNy_1,…,y_N are pairwise distinct. We first prove (Ri+1)(R_i+1). Let K≔di+1+1K d_i+1+1. Since i+1≤L−1i+1≤ L-1, we have K≤NK≤ N, so the first K vectors y1,…,yKy_1,…,y_K are pairwise distinct. Write (wj⊤,bj)(w_j ,b_j), j∈[di+1]j∈[d_i+1], for the rows of BiB_i and define Mi+1≔(1⋯1σ(w1⊤y1+b1)⋯σ(w1⊤yK+b1)σ(wdi+1⊤y1+bdi+1)⋯σ(wdi+1⊤yK+bdi+1)). M_i+1 pmatrix1&·s&1\\ σ (w_1 y_1+b_1 )&·s&σ (w_1 y_K+b_1 )\\ && \\ σ (w_d_i+1 y_1+b_d_i+1 )&·s&σ (w_d_i+1 y_K+b_d_i+1 ) pmatrix. Up to moving the row of ones to the bottom, Mi+1M_i+1 consists of the first K columns of ¯i+1θ() X_i+1^θ(P), i.e. it is a (di+1+1)×(di+1+1)(d_i+1+1)×(d_i+1+1) minor of ¯i+1θ() X_i+1^θ(P). Applying Lemma 1 in ℝdiR^d_i to the K points y1,…,yKy_1,…,y_K gives that, for Lebesgue-a.e. choice of the rows (w1⊤,b1),…,(wdi+1⊤,bdi+1)(w_1 ,b_1),…,(w_d_i+1 ,b_d_i+1) of BiB_i, the matrix Mi+1M_i+1 is invertible. Therefore, for Lebesgue-a.e. block BiB_i, we have rank¯i+1θ()=di+1+1rank X_i+1^θ(P)=d_i+1+1. Thus (Ri+1)(R_i+1) holds for Lebesgue-a.e. block BiB_i, conditional on the previous blocks satisfying (Di)(D_i). We next prove (Di+1)(D_i+1). Fix n≠n′n≠ n . Since yn≠yn′y_n≠ y_n , if xi+1(pn,θ)=xi+1(pn′,θ)x_i+1(p_n;θ)=x_i+1(p_n ;θ), then, by injectivity of σ applied coordinatewise, we get: Wiyn+bi=Wiyn′+bi. W_iy_n+b_i=W_iy_n +b_i. Equivalently, Wi(yn−yn′)=0W_i(y_n-y_n )=0. Since yn−yn′≠0y_n-y_n ≠ 0, this is a proper linear condition on the entries of WiW_i. Hence, for fixed previous blocks, it defines a Lebesgue-null subset of the space of blocks BiB_i. Taking the finite union over all pairs n≠n′n≠ n , we get that (Di+1)(D_i+1) holds for Lebesgue-a.e. block BiB_i. It remains to pass from a fixed choice of the previous blocks to all parameters. Write qk≔dk+1(dk+1)q_k d_k+1(d_k+1) for the size of block BkB_k and, for −1≤i≤L−2-1≤ i≤ L-2, let ℬi⊆ℝq0+⋯+qiB_i ^q_0+·s+q_i be the set of prefixes (B0,…,Bi)(B_0,…,B_i) for which (Rk)(R_k) or (Dk)(D_k) fails for some k≤i+1k≤ i+1; thus ℬ−1=∅B_-1= . Each ℬiB_i is a Borel set: the failure of (Rk)(R_k) is the vanishing of all (dk+1)×(dk+1)(d_k+1)×(d_k+1) minors of ¯kθ() X_k^θ(P), finitely many real-analytic functions of the prefix, and the failure of (Dk)(D_k) is a finite union of equalities between real-analytic functions. If (B0,…,Bi)∈ℬi(B_0,…,B_i) _i, then either (B0,…,Bi−1)∈ℬi−1(B_0,…,B_i-1) _i-1, or this prefix satisfies (Di)(D_i) and BiB_i belongs to the set F(B0,…,Bi−1)F(B_0,…,B_i-1) of blocks for which (Ri+1)(R_i+1) or (Di+1)(D_i+1) fails, which we have just shown to be Lebesgue-null. Hence, by Tonelli’s theorem, λ(ℬi)≤λ(ℬi−1×ℝqi)+∫(B0,…,Bi−1)∉ℬi−1λ(F(B0,…,Bi−1))d(B0,…,Bi−1)=0, λ(B_i)≤λ (B_i-1×R^q_i )+ _(B_0,…,B_i-1) _i-1λ (F(B_0,…,B_i-1) )\,d(B_0,…,B_i-1)=0, where the first term vanishes because ℬi−1B_i-1 is null by the induction hypothesis and the second because its integrand vanishes. Iterating this over i=0,…,L−2i=0,…,L-2, every ℬiB_i is Lebesgue-null. Finally, since (Ri)(R_i) only involves the blocks B0,…,Bi−1B_0,…,B_i-1, the set of parameters θ∈ℝqθ ^q_D for which (Ri)(R_i) fails for some i∈0,⋯,L−1i∈\0,·s,L-1\ is contained in ℬL−2×ℝqL−1B_L-2×R^q_L-1, which is null; its intersection with Θ _D is the required ℳM_P. This completes the proof. ∎ We now prove Theorem 9, which is a stronger version of Theorem 2. Theorem 9 (Reconstruction from hidden layer activations). Let ⊆ℝd0X ^d_0 be compact with nonempty interior and σ:ℝ→ℝσ:R be real analytic, injective, and non-polynomial. Consider fully connected MLPs with biases, widths =(d0,…,dL)D=(d_0,…,d_L), and raw parameter space Θ=ℝq _D=R^q_D, where q=∑i=0L−1di+1(di+1)q_D= _i=0^L-1d_i+1(d_i+1). Suppose N≥1+max0≤i<LdiN≥ 1+ _0≤ i<Ld_i. For Lebesgue-almost every probe tuple ∈NP ^N, there exists a nonzero real-analytic function Δ:ℝq→ℝ _P:R^q_D such that ℳ=θ:Δ(θ)=0M_P=\θ: _P(θ)=0\ and we have Ψall(θ,)=Ψall(θ0,)⇔θ=θ0,∀θ0∈Θ∖ℳ,∀θ∈Θ _all(θ,P)= _all( _0,P)\, \,θ= _0,\,∀\, _0∈ _D _P,\,∀\,θ∈ _D The exceptional set ℳM_P is closed and satisfies dimHℳ≤q−1 _HM_P≤ q_D-1; hence it is Lebesgue-null and nowhere dense. Proof. Let E⊆NE ^N be the exceptional probe set in Lemma 2, applied with the given N. Then E is Lebesgue-null. Fix ∈N∖EP ^N E and define Δ(θ)≔∏i=0L−1det(¯iθ()¯iθ()⊤),ℳ≔θ∈ℝq:Δ(θ)=0. _P(θ) _i=0^L-1 ( X_i^θ(P) X_i^θ(P) ),\,M_P \θ ^q_D: _P(θ)=0 \. Each entry of ¯iθ() X_i^θ(P) is real analytic in θ, so Δ _P is real analytic on ℝqR^q_D. For any real matrix A, the identity v⊤AA⊤v=‖A⊤v‖22v A v=\|A v\|_2^2 shows that AA⊤A is positive definite exactly when A has full row rank. Consequently, Δ(θ)>0⟺rank¯iθ()=di+1for every i=0,…,L−1. _P(θ)>0\, \,rank X_i^θ(P)=d_i+1\,for every i=0,…,L-1. Lemma 2 gives these ranks simultaneously for Lebesgue-almost every θ∈ℝqθ ^q_D. In particular, they hold at some θ∗ _*, and hence Δ(θ∗)>0 _P( _*)>0. Thus Δ _P is not identically zero. By continuity, ℳM_P is closed and dimHℳ≤q−1 _HM_P≤ q_D-1 since it forms the zero-set of an analytic function. Therefore ℳM_P is Lebesgue-null. It has empty interior, since every nonempty open subset of ℝqR^q_D has positive Lebesgue measure. Being closed with empty interior, it is nowhere dense. Fix θ0=(B00,…,BL−10)∈Θ∖ℳ _0=(B_0^0,…,B_L-1^0)∈ _D _P. By the definition of Δ _P, rank¯iθ0()=di+1,i=0,…,L−1.rank X_i _0(P)=d_i+1,\,i=0,…,L-1. (7) If θ=θ0θ= _0 then trivially Ψall(θ,)=Ψall(θ0,) _all(θ,P)= _all( _0,P). Conversely, let θ=(B0,…,BL−1)∈Θθ=(B_0,…,B_L-1)∈ _D satisfy Ψall(θ,)=Ψall(θ0,) _all(θ,P)= _all( _0,P). We show that Bi=Bi0B_i=B_i^0 layer by layer. Fix i∈0,⋯,L−1i∈\0,·s,L-1\. By definition, iθ()=iθ0()andi+1θ()=i+1θ0(),X_i^θ(P)=X_i _0(P)\,and\,X_i+1^θ(P)=X_i+1 _0(P),. The first equality gives ¯iθ()=¯iθ0() X_i^θ(P)= X_i _0(P), and the second equality reads σ(Bi¯iθ0())=σ(Bi0¯iθ0())entrywise.σ (B_i\, X_i _0(P) )=σ (B_i^0\, X_i _0(P) )\,entrywise.. Since σ is injective and applied entrywise, this implies Bi¯iθ0()=Bi0¯iθ0()B_i X_i _0(P)=B_i^0 X_i _0(P). Equivalently, we have (Wi−Wi0bi−bi0)¯iθ0()=0. (W_i-W_i^0\ \ \ b_i-b_i^0 )\, X_i _0(P)=0. By equation 7, ¯iθ0() X_i _0(P) has full row rank, so the (di+1)×(di+1)(d_i+1)×(d_i+1) matrix ¯iθ0()¯iθ0()⊤ X_i _0(P)\, X_i _0(P) is invertible and ¯iθ0()+≔¯iθ0()⊤(¯iθ0()¯iθ0()⊤)−1 X_i _0(P)^+ X_i _0(P) ( X_i _0(P)\, X_i _0(P) )^-1 is a right inverse of ¯iθ0() X_i _0(P). Multiplying the previous display by it on the right gives Wi=Wi0W_i=W_i^0 and bi=bi0b_i=b_i^0, i.e. Bi=Bi0B_i=B_i^0. Since i was arbitrary, θ=θ0θ= _0. Hence Ψall(θ,)=Ψall(θ0,)⇔θ=θ0 _all(θ,P)= _all( _0,P) θ= _0, and in particular Ψall(⋅,) _all(·,P) is injective on Θ∖ℳ _D _P. ∎ Algorithm to recover weights from Ψall _all for generic targets Our proof of Theorem 9 gives us a way to actually reconstruct the weights rather than just the functions for generic targets. We formally describe the algorithm as follows: Algorithm 1 Weights from hidden activations 1: =(p1,⋯,pN)P=(p_1,·s,p_N), 2: Ψall(θ0,)=(1,⋯,L) _all( _0,P)=(X_1,·s,X_L) 3: 0←(p1,⋯,pN)X_0←(p_1,·s,p_N) 4: for i=0,⋯,L−1i=0,·s,L-1 do 5: ¯i←(iN⊤) X_i← pmatrixX_i\\ 1_N pmatrix 6: i+1←σ−1(i+1)Z_i+1←σ^-1(X_i+1) ⊳ entrywise 7: B^i←i+1¯i† B_i _i+1\, X_i ⊳ pseudoinverse 8: (W^ib^i)←B^i( W_i\ \ b_i)← B_i 9: end for 10: return θ^≔(B^0,⋯,B^L−1) θ ( B_0,·s, B_L-1) B.2 Lower bounds on probe counts For analytic MLPs, we have established upper bounds on the number of probes required for unique function identification, using either final-layer or hidden-layer probes. We now establish nearly matching lower bounds for shallow networks. Since these lower bounds hold for one-hidden-layer networks of arbitrary width, they also provide necessary lower bounds for the general family of MLPs with analytic activations (or analytic, injective, non-polynomial activations when observing Ψall _all). The formal results are stated in Theorem 10. We now prove several lemmas needed to establish the tightness results in Theorem 10. Lemma 3. Let K≥1K≥ 1 and let T be a nonzero real trigonometric polynomial of the form T(t)=A0+∑k=1K(Akcos(kt)+Bksin(kt))T(t)=A_0+ _k=1^K (A_k (kt)+B_k (kt) ), not identically 00. Then T has at most 2K2K distinct zeros in [0,2π)[0,2π). Moreover, suppose another nonzero sum H of the same form has 2K2K distinct zeros in [0,2π)[0,2π). If T vanishes at all those points, then T=λHT=λ H for some real number λ. Proof. Write T(t)=∑k=−KKckeiktT(t)= _k=-K^Kc_ke^ikt and define PT(z)≔∑k=−KKckzk+K.P_T(z) _k=-K^Kc_kz^k+K. The polynomial PTP_T is nonzero, has degree at most 2K2K, and satisfies PT(eit)=eiKtT(t)P_T(e^it)=e^iKtT(t). Distinct zeros modulo 2π2π give distinct roots of PTP_T, proving the first assertion. For the second assertion, PHP_H has degree 2K2K and 2K2K distinct roots, all shared by PTP_T. Hence PT=λPHP_T=λ P_H. Therefore T=λHT=λ H, with λ real because both functions are real-valued and H≢0H ≡ 0. ∎ Lemma 4. Let K≥1K≥ 1 and let H(t)=∑k=1Kγksin(kt)≢0H(t)= _k=1^K _k (kt) ≡ 0 have 2K2K distinct zeros r1,…,r2K∈[0,2π)r_1,…,r_2K∈[0,2π). We define: ≔(r1,…,r2K,r1+2π,…,r2K+2π), (r_1,…,r_2K,r_1+2π,…,r_2K+2π), K _K ≔Q(t)+tR(t):Q∈span1,sin(kt),cos(kt):k∈[K],R∈spancos(kt):k∈[K]. \Q(t)+tR(t): array[]lQ \1, (kt), (kt):k∈[K]\,\\ R \ (kt):k∈[K]\ array \. Writing =(z1,…,z4K)Z=(z_1,…,z_4K), the set of sampled vectors (S(z1),…,S(z4K)):S∈K \ (S(z_1),…,S(z_4K) ):S _K \ is a 3K3K-dimensional linear subspace of ℝ4KR^4K. Moreover, a function S∈KS _K satisfies S(zn)=0S(z_n)=0 for every n=1,…,4Kn=1,…,4K if and only if there exists λ∈ℝλ such that S(t)=λH(t)S(t)=λ H(t) for every t∈ℝt . Proof. Every S∈KS _K can be written as S(t)=Q(t)+tR(t),Q(t)=a0+∑k=1K,(aksin(kt)+bkcos(kt)),R(t)=∑k=1Kckcos(kt).S(t)=Q(t)+tR(t),\,Q(t)=a_0+ _k=1^K,\, (a_k (kt)+b_k (kt) ),\,R(t)= _k=1^Kc_k (kt). We first verify that these coefficients are uniquely determined by S. It suffices to show that S(t)=0S(t)=0 for every t forces all coefficients to be zero. Since Q and R are 2π2π-periodic, subtracting the identity at t from the identity at t+2πt+2π gives 0=S(t+2π)−S(t)=2πR(t).0=S(t+2π)-S(t)=2π R(t). Thus R=0R=0, and then Q=0Q=0. For each ℓ=1,…,K =1,…,K, the elementary sine and cosine integration identities give ∫02πR(t)cos(ℓt)t=πcℓ,∫02πQ(t)t=2πa0,∫02πQ(t)sin(ℓt)t=πaℓ,∫02πQ(t)cos(ℓt)t=πbℓ. _0^2πR(t) ( t)\,dt=π c_ ,\, _0^2πQ(t)\,dt=2π a_0,\, _0^2πQ(t) ( t)\,dt=π a_ ,\, _0^2πQ(t) ( t)\,dt=π b_ ., which uniquely identifies the coefficients, implying linear independence. We next characterize the functions whose sampled values are all zero. Suppose S(zn)=0S(z_n)=0 for every n=1,…,4Kn=1,…,4K. For each j=1,…,2Kj=1,…,2K, periodicity gives 0=S(rj)=Q(rj)+rjR(rj), 0=S(rj+2π)=Q(rj)+(rj+2π)R(rj).0=S(r_j)=Q(r_j)+r_jR(r_j),\,0=S(r_j+2π)=Q(r_j)+(r_j+2π)R(r_j). Subtracting these yields R(rj)=0,j=1,…,2K.R(r_j)=0,\,j=1,…,2K. Since R is a finite cosine sum using frequencies at most K, Lemma 3 implies R=μHR=μ H for some μ∈ℝμ ; if R=0R=0, take μ=0μ=0. For every real t, cosine symmetry and sine antisymmetry give R(−t)=R(t),H(−t)=−H(t).R(-t)=R(t),\,H(-t)=-H(t). Therefore we get: μH(t)=R(t)=R(−t)=μH(−t)=−μH(t).μ H(t)=R(t)=R(-t)=μ H(-t)=-μ H(t). It follows that 2μH(t)=02μ H(t)=0 for every t. Because H is not identically zero, μ=0μ=0, and hence R=0R=0. We now have S=QS=Q and Q(rj)=0Q(r_j)=0 for all j. A second application of Lemma 3 gives S(t)=Q(t)=λH(t)∀t∈ℝS(t)=Q(t)=λ H(t)\,∀\,t for some λ∈ℝλ , including λ=0λ=0 when Q=0Q=0. Conversely, every function S=λHS=λ H vanishes at all rjr_j and rj+2πr_j+2π, since H is 2π2π-periodic. This proves the stated equivalence. It remains to determine the dimension of the set of sampled vectors. For any function ϕφ, write vϕ≔(ϕ(z1),…,ϕ(z4K))∈ℝ4K.v_φ (φ(z_1),…,φ(z_4K) ) ^4K. Choose k∗∈1,…,Kk_*∈\1,…,K\ such that γk∗≠0 _k_*≠ 0, which is possible because H is not identically zero. Consider the following collection of 3K3K functions: ℬ≔1∪sin(kt):1≤k≤K,k≠k∗∪cos(kt):1≤k≤K∪tcos(kt):1≤k≤K. \1\∪\ (kt):1≤ k≤ K,\ k≠ k_*\∪\ (kt):1≤ k≤ K\∪\t (kt):1≤ k≤ K\. We show that their sampled vectors vϕ:ϕ∈ℬ\v_φ:φ \ are linearly independent and that every sampled vector from KV_K is a linear combination of them. For independence, suppose real coefficients dϕd_φ satisfy ∑ϕ∈ℬdϕvϕ=0. _φ d_φv_φ=0. Define S(t)=∑ϕ∈ℬdϕϕ(t).S(t)= _φ d_φ(t). Then S∈KS _K and S(zn)=0S(z_n)=0 for every n. The equivalence already proved implies S=λHS=λ H. The coefficient of sin(k∗t) (k_*t) in the displayed expression for S is zero, because this function was omitted from ℬB. Its coefficient in λHλ H is λγk∗λ _k_*. Uniqueness of the coefficient representation therefore gives 0=λγk∗.0=λ _k_*. Since γk∗≠0 _k_*≠ 0, we obtain λ=0λ=0 and hence S=0S=0. The linear independence established at the beginning of the proof now forces dϕ=0d_φ=0 for every ϕ∈ℬφ . Thus the 3K3K sampled vectors are linearly independent. Finally, because H vanishes at every point of Z, 0=vH=∑k=1Kγkvsin(k⋅).0=v_H= _k=1^K _kv_ (k\,·). Solving for the sampled vector of the omitted sine function gives vsin(k∗⋅)=−∑k=1k≠k∗Kγkγk∗vsin(k⋅).v_ (k_*\,·)=- _ subarrayck=1\\ k≠ k_* subarray^K _k _k_*v_ (k\,·). Every function in KV_K is a linear combination of the functions in ℬB and this one omitted sine function. The last identity therefore shows that every sampled vector from KV_K is a linear combination of vϕ:ϕ∈ℬ\v_φ:φ \. Conversely, every linear combination of these sampled vectors is obtained by sampling the corresponding linear combination of functions in ℬB, which belongs to KV_K. Hence the set of sampled vectors is exactly the linear subspace generated by these 3K3K linearly independent vectors. Its dimension is therefore 3K3K. ∎ Lemma 5. Let ⊆ℝX be a nonempty open interval. If wj≠0w_j≠ 0 for all j∈[m]j∈[m] and bj/wj≠bk/wkb_j/w_j≠ b_k/w_k whenever j≠kj≠ k, then 1,tanh(w1x+b1),…,tanh(wmx+bm)1, (w_1x+b_1),…, (w_mx+b_m) are linearly independent on X. Proof. Suppose C+∑j=1mAjtanh(wjx+bj)=0C+ _j=1^mA_j (w_jx+b_j)=0 on X. Extend the left-hand side to the meromorphic function Φ(z)≔C+∑j=1mAjtanh(wjz+bj). (z) C+ _j=1^mA_j (w_jz+b_j). Since tanh=sinh/cosh = / and cosh′=sinh = , the poles of its jjth summand occur at zj,k=iπ(k+1/2)−bjwj,k∈ℤ, z_j,k= iπ(k+1/2)-b_jw_j,\,k , and have residue Aj/wjA_j/w_j. Their real part is −bj/wj-b_j/w_j, so no pole is shared by two distinct summands. The union of the pole sets is closed and discrete, and its complement in ℂC is connected: a segment between any two points can be detoured around the finitely many poles it meets. By the Identity Theorem of complex analysis, Φ is zero on this complement because it vanishes on X. Its residue at every pole therefore vanishes. Thus Aj/wj=0A_j/w_j=0 for every j, giving Aj=0A_j=0 and then C=0C=0. ∎ Theorem 10 (Tightness of probe complexities). Let f(u,θ)f(u,θ) denote a shallow universal MLP, with analytic activation σ, and layers of dimension =(1,m,1)D=(1,m,1) with m≥1m≥ 1. Thus, the total number of parameters is q=3m+1q=3m+1 and the maximal dimension of any layer is dmax=md_ =m. In each of the following cases, there exists a compact parameter space Θ⊆ℝq _D ^q with nonempty interior for which the stated conclusion holds. We have the following: 1. There exists a real-analytic activation σ (namely, σ=sinσ= ) for which the following holds: if N≤2q−2=6mN≤ 2q-2=6m then there exists a non-empty open interval N⊆ℝNU_N R^N such that for all ∈NP _N ∃θ1,θ2∈Θ, such that Ψfinal(θ1,)=Ψfinal(θ2,) and f(⋅,θ1)≢f(⋅,θ2).∃\, _1, _2∈ _D, such that _final( _1,P)= _final( _2,P) and f(·; _1) ≡ f(·; _2)\ . Thus, a generic shared probe of length 2q−22q-2 cannot identify f(⋅,θ)f(·;θ) from the input-output evaluations. 2. There exists a real-analytic, non-polynomial and injective activation σ (namely, σ=tanhσ= ) for which the following holds: N≤dmax≔mN≤ d_ m. Then, for every ∈NP ^N, ∃θ1,θ2∈Θ, such that Ψall(θ1,)=Ψall(θ2,) and f(⋅,θ1)≢f(⋅,θ2).∃\, _1, _2∈ _D, such that _all( _1,P)= _all( _2,P) and f(·; _1) ≡ f(·; _2)\ . Thus, probes of length N≤dmaxN≤ d_ cannot identify f(⋅,θ)f(·;θ) from the hidden neuron evaluations. Proof. We start with two observations: • To show that the lower bound of 2q−12q-1 is necessary for the family of all MLPs with analytic activations, it is sufficient to come up with one analytic activation for which the 2q−22q-2 probes don’t suffice. We use σ=sinσ= in part 1 and σ=tanhσ= in part 2. • We show the necessary lower bounds 2q−12q-1 or dmax+1d_ +1 using the layer widths =(1,m,1)D=(1,m,1), but we show the tightness for any m∈ℕm . Now by taking m→∞m→∞ we can conclude the general lower bound of 2q−12q-1 is tight. With these arguments, we now prove the two constructive counter examples that establish the lower bounds. We start with proving the first part. First we consider an interval containing [0,4π][0,4π]. We set K≔2mK 2m, ϵ≔1/(100m)ε 1/(100m), and choose base parameters θ10,θ20 _1^0, _2^0 whose affine pre-activations are g1(t) g_1(t) ≔g(t,θ10)=ϵ∑k=1msin(kt), g(t; _1^0)=ε _k=1^m (kt), g2(t) g_2(t) ≔g(t,θ20)=ϵ∑k=m+1K−1sin(kt)+sin(Kt), g(t; _2^0)=ε _k=m+1^K-1 (kt)+ (Kt), H(t) H(t) ≔g1(t)−g2(t). g_1(t)-g_2(t). Both networks have exactly m hidden neurons, positive distinct slopes, nonzero readout coefficients, and zero biases. For tj=(j+1/2)π/Kt_j=(j+1/2)π/K, j=0,…,2K−1j=0,…,2K-1, one has H(tj)=−(−1)j+ej,|ej|≤(K−1)ϵ<1/50. H(t_j)=-(-1)^j+e_j,\,|e_j|≤(K-1)ε<1/50. The signs alternate at these points, including across the final interval to t0+2πt_0+2π. The intermediate value theorem and Lemma 3 therefore give exactly 2K2K distinct zeros modulo 2π2π. We choose representatives r1,…,r2K∈[0,2π)r_1,…,r_2K∈[0,2π) and form the 4K4K-tuple Z of Lemma 4. We consider the derivative, with respect to both parameter vectors, of the difference between the pre-activations. At the base pair it ranges over the functions Q(t)+tR(t),Q+tR∈K.Q(t)+tR(t),\,Q+tR _K. This can be seen by considering the fact that, for a neuron with base slope k, base bias zero, and nonzero base readout coefficient a, the three derivatives are: ∂a(asin(wt+b))=sin(kt),∂b(asin(wt+b))=acos(kt),∂w(asin(wt+b))=atcos(kt). _a (a (wt+b) )= (kt),\, _b (a (wt+b) )=a (kt),\, _w (a (wt+b) )=at (kt). The two networks together contain each slope k∈[K]k∈[K] exactly once; the difference of their output-bias variations supplies the constant function. Thus every coefficient of Q and R can be varied independently. By Lemma 4, the Jacobian of the pre-activation difference on Z has rank 3K=2q−23K=2q-2. We choose 3K3K independent rows and let 0P_0 be their probe locations. We put N0≔2q−2N_0 2q-2 and define ℰ(θ1,θ2,)≔(g(pn,θ1)−g(pn,θ2))n=1N0. ( _1, _2,P) (g(p_n; _1)-g(p_n; _2) )_n=1^N_0. Then ℰ(θ10,θ20,0)=0E( _1^0, _2^0,P_0)=0, and its parameter Jacobian has an invertible N0×N0N_0× N_0 minor. We take the remaining 2q−N0=22q-N_0=2 parameter coordinates at their base values. The implicit function theorem gives parameter pairs θ1(),θ2() _1(P), _2(P), continuous on a nonempty open neighborhood N0U_N_0 of 0P_0, such that ℰ(θ1(),θ2(),)=0∀∈N0.E( _1(P), _2(P),P)=0\,∀\,P _N_0. Consequently their final sine outputs agree at every probe. Choose t∗∈(0,4π)t_*∈(0,4π) with H(t∗)≠0H(t_*)≠ 0. At the base pair, we have |g1(t)|≤mϵ<π/2,|g2(t)|≤1+(m−1)ϵ<π/2.|g_1(t)|≤ mε<π/2,\,|g_2(t)|≤ 1+(m-1)ε<π/2. The sine function is injective on (−π/2,π/2)(-π/2,π/2), so f(t∗,θ10)≠f(t∗,θ20)f(t_*; _1^0)≠ f(t_*; _2^0). We shrink the neighbourhood N0U_N_0 so that this inequality persists, the probe entries remain distinct, and both parameter vectors lie in compact boxes around their base values with positive distinct slopes and nonzero readout coefficients. Their union is a compact set K with nonempty interior containing every constructed pair. Thus the final functions are different for every tuple in this open set. For a general interval X, choose [a,b]⊂[a,b] , a<ba<b, and t=αx+βt=α x+β, α>0α>0, mapping [a,b][a,b] onto [0,4π][0,4π]. Replacing each hidden row (wj,bj)(w_j,b_j) by (αwj,βwj+bj)(α w_j,β w_j+b_j) is an invertible linear change of its parameters. It transports the construction, its open probe set, and the compact parameter set into X. For N<N0N<N_0, let NU_N be the projection of N0U_N_0 onto the first N probe coordinates. This set is nonempty and open. Every tuple in NU_N extends to a tuple in N0U_N_0, whose colliding pair also agrees on the shorter tuple and belongs to the same K. Since a nonempty open set has positive Lebesgue measure, generic shared separation is impossible for N≤2q−2N≤ 2q-2. Now we prove the second part. Fix any θ0=(w,b,a,c)∈int(Θ) _0=(w,b,a,c) ( _D) such that wj≠0w_j≠ 0 for every j and bjwk−bkwj≠0b_jw_k-b_kw_j≠ 0 whenever j<kj<k. These conditions hold for Lebesgue-almost every interior parameter, because their failure is a finite union of zero sets of nonzero polynomials. Fix 1≤N≤m1≤ N≤ m and an arbitrary probe tuple ∈NP ^N. The matrix A A ≔¯1θ0()=(h1(p1,θ0)⋯h1(pN,θ0)hm(p1,θ0)⋯hm(pN,θ0)1⋯1)∈ℝ(m+1)×N X_1 _0(P)= pmatrixh_1(p_1; _0)&·s&h_1(p_N; _0)\\ && \\ h_m(p_1; _0)&·s&h_m(p_N; _0)\\ 1&·s&1 pmatrix ^(m+1)× N has more rows than columns. Hence there exists δ∈ℝm+1∖0δ ^m+1 \0\ with δ⊤A=0δ A=0. Define θ(t)θ(t) by keeping w,bw,b fixed and replacing the last affine block by B1(t)≔B10+tδ⊤,B10=(a⊤c).B_1(t) B_1^0+tδ , B_1^0=(a \ c). The hidden functions are unchanged, and g(x,θ(t))−g(x,θ0) g(x;θ(t))-g(x; _0) =t(δm+1+∑j=1mδjhj(x,θ0)). =t ( _m+1+ _j=1^m _jh_j(x; _0) ). Since δ⊤A=0δ A=0, the right-hand side vanishes at every probe. Therefore the sampled final outputs also agree, and Ψall(θ(t),)=Ψall(θ0,). _all(θ(t),P)= _all( _0,P). It remains to show that the realized functions are different. By Lemma 5, the functions 1,h1(⋅,θ0),…,hm(⋅,θ0)1,h_1(·; _0),…,h_m(·; _0) are linearly independent on X. Since δ≠0δ≠ 0, the function r(x)≔δm+1+∑j=1mδjhj(x,θ0)r(x) _m+1+ _j=1^m _jh_j(x; _0) is not identically zero on X. Consequently, for every t≠0t≠ 0, the output preactivations g(⋅,θ(t))g(·;θ(t)) and g(⋅,θ0)g(·; _0) differ somewhere on X. Injectivity of the outer tanh therefore gives f(⋅,θ(t))≢f(⋅,θ0)on .f(·;θ(t)) ≡ f(·; _0) X. Finally, θ0∈int(Θ) _0 ( _D) and θ(t)→θ0θ(t)→ _0. Thus all sufficiently small nonzero t give parameters in Θ _D and in any prescribed neighborhood of θ0 _0. This proves that, for every probe tuple with N≤mN≤ m, almost every interior target admits arbitrarily close, functionally distinct competitors with identical complete probe observations. For this shallow tanh family, dmax=md_ =m hidden probes fail to identify almost every interior target, whereas the upper bound guarantees identification of almost every target from almost every tuple of dmax+1d_ +1 probes. Thus the hidden-probe threshold is sharp in the worst case over the class of analytic, injective, non-polynomial activations. For the sin family, the nonempty open set of failing probe tuples at N≤2q−2N≤ 2q-2 rules out a general almost-everywhere-in-probes identification guarantee at those cardinalities. Hence the sufficient bound 2q+12q+1 is within two probes of the lower bound 2q−12q-1 for such a guarantee. We do not claim that these bounds are tight for every individual activation. ∎ B.3 ReLU Network Identification from probes We begin with a counterexample showing that ReLU networks cannot be exactly reconstructed from finitely many probes. See 3 Proof. Consider hat functions of the form gc,ϵ(x)≔ReLU(x−c+ϵ)−2ReLU(x−c)+ReLU(x−c−ϵ)=maxϵ−|x−c|,0.g_c,ε(x) (x-c+ε)-2ReLU(x-c)+ReLU(x-c-ε)= \ε-|x-c|,0\. For any c,ϵc,ε, such a function is a ReLUReLU network with three hidden neurons, and it is supported in [c−ϵ,c+ϵ][c-ε,c+ε]. Given any probes p1,⋯,pNp_1,·s,p_N, choose c∈(0,1)c∈(0,1) off the probes and 0<ϵ1<ϵ20< _1< _2 below the distance from c to the probes and to the boundary; then gϵ1g_ _1 and gϵ2g_ _2 vanish at every probe and differ at c. Thus no finite set of probes, however chosen, can completely identify ReLUReLU networks. ∎ The probabilistic setting of identification: Here we describe the probabilistic setting of function identification in ReLU Nets. Let μ be a Borel probability measure on X. The probes are drawn from μ,=(p1,⋯,pN)μ,\,P=(p_1,·s,p_N) with p1,⋯,pNp_1,·s,p_N i.i.d. μ. We also consider the same measure to quantify agreement between functions. For two parameters θ0,θ∈ℝq _0,θ ^q_D we define the disagreement sets: Δμ(θ,θ0)≔μx∈:f(x,θ)≠f(x,θ0)Δμall(θ,θ0)≔μx∈:xi(x,θ)≠xi(x,θ0)for some i∈[L], split& _μ(θ, _0) μ \x :f(x;θ)≠ f(x; _0) \\\ & ^all_μ(θ, _0) μ \x :x_i(x;θ)≠ x_i(x; _0)\ for some i∈[L] \, split (8) as the measure of the inputs at which the two networks compute different outputs. For uniform μ these are fractions of the volume of X. Given thresholds ϵ,δε,δ, we consider guarantees of the form: for every target θ0 _0, with probability at least 1−δ1-δ over the probes, every network f(⋅,θ)f(·,θ) agreeing on those probes also agrees with f(⋅,θ0)f(·, _0) outside a set of measure at most ϵε. If we moreover assume that supθ∈Θ,x∈‖f(x,θ)‖≤B _θ∈ ,x \|f(x;θ)\|≤ B for some constant B, then we would like to show that one can reconstruct fθf_θ up to any ϵε in Ls(μ)L^s(μ) norm. Note that such boundedness assumption is automatic for the hypothesis class of ReLU networks if we assume the input space X and the parameter space Θ to be compact. Theorem 4 presents a semi-formal version of our results for ReLU networks, while Theorem 11 provides the formal statement. Intuitively, this theorem says the following. Since we cannot identify the function fθf_θ exactly everywhere from the probes, we want to identify fθf_θ from Ψall(θ,) _all(θ,P) or Ψfinal(θ,) _final(θ,P) at a large region of the input space X, namely at a set of measure at least 1−ϵ1-ε (where μ()=1μ(X)=1) by using N many probes. Also assuming that the probes are sampled independently at random under the same probability measure μ, we want the above property to hold with high probability (at least 1−δ1-δ). Thus, we want the target function fθf_θ and the reconstructed function fθ^f_ θ to be approximately equal with high probability. This is similar in spirit (but unrelated) to the PAC learning framework used in classical learning theory. Now, we prove Theorem 11 on ReLUReLU networks, which is a formal version of Theorem 4. For that we first define some notations and prove a couple of helper lemmas. Proof of the formal version of Theorem 4: We first state and prove some bounds on counting polynomials and the random-sampling bound used in the proof. The zero-pattern estimate below is a specialization of Rónyai et al. (2001, Theorem 1.1). We still prove it for completeness. Lemma 6 (Counting polynomial tests). Let g1,⋯,gJg_1,·s,g_J be real polynomials of degree at most r in P variables, where P,r,J≥1P,r,J≥ 1, and we define D≔(P+r)−1D P+rr-1. The numbers of distinct vectors ([gj(v)>0])j=1J,([gj(v)≠0])j=1J,v∈ℝP (1[g_j(v)>0] )_j=1^J,\, (1[g_j(v)≠ 0] )_j=1^J,\,v ^P, are at most ∑k=0D(Jk) _k=0^D Jk and (P+rJP) P+rJP, respectively. If J≥DJ≥ D, the first bound is at most (eJ/D)D(eJ/D)^D. Proof. We first bound the number of positive vectors. For this, we choose one parameter for each realized vector. Subtract from every gjg_j a positive number smaller than all its positive values at the chosen parameters. If none of these values is positive, any positive number works. This preserves every chosen binary vector and makes all tested values nonzero. List the D nonconstant monomials of degree at most r. Each shifted polynomial is an affine function of this list. It remains to count strict sign vectors of J affine functions on ℝDR^D. A nonempty set with fixed strict signs is an intersection of open halfspaces and is convex. Adding one zero hyperplane splits such a set only when the hyperplane meets it. The intersections of the split sets with this hyperplane have distinct strict sign vectors for the preceding functions restricted to ℝD−1R^D-1. A repeated hyperplane causes no split, and constant functions add no choices. Thus the maximum count a(J,D)a(J,D) satisfies a(J,D)≤a(J−1,D)+a(J−1,D−1). a(J,D)≤ a(J-1,D)+a(J-1,D-1). Starting from a(0,D)=a(J,0)=1a(0,D)=a(J,0)=1, induction gives a(J,D)≤∑k=0D(Jk)a(J,D)≤ _k=0^D Jk. For J≥DJ≥ D, we set t=D/Jt=D/J and we conclude by the following bounding: tD∑k=0D(Jk)≤∑k=0D(Jk)tk≤(1+t)J≤eD. t^D _k=0^D Jk≤ _k=0^D Jkt^k≤(1+t)^J≤ e^D. Now we bound the number of non-zero vectors. Let S contain the realized sets j:gj(v)≠0\j:g_j(v)≠ 0\. We choose a realizing parameter vSv_S for each S∈S , and define GS≔∏j∈SgjG_S _j∈ Sg_j, with empty product 11. Then GS(vT)≠0G_S(v_T)≠ 0 exactly when S⊆TS T. Ordering the sets by nondecreasing cardinality makes the matrix (GS(vT))S,T∈(G_S(v_T))_S,T upper triangular with nonzero diagonal. The polynomials GSG_S are therefore linearly independent. Each has degree at most rJrJ, and the space of such polynomials has (P+rJP) P+rJP monomials as a basis. This proves the second bound. ∎ For a class C of subsets of X, write Π(M) _C(M) for the largest number of distinct vectors ([pa∈C])a=1M(1[p_a∈ C])_a=1^M, as C∈C varies, on any M fixed inputs. This is the growth function of C from VC theory. Moreover, the following lemma is inspired from the classical double-sampling argument for random ϵε-nets from (Haussler and Welzl, 1987; Blumer et al., 1989). We also use the following consequence of the polynomial sign bound of Bartlett et al. (2019, Lemma 17). Lemma 7 (Counting polynomial inequalities). Let g1,⋯,gJg_1,·s,g_J be real polynomials of degree at most r in P variables, where r≥1r≥ 1 and J≥P≥1J≥ P≥ 1. Then |([gj(v)>0])j=1J:v∈ℝP|≤(4erJP)P. | \ (1[g_j(v)>0] )_j=1^J:v ^P \ |≤ ( 4erJP )^P. Proof. Choose one parameter for each realized binary vector. For each j, choose ηj>0 _j>0 smaller than every positive value of gjg_j at these finitely many parameters; if there is no positive value, choose any ηj>0 _j>0. Replacing gjg_j by gj−ηjg_j- _j preserves the chosen binary vectors and makes every tested value nonzero. The polynomial sign bound therefore gives at most 2(2erJP)P≤(4erJP)P 2 ( 2erJP )^P≤ ( 4erJP )^P such vectors. ∎ Lemma 8 (Random samples detect large disagreement sets). Let K,K,X be compact metric spaces, let μ be a Borel probability measure on X, and let h:K×→[0,∞)h:K×X→[0,∞) be continuous. Set Cθ=u:h(θ,u)>0C_θ=\u:h(θ,u)>0\ and =Cθ:θ∈KC=\C_θ:θ∈ K\. Suppose that D,Q≥1D,Q≥ 1 are integers and Π(M)≤(eMQD)D,M≥D. _C(M)≤ ( eMQD )^D,\,M≥ D. For ϵ,δ∈(0,1)ε,δ∈(0,1), if N≥6ϵ(Dlog12eQϵ+log2δ), N≥ 6ε (D 12eQε+ 2δ ), then, with probability at least 1−δ1-δ, every C∈C missed by N i.i.d. samples from μ has μ(C)≤ϵμ(C)≤ε. Proof. Let E be the event that the sample misses some C with μ(C)>ϵμ(C)>ε. Draw an independent second sample of size N. Let E′E be the event that some C∈C contains no first-sample point and at least Nϵ/2Nε/2 second-sample points. These events are measurable. To see this, we observe that the map θ↦μ(Cθ)θ μ(C_θ) is lower semicontinuous by Fatou’s lemma. In K×NK×X^N or K×2NK×X^2N, the no-hit conditions are closed, whereas the conditions μ(Cθ)>ϵμ(C_θ)>ε and the required number of second-sample hits are open. The defining sets are therefore intersections of a closed set with an open set in a compact metric space. Such sets are countable unions of compact sets, as are their coordinate projections onto the samples. Thus E,E′E,E are Borel. Suppose the first sample belongs to E, and choose a witnessing set of measure q>ϵq>ε. Its second-sample hit count Z has mean NqNq and variance Nq(1−q)Nq(1-q). If Nϵ≥8Nε≥ 8, by Chebyshev’s inequality we get ℙZ<Nϵ/2≤4(1−q)Nq<12. \Z<Nε/2\≤ 4(1-q)Nq< 12. Consequently we have ℙ(E)≤2ℙ(E′)P(E)≤ 2P(E ). Now we pair the two samples and independently swap the members of each pair with probability 1/21/2. Their joint distribution is unchanged. On the fixed 2N2N points there are at most Π(2N) _C(2N) membership vectors. For a fixed vector to witness E′E , all its marked points must be assigned to the second sample. If a pair has two marked points this is impossible; otherwise, with k marked points, its probability is 2−k2^-k. Since k≥Nϵ/2k≥ Nε/2, a union bound gives ℙ(E)≤2Π(2N)2−Nϵ/2.P(E)≤ 2 _C(2N)2^-Nε/2. It remains to check the stated threshold. Put A=Dlog(12eQ/ϵ)+log(2/δ)A=D (12eQ/ε)+ (2/δ) and N0=6A/ϵN_0=6A/ε. Then N0>DN_0>D, N0ϵ>8N_0ε>8, and Dlog2eN0QD+log2δ=A+DlogAD≤2A=N0ϵ3. D 2eN_0QD+ 2δ=A+D AD≤ 2A= N_0ε3. The function Nϵ−3Dlog(2eNQ/D)−3log(2/δ)Nε-3D (2eNQ/D)-3 (2/δ) has derivative ϵ−3D/N≥0ε-3D/N≥ 0 for N≥N0N≥ N_0. Hence, for every such N, Dlog2eNQD+log2δ≤Nϵ3. D 2eNQD+ 2δ≤ Nε3. Applying the growth bound at M=2NM=2N and using 3log2/2>13 2/2>1, we conclude that ℙ(E) (E) ≤2exp(Dlog2eNQD−log22Nϵ)≤δ. ≤ 2 (D 2eNQD- 22Nε )≤δ. ∎ We shall now state and prove Theorem 11, which is the formal version of Theorem 4. Throughout this section, ⊆ℝdX ^d is compact, L≥2L≥ 2, σ=ReLUσ=ReLU in layers 1,⋯,L−11,·s,L-1 and the output layer is linear. We use the following notations for ReLU nets, similar to what we used for analytic nets: x0(u,θ)≔u,x¯i(u,θ)≔(xi(u;θ);1)∈ℝdi+1, 0≤i<L,zi(u,θ)≔Bix¯i−1(u;θ)∈ℝdi,i∈[L],xi(u,θ)≔ReLU(zi(u;θ)),i∈[L−1],xL(u,θ)≔f(u,θ)≔zL(u,θ),H≔∑i=1L−1di,ℓ≔H+m. splitx_0(u;θ)& u,\\ x_i(u;θ)& (x_i(u;θ);1) ^d_i+1,\,0≤ i<L,\\ z_i(u;θ)& B_i x_i-1(u;θ) ^d_i,\,i∈[L],\\ x_i(u;θ)& (z_i(u;θ)),\,i∈[L-1],\\ x_L(u;θ)& f(u;θ) z_L(u;θ),\\ H& _i=1^L-1d_i,\, H+m. split (9) We denote by βij⊤ _ij the j-th row of BiB_i. We call the rows (i,j)(i,j) with i≤L−1i≤ L-1 the hidden rows. H is thus the total number of hidden neurons. Theorem 11 (PAC identification of ReLUReLU networks). Let ⊂ℝdX ^d be compact and μ be a Borel probability measure on X. Consider the family of ReLUReLU MLPs ℱΘF_ _D of ℓ neurons with widths ≔(d,d1,⋯,dL−1,m)D (d,d_1,·s,d_L-1,m) with compact parameter space Θ⊆ℝq _D ^q_D. We consider any ϵ,δ∈(0,1)ε,δ∈(0,1) and integers: N1≥6ϵ(qLlog48eL(ℓ−m+1)ϵ+log2δ),N2≥6ϵ(qlog24eℓϵ+log2δ). N_1≥ 6ε (q_DL 48eL( -m+1)ε+ 2δ ),\,N_2≥ 6ε (q_D 24e ε+ 2δ ). Let 1=(p1,⋯,pN1)∈N1,2=(p1,⋯,pN2)∈N2P_1=(p_1,·s,p_N_1) ^N_1,P_2=(p_1,·s,p_N_2) ^N_2 be probes with pip_i i.i.d. μ. We fix the target θ0∈Θ _0∈ _D. Then we have: (i) ℙ1∼μN1∀θ∈Θ,Ψfinal(θ,1)=Ψfinal(θ0,1)⟹Δμ(θ,θ0)≤ϵ≥1−δP_P_1 μ^N_1 \∀θ∈ _D,\, _final(θ,P_1)= _final( _0,P_1) _μ(θ, _0)≤ε \≥ 1-δ, i.e, with probability at least 1−δ1-δ, any parameter θ∈Θθ∈ _D having the same final output on the probe tuple 1P_1 as the target θ0 _0 has disagreement set of measure at most ϵε. (i) ℙ2∼μN2∀θ∈Θ,Ψall(θ,2)=Ψall(θ0,2)⟹Δμall(θ,θ0)≤ϵ≥1−δP_P_2 μ^N_2 \∀θ∈ _D,\, _all(θ,P_2)= _all( _0,P_2) ^all_μ(θ, _0)≤ε \≥ 1-δ i.e, with probability at least 1−δ1-δ, any parameter θ∈Θθ∈ _D having the same hidden activations on the probe tuple 2P_2 as θ0 _0 can disagree on any hidden layer neuron on an input set of measure at most ϵε. Moreover, let B>0B>0 be such that supθ∈Θ,x∈‖f(x,θ)‖2≤B _θ∈ _D,\ x \|f(x;θ)\|_2≤ B. Then we also have (a) ℙ1∼μN1∀θ∈Θ,Ψfinal(θ,1)=Ψfinal(θ0,1)⟹∥fθ−fθ0∥L2(μ)≤2Bϵ≥1−δP_P_1 μ^N_1 \∀θ∈ _D,\, _final(θ,P_1)= _final( _0,P_1) \|f_θ-f_ _0\|_L^2(μ)≤ 2B ε \≥ 1-δ and (b) ℙ2∼μN2∀θ∈Θ,Ψall(θ,2)=Ψall(θ0,2)⟹∥fθ−fθ0∥L2(μ)≤2Bϵ≥1−δP_P_2 μ^N_2 \∀θ∈ _D,\, _all(θ,P_2)= _all( _0,P_2) \|f_θ-f_ _0\|_L^2(μ)≤ 2B ε \≥ 1-δ Proof. Write p=qp=q_D, and we recall the definition of xi,x¯i,zi,Hx_i, x_i,z_i,H and the details of the ReLU net structure from equation 9. We use the following notations: Pi P_i ≔∑k=1idk(dk−1+1),i∈[L],Dfinal≔∑i=1LPi, _k=1^id_k(d_k-1+1),\,i∈[L],\,D_final _i=1^LP_i, Qfinal Q_final ≔4∑i=1L−1idi+4L,Dall≔p,Qall≔2ℓ. 4 _i=1^L-1i\,d_i+4L,\,D_all p,\,Q_all 2 . Here PiP_i is the number of parameters in the first i layers, and PL=pP_L=p. We fix a parameter θ0 _0 and define Cθfinal=u:f(u,θ)≠f(u,θ0),Cθall=u:xi(u,θ)≠xi(u,θ0) for some i∈[L]. C^final_θ=\u:f(u;θ)≠ f(u; _0)\,\,C^all_θ=\u:x_i(u;θ)≠ x_i(u; _0) for some i∈[L]\. Let ⋆C consist of these sets as θ∈Θθ∈ _D varies. We shall show that, for ⋆∈final,all ∈\final,all\, we have Π⋆(M)≤(eMQ⋆D⋆)D⋆,M≥D⋆ _C (M)≤ ( eMQ_ D_ )^D_ ,\,M≥ D_ , where Π(M) _C(M) denotes the largest number of distinct vectors ([pa∈C])a=1M(1[p_a∈ C])_a=1^M, by varying C∈C on any M fixed inputs. Fix u∈u . By (9), equality with the target at every layer is equivalent to the following 3 conditions: (i) βij⊤x¯i−1(u,θ0)=xi,j(u,θ0),(i<L,xi,j(u,θ0)>0) _ij x_i-1(u; _0)=x_i,j(u; _0),\, (i<L,\ x_i,j(u; _0)>0 ), (i) βij⊤x¯i−1(u,θ0)≤0,(i<L,xi,j(u,θ0)=0) _ij x_i-1(u; _0)≤ 0,\, (i<L,\ x_i,j(u; _0)=0 ) and (i) βLj⊤x¯L−1(u,θ0)=fj(u,θ0),(j∈[m]), _Lj x_L-1(u; _0)=f_j(u; _0),\, (j∈[m] ), where the first two conditions range over all hidden rows. Necessity follows directly from the network equations. Conversely, starting from the common input u, these conditions imply equality of successive states by induction. For fixed θ0 _0 and u, these are affine conditions in θ. Each equality g=0g=0 is determined by the two tests g>0g>0 and −g>0-g>0, and each inequality g≤0g≤ 0 by the test g>0g>0. Thus at most 2ℓ2 affine tests per input suffice. On M inputs, Lemma 6 with P=pP=p and r=1r=1 gives Πall(M)≤(2eMℓp)p,M≥p. _C^all(M)≤ ( 2eM p )^p,\,M≥ p. Now consider final-output observations. We count the hidden activation patterns successively over the layers, as in Bartlett et al. (2019, Section 4). Fix M≥DfinalM≥ D_final inputs p1,⋯,pMp_1,·s,p_M. For i=0,⋯,L−1i=0,·s,L-1, let iS_i be the partition of Θ _D obtained by grouping parameters according to the bits [zk,j(pa;θ)>0],k∈[i],j∈[dk],a∈[M]. 1[z_k,j(p_a;θ)>0],\,k∈[i], j∈[d_k], a∈[M]. Only nonempty parts are retained, and 0=ΘS_0=\ _D\. Fix i∈[L−1]i∈[L-1] and S∈i−1S _i-1. On S, every preceding ReLU equals multiplication by its fixed activation bit at each sampled input. Consequently, each zi,j(pa,θ)z_i,j(p_a;θ) agrees on S with a polynomial of degree at most i in the first PiP_i parameters. Indeed, first-layer preactivations are affine, and each subsequent affine layer increases the degree by at most one. There are MdiMd_i such polynomials. Since Mdi≥M≥Dfinal≥PiMd_i≥ M≥ D_final≥ P_i, Lemma 7 shows that S is divided into at most (4eiMdi/Pi)Pi (4eiMd_i/P_i )^P_i parts of iS_i. Hence |i| |S_i| ≤|i−1|(4eiMdiPi)Pi, ≤|S_i-1| ( 4eiMd_iP_i )^P_i, |L−1| |S_L-1| ≤∏i=1L−1(4eiMdiPi)Pi. ≤ _i=1^L-1 ( 4eiMd_iP_i )^P_i. Zero preactivations are assigned bit zero, so these partitions also include parameters at which some preactivation vanishes. Fix S∈L−1S _L-1. For each a∈[M]a∈[M] and j∈[m]j∈[m], let fa,jS(θ)f_a,j^S(θ) be the polynomial obtained by replacing every hidden ReLU at pap_a by multiplication by its bit on S. It has degree at most L and equals fj(pa,θ)f_j(p_a;θ) for θ∈Sθ∈ S. Define Ra,S(θ)≔∑j=1m(fa,jS(θ)−fj(pa,θ0))2. R_a,S(θ) _j=1^m (f_a,j^S(θ)-f_j(p_a; _0) )^2. This polynomial has degree at most 2L2L, and, for θ∈Sθ∈ S, Ra,S(θ)≠0⇔f(pa,θ)≠f(pa,θ0).R_a,S(θ)≠ 0 f(p_a;θ)≠ f(p_a; _0). The second part of Lemma 6 therefore bounds the number of disagreement vectors on S by (p+2LMp)≤(e(p+2LM)p)p≤(4eLMp)p, p+2LMp≤ ( e(p+2LM)p )^p≤ ( 4eLMp )^p, where we used M≥pM≥ p. Summing over S∈L−1S _L-1 and applying the weighted arithmetic–geometric mean inequality, with weights P1/Dfinal,⋯,PL−1/Dfinal,p/DfinalP_1/D_final,·s,P_L-1/D_final,p/D_final, gives Πfinal(M) _C^final(M) ≤[∏i=1L−1(4eiMdiPi)Pi](4eLMp)p ≤ [ _i=1^L-1 ( 4eiMd_iP_i )^P_i ] ( 4eLMp )^p ≤(eM(4∑i=1L−1idi+4L)∑i=1L−1Pi+p)∑i=1L−1Pi+p ≤ ( eM (4 _i=1^L-1i\,d_i+4L ) _i=1^L-1P_i+p ) _i=1^L-1P_i+p =(eMQfinalDfinal)Dfinal. = ( eMQ_finalD_final )^D_final. Moreover, we have Dfinal≤pL,Qfinal≤4L(H+1).D_final≤ pL,\,Q_final≤ 4L(H+1). We now apply Lemma 8. Set K=ΘK= _D and define: hfinal(θ,u)≔‖f(u,θ)−f(u,θ0)‖2,hall(θ,u)≔maxi∈[L]‖xi(u,θ)−xi(u,θ0)‖2. h_final(θ,u) \|f(u;θ)-f(u; _0)\|_2,\,h_all(θ,u) _i∈[L]\|x_i(u;θ)-x_i(u; _0)\|_2. Both functions are continuous, and their positive sets are CθfinalC^final_θ and CθallC^all_θ, respectively. For part (i), the preceding bounds give 6ϵ(Dfinallog12eQfinalϵ+log2δ)≤6ϵ(pLlog48eL(H+1)ϵ+log2δ)≤N1. 6ε (D_final 12eQ_finalε+ 2δ )\,≤ 6ε (pL 48eL(H+1)ε+ 2δ )≤ N_1. Lemma 8 therefore supplies an event 1⊆N1G_1 ^N_1 of probability at least 1−δ1-δ on which every C∈finalC ^final missed by the probes has measure at most ϵε. For every 1∈1P_1 _1 and every θ∈Θθ∈ _D, Ψfinal(θ,1)=Ψfinal(θ0,1)⟹Cθfinal∩p1,⋯,pN1=∅⟹Δμ(θ,θ0)≤ϵ. _final(θ,P_1)= _final( _0,P_1) C^final_θ∩\p_1,·s,p_N_1\= _μ(θ, _0)≤ε. This proves (i). For part (i), apply the same lemma with D=Dall=pD=D_all=p and Q=Qall=2ℓQ=Q_all=2 . The assumed bound on N2N_2 is its required threshold. Thus, on an event 2⊆N2G_2 ^N_2 of probability at least 1−δ1-δ, every θ∈Θθ∈ _D satisfies Ψall(θ,2)=Ψall(θ0,2)⟹Cθall∩p1,⋯,pN2=∅⟹Δμall(θ,θ0)≤ϵ. _all(θ,P_2)= _all( _0,P_2) C^all_θ∩\p_1,·s,p_N_2\= ^all_μ(θ, _0)≤ε. Since Cθfinal⊆CθallC^final_θ C^all_θ, the final-output disagreement is also at most ϵε. This proves (i). Finally, if the outputs are bounded by B, then, for every 1≤s<∞1≤ s<∞, we have: ‖fθ−fθ0‖Ls(μ)s \|f_θ-f_ _0\|_L^s(μ)^s ≤(2B)sμ(Cθfinal)≤(2B)sϵ. ≤(2B)^sμ(C^final_θ)≤(2B)^sε. Taking s=2s=2 gives the claimed 2Bϵ2B ε bound. ∎ B.4 Function-level identification of multi-head attention Algorithm 2 RecoverMHA for h=L=1h=L=1 1: oracle Fn(⋅,θ0)F_n(·; _0); rank0=d rankW_0=d 2: effective matrices (0,0)(M_0,W_0) 3: query yi←F1(ei⊤,θ0)y_i← F_1(e_i ; _0), i=1,…,di=1,…,d 4: ←[y1;…;yd]W←[y_1;…;y_d] ⊳ Value matrix 5: P←[0⊤;e1⊤;…;ed⊤]P←[0 ;e_1 ;…;e_d ] 6: query Y←Fd+1(P,θ0)Y← F_d+1(P; _0) 7: Z←PZ← PW 8: A←[Y 1d+1][Z 1d+1]−1A←[Y\ 1_d+1][Z\ 1_d+1]^-1 ⊳ Attn. wts. 9: ij←log(Ai+1,j+1/Ai+1,1)M_ij← \! (A_i+1,j+1/A_i+1,1 ), i,j∈[d]i,j∈[d] 10: return (,)(M,W) We use the architecture, parameter spaces, and probe maps defined in equations 2–(3). In particular, Θ is the raw parameter space and D=dimΘD= . The canonical representation of a layer is obtained by merging heads with identical score matrices aM_a, summing their value matrices aW_a, and removing groups whose summed value matrix is zero. We start the section with a brief discussion on related works and then prove our results. Relation to query-based attention recovery. Query based recovery of attention blocks has been studied recently and we acknowledge the progress of those works and discuss what we contribute. Bhattamishra et al. (2026) study recovery of single-head attention from scalar-output queries, while Kim and Kim (2026) recover canonical multi-head representations. Our three-token separation argument is a matrix-output adaptation of the canonical-identifiability result (Kim and Kim, 2026, Proposition 3.1). However, our main focus is generic identification of multilayer MHA stacks from a shared, nonadaptive probe probes with hidden access. We also consider different observation models: our queries return complete tokenwise layer outputs for Ψall _all, whereas their attention oracles return a scalar at the final token only. Softmax attention setup: Now we setup some detailed notation, some of which overlap with the same from main text, but in more details. We begin with some definitions. We consider d dimensional inputs and h attention heads, with h|dh d. Set r=d/hr=d/h, the value dimension of each head, and let k be the query/key dimension with 1≤k≤d1≤ k≤ d . A bias-free, unmasked MHA layer has parameters θ≔(a,a,a,a)a=1h,a,a∈ℝd×k,a∈ℝd×r,a∈ℝr×d.θ \(Q_a,K_a,V_a,O_a)\_a=1^h,\,Q_a,K_a ^d× k,\,V_a ^d× r,\,O_a ^r× d. For an input probe P∈ℝn×dP ^n× d whose rows are the token vectors p1⊤,…,pn⊤p_1 ,…,p_n , the layer computes Fn(P,θ)=∑a=1hsoftmax(PaP⊤)Pa,a≔aa⊤k,a≔aa.F_n(P;θ)= _a=1^hsoftmax(PM_aP )\,PW_a,\,M_a Q_aK_a k,\,W_a _aO_a. The softmax acts rowwise:[softmax(PaP⊤)]ij=exp(pi⊤apj)∑s=1nexp(pi⊤aps). [softmax(PM_aP ) ]_ij= (p_i M_ap_j) _s=1^n (p_i M_ap_s). The matrices a,a∈ℝd×dM_a,W_a ^d× d thus have ranks at most k and r, respectively. A probe specifies P and returns the complete matrix Fn(P,θ)∈ℝn×dF_n(P;θ) ^n× d. An L-layer MHA stack has parameters =(θ1,…,θL) θ=( _1,…, _L) and we consider the case where the token dimension d is preserved across layers. Layer ℓ has hℓh_ heads, query/key dimension kℓk_ , and value dimension rℓ=d/hℓr_ =d/h_ . Our proofs only need manifold doms of parameters. Let Θ=∏ℓ=1LΘℓ = _ =1^L _ denote the parameter space of the L-layer MHA architecture, with Θℓ _ being the parameter space of the ℓ -th layer. Let qℓ≔hℓ(kℓ(2d−kℓ)+rℓ(2d−rℓ)),q≔∑ℓ=1Lqℓq_ h_ (k_ (2d-k_ )+r_ (2d-r_ ) ),\,q _ =1^Lq_ . Then qℓ=dim(Θℓ)q_ = ( _ ) since the set of d×d× d matrices of rank at most k is an irreducible algebraic variety of dimension k(2d−k)k(2d-k). Thus, dim(Θℓ)=qℓ,dim(Θ)=q ( _ )=q_ ,\, ( )=q. Proof of Stronger version of Theorem 5 We use two elementary facts: a discrete subset of a Euclidean space is countable, and a nonzero analytic function on a connected open subset of ℝmR^m has a zero set of Hausdorff dimension at most m−1m-1. Lemma 9 (Three-token separation). Two MHA layers agree on every three-token input if and only if they agree at every sequence length. Proof. Define ρ:ℝ→ℝ:ρ(t)=et/(2+et)ρ:R :ρ(t)=e^t/(2+e^t). It is not hard to see that the functions ρ(gt)ρ(gt) for distinct “rates” g∈ℝg are linearly independent. Now, suppose the three-token maps agree. Equal-token inputs give equal value sums. List their distinct score matrices as BjB_j, and let Δj _j be the difference between the associated value sums. The repeated-token row of (q+tv,q,q)(q+tv,q,q) then gives ∑jρ(tq⊤Bjv)v⊤Δj=0. _jρ(tq B_jv)\,v _j=0. For generic (q,v)(q,v) the rates are distinct, since q⊤(Bj−Bj′)vq (B_j-B_j )v is a nonzero polynomial for j≠j′j≠ j . Linear independence gives v⊤Δj=0v _j=0, hence Δj=0 _j=0. The converse, at every length, follows by merging the summands. ∎ We make a note that the above is the matrix-output counterpart of Kim and Kim (2026, Proposition 3.1). Lemma 10 (Countably many candidates from a fixed probe batch). Let H≥2H≥ 2, H|dH d, and s=d/Hs=d/H. Consider probe tuples containing d(2H+1)d(2H+1) probes: d singletons, d(H+1)d(H+1) triples, and d(H−1)d(H-1) sequences of length s+1s+1. Let 0P_0 be the set of all such probes. Fix an MHA layer with h≤Hh≤ H heads, h|dh d, query/key dimension 1≤k≤d1≤ k≤ d, and value dimension r=d/hr=d/h. Its effective parameter space is Θeff≔θeff=((a,a))a=1h:ranka≤k,ranka≤r, _eff \ _eff=((M_a,W_a))_a=1^h:rankM_a≤ k,\,rankW_a≤ r \, with q≔dimΘeff=h[k(2d−k)+r(2d−r)].q _eff=h [k(2d-k)+r(2d-r) ]. Then, for almost all tuples (,θ)∈0×Θeff∖(P,θ) _0× _eff (with dimH≤dim0+q−1 _HN≤ _0+q-1) and almost all θ∈θeffθ∈ _eff the set θ′∈Θeff:Ψfinal(θ′,)=Ψfinal(θ,) \θ ∈ _eff: _final(θ ,P)= _final(θ,P) \ is at most countable. Proof. We first choose parameters and probes for which no nonzero first-order change in the parameters leaves all outputs unchanged. We then show that this property holds almost everywhere and implies the claimed countability. Let Vij=αij−1, 0<α1<⋯<αd,V_ij= _i^j-1,\,0< _1<·s< _d, and let PaP_a keep the aath consecutive coordinate block of size r and set the other coordinates to zero. Choose a=VPa,a=caM∗,M∗=e1⊤+∑j=2kejej⊤,W_a=VP_a,\,M_a=c_aM_*,\,M_*=e_11 + _j=2^ke_je_j , where 0<c1<⋯<ch0<c_1<·s<c_h. Partition [d][d] into H blocks J of size s, and set uj=V:,ju_j=V_:,j for j≤d−sj≤ d-s. Use the probes ei,(e1+tνei,e1,e1),(uj,(uj+eb)b∈J),e_i,\,(e_1+t_νe_i,e_1,e_1),\,(u_j,(u_j+e_b)_b∈ J), for i∈[d]i∈[d], ν∈[H+1]ν∈[H+1], and j∈[d−s]j∈[d-s], stacking the listed tokens as rows. Their counts are exactly those specified by 0P_0. We will repeatedly use the following elementary fact: a nonzero polynomial of degree below q cannot vanish at q distinct points. Consequently, the matrix (αij−1)i∈I,j∈[q]( _i^j-1)_i∈ I,\,j∈[q] is invertible whenever |I|=q|I|=q. We start with showing that the singleton and triple outputs constrain the value matrices. Write ρ(t)=et/(2+et)ρ(t)=e^t/(2+e^t). Singletons determine S=∑aaS= _aW_a. For a fixed direction eie_i, subtract e1⊤Se_1 S from a repeated output row of the triple and divide by tνt_ν. The result is Yν=∑aρ(tνga)ba⊤,ga=e1⊤aei=ca,ba=a⊤ei.Y_ν= _aρ(t_νg_a)b_a ,\,g_a=e_1 M_ae_i=c_a,\,b_a=W_a e_i. We define: Cνa=ρ(tνca),B=(ba⊤)a=1h,(Da)ν=tνρ′(tνca).C_ν a=ρ(t_νc_a),\,B=(b_a )_a=1^h,\,(D_a)_ν=t_νρ (t_νc_a). Each row of B is nonzero and occupies a different coordinate block, so its rows are independent. Choose the positive scales so that rank[CDa]=h+1rank[C\ D_a]=h+1 for every a. Here is a real-variable justification that such scales exist. For c>0c>0, as t→∞t→∞, we get ρ(ct)−1=−2e−ct+O(e−2ct),tρ′(ct)=2te−ct+O(te−2ct).ρ(ct)-1=-2e^-ct+O(e^-2ct),\,tρ (ct)=2te^-ct+O(te^-2ct). Suppose ∑bβbρ(cbt)+γtρ′(cat)=0 _b _bρ(c_bt)+γ tρ (c_at)=0. Taking the limit gives ∑bβb=0 _b _b=0. After subtracting this constant, consider the smallest rate with a nonzero coefficient. Dividing by its exponential leaves a nonzero constant or a nonzero linear function of t, plus a term tending to zero. This is impossible. Thus the h+1h+1 functions are independent. Their evaluation matrices therefore have full rank for almost every choice of the scales, simultaneously for all a. Now write δa,δa _a, _a for first-order parameter changes, and suppose every observed output has zero first-order change. Then δS=0,CδB+∑aDaδgaba⊤=0.δ S=0,\,Cδ B+ _aD_a\,δ g_a\,b_a =0. Taking row combinations that cancel C, and multiplying by a right inverse of B, gives δga=0δ g_a=0 for every a, because DaD_a is not a linear combination of the columns of C. Then CδB=0Cδ B=0, so δB=0δ B=0. Doing this for every i gives δa=0 _a=0. Now we show that the remaining outputs constrain the score matrices corresponding to the softmax attention. Since bPa=a=baW_bP_a=1_a=bW_a, multiplying the output on the right by PaP_a isolates head a. For a block input Z, we therefore obtain δAaZa=0,Aa=softmax(ZaZ⊤).δ A_a\,ZW_a=0,\,A_a=softmax(ZM_aZ ). Every s selected rows of aW_a are independent. Indeed, after removing nonzero row factors, their first s nonzero columns form the polynomial-evaluation matrix described above. Hence the rows of ZaZW_a are affinely independent. Since also δAa=0δ A_a1=0, it follows that δAa=0δ A_a=0. Differentiating the attention log-ratios gives uj⊤δaeb=0,eb′⊤δaeb=0(j≤d−s,b,b′∈J).u_j _ae_b=0,\,e_b _ae_b=0\,(j≤ d-s,\ b,b ∈ J). For each J, the vectors (uj)j≤d−s(u_j)_j≤ d-s and (eb)b∈J(e_b)_b∈ J span ℝdR^d: deleting rows J from the first d−sd-s columns of V leaves an invertible polynomial-evaluation matrix. Thus δaeb=0 _ae_b=0 for every b and every a. We have proved that all first-order parameter changes vanish. Now we use this example to conclude the result for almost every parameter and probe tuple. A rank-t matrix can locally be written as (U11U12U21U21U11−1U12),detU11≠0. pmatrixU_11&U_12\\ U_21&U_21U_11^-1U_12 pmatrix,\, U_11≠ 0. Its three displayed blocks provide t(2d−t)t(2d-t) independent coordinates. Thus the effective parameters with ranks exactly k and r have q independent coordinates; smaller ranks require fewer coordinates. The previous discussion shows that the matrix of first derivatives of the outputs has rank q at our chosen parameters and probes. The same is true when differentiating with respect to the raw factors: full-rank factors allow every first-order change in the independent matrix coordinates. Some square submatrix of these derivatives therefore has a nonzero determinant. This determinant is analytic on the connected raw parameter and probe space, so its zero set has measure zero. Passing to the independent matrix coordinates preserves this almost-everywhere conclusion. We note that full-rank factorizations are locally described by these coordinates together with an arbitrary invertible change of basis in the factors. Fubini applied to these extra coordinates gives the assertion. Now, finally we bound the exceptional set. Record both the probes and their outputs: (,θ)=(,Ψfinal(θ,)),m=dim0+q.G(P,θ)= (P, _final(θ,P) ),\,m= _0+q. Let ℬB be the union of the pairs for which a parameter matrix has smaller rank than allowed and the remaining pairs where the matrix of first derivatives of G has rank below m. The smaller-rank matrices require fewer independent coordinates. For matrices of the full allowed ranks, the derivative condition fails on proper analytic zero sets, by the preceding construction. Hence dimHℬ≤m−1. _HB≤ m-1. Here we use the elementary fact that the zero set of a nonzero analytic function on an open subset of ℝmR^m has Hausdorff dimension at most m−1m-1. We must also exclude parameters whose observations match those of a point in ℬB. Define ≔−1((ℬ)).N ^-1 (G(B) ). Smooth maps do not increase Hausdorff dimension. Therefore we have: dimH(ℬ)≤m−1 _HG(B)≤ m-1. Outside ℬB, select m coordinates of G whose derivative matrix is invertible. The inverse function theorem gives a smooth local inverse. Applying this inverse to the corresponding coordinates of (ℬ)G(B) gives dimH(∖ℬ)≤dimH(ℬ)≤m−1. _H(N )≤ _HG(B)≤ m-1. A countable collection of these neighborhoods suffices. Since ℬ⊆B , we conclude that dimH≤m−1. _HN≤ m-1. The set ℬB is closed: having the full allowed matrix ranks and an invertible selected derivative matrix are open conditions. Thus ℬB is a countable union of compact sets. Its continuous image (ℬ)G(B) is likewise such a union, so N is measurable. If (,θ)∉(P,θ) , every parameter with the same observations lies outside ℬB. The inverse function theorem therefore isolates each matching parameter from all other matches. There are at most countably many such isolated points. Finally, Fubini in the independent matrix coordinates gives the almost-everywhere conclusion in the stated order. ∎ Lemma 11 (Prefixes are generically locally invertible). For each fixed length n and prefix depth ℓ , the input Jacobian of xℓ,nx_ ,n is nonsingular almost everywhere in the raw prefix parameters and input. Consequently, for almost every prefix, its image contains a nonempty open set. Proof. It suffices to construct one nonsingular Jacobian. In each head take a=εe1e1⊤M_a= e_1e_1 and let aW_a be coordinate-block projections summing to IdI_d. Writing X=(z,X′)X=(z,X ), the layer is (z,X′)⟼(fε(z),Aε(z)X′),Aε=softmax(εzz⊤),fε=Aεz.(z,X ) (f_ (z),A_ (z)X ),\,A_ =softmax( z ),\,f_ =A_ z. Choose distinct ziz_i. For ε>0 >0, the matrix Kij=eεzizjK_ij=e z_iz_j is positive definite, because c⊤Kc=∑m≥0εm!(∑icizim)2>0(c≠0)c Kc= _m≥ 0 ^mm! ( _ic_iz_i^m )^2>0\,(c≠ 0) by the Vandermonde determinant. Thus AεA_ is invertible. For n≥2n≥ 2, put w=z−z¯w=z- z1 and v=‖w‖2/n>0v=\|w\|^2/n>0. Expansion at ε=0 =0 gives Dfε=⊤n+ε(vI+2zw⊤n)+O(ε2).Df_ = 11 n+ (vI+ 2zw n )+O( ^2). Its first-order block on ⟂1 is vI+2ww⊤/n≻0vI+2w /n 0, so DfεDf_ is invertible for sufficiently small ε>0 >0. The full input Jacobian is block triangular with invertible diagonal blocks DfεDf_ and d−1d-1 copies of AεA_ . The output first coordinates remain distinct: the function q↦∑jzjeεqzj/∑jeεqzjq _jz_je qz_j/ _je qz_j has derivative ε times a positive weighted variance. We can therefore repeat the construction through each layer. For n=1n=1 the witness is the identity. Each prefix determinant is consequently a nonzero analytic function. Its zero set is null; Fubini and the inverse function theorem give the conclusion. ∎ The geometric strengthening. The following standard result is the only additional external ingredient used to obtain the open-probes and closed-exception claims. We state it explicitly rather than treating those claims as consequences of analyticity. Theorem 12 (Real-exponential geometry; external input). Call a set definable if it can be described by a finite formula using real constants, arithmetic, exponentials, equalities, inequalities and quantifiers over real variables. Such sets have integer Hausdorff dimension, satisfy dimHA¯=dimHA,A⊆ℝm is null⟺dimHA<m, _H A= _HA,\,A ^m is null\ \ _HA<m, and, for a definable family B⊆ℝs×ℝDB ^s×R^D, each set u:dimHBu=j\u: _HB_u=j\ is definable. Here dim∅=−∞ =-∞. This is a consequence of Wilkie’s o-minimality theorem for ℝexpR_ and its dimension theory (Wilkie, 2009); we do not reprove that foundational theorem. Theorem 13 (Structured probing for softmax attention). Define H≔maxℓhℓH _ h_ and N=d(2H+1)+1N=d(2H+1)+1. There exist lengths nj≤max3,1+d/Hn_j≤ \3,1+d/H\ and an open, dense, full-measure set ⊆=∏j=1Nℝnj×dU = _j=1^NR^n_j× d such that every ∈P admits a closed exceptional set E⊆ΘE_P , with dimHE≤D−1 _HE_P≤ D-1, satisfying Ψall(,)=Ψall(0,)→xℓ,n(⋅,)≡xℓ,n(⋅,0)(ℓ∈[L],n≥1) _all( θ;P)= _all( θ_0;P)\,→\,x_ ,n(·; θ)≡ x_ ,n(·; θ_0)\,( ∈[L],\ n≥ 1) for every 0∉E θ_0∉ E_P and every ∈Θ θ∈ . The probe set P is fixed independently of the target. Proof. We first state and prove an equivalent result. That with some more argument proves Theorem 13. Proposition 14 (Fixed-probes identification). Put H=maxℓhℓH= _ h_ and N=d(2H+1)+1N=d(2H+1)+1. There exist lengths nj≤max3,1+d/Hn_j≤ \3,1+d/H\ such that, for almost every probe tuples ∈=∏j=1Nℝnj×dP = _j=1^NR^n_j× d and almost every target 0∈Θ θ_0∈ , every competitor with the same complete hidden transcript has the same canonical representation in every layer. Consequently, Ψall(,)=Ψall(0,)→xℓ,n(⋅,)≡xℓ,n(⋅,0)(ℓ∈[L],n≥1). _all( θ;P)= _all( θ_0;P)\ →\ x_ ,n(·; θ)≡ x_ ,n(·; θ_0)\,( ∈[L],\ n≥ 1). Proof. Assume first H≥2H≥ 2. Take the length profile of Lemma 10 and append one triple. Thus N=d+d(H+1)+d(H−1)+1=d(2H+1)+1.N=d+d(H+1)+d(H-1)+1=d(2H+1)+1. We first use the hidden transcript to restrict each layer. Let Gℓ(ζ,<ℓ)G_ (ζ; θ_< ) be the first batch after the target prefix, and let πℓ _ map raw factors to effective parameters. The map (ζ,)⟼(Gℓ(ζ,<ℓ),πℓ(θℓ))(ζ, θ) (G_ (ζ; θ_< ), _ ( _ ) ) is a submersion almost everywhere: its probes derivative is invertible by Lemma 11, and its independent current-layer factor map is a submersion on the full-rank stratum. Pulling back the exceptional set in Lemma 10 therefore gives a null set. For almost every first batch and target, each layer has only countably many consistent effective candidates. This includes every matching competitor: its hidden transcript fixes the same layer inputs as those observed from the target. Then we use the last triple to separate these candidates. Fix such a first batch and target, with open prefix images at length three. For a candidate η at layer ℓ not canonically equivalent to the target, the discrepancy X⟼F3(ℓ)(xℓ−1,3(X,0),η)−xℓ,3(X,0)X F_3^( )(x_ -1,3(X; θ_0);η)-x_ ,3(X; θ_0) is nonzero and analytic. Otherwise the standalone layer maps would agree on an open set, hence everywhere, contradicting Lemma 9. Almost every last triple avoids the zero sets of all these countably many discrepancies. Every matching stack then has the target’s canonical layers and agrees at every length. If H=1H=1, instead use d singletons, one (d+1)(d+1)-token input, and 2d2d additional triples. The singleton states satisfy Cℓ=Cℓ−1ℓC_ =C_ -1W_ , so generically they determine every ℓW_ . By Lemma 11, the prefix image cannot lie in det[Z]=0 [1\ Z]=0. Analyticity therefore makes the propagated (d+1)(d+1)-token input Z affinely independent at every depth, almost everywhere. From its layer output Y recover A=[Yℓ−1][Z]−1.A=[1\ \ YW_ ^-1][1\ \ Z]^-1. The equations log(Aij/Ai1)=zi⊤ℓ(zj−z1) (A_ij/A_i1)=z_i M_ (z_j-z_1) determine ℓM_ : the differences zj−z1z_j-z_1, j=2,…,d+1j=2,…,d+1, form a basis and the tokens span ℝdR^d. Finally we show that the failure set is measurable. At each layer, give the competitor’s heads positive signs and the target’s heads negative signs. Canonical agreement means that these signed heads can be partitioned into groups with equal score matrices and zero signed value sums. There are finitely many partitions, each imposing polynomial equalities. Canonical agreement is therefore closed. The incidence set of matching transcripts but canonical disagreement is closed intersected with open, hence a countable union of compact sets. Its projection B=(,0):∃,Ψall(;)=Ψall(0;),≁can0B=\(P, θ_0):∃ θ, _all( θ;P)= _all( θ_0;P), θ _ can θ_0\ is likewise a countable union of compact sets and is measurable. The preceding argument and Fubini show that B is null. A second application of Fubini gives the stated quantifier order. ∎ Take the length profile of Proposition 14 and its null failure set B. The set B is definable: finite hidden transcripts involve arithmetic and exponentials, and canonical agreement is given by the finite polynomial description above. By Fubini, almost every fibre B_P is null. Therefore A=∈:dimHB=DA=\P : _HB_P=D\ is null and definable by Theorem 12. We define =∖A¯,E=B¯(∈).U=S A,\,E_P= B_P\,(P ). The same theorem makes U open, dense and full measure, and E_P closed with Hausdorff dimension at most D−1D-1. Outside E_P there is no canonically different competitor with the same transcript. Canonical agreement implies equality of all layer maps, and thus of all composed functions at every length. ∎ Multilayer MHA reconstruction from a fixed set of probes The observations already supply each layer’s training inputs: they are the preceding target states, not states generated by the reconstructed prefix. Each layer can consequently be fitted independently. Define its cached-data loss by ℒℓ(θℓ)=∑j=1N‖Fnj(ℓ)(Zℓ(j),θℓ)−Yℓ(j)‖F2.L_ ( _ )= _j=1^N \|F_n_j^( )(Z_ ^(j); _ )-Y_ ^(j) \|_F^2. (10) Optimization is over the specified raw factors, so the prescribed head counts and rank bounds are maintained automatically. Algorithm 3 Fixed-probe set multilayer reconstruction 1: Architecture; probe set =(X(j))j=1N∈P=(X^(j))_j=1^N ; complete hidden access to a target 0∉E θ_0∉ E_P 2: A stack with the target’s composed hidden functions at every length 3: for j=1,…,Nj=1,…,N do 4: Query X(j)X^(j) once; record H0(j)←X(j)H_0^(j)← X^(j) and all returned states H1(j),…,HL(j)H_1^(j),…,H_L^(j) 5: end for 6: for ℓ=1,…,L =1,…,L do ⊳ These fits may run in parallel 7: Set Zℓ(j)←Hℓ−1(j)Z_ ^(j)← H_ -1^(j) and Yℓ(j)←Hℓ(j)Y_ ^(j)← H_ ^(j) for every j 8: Find any θ^ℓ θ_ in the layer’s raw parameter space with ℒℓ(θ^ℓ)=0L_ ( θ_ )=0 ⊳ Fit nonlinear solver 9: end for 10: return ^=(θ^1,…,θ^L) θ=( θ_1,…, θ_L) Correctness and scope. Each feasibility problem has a solution, the true layer. Induction over ℓ shows that the returned stack reproduces every recorded state. Theorem 13 therefore gives the all-length conclusion. In numerical implementations it is replaced by layerwise least-squares optimization. It should be mentioned that neither its global convergence nor an OOD error bound from a nonzero residual follows from the theorem. The theorem provides a guarantee, the implementation requires implementing a solver or independently fitting the layers. No head alignment between layers is required. The special case of one layer: The above statement in Theorem 13 discusses identification of generic targets from probes in the multilayer case. However, we show in the following lemma that for a single layer, we can exactly identify every target at a function level. Lemma 12 (One layer: every target from fixed probes). Let L=1L=1 and N=4dh+1N=4dh+1. There exist an integer n∗=n∗(d,h)n_*=n_*(d,h) and a fixed probe set ∈(ℝn∗×d)NP∈(R^n_*× d)^N such that, simultaneously for all θ1,θ2∈Θ _1, _2∈ , we have: Ψfinal(θ1;)=Ψfinal(θ2;)⇔[Fn(⋅;θ1)≡Fn(⋅;θ2) for every n≥1]. _final( _1;P)= _final( _2;P)\, \, [F_n(·; _1)≡ F_n(·; _2) for every n≥ 1 ]. Proof. Put K=(d−1)(2h−1)+1K=(d-1)(2h-1)+1 and qt=(1,t,…,td−1)⊤q_t=(1,t,…,t^d-1) for t=1,…,Kt=1,…,K, and let D contain the distinct vectors among 0,e1,…,ed,q1,…,qK0,e_1,…,e_d,q_1,…,q_K. For positive integer multiplicities m=(mx)x∈m=(m_x)_x , let X(m)X(m) contain mxm_x copies of each x. At a copy of x0x_0, output coordinate c equals R(m,ξ)=∑a=1h∑x∈mxτaxx⊤wa∑x∈mxτax,τa=exp(a⊤x0),wa=aec,R(m;ξ)= _a=1^h _x m_x _a^xx w_a _x m_x _a^x,\, _a= (M_a x_0),\,w_a=W_ae_c, (11) where τx=∏iτixiτ^x= _i _i^x_i and ξ=((τa,wa))a∈Ξ=((0,∞)d×ℝd)hξ=(( _a,w_a))_a∈ =((0,∞)^d×R^d)^h. Clearing the positive denominators of R(m,ξ)−R(m,ξ′)R(m;ξ)-R(m;ξ ) gives a polynomial P(m,ξ,ξ′)P(m;ξ,ξ ), since every x∈x has nonnegative integer coordinates. The Finite Witness Theorem (Amir et al., 2023), applied to the 4dh4dh-dimensional pair (ξ,ξ′)(ξ,ξ ), implies that almost every choice of N=4dh+1N=4dh+1 generic positive multiplicity vectors detects every nonzero discrepancy. The bad witness set is semialgebraic: writing P=∑αcα(ξ,ξ′)mαP= _αc_α(ξ,ξ )m^α, it is the projection of the set defined by ∑αcα2>0 _αc_α^2>0 and P(m(j),ξ,ξ′)=0P(m^(j);ξ,ξ )=0 for every j. Its nullity implies that its closure has empty interior. We choose a good rational tuple. Since R(λm,ξ)=R(m,ξ)R(λ m;ξ)=R(m;ξ), normalize each vector to sum one and clear all denominators by a common integer. This gives positive integer multiplicities of a common sum n∗n_*. The resulting probe set works for every row and coordinate, because their parameters all belong to Ξ . Suppose two layers agree on this set of probes. Let B1,…,BJB_1,…,B_J, J≤2hJ≤ 2h, be their distinct score matrices, and let Δj _j be the difference of their summed value matrices at BjB_j. For each j, some qtq_t satisfies qt⊤Bj≠qt⊤Blq_t B_j≠ q_t B_l for all l≠jl≠ j: any d moment-curve vectors are independent, so each proper subspace q:q⊤(Bj−Bl)=0\q:q (B_j-B_l)=0\ contains at most d−1d-1 of them, fewer than K altogether. Fix this x0=qtx_0=q_t and a coordinate c. The witness property gives ∑l=1JAl(m)Dl(m)≡0,Dl=∑xmxex0⊤Blx,Al=∑xmxex0⊤Blxx⊤Δlec. _l=1^J A_l(m)D_l(m)≡ 0,\,D_l= _xm_xe^x_0 B_lx,\,A_l= _xm_xe^x_0 B_lxx _le_c. Each denominator has coefficient one at m0m_0, and its coefficients at meim_e_i determine x0⊤Blx_0 B_l. After merging equal denominators, DjD_j is a distinct linear factor. Clearing denominators shows that this prime factor divides AjA_j, so Aj=λDjA_j=λ D_j. The coefficient at m0m_0 gives λ=0λ=0; the coefficients at meim_e_i then give (Δj)ic=0( _j)_ic=0. Thus all Δj=0 _j=0, proving functional agreement at every length. ∎ B.5 Learning Functionals and Operators from Probes We first prove a common approximation lemma. We then give formal versions of Theorems 6 and 7, in that order. All function spaces below carry the uniform norm. Matrix inputs, outputs, and observations are identified with vectors by listing their entries in a fixed order. All MLP readouts have biases, an affine output layer, and a fixed continuous, non-polynomial hidden activation τ. Separation and approximation. For a compact parameter set K and a jointly continuous family f:×K→ℝmf:X× K ^m on compact X, write ℱK≔fθ=f(⋅,θ):θ∈K. _K \f_θ=f(·;θ):θ∈ K\. A map R:K→ℝsR:K ^s separates the represented functions if R(θ)=R(θ′)⟹fθ=fθ′,θ,θ′∈K. R(θ)=R(θ )\, \,f_θ=f_θ ,\,θ,θ ∈ K. Note that equality of functions need not imply equality of R, because R acts on weights. Lemma 13 (Separation implies universal approximation). Let ⊆ℝdX ^d and K⊆ℝqK ^q be compact, let f:×K→ℝmf:X× K ^m be continuous, and let R:K→ℝsR:K ^s be continuous and separate the represented functions. Then ℱKF_K is compact and there is a continuous map g:R(K)→ℱK,g(R(θ))=fθ. g:R(K) _K,\,g(R(θ))=f_θ. For every continuous Λ:ℱK→ℝk :F_K ^k, every compact ⊆ℝaY ^a, every continuous :ℱK→C(,ℝk)T:F_K→ C(Y,R^k), and every ϵ>0ε>0, there are MLPs ϕ:ℝs→ℝkφ:R^s ^k and ψ:ℝs+a→ℝkψ:R^s+a ^k such that supθ∈K‖Λ(fθ)−ϕ(R(θ))‖2<ϵ,supθ∈Ksupy∈‖(fθ)(y)−ψ(R(θ),y)‖2<ϵ. _θ∈ K\| (f_θ)-φ(R(θ))\|_2<ε,\, _θ∈ K _y \|T(f_θ)(y)-ψ(R(θ),y)\|_2<ε. Proof. Uniform continuity of f on ×KX× K implies continuity of the realization map :K→C(,ℝm),(θ)=fθ, :K→ C(X,R^m),\, (θ)=f_θ, in the uniform norm. Thus ℱK=(K)F_K= (K) is compact. Put S=R(K)S=R(K) and define g(R(θ))=fθg(R(θ))=f_θ. Separation makes g well defined. For every closed C⊆ℱKC _K, g−1(C)=R((C)−1) g^-1(C)=R ( ^-1(C) ) is compact and therefore closed in S. Hence g is continuous. The map Λ∘g g is continuous on the compact set S. The universal approximation theorem Leshno et al. (1993) gives an MLP ϕφ such that sups∈S‖Λ(g(s))−ϕ(s)‖2<ϵ. _s∈ S\| (g(s))-φ(s)\|_2<ε. Substituting s=R(θ)s=R(θ) proves the functional conclusion. The compact-set, vector-valued form follows by applying the Tietze extension theorem and the scalar approximation theorem coordinatewise. For the operator result, suppose G(s,y)≔(g(s))(y),(s,y)∈S×.G(s,y) (g(s))(y),\,(s,y)∈ S×Y. If (sj,yj)→(s,y)(s_j,y_j)→(s,y), then we have ‖G(sj,yj)−G(s,y)‖2 \|G(s_j,y_j)-G(s,y)\|_2 ≤‖(g(sj))−(g(s))‖∞ ≤\|T(g(s_j))-T(g(s))\|_∞ +‖(g(s))(yj)−(g(s))(y)‖2→0. \,+\|T(g(s))(y_j)-T(g(s))(y)\|_2→ 0. Thus G is continuous on the compact set S×S×Y. We thus apply the same approximation theorem to G, and then substitute s=R(θ)s=R(θ). ∎ Final-output observations. The following Theorem 15 is the formal version of Theorem 6. Theorem 15 (Approximation from final-output probes). Let f:ℝd×ℝq→ℝmf:R^d×R^q ^m be real analytic, let ⊆ℝdX ^d be compact with nonempty interior, and let Θ⊆ℝq ^q be compact. For every N≥2q+1N≥ 2q+1, and for almost every =(p1,⋯,pN)∈NP=(p_1,·s,p_N) ^N, the map T:ℱΘ→ℝmN,T(g)=(g(p1),⋯,g(pN)), T_P:F_ ^mN,\,T_P(g)=(g(p_1),·s,g(p_N)), is one-to-one and has a continuous inverse on its image. Then, for every continuous Λ:ℱΘ→ℝk :F_ ^k, every compact ⊆ℝaY ^a, every continuous :ℱΘ→C(,ℝk)T:F_ → C(Y,R^k), and every ϵ>0ε>0, there are MLPs ϕ:ℝmN→ℝkφ:R^mN ^k and ψ:ℝmN+a→ℝkψ:R^mN+a ^k satisfying supθ∈Θ‖Λ(fθ)−ϕ(Ψfinal(θ,))‖2<ϵ,supθ∈Θsupy∈‖(fθ)(y)−ψ(Ψfinal(θ,),y)‖2<ϵ. _θ∈ \| (f_θ)-φ( _final(θ,P))\|_2<ε,\, _θ∈ _y \|T(f_θ)(y)-ψ( _final(θ,P),y)\|_2<ε. Proof. Since f is globally analytic, we may apply Theorem 1 on sufficiently large input and parameter balls containing X and Θ in their interiors. Restricting the parameters to Θ and intersecting the exceptional probe set with 2q+1X^2q+1 gives a Lebesgue-null 0⊆2q+1N_0 ^2q+1 outside which final-output observations separate all represented functions. For N>2q+1N>2q+1, take =0×N−2q−1N=N_0×X^N-2q-1; for N=2q+1N=2q+1, take =0N=N_0. Fix ∉P . The map R(θ)≔Ψfinal(θ,)R(θ) _final(θ,P) is continuous and separates functions on Θ . Lemma 13, with K=ΘK= , gives both approximation conclusions and a continuous map g:R(Θ)→ℱΘg:R( ) _ . Separation makes T_P one-to-one, and g(T(fθ))=fθg(T_P(f_θ))=f_θ, so g=T−1g=T_P^-1. Also, ‖T(f)−T(f′)‖2≤N‖f−f′‖∞,f,f′∈ℱΘ. \|T_P(f)-T_P(f )\|_2≤ N\|f-f \|_∞,\,f,f _ . The exceptional set comes solely from the identification theorem. ∎ Hidden-layer observations. The following Theorem 16 is the formal version of Theorem 7. We retain the complete final probe responses in their coordinate order, but discard the ordering of the hidden neurons or channels. For MLPs and MHA stacks, respectively, define vi,k(θ) v^i,k(θ) ≔(xi,k(pj,θ))j∈[N]∈ℝN,i∈[L−1],k∈[di], (x_i,k(p_j;θ) )_j∈[N] ^N,\,i∈[L-1],\,k∈[d_i], vi,k(θ) v^i,k(θ) ≔([xi,nj(Pj,θ)]a,k)j∈[N],a∈[nj]∈ℝ∑jnj,i∈[L−1],k∈[d]. ([x_i,n_j(P_j;θ)]_a,k )_j∈[N],\,a∈[n_j] _jn_j,\,i∈[L-1],\,k∈[d]. The coordinates within each vector retain the probe order and, for attention, the token order. Let Si(θ)S_i(θ) be the multiset of these vectors, including repetitions Thus Si(θ)S_i(θ) contains rir_i vectors in ℝtR^t, where ri=dir_i=d_i, t=Nt=N for MLPs and ri=dr_i=d, t=∑jnjt= _jn_j for MHA. For a multiset S of r vectors in ℝtR^t, we define ℳr(S) _r(S) ≔(∑v∈Svα)α∈ℕ0t, 1≤|α|≤r,ℕ0≔0,1,⋯, ( _v∈ Sv^α )_α _0^t,\,1≤|α|≤ r,\,N_0 \0,1,·s\, |α| |α| ≔∑b=1tαb,vα≔∏b=1tvbαb, _b=1^t _b,\,v^α _b=1^tv_b _b, R(θ) R_P(θ) ≔[ℳr1(S1(θ));⋯;ℳrL−1(SL−1(θ));Ψfinal(θ,)]. [M_r_1(S_1(θ));·s;M_r_L-1(S_L-1(θ)); _final(θ,P) ]. Every occurrence of a repeated vector contributes to the sum. For the MLP variant of HiddenProbe, we use layerwise Set Transformers followed by a final MLP, as described in Section A. For the pure-MHA specialization of HiddenProbe, we consider the shared trajectory encoder and layerwise pooling given by R^,φ(θ) R_P, (θ) ≔[(∑k=1dφ(vi,k(θ)))i=1L−1;Ψfinal(θ,)], [ ( _k=1^d (v^i,k(θ)) )_i=1^L-1; _final(θ,P) ], ℋ,ϕ(θ) _P,φ(θ) ≔ρ(R^,φ(θ)),ϕ≔(φ,ρ). ρ ( R_P, (θ) ),\,φ ( ,ρ). Here φ and ρ are MLPs satisfying the preceding activation and bias assumptions, with unrestricted hidden widths and encoder output dimension. The same φ is used for every hidden channel and layer. The final responses Ψfinal(⋅,) _final(·,P) retain their coordinate order. For operators, the prediction MLP additionally receives the query input. Theorem 16 (Approximation from hidden-layer probes). Consider either of the following settings. 1. MLPs. Consider the MLP family of Theorem 2. Its widths are =(d0,⋯,dL)D=(d_0,·s,d_L) and its activation is real analytic, injective, and non-polynomial. Let ⊆ℝd0X ^d_0 be compact with nonempty interior, let N≔1+max0≤i<Ldi,N 1+ _0≤ i<Ld_i, and let ⊆NU ^N consist of pairwise distinct tuples whose one-padded input matrix has rank d0+1d_0+1. This set has full measure by Lemma 2. Set fθ=xL(⋅,θ)|f_θ=x_L(·;θ)|_X. 2. Softmax attention stacks. Let the family satisfy the assumptions of Theorem 13, and define H≔maxℓ∈[L]hℓ,N≔d(2H+1)+1.H _ ∈[L]h_ ,\,N d(2H+1)+1. Let n1,⋯,nNn_1,·s,n_N and ⊆∏j=1Nℝnj×dU _j=1^NR^n_j× d be the length profile and full-measure good-bank set supplied by that theorem. For an evaluation length n≥1n≥ 1 and compact n⊆ℝn×dX_n ^n× d, set fθ=xL,n(⋅,θ)|nf_θ=x_L,n(·;θ)|_X_n. Let Θ=ℝD =R^D be the respective full raw parameter space. For every ∈P , there is a closed set E⊆ΘE_P with dimHE≤D−1 _HE_P≤ D-1 such that the following holds. For every compact K⊆Θ∖EK E_P, the representation R_P defined above admits a continuous map g:R(K)→ℱKg_P:R_P(K) _K with g(R(θ))=fθ,θ∈K.g_P(R_P(θ))=f_θ,\,θ∈ K. For every continuous Λ:ℱK→ℝk :F_K ^k, every compact ⊆ℝaY ^a, every continuous :ℱK→C(,ℝk)T:F_K→ C(Y,R^k), and every ϵ>0ε>0, there are models from the respective HiddenProbe or HiddenProbe family described above such that supθ∈K‖Λ(fθ)−ℋ,ϕ(θ)‖2<ϵ,supθ∈Ksupy∈‖(fθ)(y)−ℋ,ψop(θ,y)‖2<ϵ. _θ∈ K \| (f_θ)-H_P,φ(θ) \|_2<ε,\, _θ∈ K _y \|T(f_θ)(y)-H^op_P,ψ(θ,y) \|_2<ε. Proof. Fix ∈P . For MLPs, we define E≔θ∈Θ:∏i=0L−1det(¯iθ()(¯iθ())⊤)=0. E_P \θ∈ : _i=0^L-1 ( X_i^θ(P) ( X_i^θ(P) ) )=0 \. By Lemma 2, the displayed product is a nonzero real-analytic function of θ. Its zero set is closed and has Hausdorff dimension at most D−1D-1. For θ0∉E _0∉ E_P, equality of the ordered observations with any raw competitor gives, by injectivity, (Bi−Bi0)¯iθ0()=0, 0≤i<L.(B_i-B_i^0) X_i _0(P)=0,\,0≤ i<L. Full row rank gives Bi=Bi0B_i=B_i^0 for every i. For an affine output layer, its equality follows directly without inverting an activation. For attention, use E_P from Theorem 13. That theorem also identifies every target outside E_P against every raw competitor, at every sequence length. We first show that ℳrM_r determines an r-element multiset. If ℳr(S)=ℳr(S′)M_r(S)=M_r(S ), then for every a∈ℝta ^t and b∈[r]b∈[r], expansion of (a⊤v)b(a v)^b gives ∑v∈S(a⊤v)b=∑v∈S′(a⊤v)b. _v∈ S(a v)^b= _v∈ S (a v)^b. For real numbers c1,⋯,crc_1,·s,c_r, their power sums pb=∑jcjbp_b= _jc_j^b determine the coefficients of ∏j(z−cj) _j(z-c_j): writing ebe_b for the sum of all products of b distinct cjc_j and e0=1e_0=1, Newton’s identities give beb=∑j=1b(−1)j−1eb−jpj,b∈[r],∏j=1r(z−cj)=∑b=0r(−1)bebzr−b. be_b= _j=1^b(-1)^j-1e_b-jp_j,\,b∈[r],\, _j=1^r(z-c_j)= _b=0^r(-1)^be_bz^r-b. Consequently the projected multisets agree, including multiplicities. Choose a outside the finitely many hyperplanes a:a⊤(v−v′)=0\a:a (v-v )=0\ indexed by distinct vectors in S∪S′S∪ S . This projection is one-to-one on that union, so S=S′S=S . Now fix compact K⊆Θ∖EK E_P and θ,θ0∈Kθ, _0∈ K with R(θ)=R(θ0)R_P(θ)=R_P( _0). The preceding argument gives equality of the hidden trajectory multisets. Thus one permutation per hidden layer aligns the competitor’s responses at all probes. For MLPs, choose these permutation matrices PiP_i, put P0=PL=IP_0=P_L=I, and transform the competing parameter by Wi′=Pi+1WiPi⊤,bi′=Pi+1bi, 0≤i<L. W_i =P_i+1W_iP_i ,\,b_i =P_i+1b_i,\,0≤ i<L. Induction gives xi(u,θ′)=Pixi(u,θ)x_i(u;θ )=P_ix_i(u;θ). Hence Ψall(θ′,)=Ψall(θ0,) _all(θ ,P)= _all( _0,P) while fθ′=fθf_θ =f_θ. For attention, states are row matrices. Choose permutations Πi _i that align their columns. Let Π0=ΠL=I _0= _L=I, and we define Qia′=Πi−1⊤Qia,Kia′=Πi−1⊤Kia,Via′=Πi−1⊤Via,Oia′=OiaΠi. Q_ia = _i-1 Q_ia,\,K_ia = _i-1 K_ia,\,V_ia = _i-1 V_ia,\,O_ia =O_ia _i. Then ia′=Πi−1⊤iaΠi−1M_ia = _i-1 M_ia _i-1 and ia′=Πi−1⊤iaΠiW_ia = _i-1 W_ia _i, so we get: (XΠi−1)ia′(XΠi−1)⊤=XiaX⊤,(XΠi−1)ia′=XiaΠi. (X _i-1)M_ia (X _i-1) =XM_iaX ,\,(X _i-1)W_ia =XW_ia _i. The attention scores are unchanged, and induction gives us xi,n(X,θ′)=xi,n(X,θ)Πi.x_i,n(X;θ )=x_i,n(X;θ) _i. Again, Ψall(θ′,)=Ψall(θ0,) _all(θ ,P)= _all( _0,P) and fθ′=fθf_θ =f_θ. The all-competitor identification established above implies fθ=fθ0f_θ=f_ _0 in either case. It is unnecessary that θ′θ belong to K. The map R_P is continuous, since its coordinates are polynomials in finitely many continuous observations. The realization is jointly continuous on the chosen compact input domain and K. Lemma 13 therefore gives g_P and continuous functional and operator readouts on R(K)R_P(K) and R(K)×R_P(K)×Y. It remains to realize their approximations by the specified architectures. For HiddenProbe, Set Transformer universality Lee et al. (2019), with sufficient capacity and output dimension, approximates each layer’s continuous invariant map ℳriM_r_i. The final MLP approximates the resulting continuous readout. For HiddenProbe, the case L=1L=1 follows directly from Lemma 13, since there are no hidden trajectories. Suppose L≥2L≥ 2, and define ν(v) ν(v) ≔(vα)α∈ℕ0t, 1≤|α|≤d,t≔∑j=1Nnj,C≔vi,k(θ):θ∈K,i∈[L−1],k∈[d], (v^α )_α _0^t,\,1≤|α|≤ d, t _j=1^Nn_j,\,C \v^i,k(θ):θ∈ K,\ i∈[L-1],\ k∈[d] \, ≔R(K),1≔z:dist(z,)≤1. R_P(K),\,S^1 \z:dist(z,S)≤ 1\. Both C and 1S^1 are compact. By Lemma 13, there are MLPs ρ,ψρ,ψ satisfying supθ∈K‖Λ(fθ)−ρ(R(θ))‖2<ϵ/2,supθ∈Ksupy∈‖(fθ)(y)−ψ(R(θ),y)‖2<ϵ/2. _θ∈ K\| (f_θ)-ρ(R_P(θ))\|_2<ε/2,\, _θ∈ K _y \|T(f_θ)(y)-ψ(R_P(θ),y)\|_2<ε/2. Uniform continuity on 1S^1 and 1×S^1×Y gives δ∈(0,1)δ∈(0,1) such that, whenever z,z′∈1z,z ^1 and ‖z−z′‖2<δ\|z-z \|_2<δ, we have: ‖ρ(z)−ρ(z′)‖2<ϵ/2,supy∈‖ψ(z,y)−ψ(z′,y)‖2<ϵ/2. \|ρ(z)-ρ(z )\|_2<ε/2,\, _y \|ψ(z,y)-ψ(z ,y)\|_2<ε/2. Choose an MLP φ with supv∈C‖φ(v)−ν(v)‖2<δ1+d(L−1). _v∈ C\| (v)-ν(v)\|_2< δ1+d(L-1). Since ℳd(Si)=∑k=1dν(vi,k)M_d(S_i)= _k=1^dν(v^i,k), we have: supθ∈K‖R^,φ(θ)−R(θ)‖2 _θ∈ K\| R_P, (θ)-R_P(θ)\|_2 ≤∑i=1L−1∑k=1dsupθ∈K‖φ(vi,k(θ))−ν(vi,k(θ))‖2 ≤ _i=1^L-1 _k=1^d _θ∈ K\| (v^i,k(θ))-ν(v^i,k(θ))\|_2 <δ. <δ. In particular, R^,φ(K)⊆1 R_P, (K) ^1. Thus we get: supθ∈K‖Λ(fθ)−ρ(R^,φ(θ))‖2 _θ∈ K\| (f_θ)-ρ( R_P, (θ))\|_2 ≤supθ∈K‖Λ(fθ)−ρ(R(θ))‖2+supθ∈K‖ρ(R(θ))−ρ(R^,φ(θ))‖2<ϵ. ≤ _θ∈ K\| (f_θ)-ρ(R_P(θ))\|_2+ _θ∈ K\|ρ(R_P(θ))-ρ( R_P, (θ))\|_2<ε. Replacing ρ(z)ρ(z) by ψ(z,y)ψ(z,y) and taking the supremum over y∈y gives the operator result. The independence assertions follow from the identification theorems. ∎ Appendix C Architectural details of HiddenProbe and HiddenProbe We describe the architectures and training procedures of HiddenProbe for MLPs and HiddenProbe for Transformers in separate subsections. C.1 HiddenProbe for MLPs We describe the architectures used for INR classification and Model Zoo regression tasks. C.1.1 INR Classification Target networks and learned probes. For INR classification, each target is a SIREN with sine activations. We use N=128N=128 learned coordinate probes. Each probe has a trainable code ζq∈ℝ32 _q ^32, initialized from (0,I32)N(0,I_32). Two shared affine maps generate the coordinates: uq=tanh(A2(A1ζq+b1)+b2)∈(−1,1)2,A1∈ℝ32×32,A2∈ℝ2×32. u_q= (A_2(A_1 _q+b_1)+b_2 )∈(-1,1)^2, A_1 ^32× 32, A_2 ^2× 32. There is no activation between the affine maps. The same ordered probe bank is presented to every target; generation is not conditioned on the target’s weights or responses. Observed responses and neuron trajectories. For INR classification, during one forward pass, we record the two hidden layer responses Xiθ∈ℝN×32X_i^θ ^N× 32, i=1,2i=1,2, and final outputs Oθ∈ℝN×3O^θ ^N× 3. For each hidden neuron, we collect its responses across the complete bank: vxi,c(θ)=([Xiθ]q,c)n=1N∈ℝ128,i∈1,2,c∈[32]. v_x^i,c(θ)= ([X_i^θ]_q,c )_n=1^N ^128, i∈\1,2\, c∈[32]. We design features by concatenating the activations with global features: mean, standard deviation, maximum, minimum. Denoting these statistics by s(v)s(v), the resulting feature vector is χ(v)=[v;s(v)].χ(v)=[v;s(v)]. No normalization is applied before the token projection. Probe order is retained within every trajectory. Neuron tokens and joint readout. A shared affine map projects every neuron feature to ℝdeR^d_e, and a learned embedding eie_i identifies its hidden layer: ti,cθ t_i,c^θ =Aχ(vxi,c(θ))+a+ei, =Aχ (v_x^i,c(θ) )+a+e_i, toutθ t_out^θ =Aoutvec(Oθ)+aout. =A_outvec(O^θ)+a_out. One joint Transformer processes all tokens. We use three blocks: each block has eight attention heads, feedforward, ReLU, dropout 0.10.1 and LayerNorm before its attention and feedforward sublayers. There is no final LayerNorm after the last block. An MLP maps the final CLS token to class logits. There are no neuron-index embeddings or sequence positional encodings. At evaluation, when dropout is disabled, permuting the neurons within a hidden layer only permutes their tokens and leaves the prediction unchanged. Probe order, in contrast, is retained in both the trajectories and final-output vector. Relation to the theory. The neuron tokens preserve correspondence across probes, as in the theoretical trajectory representation. The implementation processes the layers jointly rather than using separate layerwise set encoders; layer embeddings retain their identities. It also observes preactivations, from which the theoretical postactivation responses can be computed. Sine is analytic but not injective, so the parameter-identification theorem stated for injective activations does not apply directly to these targets. The learned probes and finite-width token projections are not assumed to satisfy exact separation conditions. The universality results concern capacity-growing architecture families under their stated assumptions, not every fixed-width encoder or trained parameter choice. C.1.2 HiddenProbe for CNN-zoo accuracy regression Target networks and learned probes The regression targets are image-classification CNNs. The grayscale zoos use three convolutional layers with channels 1→16→16→161→ 16→ 16→ 16, kernel size 33 and stride 22, followed by global average pooling and a ten-output affine classifier. The Wild Park zoo contains targets with different depths, channel counts, convolutional settings and activations. Here, we use N=128N=128 learned image probes. Each probe is generated from a trainable code in ℝ32×1×1R^32× 1× 1 and has the appropriate image dimensions. A final tanh bounds the image values to (−1,1)(-1,1). The same learned probe bank is used for every target network. The output and hidden-response branches described below share this probe bank and the same target forward pass. We follow a protocol similar to that of Kahana et al. (2025) for learning probes in classification and regression tasks. Hidden aggregation and Prediction The regressor stacks the probes’ output at each channel and then pools the channels within each layer. Like in the case of INR classification, we also use additional global statistics like channelwise means, maxima, standard deviation etc over probes to design extra features to aggregate all the information from hidden layers. One self-attention block mixes the layer vectors using a learned relative-layer distance bias, followed by a residual pre-LayerNorm feedforward block with GELU. A final attention pool over layers, funal output and a projection give zH∈ℝdzz_H ^d_z. The attention projections in this aggregation have inner dimension 1616; large hidden-branch projections use factorized weights to reduce the number of parameters, and match the parameter count with that of ProbeGen. The predictor is an MLP that taken in the hidden combined representation and gives a scalar that predicts the accuracy. Relation to the theory The regressor follows the shared-probe principle and uses hidden responses without additional target-network queries. Its channel pooling respects channel permutations. It is an empirical adaptation of hidden probing in the context of CNN Model Zoo. C.2 HiddenProbe for Transformers Details on the learned probes. For MNIST, we use N=256N=256 probes divided into two families. The first contains 192192 learned sequences, each consisting of 1717 token vectors in ℝ32R^32. The remaining 6464 probes are learned 28×2828× 28 images. Each encoder probe is parameterized by independently learned 3232-dimensional token codes. Each image is parameterized by 4949 learned 1616-dimensional codes, corresponding to its 4×44× 4 patches. Within each route, a shared residual two-layer MLP transforms these codes: G(z)=z+W2GELU(W1z+b1)+b2.G(z)=z+W_2GELU(W_1z+b_1)+b_2. The transformed image patches are assembled into a 28×2828× 28 image. The same learned bank is presented to every target: generation is not conditioned on the target’s weights or its responses. Observed responses and channel trajectories. For MNIST and AGNews datasets, the Transformers have two blocks. For every probe, we record the outputs of the target’s two transformer blocks, its final normalized token states, and its classifier logits. Denote the three hidden layers by X1X_1, X2X_2, and XnormX_norm. Each hidden layer has 3232 channels, and each probe produces a 1010-dimensional logit vector. For each layer i∈1,2,normi∈\1,2,norm\ and channel c∈[32]c∈[32], we concatenate that channel’s responses across all probes and token positions: vi,c(θ)=([Xi(Pj,θ)]t,c)j∈[N],t∈[nj]∈ℝT,v^i,c(θ)= ([X_i(P_j;θ)]_t,c )_j∈[N],\,t∈[n_j] ^T,\, The concatenation uses a fixed order: encoder probes first, then image probes, with token positions kept in order within each probe. Consequently, each channel is represented by its responses to the complete bank. Encoding and combining channel features. One shared trajectory encoder E is applied independently to every channel in all three blocks: ui,c=E(vi,c)∈ℝ192.u^i,c=E(v^i,c) ^192. The encoder consists of LinearLinear, GELU, LinearLinear, and LayerNorm. It produces 3232 feature vectors for each view; it does not pool across channels. We then combine each block-2 feature with the final-normalized feature of the same channel: u~2,c=Fuse([u2,c;unorm,c])∈ℝ192. u^2,c=Fuse ([u^2,c;u^norm,c] ) ^192. This is concatenated with the 3232 block-1 features, giving 6464 hidden-response tokens in total. Joint readout and pooling. Each probe’s classifier logits are projected from ℝ10R^10 to ℝ192R^192. Together with a learned CLS token, the readout input contains 1+32+32+256=3211+32+32+256=321 tokens of dimension 192192. Two pre-Layernorm Transformer blocks process these tokens jointly, a final LayerNorm is applied to their outputs. Write the resulting tokens as zCLS,z1,…,z320∈ℝ192z_CLS,z_1,…,z_320 ^192. We form rθ=[zCLS;1320∑a=1320za;maxa∈[320]za]∈ℝ576,r_θ= [z_CLS; 1320 _a=1^320z_a; _a∈[320]z_a ] ^576, where the maximum is coordinatewise. Thus, mean and maximum pooling act jointly over all non-CLS tokens, including the logit tokens, rather than separately within each layer. A prediction head with dimensions 576→192→1576→ 192→ 1 and a GELU activation maps rθr_θ to standardized accuracy. The predictor receives only the recorded responses and fixed semantic identifiers, not target weights, target identities, or generated probe coordinates. Relation to the theory. The theoretical constructions motivate a shared bank with different sequence lengths and the use of intermediate responses. The experiments apply these principles to complete transformer classifiers containing feedforward and normalization layers. They do not assume that the learned probes satisfy the exact reconstruction conditions or identify the target function. Likewise, the universality results concern capacity-growing architecture families, not a guarantee for this particular finite-width configuration or its learned parameters. Appendix D Experimental details D.1 Function-level reconstruction on synthetic neural networks To illustrate the benefits of hidden activations, we study function-level reconstruction of frozen neural networks from a limited number of input probes. We compare output-only regression, which observes only the final network outputs, with intermediate-layer regression, which independently fits each layer using observed teacher activations as its inputs and targets. Our goal is to examine how well each approach reconstructs the target function, both between the observed probes and outside the training interval. Common Setup We consider function-level reconstruction of MLPs with different activations and stacks of MHA blocks, using either final outputs alone or complete hidden-layer observations. Let ≔(θ1,…,θL) θ ( _1,…, _L) denote the parameters of a sequential model f=fθL∘⋯∘fθ1f_ θ=f_ _L ·s f_ _1. Given a set of probes =(p1,…,pN)P=(p_1,…,p_N), we observe either the final outputs Ψfinal(,) _final( θ,P) or the complete hidden-layer observations Ψall(,) _all( θ,P). For the output-only case we do MSE regression to interpolate on the probes. If f^f_ θ is the “student” network trying to reconstruct the “teacher” network, we minimize the following losses: (i) For learning from Ψfinal _final, we minimize min∑i=1N^‖f^(pi)−f(pi)‖2 _ θ _i=1^N\|f_ θ(p_i)-f_ θ(p_i)\|^2 (i) For learning from Ψall _all, we fit sum of MSE on each of the L layers, given by ∑ℓ∑i=1N‖fθℓ^(Xiℓ−1)−Xiℓ‖2 _ _i=1^N\|f_ _ (X -1_i)-X _i\|^2, where Xiℓ≔fθℓ∘⋯∘fθ1(pi)X _i f_ _ ·s f_ _1(p_i), i.e Xiℓ\X _i\ are the observed outputs from the ℓ -th layer. Here we fit each hidden layer independently but simultaneously using MSE regression. Analytic MLP: We used a neural network with tanh activation of widths (1,4,4,1)(1,4,4,1), the training interval was [−1,1][-1,1], and the evaluation interval is [−3,3][-3,3]. As shown in Figure 4, access to the hidden activations enables essentially exact reconstruction once sufficiently many observations are available, whereas reconstruction from final outputs alone improves much more gradually and retains a substantially larger error even as the number of observed samples increases. (a) OOD MSE from Ψfinal _final (b) OOD MSE from Ψall _all (c) Reconctruction from Ψall,Ψfinal _all, _final Figure 4: Reconstruction of Analytic Net from Ψfinal,Ψall _final, _all. ReLU MLP: As in the analytic activation case, Theorem 4 suggests that hidden activations can reduce the number of probes necessary to (almost) identify a ReLU network. We take a target ReLU network with 2 hidden layers, width 4 and weights that represent a member of the “sawtooth” function class, having a large number of breakpoints. As discussed in Hanin and Rolnick (2019), such a large number of breakpoints is atypical. This makes the target a challenging example for reconstruction from final outputs alone using MSE regression from random initialization. This matches the observation: we can’t reconstruct the target with just the final output. But with hidden activations, we can reconstruct the target almost exactly with far fewer probes. The same is also reflected in the out-of-distribution MSE plots (a) OOD MSE from Ψfinal _final (b) OOD MSE from Ψall _all (c) Reconctruction from Ψall,Ψfinal _all, _final Figure 5: Reconstruction of ReLU Net from Ψfinal,Ψall _final, _all. Multi Head Attention Stack Here, we want to reconstruct an MHA stack. We randomly initialize a two-layer multi-head attention network with input width d=4d=4, two heads per layer and query/key and value widths k=r=2k=r=2 (bias-free softmax attention, without residual connections, normalization or feed-forward blocks). We sample four single-token inputs followed by generic three-token inputs, and a budget of N probes uses the first N of them, for N from 5 to 40. We fit the “student” MHA block jointly using Ψfinal _final, and each layer independently using Ψall _all, both using MSE regression as described above. Figure 6: Reconstruction of MHA Stack from Ψfinal,Ψall _final, _all. As shown in Figure 6, access to the hidden activations enables essentially exact reconstruction from as few as nine probes, below the N=21N=21 probes that Theorem 5 predicts, whereas reconstruction from final outputs alone becomes exact only at about 31 probes. Together, these three experiments suggest that hidden activations enable more accurate function reconstruction with fewer probes than final outputs alone. This observation is consistent with our empirical results on neural functional learning, where HiddenProbe, which uses hidden activations, outperforms ProbeGen, which relies only on final outputs. D.2 Full INR Classification Comparison Table 4 provides an extended comparison on the INR classification benchmarks, including both recent weight-space methods and several earlier baselines omitted from the main table for brevity. The MLP, MLP with permutation augmentation, MLP with alignment, INR2Vec, Transformer, and DWS baselines are taken from the DWSNets evaluation (Navon et al., 2023); these results were originally reported on the MNIST and Fashion-MNIST INR benchmarks. For methods evaluated on CIFAR-10 and augmented CIFAR-10, we additionally use the comparison reported by ScaleGMN (Kalogeropoulos et al., 2024), which includes DWS, NFN, NG-GNN, ScaleGMN, and ScaleGMN-B on all four INR benchmarks. We further include results from Neural Graphs (Kofinas et al., 2024), NFT (Zhou et al., 2023b), Monomial-NFN (Tran et al., 2024), Quasi-Equivariant Monomial-NFN (Tran et al., 2026), MAGEP-NFN (Vo et al., 2025), and ProbeGen (Kahana et al., 2025). Results unavailable in the corresponding source are marked with “–”. Table 4: Classification accuracy (↑ ) on INR weight-space benchmarks. Method MNIST FMNIST CIFAR-10 CIFAR-10 Aug StatNN 0.398±0.0010.398± 0.001 0.418±0.0020.418± 0.002 – – DWS 0.857±0.0060.857± 0.006 0.671±0.0030.671± 0.003 0.3445±0.0040.3445± 0.004 0.4127±0.0010.4127± 0.001 NFN 0.791±0.0080.791± 0.008 0.689±0.0060.689± 0.006 0.3341±0.0010.3341± 0.001 0.4660±0.0010.4660± 0.001 ScaleGMN 0.966±0.0020.966± 0.002 0.808±0.0010.808± 0.001 0.3882±0.0010.3882± 0.001 0.5695±0.0050.5695± 0.005 NG-GNN 0.914±0.0060.914± 0.006 0.680±0.0020.680± 0.002 0.3604±0.0040.3604± 0.004 0.4570±0.0020.4570± 0.002 NG-T 0.924±0.0030.924± 0.003 0.727±0.0060.727± 0.006 – – NFT – – – 0.6340±0.0000.6340± 0.000 Neural Graphs(128 probes) 0.976±0.0010.976± 0.001 0.745±0.0080.745± 0.008 - - Monomial-NFN – – 0.3423±0.0030.3423± 0.003 – Monomial-NFN Quasi – – 0.3532±0.0050.3532± 0.005 – MAGEP-NFN – – 0.3718±0.0030.3718± 0.003 – ProbeGen 0.984±0.0010.984± 0.001 0.877±0.0030.877± 0.003 0.573±0.0070.573± 0.007 0.563±0.0030.563± 0.003 HiddenProbe 0.986±0.00060.986± 0.0006 0.886±0.0050.886± 0.005 0.580±0.0020.580± 0.002 0.606±0.0020.606± 0.002 D.3 Full Test-Accuracy Prediction Comparison Table 5 provides an extended comparison on the neural-network test-accuracy prediction benchmarks. We include a broad range of approaches for learning from neural-network weights. MLP is a simple baseline operating directly on flattened network parameters, while StatNN (Unterthiner et al., 2020) predicts performance from summary statistics of the weights and biases. SNE (Andreis et al., 2024) represents neural networks using a hierarchical set-based encoder. We additionally include the permutation-equivariant NFN variants, NFNNP and NFNHNP (Zhou et al., 2023a), as well as the scale- and permutation-aware ScaleGMN and ScaleGMN-B architectures (Kalogeropoulos et al., 2024). We further compare with graph-based weight-space models. NG-GNN and NG-T (Kofinas et al., 2024) represent the input network as a graph and process it using a graph neural network or relational Transformer, respectively. DNG-Encoder (Wu et al., 2026) extends this approach by constructing a dynamic neural graph that models inference-time information flow. We also report the probe-enhanced Neural Graphs variants with 64 and 128 probes, as well as vanilla probing with 64 and 128 probes, following the evaluation of ProbeGen (Kahana et al., 2025). Finally, we include NFT (Zhou et al., 2023b), ProbeGen (Kahana et al., 2025), and our HiddenProbe model. Results that were not reported for a particular benchmark are marked with “–”. Method MNIST FMNIST SVHN CIFAR-WP CIFAR-GS MLP 0.878±0.0010.878± 0.001 0.874±0.0010.874± 0.001 0.809±0.0030.809± 0.003 – 0.880±0.0000.880± 0.000 StatNN 0.926±0.0000.926± 0.000 0.915±0.0000.915± 0.000 0.843±0.0000.843± 0.000 0.719±0.0100.719± 0.010 0.914±0.0000.914± 0.000 SNE 0.941±0.0000.941± 0.000 0.928±0.0010.928± 0.001 0.858±0.0030.858± 0.003 – 0.927±0.0000.927± 0.000 NFNNP 0.937±0.0000.937± 0.000 0.922±0.0010.922± 0.001 0.856±0.0010.856± 0.001 – 0.922±0.0010.922± 0.001 NFNHNP 0.942±0.0010.942± 0.001 0.935±0.0000.935± 0.000 0.931±0.0050.931± 0.005 – 0.934±0.0010.934± 0.001 NFT – – 0.858±0.0000.858± 0.000 – 0.926±0.0010.926± 0.001 ScaleGMN – – – – 0.941±0.0060.941± 0.006 ScaleGMN-B – – – – 0.941±0.0000.941± 0.000 NG-GNN – – 0.863±0.0020.863± 0.002 0.804±0.0090.804± 0.009 0.930±0.0010.930± 0.001 NG-T – – 0.872±0.0010.872± 0.001 0.817±0.0070.817± 0.007 0.935±0.0000.935± 0.000 DNG-Encoder – – 0.867±0.0020.867± 0.002 0.8743±0.00210.8743± 0.0021 0.936±0.0000.936± 0.000 Neural Graphs (64 probes) – – – 0.888±0.0090.888± 0.009 0.938±0.0010.938± 0.001 Neural Graphs (128 probes) – – – 0.885±0.0050.885± 0.005 0.938±0.0000.938± 0.000 Vanilla Probing (64 probes) – – – 0.885±0.0080.885± 0.008 0.933±0.0010.933± 0.001 Vanilla Probing (128 probes) – – – 0.889±0.0080.889± 0.008 0.936±0.0010.936± 0.001 ProbeGen 0.955±0.0000.955± 0.000 0.945±0.0040.945± 0.004 0.873±0.0050.873± 0.005 0.932±0.0060.932± 0.006 0.957±0.0010.957± 0.001 HiddenProbe 0.966±0.0040.966± 0.004 0.957±0.0020.957± 0.002 0.957±0.0030.957± 0.003 0.940±0.0060.940± 0.006 0.965±0.0000.965± 0.000 Table 5: Full comparison of test-accuracy prediction performance measured by Kendall’s τ (↑ ). Results that were not reported for a particular benchmark are denoted by “–”. D.4 Transformer Accuracy Prediction Across Thresholds Following the evaluation protocol of Tran et al. (2026), we additionally evaluate HiddenProbe on subsets of networks obtained by applying accuracy thresholds of 20%20\%, 40%40\%, 60%60\%, and 80%80\%. Tables 6 and 7 report the results for MNIST-Transformers and AGNews-Transformers, respectively. HiddenProbe consistently outperforms the previous state-of-the-art method, Transformer-NFN Quasi, achieving the highest Kendall’s τ across all evaluated thresholds on both benchmarks. Accuracy threshold No threshold 20% 40% 60% 80% MLP 0.866±0.0020.866± 0.002 0.873±0.0010.873± 0.001 0.874±0.0030.874± 0.003 0.874±0.0060.874± 0.006 0.873±0.0070.873± 0.007 STATNN (Unterthiner et al., 2020) 0.881±0.0010.881± 0.001 0.872±0.0010.872± 0.001 0.868±0.0010.868± 0.001 0.860±0.0010.860± 0.001 0.856±0.0010.856± 0.001 XGBoost (Chen and Guestrin, 2016) 0.860±0.0020.860± 0.002 0.839±0.0040.839± 0.004 0.869±0.0030.869± 0.003 0.846±0.0010.846± 0.001 0.884±0.0010.884± 0.001 LightGBM (Ke et al., 2017) 0.858±0.0020.858± 0.002 0.835±0.0010.835± 0.001 0.847±0.0010.847± 0.001 0.822±0.0010.822± 0.001 0.830±0.0010.830± 0.001 Random Forest (Breiman, 2001) 0.772±0.0020.772± 0.002 0.758±0.0040.758± 0.004 0.769±0.0010.769± 0.001 0.752±0.0010.752± 0.001 0.759±0.0010.759± 0.001 Transformer-NFN (Tran et al., 2025) 0.905±0.0020.905± 0.002 0.899±0.0010.899± 0.001 0.895±0.0010.895± 0.001 0.895±0.0020.895± 0.002 0.888±0.0020.888± 0.002 Transformer-NFN large (Tran et al., 2026) 0.907±0.0010.907± 0.001 0.904±0.0020.904± 0.002 0.897±0.0020.897± 0.002 0.897±0.0020.897± 0.002 0.890±0.0010.890± 0.001 Transformer-NFN Quasi (Tran et al., 2026) 0.911±0.0010.911± 0.001 0.905±0.0010.905± 0.001 0.898±0.0020.898± 0.002 0.897±0.0010.897± 0.001 0.892±0.0010.892± 0.001 ProbeGen 0.898±0.0010.898± 0.001 0.885±0.0010.885± 0.001 0.873±0.0010.873± 0.001 0.872±0.0010.872± 0.001 0.863±0.0010.863± 0.001 HiddenProbe 0.920±0.0030.920± 0.003 0.910±0.0020.910± 0.002 0.904±0.0010.904± 0.001 0.900±0.0020.900± 0.002 0.898±0.0020.898± 0.002 Table 6: Performance measured by Kendall’s τ on the MNIST-Transformers dataset. Uncertainties indicate the standard error over 5 seeds. Accuracy threshold No threshold 20% 40% 60% 80% MLP 0.879±0.0060.879± 0.006 0.875±0.0010.875± 0.001 0.841±0.0120.841± 0.012 0.842±0.0010.842± 0.001 0.862±0.0060.862± 0.006 STATNN (Unterthiner et al., 2020) 0.841±0.0020.841± 0.002 0.839±0.0030.839± 0.003 0.812±0.0030.812± 0.003 0.813±0.0010.813± 0.001 0.812±0.0010.812± 0.001 XGBoost (Chen and Guestrin, 2016) 0.859±0.0010.859± 0.001 0.852±0.0020.852± 0.002 0.872±0.0020.872± 0.002 0.874±0.0010.874± 0.001 0.872±0.0010.872± 0.001 LightGBM (Ke et al., 2017) 0.835±0.0010.835± 0.001 0.845±0.0010.845± 0.001 0.837±0.0010.837± 0.001 0.835±0.0010.835± 0.001 0.820±0.0010.820± 0.001 Random Forest (Breiman, 2001) 0.774±0.0030.774± 0.003 0.801±0.0010.801± 0.001 0.797±0.0010.797± 0.001 0.798±0.0020.798± 0.002 0.773±0.0010.773± 0.001 Transformer-NFN (Tran et al., 2025) 0.910±0.0010.910± 0.001 0.908±0.0010.908± 0.001 0.897±0.0010.897± 0.001 0.896±0.0010.896± 0.001 0.890±0.0010.890± 0.001 Transformer-NFN large (Tran et al., 2026) 0.913±0.0010.913± 0.001 0.910±0.0020.910± 0.002 0.898±0.0020.898± 0.002 0.898±0.0010.898± 0.001 0.893±0.0020.893± 0.002 Transformer-NFN Quasi (Tran et al., 2026) 0.914±0.0010.914± 0.001 0.913±0.0020.913± 0.002 0.901±0.0010.901± 0.001 0.903±0.0020.903± 0.002 0.896±0.0010.896± 0.001 ProbeGen 0.885±0.0010.885± 0.001 0.883±0.0020.883± 0.002 0.860±0.0020.860± 0.002 0.857±0.0010.857± 0.001 0.854±0.0010.854± 0.001 HiddenProbe 0.916±0.0020.916± 0.002 0.918±0.0020.918± 0.002 0.906±0.0020.906± 0.002 0.904±0.0020.904± 0.002 0.901±0.0020.901± 0.002 Table 7: Performance measured by Kendall’s τ on the AGNews-Transformers dataset. Error bars indicate the standard error over 5 seeds. D.5 Number of parameters comparison Here we show the comparison of the number of parameters for different popular models: Method INR classification CNN regression Transformer zoo MNIST FMNIST CIFAR-10 CIFAR-10-GS MNIST-Trans. MLP (Monomial-NFN setup) 2.002.00 2.002.00 2.002.00 – – MLP (Transformer setup) – – – – 0.9330.933 StatNN – – – 1.061.06 0.2030.203 DWS – 0.5530.553 – – – NG-GNN (no probes) – 0.3310.331 – – – NFNNP 15.015.0 15.015.0 16.016.0 2.032.03 – NFNHNP 22.022.0 22.022.0 42.042.0 2.812.81 – ScaleGMN – 1.1281.128 – – – ScaleGMN-B – 1.6891.689 – – – Monomial-NFN 22.022.0 20.020.0 16.016.0 0.250.25 – Monomial-NFN Quasi 22.222.2 20.520.5 16.316.3 0.260.26 – Transformer-NFN – – – – 1.8121.812 Transformer-NFN Large – – – – 2.8572.857 Transformer-NFN Quasi – – – – 1.8941.894 ProbeGen 0.40.4 0.40.4 0.450.45 1.01.0 1.8 HiddenProbe 0.40.4 0.40.4 0.450.45 1.01.0 1.8 Table 8: Number of trainable parameters (millions, M) for published baselines on three INR classification tasks, CIFAR-10-GS regression, and MNIST-Transformers. INR and CNN counts are taken from Tran et al. (2024); Tran et al. (2026) and the ScaleGMN supplement (Kalogeropoulos et al., 2024); MNIST-Transformers counts are from Tran et al. (2025); Tran et al. (2026). Published model configurations may vary across sources and need not exactly match those used in our accuracy comparisons. ProbeGen and HiddenProbe counts are provided by the authors. A dash indicates that a count has not been verified for the specified task. D.6 Time measurement Here we show the time comparison of popular deep weight space models and ProbeGen and HiddenProbe on the MNIST INR benchmark. Table 9: Runtime on the full MNIST-INR training split (55,000 target INRs, batch size 32). Method Time (s) Relative to MLP MLP 103.461 1.00× NFN 90.755 0.88× ProbeGen 109.122 1.05× HiddenProbe 155.508 1.50× NFT 164.774 1.59× DWS 174.766 1.69× D.7 Tuning budget For benchmarks on which ProbeGen had not previously been evaluated, we ran ProbeGen ourselves. We used the same hyper-parameter selection procedure for ProbeGen and for HiddenProbe, ensuring that both methods were tuned under the same validation-based protocol. In all cases, we used the original benchmark train/validation/test splits and selected hyper-parameters using only the validation split. For both methods, we fixed the batch size to 3232, the number of learned probes to 128128, and the weight decay to zero. All models were optimized with Adam. For Transformer experiments, we used 256256 probes. For each dataset, we performed a 27-configuration sweep over the learning rate and learning-rate scheduler. We considered learning rates 3×10−4, 5×10−4, 7×10−4.\3× 10^-4,\,5× 10^-4,\,7× 10^-4\. For the ReduceLROnPlateau scheduler, we evaluated all combinations of patience∈3,5,factor∈0.2,0.3,0.5,0.7,patience∈\3,5\,\,factor∈\0.2,0.3,0.5,0.7\, for each learning rate, resulting in 2424 configurations. We additionally evaluated a cosine-annealing schedule for each of the three learning rates, yielding 33 further configurations and 2727 configurations in total. Each hyper-parameter configuration was first trained for 2020 epochs using seed 00. For test-accuracy prediction, configurations were ranked exclusively according to validation Kendall’s τ; test performance was not used for model selection. The best configuration for each method was then retrained using five seeds, 0,…,40,…,4, for 6060 epochs, and we report the mean and standard error of the resulting test Kendall’s τ. For CIFAR-10 INR classification, we use the same 27-configuration optimization grid for both ProbeGen and HiddenProbe, but select the best configuration according to validation classification accuracy. The selected configuration is subsequently evaluated over five seeds. The final CIFAR-10 INR runs are trained for 3030 epochs.