Paper deep dive
A concentration result for multilayer feedforward neural networks
Vera Koponen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/18/2026, 6:14:52 AM
Summary
This paper presents a mathematical proof of a concentration result for multilayer feedforward neural networks. It demonstrates that under specific conditions—where weight distributions between layers approximate a fixed continuous curve and input neurons have independent, identically distributed values with a continuous density—the value of the output neuron converges in probability to a specific constant as the number of input neurons approaches infinity. The study utilizes model theory and continuous structures to formalize these properties.
Entities (7)
Relation Signals (5)
Vera Koponen → affiliatedwith → Uppsala University
confidence 100% · Vera Koponen Address: Vera Koponen, Department of Mathematics, Uppsala University, Sweden.
Output Neuron → convergesto → Constant Value
confidence 95% · there is a number ψ such that... the probability that the value of the output neuron is in [ψ−ε,ψ+ε] tends to 1
Multilayer Feedforward Neural Network → hasproperty → Concentration Result
confidence 95% · We consider... a multilayer feedforward artificial neural network... the main result is that... the probability that the value of the output neuron... tends to 1
Weight Distribution → approximates → Continuous Curve
confidence 90% · the distribution of weights of connections from a layer to the next are... approximated well by a fixed continuous... curve
Input Neurons → havedistribution → Continuous Probability Density
confidence 90% · the values of the n input neurons are independently and identically distributed with a continuous probability density function
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We consider for an arbitrary fixed $\rho$ and for each positive integer $n$ a multilayer feedforward artificial neural network with $\rho$ layers, $n$ neurons in the first layer (the input layer) and only one neuron, the output neuron, in the last layer. Very roughly formulated, the main result is that if the distribution of weights of connections from a layer to the next are, for all large $n$, approximated well by a fixed continuous (but otherwise arbitrary) curve which does not depend on $n$, and if the values of the $n$ input neurons are independently and identically distributed with a continuous probability density function, then there is a number $\psi$ such that for all $\varepsilon > 0$ the probability that the value of the output neuron is in $[\psi - \varepsilon, \psi + \varepsilon]$ tends to 1 as $n$ tends to infinity.
Tags
Links
- Source: https://arxiv.org/abs/2608.15335v1
- Canonical: https://arxiv.org/abs/2608.15335v1
Trouble viewing inline? Open PDF directly →
Full Text
72,389 characters extracted from source content.
Expand or collapse full text
A concentration result for multilayer feedforward neural networks Vera Koponen Address: Vera Koponen, Department of Mathematics, Uppsala University, Sweden. Email address: vera.koponen@math.u.se Date: 15 August, 2026 Abstract. We consider for an arbitrary fixed ρ and for each positive integer n a multilayer feedforward artificial neural network with ρ layers, n neurons in the first layer (the input layer) and only one neuron, the output neuron, in the last layer. Very roughly formulated, the main result is that if the distribution of weights of connections from a layer to the next are, for all large n, approximated well by a fixed continuous (but otherwise arbitrary) curve which does not depend on n, and if the values of the n input neurons are independently and identically distributed with a continuous probability density function, then there is a number ψ such that for all ε>0 >0 the probability that the value of the output neuron is in [ψ−ε,ψ+ε][ψ- ,ψ+ ] tends to 1 as n tends to infinity. 1. Introduction Artificial neural networks (NNs) [24] are widely used in many different contexts and often successfull in predicting outcomes or classifying objects (e.g. images). But they do not produce any explanations, or justifications, of their predictions, or classifications [28]. This raises ethical and legal questions about their use, and it motivates efforts to understand the internal behaviour of NNs. We will consider the most basic type of N, the multilayer feedforward N in which every neuron is assigned a unique nonnegative real number, its value (or signal strength), once all of its input neurons have been assigned values. We also assume that the N has only one output neuron, and that every neuron, except the input neurons, is connected to all neurons in the previous layer. In this setting three things determine the value of the output neuron. Firstly, the values of the input neurons. Secondly, the weights (which are nonnegative real numbers) of the connections from one layer to the next. Thirdly, the so-called activation function which is a specification of how to assign a value to a (not input) neuron based on the values of neurons in the previous layer and the weights of connections from neurons in the previous layer. We wish to understand how the value of the output neuron correlates with the properties of the N such as: the number of layers, the number of neurons in each layer, the distribution of weights of connections from one layer to the next, and the values of the input neurons. For a specific N the value out the output neuron can be exactly computed given complete information about the values of the input neurons and complete information about the N itself, including knowing the weight of each individual connection. But here we are interested in finding more general correlations for large NNs. The focus on large NNs is motivated by the fact that such are common and that they are harder to analyse as many weighted connections are involved in producing the output of the N. That is, we wish to be able to make conclusions about the distribution of the values of the output neuron given only partial information about the N. In this study such partial information will be the number of layers, some very rough bounds on the number of neurons in each layer, information about the approximate distribution of weights. In addition we will assume that the activation functions of neurons are “sufficently nice”, where the use of averages and taking maxima or minima falls in the category of “sufficiently nice”. When using an N we probably do not know the exact values given to the input neurons. But we may have some knowledge, or at least an educated guess, about the probabilistic distribution of values to the input neurons. So given partial information about an N (ask explained above), a probability distribution on the set of possible combinations of values to the input neurons and an interval I, we can ask what the probability is that the value of the output neuron belongs to I. In this article we prove a “concentration result” for the output neuron, formally stated as Theorem 3.3 and informally stated below. We assume throughout that for some constant ≥1 m≥ 1 the values of neurons and weights of connections are always in the interval [0,][0, m]. Main result, in a simplified formulation. Suppose that for every large enough integer n, ℬnB_n is an N with ρ≥2ρ≥ 2 layers, n neurons in the first layer, the input layer, and with a single output neuron in layer ρ. For i=2,…,ρi=2,…,ρ, let Wi:[0,]→[0,]W_i:[0, m]→[0, m] be continuous with∫[0,]Wi(x)x=1 _[0, m]W_i(x)dx=1, and suppose that the following hold: (1) For every interval I⊆[0,]I [0, m] any non input neuron b, the proportion of connections to b with weight in I from neurons in the previous layer is between ∫IWi(x)x−δ _IW_i(x)dx-δ and ∫IWi(x)x+δ _IW_i(x)dx+δ. (2) For all i=2,…,ρi=2,…,ρ, the activation function for neurons in layer i is defined by using a continuous aggregation function FiF_i, in the sense of Definition 2.2 below, and a continuous function gi:[0,]2→[0,]g_i:[0, m]^2→[0, m], as explained in (A8) If the values of the neurons of the input layer are independently and identically distributed with a continuous probability density function, then there is ψ∈[0,]ψ∈[0, m] such that for all ε>0 >0, the probability that the value of the output neuron is in [ψ−ε,ψ+ε][ψ- ,ψ+ ] tends to 1 as n→∞n→∞. The proof of the main result shows that if ℬnB_n is as assumed above and n is large, then there is an N ℬn′B _n with only 2 layers and n input neurons such that with high probability the value of the output neuron of ℬn′B _n is in [ψ−ε,ψ−ε][ψ- ,ψ- ] for the same ψ as above. This is Corollary 3.4. It seems to be impossible to get the conclusion of the theorem above without having some knowledge about the weights of the N, as for example expressed by (1) above, because the values of non input neurons depend on the weights of connections and the values of the input neurons. One can find out the distribution of weights of connections to any neuron from neurons in the previous layer with relatively small computational resources: Suppose that the N has ρ layers (where ρ is fixed) and at most n neurons in each layer. Given any non input neuron we can sort, by size, the weights of the at most n incomming connections in time O(nlogn)O(n n). There are at most (ρ−1)n(ρ-1)n non input neurons so the time needed for repeating this for all non input neurons is bounded by O(n2logn)O(n^2 n). After sorting the weights of the incomming connections to a neuron b we can inspect if their distribution follows, up to a small deviation, a sufficiently uncomplicated continuous curve (represented by WiW_i if b belongs to the ithi^th layer) relative to the number of incomming connections to b. Thus checking if condition (1) holds is computationally feasible. Remark 4.6 explains how the main theorem can straightforwardly be generalized to a context where each neuron has several features where each feature has its own value. Related work and further research Convergence properties of graph neural networks (GNNs) have been studied by Adam-Day et. al. in [1, 2] for certain probability distributions on the underlying graph. However the GNNs of [1, 2] do not consider weights of connections from one layer to the next. Once the random graph is chosen in [1, 2] the values of the node features are updated a certain number of steps by means of aggregating (locally or globally) over the values of node features in the previous step and using a Lipschitz continuous updating function. From a technical point of view the most closely related work is probably [19]. This study is also inspired by studies on convergence phenomena such as [8, 14, 16, 17, 18, 20, 21, 22, 23, 25, 26, 27] in the field of Statistical Relational Artificial Intelligence [5, 9, 11, 15]. The main result of the present study has rather strong assumptions. For example, it is assumed that if b1b_1 and b2b_2 are neurons in the same non input layer then the weights of the incomming connections to b1b_1 follow (roughly) the same distribution as the weights of the incomming connection sto b2b_2. I believe that this assumption can be replaced by weaker ones which allow neurons in the same layer to have different distributions of incomming weights. Another direction of generalization is obviously to consider other probability distributions on the values of the input neurons. Under such weaker assumptions, the value of the output neuron need not converge (as n→∞n→∞) almost surely to a single value. Instead, if we are given an interval I⊆[0,]I [0, m] we may be able to estimate the probability that the value of the output neuron belongs to I. Preliminaries By ℕ+N^+ we denote the set of positive integers. For a positive integer n, [n]:=1,…,n[n]:=\1,…,n\. If S is a set then |S||S| is its cardinality which we also call size if S is finite. Finite sequence may be denoted by s¯ s, for some symbol s, and then |s¯|| s| denotes the length of the sequence. In order to formalize the notions of a multilayer feedforward neural network (N) and its states the concept of continuous, or general, structure (see e.g. [4, 6]) will be used. This concept generalizes the concept of “discrete” structure (see e.g. [10, 12, 13]) which includes for example graphs in the usal sense of discrete mathematics. But besides the basic definitions given below no further knowledge of model theory is necessary to understand the results and proofs. By a signature we mean a finite set of so-called relation symbols, each with an associated arity which is a positive integer. By a continuous structure, or just structure, say A, we mean a nonempty set A called the domain of A combined with an interpretation of every R∈σR∈σ which, if R has arity k, is a function from AkA^k to the set of real numbers ℝR; this function, the interpretation of R in A, is denoted by R^A. In this study the range of R^A will always be included in [0,][0, m] for some fixed, but arbitrary, ≥1 m≥ 1. The following is a direct consequence of [3, Corollary A.1.14] which in turn follows from a bound given by Chernoff [7]: Lemma 1.1. Let Z be the sum of n independent 0/1-valued random variables, each one with probability p of having the value 1, where p>0p>0. For every ε>0 >0 there is cε>0c_ >0, depending only on ε , such that the probability that |Z−pn|>εpn|Z-pn|> pn is less than 2e−cεpn2e^-c_ pn. (If p=0p=0 then the same statement holds if ‘2e−cεpn2e^-c_ pn’ is replaced by ‘e−ne^-n’.) 2. Aggregation functions In order to define activation functions for neurons in a way that does not depend on the number of neurons of an N we will use the concept of aggregation function. We fix some real ≥1 m≥ 1. Definition 2.1. (a) Let [0,]fin[0, m]^fin denote the set of all finite nonempty sequences of reals from the interval [0,][0, m], or equivalently, [0,]fin=⋃k=1∞[0,]k[0, m]^fin= _k=1^∞[0, m]^k. (b) A function F:[0,]fin→[0,]F:[0, m]^fin→[0, m] will be called an aggregation function if it is symmetric in the sense that if (x1,…,xk)∈[0,]k(x_1,…,x_k)∈[0, m]^k and (y1,…,yk)(y_1,…,y_k) is a permutation (reordering) of (x1,…,xk)(x_1,…,x_k) then F(x1,…,xk)=F(y1,…,yk)F(x_1,…,x_k)=F(y_1,…,y_k). Examples of aggregation functions include the average of a sequence, av(x1,…,xn):=x1+…+xnnav(x_1,…,x_n):= x_1+…+x_nn and the maximum of a sequence, max(x1,…,xn):= the largest number among x1,…,xn.max(x_1,…,x_n):= the largest number among $x_1,…,x_n$. We will require that aggregation functions used by NNs are continuous in the sense defined below. The intuition behind the definition is that if the entries of two sequences (not necessarily of the same length) have similar distribution (if one forgets about the ordering), then a continuous aggregation function should give roughly the same output for both sequences. The definition below is, in theory at least, slightly stronger than the definition of continuous aggregation function in [19] since conditions (c) and (d) in [19, Definition 2.4] are replaced below by a more general condition, labelled (c). Definition 2.2. Let F:[0,]fin→[0,]F:[0, m]^fin→[0, m] be an aggregation function. We call F continuous if the following two conditions hold: (1) For every ε>0 >0 there is δ>0δ>0 such that for all n, if (q1,…,qn)(q_1,…,q_n), (q1′,…,qn′)(q _1,…,q _n) ∈[0,]n∈[0, m]^n and |qi−qi′|≤δ|q_i-q _i|≤δ for all i=1,…,ni=1,…,n, then |F(q1,…,qn)−F(q1′,…,qn′)|≤ε|F(q_1,…,q_n)-F(q _1,…,q _n)|≤ . (2) For every ε>0 >0 there are δ>0δ>0 and M,N∈ℕ+M,N ^+ such that if α0,…,αM−1∈[0,1] _0,…, _M-1∈[0,1] and (q1,…,qn),(q1′,…,qm′)∈[0,1]<ω(q_1,…,q_n),(q _1,…,q _m)∈[0,1]^<ω (where we may have n≠mn≠ m) are such that conditions (a)–(d) below hold, then |F(q1,…,qn)−F(q1′,…,qm′)|≤ε|F(q_1,…,q_n)-F(q _1,…,q _m)|≤ : (a) n,m≥Nn,m≥ N, (b) for all i=0,…,M−1i=0,…,M-1, if αi>0 _i>0 then αi>δ _i>δ, and (regardless of whether αi>0 _i>0 or not) |l:ql∈[iM,i+1M]|n∈(αi−δ,αi+δ), | \l:q_l∈ [ iM, i+1M ] \ |n∈( _i-δ, _i+δ), |l:ql′∈[iM,i+1M]|m∈(αi−δ,αi+δ), and | \l:q _l∈ [ iM, i+1M ] \ |m∈( _i-δ, _i+δ),\ and (c) if 0≤i<M−20≤ i<M-2 and αi=αi+1=αi+2=0 _i= _i+1= _i+2=0, then l:ql∈[i+1M,i+2M]=l:ql′∈[i+1M,i+2M]=∅. \l:q_l∈ [ i+1M, i+2M ] \=\l:q _l∈ [ i+1M, i+2M ] \= . Lemma 2.3. The aggregation functions maximum, minimum and average are continuous. Proof. For the average this is proved in [19, Lemma 2.5] since only conditions (a) and (b) of part (2) are used and conditions (a) and (b) above are indentical with (a) and (b) in [19, Definition 2.4]. For maximum and minimum the proof of the corresponding result in [19, Lemma 2.5] can easily be modified to yield continuity of maximum and minimum with Definition 2.2 of continuity. □ Lemma 2.4. If F:[0,]fin→[0,]F:[0, m]^fin→[0, m] is a continuous aggregation function and h:[0,]→[0,]h:[0, m]→[0, m] is continuous, then G:[0,]fin→[0,]G:[0, m]^fin→[0, m] defined by G(q1…,qn):=h(F(q1,…,qn))G(q_1…,q_n):=h(F(q_1,…,q_n)) is a continuous aggregation function. 3. Model theoretic representation We fix some ρ≥2ρ≥ 2 and consider a sequence of neural networks (NNs), denoted ℬnB_n (‘ℬB’ for “base structure”) for n∈ℕ+n ^+, such that each ℬnB_n has exactly ρ layers and the first layer of ℬnB_n (the layer of input neurons) has exactly n neurons. For each n and i=1,…,ρi=1,…,ρ, the ithi^th layer of ℬnB_n is denoted by Ln,iL_n,i. We also assume that the last layer, Ln,ρL_n,ρ, is a singleton. We let Ln,ρ:=onL_n,ρ:=\o_n\ and call ono_n the output neuron of ℬnB_n. Fix some real number ≥1 m≥ 1, where we think of m as the maximal value that neurons or connections betwen them can have. For all n, i=1,…,ρ−1i=1,…,ρ-1 and all choices of a∈Ln,ia∈ L_n,i and b∈Ln,i+1b∈ L_n,i+1 there is a connection from a to b which carries a weight which is a real number in [0,][0, m]. Once values are assigned to all input neurons of ℬnB_n (the neurons in Ln,1L_n,1) all other neurons should automatically be assigned values that depend only on the values of the input neurons, the weight of the connections, and some activation function. The collection of values of all neurons obtained in this way will be thought of as a state of ℬnB_n, and we are interested in the set of all states that ℬnB_n can be in. To make these ideas mathematically precise we formulate them in a model theoretic language. Since values of neurons and weights of connections are real numbers we will consider continuous structures in which relation symbols are interpreted as functions taking values in [0,][0, m]. So if σ is a signature (i.e. set of relation symbols), A is a continuous σ-structure, and R∈σR∈σ has arity k, then the interpretation of R in A, denoted R^A, is a function from AkA^k (where A is the domain of A) to ℝR. Since we have, in addition, stipulated that all values must be in [0,][0, m] we also require R^A to be a mapping into [0,][0, m]. Let τ=L1,…,Lρ,Eτ=\L_1,…,L_ρ,E\ where L1,…,LρL_1,…,L_ρ have arity ρ and E has arity 2. For each n, ℬnB_n is a τ-structure with domain BnB_n such that (A1) For all b∈Bnb∈ B_n, there is exactly one i∈[ρ]i∈[ρ] such that Liℬn(b)=1L_i^B_n(b)=1, and if j≠ij≠ i then Ljℬn(b)=0L_j^B_n(b)=0. This expresses that each neuron belongs to exactly one layer, and we define Ln,i:=b∈Bn:Liℬn(b)=1.L_n,i:=\b∈ B_n:L_i^B_n(b)=1\. (A2) If i,j∈[ρ]i,j∈[ρ], j≤ij≤ i or j−i>1j-i>1, then for all a∈Ln,ia∈ L_n,i and b∈Ln,jb∈ L_n,j we have Eℬn(a,b)=0E^B_n(a,b)=0. This expresses that a synapse with positive weight must go from some layer Ln,iL_n,i to the next layer Ln,i+1L_n,i+1. We will assume that all ℬnB_n, n∈ℕ+n ^+, have “similar design” which besides having the same number of layers will mean that they use the same activation functions and that the weights of connections between two fixed consecutive layers have similar distributions for all sufficiently large n. First we fix some functions that will specify a “similarity class” of sequences of NNs. (A3) Let ζ(x)ζ(x) be a polynomial such that ζ(x)>0ζ(x)>0 for x>0x>0 and limx→∞ζ(x)=∞ _x→∞ζ(x)=∞. (A4) Let W2,…,Wρ:[0,]→[0,∞)W_2,…,W_ρ:[0, m]→[0,∞) and g2,…,gρ:[0,]2→[0,]g_2,…,g_ρ:[0, m]^2→[0, m] be continuous functions such that ∫[0,]Wi(x)x=1 _[0, m]W_i(x)dx=1 for all i=2,…,ρi=2,…,ρ. (A5) Let F2,…,Fρ:[0,]fin→[0,]F_2,…,F_ρ:[0, m]^fin→[0, m] be continuous aggregation functions in the sense of Defintion 2.2. Next, we assume that the following conditions hold: (A6) For all i=1,…,ρi=1,…,ρ, |Ln,i|≤ζ(n)|L_n,i|≤ζ(n), so layer i of ℬnB_n has at most ζ(n)ζ(n) neurons. (A7) For all i=2,…,ρi=2,…,ρ, for every interval I⊆[0,]I [0, m] with different endpoints, and for every ε>0 >0, if n is sufficiently large and b∈Ln,ib∈ L_n,i, then ∫IWi(x)dx−ε≤|a∈Ln,i−1:Eℬn(a,b)∈I||Ln,i|≤∫IWi(x)dx+ε, _IW_i(x)dx- ≤ | \a∈ L_n,i-1:E^B_n(a,b)∈ I \ | |L_n,i |≤ _IW_i(x)dx+ , and if ∫IWi(x)x=0 _IW_i(x)dx=0, then the proportion above is 0. Informally, this means that the distribution of weights of connections from Ln,i−1L_n,i-1 to b∈Ln,ib∈ L_n,i is approximated well by WiW_i for large n. (A8) For i=2,…,ρi=2,…,ρ and any b∈Ln,ib∈ L_n,i, if t(n,i−1)t(n,i-1) denotes the number of neurons in layer i−1i-1, vjv_j (j=1,…,t(n,i)j=1,…,t(n,i)) is the value of the jthj^th neuron in Ln,i−1L_n,i-1 (in some arbitrary order) and wjw_j is the weight of the connection from the jthj^th neuron to b, then the value of b is assigned to be Fi(gi(v1,w1),…,gi(vt(n,i−1),wt(n,i−1))).F_i (g_i(v_1,w_1),…,g_i(v_t(n,i-1),w_t(n,i-1)) ). Hence the above expression defines the activation function for neurons in layer i of ℬnB_n for all n. Remark 3.1. Fix any i∈2,…,ρi∈\2,…,ρ\. Since ∫[0,]Wi(x)x=1 _[0, m]W_i(x)dx=1 and WiW_i is continuous on [0,][0, m], hence uniformly continuous on the same interval, there are α,β∈⊆[0,]α,β∈ [0, m] such that β−α>0β-α>0 and Wi(x)>0W_i(x)>0 for all x∈[α,β]x∈[α,β]. Let M∈ℕ+M ^+. It follows that if, for all i=0,…,M−1i=0,…,M-1, we let Ii:=[α+i(β−α)M,α+(i+1)(β−α)M]I_i:= [α+ i(β-α)M,\ α+ (i+1)(β-α)M ], then ∫IiWi(x)x>0 _I_iW_i(x)dx>0. It now follows from Assumption (A7) that if n is sufficiently large, then |Ln,i−1|≥M|L_n,i-1|≥ M. Thus, as n→∞n→∞, then number of neurons in layer i−1i-1 tends to infinity. So the number of neurons in all layers except for the output layer tend to infinity as n→∞n→∞. We assumed that the input layer Ln,1L_n,1 of ℬnB_n has exactly n neurons. Without loss of generality we may (to simplify notation) assume that Ln,1=[n]:=1,…,nL_n,1=[n]:=\1,…,n\. We now formally define the notion of state of ℬnB_n, the state space of ℬnB_n, and the probability measure on the state space. Let σ:=τ∪Pσ:=τ∪\P\ where P has arity 1. The value of a neuron a of ℬnB_n in a particular state will be represented by the value of P(a)P(a) in an expansion to σ of ℬnB_n. More precisely, let nS_n consist of all σ-structures A such that A is an expansion of ℬnB_n and (A9) For all i=2,…,ρi=2,…,ρ and all b∈Ln,ib∈ L_n,i, if a1,…,at(n,i−1)a_1,…,a_t(n,i-1) is a list without repetitions of all neurons in Ln,i−1L_n,i-1, in any order, then P(b)=Fi(gi(P(a1),w1),…,gi(P(at(n,i−1)),wt(n,i−1))).P^A(b)=F_i (g_i (P^A(a_1),w_1 ),…,g_i (P^A(a_t(n,i-1)),w_t(n,i-1) ) ). For each (x1,…,xn)∈[0,]n(x_1,…,x_n)∈[0, m]^n let n(x1,…,xn) S_n(x_1,…,x_n) be the unique ∈nA _n such that xk=P(k)x_k=P^A(k) for all k∈Ln,1:=[n]k∈ L_n,1:=[n]. By (A9), n S_n is a bijection. Let μ:[0,]→[0,∞)μ:[0, m]→[0,∞) be a probability density function. Define μn:[0,]n→[0,∞) _n:[0, m]^n→[0,∞) by μn(x1,…,xn)=μ(x1)⋅…⋅μ(xn) _n(x_1,…,x_n)=μ(x_1)·…·μ(x_n), so μn _n is a probability density function on [0,]n[0, m]^n. Let ℙnP_n be the probability measure on [0,]n[0, m]^n which is induced by μn _n in the sense that if X⊆[0,]nX [0, m]^n is (Lebesque) measurable then ℙn(X)=∫Xμn(x1,…,xn)dx1…dxn.P_n(X)= _X _n(x_1,…,x_n)dx_1… dx_n. Call ⊆nX _n measurable if the inverse image −1() S^-1(X) is a measurable subset of [0,]n[0, m]^n. By mild abuse of notation we also view ℙnP_n as a probability measure on nS_n via the bijection n S_n as follows: for measurable ⊆nX _n, ℙn():=ℙn(n−1()):=∫n−1()μn(x1,…,xn)dx1…dxn.P_n(X):=P_n( S_n^-1(X)):= _ S_n^-1(X) _n(x_1,…,x_n)dx_1… dx_n. For all (x1,…,xn)∈[0,]n(x_1,…,x_n)∈[0, m]^n let Outn(x1,…,xn)Out_n(x_1,…,x_n) be the value of the output neuron of ℬnB_n in the state n(x1,…,xn) S_n(x_1,…,x_n), or formally, Outn(x1,…,xn):=Pn(x1,…,xn)(on).Out_n(x_1,…,x_n):=P S_n(x_1,…,x_n)(o_n). Lemma 3.2. For all n the function Outn:[0,]n→[0,]Out_n:[0, m]^n→[0, m] is continuous. It follows that, for every interval I⊆[0,]I [0, m], Outn−1(I)Out_n^-1(I) is Lebesque measurable. Proof. Fix any n∈ℕ+n ^+. For any fixed w∈[0,]w∈[0, m] and i=2,…,ρi=2,…,ρ, let gi,w(x)=gi(x,w)g_i,w(x)=g_i(x,w) so gi,w:[0,]→[0,]g_i,w:[0, m]→[0, m] is continuous. We prove the claim by induction on ρ. The base case is when ρ=2ρ=2. If w1,…,wnw_1,…,w_n enumerate the weights of connections from Ln,1=[n]L_n,1=[n] to the unique neuron ono_n in L2,2L_2,2 then, by (A9), Pn(x1,…,xn)(on)=F2(g2,w1(x1),…,g2,wn(xn))P S_n(x_1,…,x_n)(o_n)=F_2(g_2,w_1(x_1),…,g_2,w_n(x_n)) where the right hand side is a composition of continuous functions in the usual sense of mathematical analysis, since the aggregation function F2F_2 restricted to [0,]n[0, m]^n is continuous in this usual sense because of part (1) of Definition 2.2. Hence OutnOut_n is continuous if ρ=2ρ=2. By the same argument it follows that if ρ>2ρ>2, b∈Ln,2b∈ L_n,2, and w1,…,wnw_1,…,w_n enumerate the weights of connections from Ln,1=[n]L_n,1=[n] to b then Pn(x1,…,xn)(b)=F2(g2,w1(x1),…,g2,wn(xn))P S_n(x_1,…,x_n)(b)=F_2(g_2,w_1(x_1),…,g_2,w_n(x_n)) Then H2b(x1,…,xn):=F2(g2,w1(x1),…,g2,wn(xn))H_2^b(x_1,…,x_n):=F_2(g_2,w_1(x_1),…,g_2,w_n(x_n)) is continuous. where the right hand side is a continuous function from Now suppose that 2≤i<ρ2≤ i<ρ and that, for all b∈Ln,ib∈ L_n,i, there is continuous Hib:[0,]n→[0,]H_i^b:[0, m]^n→[0, m] such that Pn(x1,…,xn)(b)=Hib(x1,…,xn).P S_n(x_1,…,x_n)(b)=H_i^b(x_1,…,x_n). Note that we showed that this holds if i=2i=2. Let b∈Ln,i+1b∈ L_n,i+1, let a1,…,at(n,i)a_1,…,a_t(n,i) be an enumeration of Ln,iL_n,i, and let wjw_j be the weight of the connection from aja_j to b. By (A9) and the assumption we have Pn(x1,…,xn)(b)= P S_n(x_1,…,x_n)(b)= Fi+1(gi+1,w1(Hia1(x1,…,xn)),…,gi+1,wt(n,i)(Hiat(n,i)(x1,…,xn))) F_i+1 (g_i+1,w_1(H_i^a_1(x_1,…,x_n)),…,g_i+1,w_t(n,i)(H_i^a_t(n,i)(x_1,…,x_n)) ) where the right hand side is (using part (1) of Definition 2.2) a composition of continuous functions, so the function Hi+1b(x1,…,xn):=Pn(x1,…,xn)(b)H_i+1^b(x_1,…,x_n):=P S_n(x_1,…,x_n)(b) is continuous. This completes the inductive step. □ Theorem 3.3. Suppose that ℬnB_n, n∈ℕ+n ^+, is a sequence of NNs subject to the assumptions stated above ((A1) – (A8)) and that the set of states nS_n and the probability distribution ℙnP_n are as defined above. Then there is ψ∈[0,]ψ∈[0, m] such that for all ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn(∈n:P(on)∈(ψ−ε,ψ+ε))≥1−e−cn.P_n ( \A _n:P^A(o_n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn. Briefly formulated, the next theorem states that under the above stated assumptions only two layers suffice to get, up to an arbitrarily small error, the same distribution of the output neuron for large n. Corollary 3.4. Suppose ℬnB_n, n∈ℕ+n ^+, is a sequence of NNs subject to the assumptions stated above and that ℙnP_n is as stated above. Let ψ be as in Theorem 3.3. Then there is a sequence ℬn′B _n, n∈ℕ+n ^+, of NNs such that the following hold: • Each ℬn′B _n has exactly 2 layers and n neurons in the first layer. • All assumptions above hold for ℬn′B _n in place of ℬnB_n with the same W2W_2 and g2g_2 but with a modified continuous aggregation function F2′F _2 in place of F2F_2 which ℬnB_n uses. • If ℙn′P _n is the probability distribution induced by μn _n on the state space n′S _n of ℬn′B _n and on′o _n denotes the output neuron of ℬn′B _n, then for all ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn′(∈n′:P(on′)∈(ψ−ε,ψ+ε))≥1−e−cn.P _n ( \A _n:P^A(o _n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn. 4. Proofs We prove Theorem 3.3 by induction on the number ρ of layers, where ρ≥2ρ≥ 2 (by assumption). 4.1. The base case: 2 layers In this section we prove the following lemma which gives Theorem 3.3 in the case when ρ=2ρ=2: Lemma 4.1. Suppose that ρ=2ρ=2, so each ℬnB_n has two layers, Ln,1L_n,1 and Ln,2=onL_n,2=\o_n\. There is ψ∈[0,]ψ∈[0, m] such that for all ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn(∈n:P(on)∈(ψ−ε,ψ+ε))≥1−e−cn.P_n ( \A _n:P^A(o_n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn. Moreover, ψ depends only on W2,F2W_2,F_2, and g2g_2. IΔ,i:=[iΔ,(i+1)Δ].I_ ,i:= [ i m ,\ (i+1) m ]. Lemma 4.2. Let n,Δ∈ℕ+n, ^+, i∈0,…,Δ−1i∈\0,…, -1\, and k∈[n]k∈[n]. Then ℙn(∈n:P(k)∈IΔ,i)= _n ( \A _n:P^A(k)∈ I_ ,i \ )= ℙn((x1,…,xn)∈[0,]n:xk∈IΔ,i)=∫IΔ,iμ(x)x. _n (\(x_1,…,x_n)∈[0, m]^n:x_k∈ I_ ,i \ )= _I_ ,iμ(x)dx. Hence the probability does not depend on n or on k, but only on Δ and i (and μ which we have fixed). Proof. From the definitions it directly follows that xk∈IΔ,ix_k∈ I_ ,i if and only if Pn(x1,…,xn)(k)∈IΔ,iP S_n(x_1,…,x_n)(k)∈ I_ ,i. Since for all X⊆[0,]nX [0, m]^n, ℙn(X)=ℙn(n(X))P_n(X)=P_n( S_n(X)) it suffices to prove the second equality of the lemma. By the definition of ℙnP_n and since μn(x1,…,xn)=μ(x1)⋯…⋅μ(xn) _n(x_1,…,x_n)=μ(x_1)·s…·μ(x_n) where μ:[0,]→[0,∞)μ:[0, m]→[0,∞) is a probability density function, we get, by using Fubini’s theorem, ℙn((x1,…,xn)∈[0,]n:xk∈IΔ,i)= _n (\(x_1,…,x_n)∈[0, m]^n:x_k∈ I_ ,i \ )= ∫[0,]k−1×IΔ,i×[0,]n−kμn(x1,…,xn)dx1…dxn= _[0, m]^k-1× I_ ,i×[0, m]^n-k _n(x_1,…,x_n)dx_1… dx_n= ∫[0,]k−1×IΔ,i×[0,]n−kμ(x1)⋯…⋅μ(xn)dx1…dxn _[0, m]^k-1× I_ ,i×[0, m]^n-kμ(x_1)·s…·μ(x_n)dx_1… dx_n (∏i=1k−1∫[0,]μ(xi)dxi)⋅∫IΔ,iμ(xk)dxk⋅(∏i=k+1n∫[0,]μ(xi)dxi)= ( _i=1^k-1 _[0, m]μ(x_i)dx_i )· _I_ ,iμ(x_k)dx_k· ( _i=k+1^n _[0, m]μ(x_i)dx_i )= ∫IΔ,iμ(xk)dxk. _I_ ,iμ(x_k)dx_k. □ Lemma 4.3. Let n∈ℕ+n ^+, k∈[n]k∈[n], and let Ik⊆[0,]I_k [0, m] be an interval. The event ∈n:P(k)∈I\A _n:P^A(k)∈ I\ is independent from from all events ∈n:P(l)∈Ii\A _n:P^A(l)∈ I_i\ where i≠ki≠ k and Ii⊆[0,]I_i [0, m] is an interval. Proof. By the definitions, ℙn(∈n:P(i)∈Ii for all i∈[n])= _n ( \A _n:P^A(i)∈ I_i for all i∈[n] \ )= ℙn((x1,…,xn)∈[0,]n:xi∈Ii for all i∈[n]). _n (\(x_1,…,x_n)∈[0, m]^n:x_i∈ I_i for all i∈[n] \ ). Therefore it suffices to prove that the event (x1,…,xn)∈[0,]n:xk∈Ik\(x_1,…,x_n)∈[0, m]^n:x_k∈ I_k\ is independent from from all events (x1,…,xn)∈[0,]n:xi∈Ii\(x_1,…,x_n)∈[0, m]^n:x_i∈ I_i\ where i≠ki≠ k and Ii⊆[0,]I_i [0, m] is an interval. But this follows from by using Fubini’s therem since μn(x1,…,xn)=μ(x1)⋅…⋅μ(xk) _n(x_1,…,x_n)=μ(x_1)·…·μ(x_k). □ Let βΔ,i:=∫IΔ,iμ(x)x and γΔ,i:=∫IΔ,iW2(x)x. _ ,i:= _I_ ,iμ(x)dx and _ ,i:= _I_ ,iW_2(x)dx. Define (4.1) Bn(Δ,i):=k∈[n]:Eℬn(k,on)∈IΔ,i.B_n( ,i):= \k∈[n]:E^B_n(k,o_n)∈ I_ ,i \. It follows from (A7) that for all ε>0 >0, if n is large enough then (4.2) γΔ,i−ε≤|Bn(Δ,i)|n≤γΔ,i+ε. _ ,i- ≤ |B_n( ,i)|n≤ _ ,i+ . So if γΔ,i>0 _ ,i>0, then |Bn(Δ,i)|→∞|B_n( ,i)|→∞ as n→∞n→∞ and the rate och growth is linear in n. Also, by (A7), if γΔ,i=0 _ ,i=0, then Bn(Δ,i)=∅B_n( ,i)= . For all ∈nA _n, Δ∈ℕ+ ^+ and i,j∈0,…,Δ−1i,j∈\0,…, -1\, let (4.3) Cn(,Δ,i,j):=k∈Bn(Δ,i):P(k)∈IΔ,j.C_n(A, ,i,j):= \k∈ B_n( ,i):P^A(k)∈ I_ ,j \. For Δ∈ℕ+ ^+ and ε>0 >0 define (4.4) n(Δ,ε):= _n( , ):= ∈n: for all i,j∈0,…,Δ−1 such that γΔ,i>0, \A _n: for all $i,j∈\0,…, -1\$ such that $ _ ,i>0$, (βΔ,j−ε)|Bn(Δ,i)|≤|Cn(,Δ,i,j)|≤(βΔ,j+ε)|Bn(Δ,i)|. ( _ ,j- )|B_n( ,i)|≤ |C_n(A, ,i,j) |≤( _ ,j+ )|B_n( ,i)| \. Lemma 4.4. For all n,Δ∈ℕ+n, ^+ and ε>0 >0, n(Δ,ε)X_n( , ) is measurable. If n is sufficiently large, then ℙn(n(Δ,ε))≥1−e−cnP_n (X_n( , ) )≥ 1-e^-cn for some constant c>0c>0 that depends only on ε . Proof. Fix any i,j∈Δi,j∈ . Define n(Δ,ε,i,j):= _n( , ,i,j):= ∈n:(βΔ,j−ε)|Bn(Δ,i)|≤|Cn(,Δ,i,j)|≤(βΔ,j+ε)|Bn(Δ,i)|. \A _n:( _ ,j- )|B_n( ,i)|≤ |C_n(A, ,i,j) |≤( _ ,j+ )|B_n( ,i)| \. From the definition of measurable subset of nS_n it follows that every subset of nS_n of the form n(k):=∈n:P(k)∈IΔ,i for some k∈[n]X_n(k):= \A _n:P^A(k)∈ I_ ,i \\ for some k∈[n] is measurable. Since n(Δ,ε,i,j)X_n( , ,i,j) can be formed from sets of the form n(k)X_n(k) by taking complements, intersections, and unions, a finite number of times, it follows that n(Δ,ε,i,j)X_n( , ,i,j) is measurable. As n(Δ,ε)X_n( , ) is a finite union of sets of the form n(Δ,ε,i,j)X_n( , ,i,j) it follows that also n(Δ,ε)X_n( , ) is measurable. From Lemma 4.2 it follows that, for all k∈[n]k∈[n], (4.5) ℙn(∈n:P(k)∈IΔ,i)=βΔ,j:=∫IΔ,iμ(x)x.P_n ( \A _n:P^A(k)∈ I_ ,i \ )= _ ,j:= _I_ ,iμ(x)dx. First suppose that such that γΔ,i>0 _ ,i>0. Then, as noted above, limn→∞|Bn(Δ,i)|=∞ _n→∞|B_n( ,i)|=∞ so it follows from (4.5), Lemma 4.3, and Lemma 1.1 that there is d>0d>0 such that if n is large enough, then (4.6) ℙn(n(Δ,ε,i,j))≥1−e−dn.P_n (X_n( , ,i,j) )≥ 1-e^-dn. Now suppose that γΔ,i=0 _ ,i=0. Then, as noted above, Bn(Δ,i)=∅B_n( ,i)= and as, for all ∈nA _n, Cn(,Δ,i,j)⊆Bn(Δ,i)C_n(A, ,i,j) B_n( ,i) we get Cn(,Δ,i,j)=∅C_n(A, ,i,j)= . Hence n(Δ,ε,i,j)=nX_n( , ,i,j)=S_n so (4.6) holds for any choice of d>0d>0. Since n(Δ,ε)=⋃0≤i,j≤Δ−1n(Δ,ε,i,j)X_n( , )= _0≤ i,j≤ -1X_n( , ,i,j) and there are only Δ2 ^2 choices of i and j it follows that there is c>0c>0 such that if n is sufficiently large, then ℙn(n(Δ,ε))≥1−e−cn.P_n (X_n( , ) )≥ 1-e^-cn. □ Note that Ln,1=⋃i=0Δ−1Bn(Δ,i)L_n,1= _i=0 -1B_n( ,i) and observe that for every ∈nA _n we have Bn(Δ,i)=⋃j=0Δ−1Cn(,Δ,i,j)andLn,1=⋃i=0Δ−1⋃j=0Δ−1Cn(,Δ,i,j).B_n( ,i)= _j=0 -1C_n(A, ,i,j) L_n,1= _i=0 -1 _j=0 -1C_n(A, ,i,j). For ∈n(Δ,δ)A _n( ,δ) we have (4.7) βΔ,j−δ≤|Cn(,Δ,i,j)||Bn(Δ,i)|≤βΔ,j+δ. _ ,j-δ≤ |C_n(A, ,i,j)||B_n( ,i)|≤ _ ,j+δ. Note that if βΔ,j,γΔ,i>0 _ ,j, _ ,i>0 then |Cn(,Δ,i,j)|n=|Bn(Δ,i)|n⋅|Cn(,Δ,i,j)||Bn(Δ,i)|≤γΔ,iβΔ,j+3δ. |C_n(A, ,i,j)|n= |B_n( ,i)|n· |C_n(A, ,i,j)||B_n( ,i)|≤ _ ,i _ ,j+3δ. In a similar way we get |Cn(,Δ,i,j)|n≥γΔ,iβΔ,j−3δ |C_n(A, ,i,j)|n≥ _ ,i _ ,j-3δ. Thus βΔ,j,γΔ,i>0⟹γΔ,iβΔ,j−3δ≤|Cn(,Δ,i,j)|n≤γΔ,iβΔ,j+3δ. _ ,j, _ ,i>0\ \ _ ,i _ ,j-3δ≤ |C_n(A, ,i,j)|n≤ _ ,i _ ,j+3δ. On the other hand, if βΔ,j=0 or γΔ,i=0 then |Cn(,Δ,i,j)|n=0.if $ _ ,j=0$ or $ _ ,i=0$ \ \ then \ \ |C_n(A, ,i,j)|n=0. Fix M and l∈0,…,M−1l∈\0,…,M-1\. For ∈n(Δ,δ)A _n( ,δ) and k∈[n]k∈[n] let (4.8) qk():=g2(P(k),Eℬn(k,on))q_k(A):=g_2(P^A(k),E^B_n(k,o_n)) and (4.9) ω(,M,l):=|k∈[n]:qk()∈IM,l|n.ω(A,M,l):= | \k∈[n]:q_k(A)∈ I_M,l \ |n. Define Γ1:=(i,j)∈0,…,Δ−1: _1:= \(i,j)∈\0,…, -1\:\ βΔ,j,γΔ,i>0 and g2(IΔ,j×IΔ,i)⊆IM,l and _ ,j, _ ,i>0 and g_2(I_ ,j× I_ ,i) I_M,l \\ \ and Γ2:=(i,j)∈0,…,Δ−1: _2:= \(i,j)∈\0,…, -1\:\ βΔ,j=0 or γΔ,i=0, and _ ,j=0 or _ ,i=0, and g2(IΔ,j×IΔ,i)∩IM,l≠∅ and g2(IΔ,j×IΔ,i)⊈IM,l. g_2(I_ ,j× I_ ,i)∩ I_M,l≠ and g_2(I_ ,j× I_ ,i) I_M,l \. Suppose that ∈n(Δ,δ)A _n( ,δ). Then we have (4.10) ω(,M,l) ω(A,M,l) ≥∑(i,j)∈Γ1|Cn(,Δ,i,j)|n≥∑(i,j)∈Γ1(γΔ,iβΔ,j−3δ) ≥ _(i,j)∈ _1 |C_n(A, ,i,j)|n≥ _(i,j)∈ _1( _ ,i _ ,j-3δ) ≥∑(i,j)∈Γ1γΔ,iβΔ,j. ≥ _(i,j)∈ _1 _ ,i _ ,j. Note that it follows from the definitions that if (i,j)∈0,…,Δ−12(i,j)∈\0,…, -1\^2 and (i,j)∉Γ1∪Γ2(i,j)∉ _1∪ _2, then |Cn(,Δ,i,j)|n=0 |C_n(A, ,i,j)|n=0. So we also have (4.11) ω(,M,l) ω(A,M,l) ≤∑(i,j)∈Γ1|Cn(,Δ,i,j)|n+∑(i,j)∈Γ2|Cn(,Δ,i,j)|n ≤ _(i,j)∈ _1 |C_n(A, ,i,j)|n+ _(i,j)∈ _2 |C_n(A, ,i,j)|n +∑(i,j)∈0,…,Δ−12∖(Γ1∪Γ2)|Cn(,Δ,i,j)|n + _(i,j)∈\0,…, -1\^2 ( _1∪ _2) |C_n(A, ,i,j)|n ≤∑(i,j)∈Γ1(γΔ,iβΔ,j+3δ)+∑(i,j)∈Γ2(γΔ,iβΔ,j+3δ) ≤ _(i,j)∈ _1( _ ,i _ ,j+3δ)+ _(i,j)∈ _2( _ ,i _ ,j+3δ) =∑(i,j)∈Γ1γΔ,iβΔ,j+∑(i,j)∈Γ2γΔ,iβΔ,j+3δ|Γ1|+3δ|Γ2| = _(i,j)∈ _1 _ ,i _ ,j+ _(i,j)∈ _2 _ ,i _ ,j+3δ| _1|+3δ| _2| ≤∑(i,j)∈Γ1γΔ,iβΔ,j+∑(i,j)∈Γ2γΔ,iβΔ,j+6δΔ2. ≤ _(i,j)∈ _1 _ ,i _ ,j+ _(i,j)∈ _2 _ ,i _ ,j+6δ ^2. Let Ω0:=(x,y)∈[0,]2:μ(x)>0 and W(x)>0, and _0:= \(x,y)∈[0, m]^2:μ(x)>0 and W(x)>0 \,\ and let Ω be the closure of Ω0 _0. Then Ω is a finite union of closed rectangles. Now let Xl:=g2−1(IM,l)∩Ω.X_l:=g_2^-1(I_M,l)∩ . By λ2λ^2 we denote the Lebesgue measure on ℝ2R^2 restricted to [0,]2[0, m]^2. Case 1: λ2(Xl)>0λ^2(X_l)>0. As g2g_2 is continuous on [0,]2[0, m]^2 it follows that XlX_l is a compact subset of [0,]2[0, m]^2. Let YlY_l be the boundary of XlX_l relative to [0,]2[0, m]^2. That is, YlY_l contains all (x,y)∈[0,]2(x,y)∈[0, m]^2 such that for every δ′>0δ >0, the ball with radius δ′δ centered at (x,y)(x,y) contains a point in XlX_l and a point not in XlX_l. Then YlY_l is a finite union of curves, so λ2(Yl)=0λ^2(Y_l)=0. Note that (i,j)∈Γ2(i,j)∈ _2 if and only if IΔ,j×IΔ,i∩Yl≠∅I_ ,j× I_ ,i∩ Y_l≠ . Let Z:=⋃(i,j)∈Γ2IΔ,j×IΔ,i.Z:= _(i,j)∈ _2I_ ,j× I_ ,i. Since each IΔ,j×IΔ,iI_ ,j× I_ ,i is a square with side /Δ m/ it follows that we can make λ2(Z)λ^2(Z) as small as we like by choosing Δ sufficiently large. Let V:=⋃(i,j)∈Γ1IΔ,j×IΔ,iV:= _(i,j)∈ _1I_ ,j× I_ ,i Then, for any δ′>0δ >0, if Δ is sufficiently large then λ2(V)≥λ2(Xl)−δ′.λ^2(V)≥λ^2(X_l)-δ . As we assume that λ2(Xl)>0λ^2(X_l)>0 and since λ2(Xl)λ^2(X_l) does not depend on Δ or n it follows that if Δ is sufficiently large then λ2(V)λ^2(V) is larger than some positive contant that does not depend on Δ or n. But as noted above, we can make λ2(Z)λ^2(Z) as small as we like if Δ is sufficiently large, so the proportion λ2(Z)/λ2(Vl)λ^2(Z)/λ^2(V_l) can be made as small as we like if Δ is large enough. Since both Z and V are unions of squares with side length /Δ m/ it follows that the proportion |Γ2|/|Γ1|| _2|/| _1| can be made as large as we like if Δ is large enough. Let ξΔ:=minγΔ,iβΔ,j:(i,j)∈Γ1. _ := \ _ ,i _ ,j:(i,j)∈ _1 \. By the definition of Γ1 _1 it follows that γΔ,iβΔ,j>0 _ ,i _ ,j>0 for all (i,j)∈Γ1(i,j)∈ _1. Hence ξΔ>0 _ >0. As |Γ2|/|Γ1|| _2|/| _1| can be made as large as we like if Δ is large enough it follows that for every δ′>0δ >0, if Δ is large enough then ∑(i,j)∈Γ2γΔ,iβΔ,j≤δ′∑(i,j)∈Γ1ξΔ≤δ′∑(i,j)∈Γ1γΔ,iβΔ,j. _(i,j)∈ _2 _ ,i _ ,j\ ≤\ δ _(i,j)∈ _1 _ \ ≤\ δ _(i,j)∈ _1 _ ,i _ ,j. From the definition of βΔ,i _ ,i and γΔ,i _ ,i it follows that ∑i=0Δ−1βΔ,i=1 _i=0 -1 _ ,i=1 and ∑j=0Δ−1γΔ,j=1 _j=0 -1 _ ,j=1. Hence ∑(i,j)∈Γ1γΔ,iβΔ,j≤1 _(i,j)∈ _1 _ ,i _ ,j≤ 1 and we get ∑(i,j)∈Γ2γΔ,iβΔ,j≤δ′. _(i,j)∈ _2 _ ,i _ ,j≤δ . Substituting this in (4.11) gives ω(,M,l)≤∑(i,j)∈Γ1γΔ,iβΔ,j+δ′+6δΔ2.ω(A,M,l)≤ _(i,j)∈ _1 _ ,i _ ,j+δ +6δ ^2. Combining with (4.10) gives ∑(i,j)∈Γ1γΔ,iβΔ,j≤ω(,M,l)≤∑(i,j)∈Γ1γΔ,iβΔ,j+δ′+6δΔ2 _(i,j)∈ _1 _ ,i _ ,j≤ω(A,M,l)≤ _(i,j)∈ _1 _ ,i _ ,j+δ +6δ ^2 where δ′:=δ′+6δΔ2δ :=δ +6δ ^2 can be made as small as we like by taking δ′>0δ >0 small enough, then Δ large enough, and finally δ>0δ>0 small enough to make 6δΔ26δ ^2 as small as we like. Observe that we have also proved that αl:=∑(i,j)∈Γ1γΔ,iβΔ,j _l:= _(i,j)∈ _1 _ ,i _ ,j is positive if λ2(Xl)>0λ^2(X_l)>0. Case 2: λ2(Xl)=0λ^2(X_l)=0. We can argue similarly as in Case 1 with XlX_l in the role of YlY_l and conclude that ω(,M,l)ω(A,M,l) can be made as small as we like if δ′δ (as in Case 1) is chosen small enough and Δ large enough. Therefore we define αl:=0 _l:=0 and it follows that αl−δ′≤ω(,M,l)≤αl+δ′ _l-δ ≤ω(A,M,l)≤ _l+δ with δ′δ as above. We have proved that, for all l=0,…,M−1l=0,…,M-1, and regardless of whether λ2(Xl)λ^2(X_l) is positive or zero, we have αl−δ′≤ω(,M,l)≤αl+δ′. _l-δ ≤ω(A,M,l)≤ _l+δ . We assume that the aggregation function F2F_2 is continuous. Let ε>0 >0. Then there are δ∗>0δ^*>0, M,N∈ℕ+M,N ^+ such that if α0,…,αM−1∈[0,1] _0,…, _M-1∈[0,1] and (q1,…,qn),(q1′,…,qm′)∈[0,]fin(q_1,…,q_n),(q _1,…,q _m)∈[0, m]^fin are such that conditions (a)–(c) of Definition 2.2 are satisfied with δ∗δ^* in place of δ, then |F2(q1,…,qn)−F2(q1′,…,qm′)|≤ε|F_2(q_1,…,q_n)-F_2(q _1,…,q _m)|≤ . Let δ∗>0δ^*>0, M and N be such that the conclusion of the previous sentence holds. Then we choose δ′>0δ >0 small enough, Δ large enough and then δ>0δ>0 small enough so that δ′:=δ′+6δΔ2δ :=δ +6δ ^2 is smaller than δ∗δ^*. For all sufficiently large n, n(Δ,δ)X_n( ,δ) is nonempty. Let α1,…,αM−1 _1,…, _M-1 be chosen as above. If ∈n(Δ,δ)A _n( ,δ), and ′∈m(Δ,δ)A _m( ,δ) then, with ω(′,M,l):=|k∈[m]:qk(′)∈IM,l|m,ω(A ,M,l):= |k∈[m]:q_k(A )∈ I_M,l|m, we get, for all sufficiently large n,mn,m, αl−δ∗≤ω(,M,l),ω(′,M,l)≤αl+δ∗ for all l=0,…,M−1. _l-δ^*≤ω(A,M,l),ω(A ,M,l)≤ _l+δ^* for all l=0,…,M-1. Without loss of generality we can assume that n,m≥Nn,m≥ N, so conditions (a) and (b) of Definition 2.2 are satisfied if qk:=qk()q_k:=q_k(A) and qk′:=qk(′)q _k:=q_k(A ). Suppose for a moment that also condition (c) of Definition 2.2 holds for sequences constructed like these. Then |F2(q1,…,qn)−F2(q1′,…,qm′)|≤ε|F_2(q_1,…,q_n)-F_2(q _1,…,q _m)|≤ . Since P(on)=F2(q1,…,qn)P^A(o_n)=F_2(q_1,…,q_n) and P′(om)=F2(q1′,…,qm′)P^A (o_m)=F_2(q _1,…,q _m) we get |P(on)−P′(om)|≤ε|P^A(o_n)-P^A (o_m)|≤ . As ℙn(n(Δ,δ))≥1−e−cnP_n (X_n( ,δ) )≥ 1-e^-cn (where c>0c>0) for all choices of Δ and δ and all sufficiently large n, and since ε can be taken as small as we like, it follows that there is ψ∈[0,]ψ∈[0, m] such that, for every open interval I containing ψ, ℙn(∈n:P(on)∈I)≥1−e−cn.P_n ( \A _n:P^A(o_n)∈ I \ )≥ 1-e^-cn. It remains to show that if q¯:=(q1,…,qn) q:=(q_1,…,q_n) and q¯′:=(q1′,…,qm′) q :=(q _1,…,q _m) where qkq_k and qk′q _k are defined as above, then condition (c) of Definition 2.2 holds. Since the verification is the same for q¯ q and q¯′ q we only do it for q¯ q. Verification of condition (c) in Definition 2.2 Suppose that αl=αl+1=αl+2=0 _l= _l+1= _l+2=0. Suppose for a contradiction that g2(Ω)∩IM,l+1≠∅g_2( )∩ I_M,l+1≠ . Then there is z∈Ωz∈ such that g2(z)∈IM,l+1g_2(z)∈ I_M,l+1. Recall that Ω is a finite union of closed rectangles. Let Ωz _z denote the connected component to which z belongs. If there would be z′∈Ωz ∈ _z such that g2(z′)∈IM,l+1g_2(z )∈ I_M,l+1 and g2(z′)≠g2(z)g_2(z )≠ g_2(z), then, by the continuity of g2g_2, g2−1(IM,l+1)∩Ωzg_2^-1(I_M,l+1)∩ _z would have positive measure, hence also XlX_l would have have positive measure which contradicts that αl+1=0 _l+1=0. Hence g2(Ωz)∩IM,l+1g_2( _z)∩ I_M,l+1 is a singleton. There is some k such that g2−1(IM,k)∩Ωzg_2^-1(I_M,k)∩ _z has positive measure, and hence αk>0 _k>0 and g2(Ωz)∩IM,k≠∅g_2( _z)∩ I_M,k≠ . Suppose that k<lk<l. (The case l+2<kl+2<k is treated analogously.) Since we assume that αl=0 _l=0 we can argue as we did for αl+1 _l+1 and conclude that g2(Ωz)∩IM,lg_2( _z)∩ I_M,l is either empty or a singleton. Since g2(Ωz)∩IM,k≠∅g_2( _z)∩ I_M,k≠ , g2(Ωz)∩IM,l+1≠∅g_2( _z)∩ I_M,l+1≠ , and g2g_2 is continuous, it follows that g2↾Ωzg_2 _z must assume all values in IM,lI_M,l. This contradicts the previous conclusion that g2(Ωz)∩IM,lg_2( _z)∩ I_M,l is either empty or a singleton. Hence we conclude that g2(Ω)∩IM,l+1=∅g_2( )∩ I_M,l+1= . It follows that qk()∉IM,l+1q_k(A)∉ I_M,l+1 for all k∈[n]k∈[n], so condition (c) of Definition 2.2 holds. 4.2. The induction step Suppose that ρ≥3ρ≥ 3. Induction hypothesis. If ℬnB_n, n∈ℕ+n ^+, is a sequence of NNs with ρ−1ρ-1 layers, then there is ψ∈[0,]ψ∈[0, m] such that for every ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn(∈n:P(on)∈(ψ−ε,ψ+ε))≥1−e−cn.P_n ( \A _n:P^A(o_n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn. Moreover, ψ depends only on Wi,FiW_i,F_i and gig_i for i=2,…,ρ−1i=2,…,ρ-1. Suppose that ℬnB_n, n∈ℕ+n ^+, s a sequence of NNs with ρ layers. Lemma 4.5. There is ψ∈[0,]ψ∈[0, m] such that for every ε>0 >0 there is c>0c>0 such that for all sufficently large n, ℙn(∈n: for all a∈Ln,ρ−1,P(a)∈(ψ−ε,ψ+ε))≥1−e−cn.P_n ( \A _n: for all a∈ L_n,ρ-1,P^A(a)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn. Moreover, ψ depends only on Wi,FiW_i,F_i and gig_i for i=2,…,ρ−1i=2,…,ρ-1. Proof. For every n and an arbitrary choice of an∈Ln,ρ−1a_n∈ L_n,ρ-1 we construct an N ℬn′B _n with only ρ−1ρ-1 layers as follows. Remove layer ρ (i.e. Ln,ρL_n,ρ) from ℬnB_n and remove all neurons in Ln,ρ−1L_n,ρ-1 except for ana_n. So the layer ρ−1ρ-1 of ℬn′B _n is Ln,ρ−1′:=anL _n,ρ-1:=\a_n\. All other layers are left unchanged and ℬn′B _n will keep Wi,FiW_i,F_i and gig_i for i=2,…,ρ−1i=2,…,ρ-1. So each ℬn′B _n is an N with ρ−1ρ-1 layers and a unique output neuron. Let n′S _n be the set of states of ℬn′B _n and ℙn′P _n the probability measure on n′S _n defined just as ℙnP_n was defined on nS_n. By the induction hypothesis there is ψ∈[0,]ψ∈[0, m] such that for every ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn′(∈n′:P(an)∈(ψ−ε,ψ+ε))≥1−e−cnP _n ( \A _n:P^A(a_n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn and ψ depends only on Wi,FiW_i,F_i and gig_i for i=2,…,ρ−1i=2,…,ρ-1. Because of the definitions of ℬn′B _n, n′S _n and ℙn′P _n, we have ℙn′(∈n′:P(an)∈(ψ−ε,ψ+ε))= _n ( \A _n:P^A(a_n)∈(ψ- ,ψ+ ) \ )= ℙn(∈n:P(an)∈(ψ−ε,ψ+ε)), _n ( \A _n:P^A(a_n)∈(ψ- ,ψ+ ) \ ), so ℙn(∈n:P(an)∈(ψ−ε,ψ+ε))≥1−e−cnP_n ( \A _n:P^A(a_n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn Since ana_n can be any neuron in Ln,ρ−1L_n,ρ-1 and |Ln,ρ−1|≤ζ(n)|L_n,ρ-1|≤ζ(n) it follows that ℙn(∈n:∃a∈Ln,ρ−1,P(a)∉(ψ−ε,ψ+ε))≤ζ(n)⋅e−cn.P_n ( \A _n:∃ a∈ L_n,ρ-1,P^A(a)∉(ψ- ,ψ+ ) \ )\ ≤\ ζ(n)· e^-cn. Since ζ is a polynomial there is c′>0c >0 such that for all sufficiently large n, ζ(n)⋅e−cn≤e−c′nζ(n)· e^-cn≤ e^-c n, and therefore ℙn(∈n:∀a∈Ln,ρ−1,P(a)∈(ψ−ε,ψ+ε))≥1−e−c′n.P_n ( \A _n:∀ a∈ L_n,ρ-1,P^A(a)∈(ψ- ,ψ+ ) \ )≥ 1-e^-c n. This completes the proof of the lemma. □ We will prove that there is φ∈[0,1] ∈[0,1] such that for every ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn(∈n:P(on)∈(φ−ε,φ+ε))≥1−e−cn.P_n ( \A _n:P^A(o_n)∈( - , + ) \ )≥ 1-e^-cn. Moreover, φ will only depend on Wi,FiW_i,F_i and gig_i for i=2,…,ρi=2,…,ρ. Let Ln,ρ−1=a1,…,at(n)L_n,ρ-1=\a_1,…,a_t(n)\, so |Ln,ρ−1|=t(n)|L_n,ρ-1|=t(n). Let us use the following abbreviations: W:=Wρ,F:=FρW:=W_ρ,\ F:=F_ρ, and g:=g2g:=g_2. For any n, ∈nA _n and k∈[t(n)]k∈[t(n)], let qk():=g(P(ak),Eℬn(ak,on))q_k(A):=g(P^A(a_k),E^B_n(a_k,o_n)) and, for M∈ℕ+M ^+ and l∈0,…,M−1l∈\0,…,M-1\, ω(,M,l):=|k∈[t(n)]:qk()∈IM,l|t(n).ω(A,M,l):= | \k∈[t(n)]:q_k(A)∈ I_M,l \ |t(n). Let ψ be as in Lemma 4.5, so ψ depends only on Wi,FiW_i,F_i and gig_i for i=2,…,ρ−1i=2,…,ρ-1. For n∈ℕ+n ^+ and δ>0δ>0 define n(δ):=∈n: for all a∈Ln,ρ−1,P(a)∈(ψ−δ,ψ+δ).X_n(δ):= \A _n: for all a∈ L_n,ρ-1,\ P^A(a)∈(ψ-δ,ψ+δ) \. By Lemma 4.5, there is c>0c>0 such that ℙn(n(δ))≥1−e−cn for all sufficiently large n.P_n (X_n(δ) )≥ 1-e^-cn for all sufficiently large $n$. Recall that F is a continuous aggregation function and, for ∈nA _n, P(on)=F(q1(),…,qt(n)()).P^A(o_n)\ =\ F (q_1(A),…,q_t(n)(A) ). Therefore it suffices to prove that for every M∈ℕ+M ^+ there are α0,…,αM−1∈[0,1] _0,…, _M-1∈[0,1] such that condition (c) of Definition 2.2 holds and, for every δ>0δ>0 such that δ<αlδ< _l if αl>0 _l>0, and if δ′>0δ >0 is sufficiently small, n is sufficiently large, and ∈n(δ′)A _n(δ ), then for all l=0,…,M−1l=0,…,M-1 (4.12) αl−δ≤ω(,M,l)≤αl+δ. _l-δ≤ω(A,M,l)≤ _l+δ. Since P(a)∈(ψ−δ′,ψ+δ′)P^A(a)∈(ψ-δ ,ψ+δ ) if ∈n(δ′)A _n(δ ) and a∈Ln,ρ−1a∈ L_n,ρ-1, where δ′>0δ >0 can be chosen as small as we like, we essentially get a one dimensional problem, in contrast to the corresponding two dimensional problem in the base case. Let Ω0:=x∈[0,]:W(x)>0 and let Ω be the closure of Ω0. _0:= \x∈[0, m]:W(x)>0 \\ and \ let $ $ be the closure of $ _0$. Note that Ω is a finite union of closed intervals. Define gψ:[0,]→[]g_ψ:[0, m]→[ m] by gψ(x):=g(ψ,x)g_ψ(x):=g(ψ,x). For l=0,…,M−1l=0,…,M-1, let Xl:=gψ−1(IM,l)∩Ω and αl:=∫XlW(x)x.X_l:=g_ψ^-1(I_M,l)∩ and _l:= _X_lW(x)dx. Note that XlX_l is a finite union of closed intervals. By Assumption (A7), the proportion of ak∈Ln,ρa_k∈ L_n,ρ such that Eℬn(ak,on)∈XlE^B_n(a_k,o_n)∈ X_l is in (αl−δ′,αl+δ′)( _l-δ , _l+δ ) if n is large enough. Since g is continuous we can choose δ′>0δ >0 small enough so that if n is large enough and ∈n(δ′)A _n(δ ), then (4.12) holds for all l=0,…,M−1l=0,…,M-1. Hence conditions (a) and (b) of Definition 2.2 are satisfied by q¯:=(q1(),…,qt(n)) q:=(q_1(A),…,q_t(n)) if n is sufficiently large. It remains to check that condition (c) of Definition 2.2 holds. By a similar argument as in the base case we get that if αl=αl+1=αl+2 _l= _l+1= _l+2, then gψ(Xl+1)∩IM,l+1=∅g_ψ(X_l+1)∩ I_M,l+1= , so g(ψ,Xl+1)∩IM,l+1=∅g(ψ,X_l+1)∩ I_M,l+1= . As g(ψ,Xl+1)g(ψ,X_l+1) and IM,l+1I_M,l+1 are closed the distance between them is positive. By taking δ′>0δ >0 sufficiently small we may assume that if x∈[ψ−δ′,ψ+δ′]x∈[ψ-δ ,ψ+δ ] then g(x,Xl+1)∩IM,l+1=∅g(x,X_l+1)∩ I_M,l+1= . It follows that qk()∉IM,l+1q_k(A)∉ I_M,l+1 for all ∈n(δ′)A _n(δ ) and k∈[t(n)]k∈[t(n)]. Thus condition (c) of Definition 2.2 holds for q¯ q. 4.3. Proof of Corollary 3.4 Suppose ℬnB_n, n∈ℕ+n ^+, is a sequence of NNs subject to the assumptions stated in Section 3 and that ℙnP_n is as stated in the same section. Let ψ be as in Theorem 3.3. The proof of Theorem 3.3 (see Lemma 4.5) shows that there is φ∈[0,] ∈[0, m] such that for all δ>0δ>0 there is c>0c>0 such that (4.13) ℙn(∈n: for all a∈Ln,2,P(a)∈(φ−δ,φ+δ))≥1−e−cn.P_n ( \A _n: for all a∈ L_n,2,P^A(a)∈( -δ, +δ) \ )≥ 1-e^-cn. Let h:[0,]→[0,]h:[0, m]→[0, m] be any continuous function such that h(φ)=ψh( )=ψ. Let F2′:[0,]fin→[0,]F _2:[0, m]^fin→[0, m] be the aggregation function defined by F2′(x1,…,xn)=h(F2(x1,…,xn))F _2(x_1,…,x_n)=h(F_2(x_1,…,x_n)) for all n and (x1,…,xn)∈[0,]n(x_1,…,x_n)∈[0, m]^n. By Lemma 2.4, F2′F _2 is a continuous aggregation function. Define a sequence ℬn′B _n, n∈ℕ+n ^+, of NNs with only two layers as follows: (1) Layer 1 of ℬn′B _n is identical with layer 1 of ℬnB_n and layer 2 of ℬn′B _n is Ln,2′:=on′L _n,2:=\o _n\ where on′o _n is an aribtrary choice of neuron in layer 2 of ℬnB_n. (2) The weights of connections from Ln,1L_n,1 to on′o _n in ℬn′B _n are the same as the weights of the same connections in ℬnB_n. (3) The function g2:[0,]2→[0,]g_2:[0, m]^2→[0, m] of ℬnB_n is also used by ℬn′B _n, but instead of F2F_2, ℬn′B _n uses the continuous aggregation function F2′F _2, so to determine the value of on′o _n, ℬn′B _n uses the activation function F2′(g2(v1,w1),…,g2(vn,wn))F _2(g_2(v_1,w_1),…,g_2(v_n,w_n)) where viv_i is the value of the ithi^th neuron in layer 1 and wiw_i is the weight of the connection from the ithi^th neuron in layer 1 to on′o _n We let the state space n′S _n of ℬn′B _n be defined as in Section 3 (but with ′S and ℬn′B _n in place of nS_n and ℬnB_n) and we let ℙn′P _n be the probability distribution induced by μn _n on n′S _n in the same way as in that section. Let ε>0 >0. As h is continuous and maps φ to ψ, it follows from (4.13) and the construction of ℬn′B _n that if δ>0δ>0 is chosen sufficiently small in (4.13), then ℙn′(∈n′:P(on′)∈(ψ−ε,ψ+ε))≥1−e−cn.P _n ( \A _n:P^A(o _n)∈(ψ- ,ψ+ ) \ )≥ 1-e^-cn. Remark 4.6. (Generalization to the context of several features with their own values) Theorem 3.3 and Corollary 3.4 can without any new ideas, but at the expense of messier book keeping and notation, be generalized to the following context. Let every neuron have κ “features” where each feature has a value in [0,][0, m]. The features can be represented by relation symbols P1,…,PκP_1,…,P_κ. For each layer i=2,…,ρi=2,…,ρ and index j=1,…,κj=1,…,κ of a feature, there are continuous aggregation functions Fi,j:[0,]fin→[0,]F_i,j:[0, m]^fin→[0, m] and continuous gi,j:[0,]κ+1→[0,]g_i,j:[0, m]^κ+1→[0, m] such that the value of the feature j of a neuron b of ℬnB_n in layer i is equal to Fi,j(gi,j(v1,1,…,vκ,1,w1),…,gi,j(v1,t(n,i−1),…,vκ,t(n,i−1),wt(n,i−1)))F_i,j (g_i,j(v_1,1,…,v_κ,1,w_1),…,g_i,j(v_1,t(n,i-1),…,v_κ,t(n,i-1),w_t(n,i-1)) ) if vj,1,…,vj,t(n,i−1)v_j,1,…,v_j,t(n,i-1) (j=1,…,κj=1,…,κ) enumerates the values of the feature j of neurons in layer i−1i-1 and w1,…,wt(n,i−1)w_1,…,w_t(n,i-1) enumerates the weights of connections from the corresponding neurons in layer i−1i-1 to b. In this context we assume that for all j=1,…,κj=1,…,κ, there is a continuous probability density function μj:[0,]→[0,∞) _j:[0, m]→[0,∞) and we replace μn:[0,]n→[0,∞) _n:[0, m]^n→[0,∞) defined as in Section 3 by the probability density function μn:[0,]κn→[0,∞) _n:[0, m]^κ n→[0,∞) defined by μn(x1,…,xκn=μ1(x1)⋅…⋅μ1(xn)⋅…⋅μκ(xκ(n−1)+1)⋅…⋅μκ(xκn)CLOSE. _n(x_1,…,x_κ n= _1(x_1)·…· _1(x_n)·…· _κ(x_κ(n-1)+1)·…· _κ(x_κ n). The version of Theorem 3.3 in this context states that for every feature PjP_j, j=1,…,κj=1,…,κ, there is ψj∈[0,] _j∈[0, m] such that for all ε>0 >0 there is c>0c>0 such that for all sufficiently large n, ℙn(∈n:Pj(on)∈(ψj−ε,ψj+ε))≥1−e−cn.P_n ( \A _n:P_j^A(o_n)∈( _j- , _j+ ) \ )≥ 1-e^-cn. References [1] Sam Adam-Day, Theodor-Mihai Iliant, İsmail İlkan Ceylan, Zero-One Laws of Graph Neural Networks, in Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023, 70733–70756, https://doi.org/10.52202/075280-3099. [2] Sam Adam-Day, Michael Benedikt, İsmail İlkan Ceylan, Ben Finkelshtein, Almost Surely Asymptotically Constant Graph Neural Networks, in Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024, 124843–124886, https://doi.org/10.52202/079017-3965. [3] Noga Alon and Joel H. Spencer, The Probabilistic Metod, Second Edition, John Wiley & Sons (2000). [4] I. Ben Yaacov, A. Berenstein, C. W. Henson and A. Usvyatsov, Model theory for metric structures, in Z. Chatzidakis, D. Machpherson, A. Pillay and A. Wilkie (editors), Model theory with applications to algebra and analysis, London Mathematical Society Lecture Notes Series, Vol. 350 (2008) 315–427. [5] Guy Van den Broeck, Kristian Kersting, Sriraam Natarajan, David Poole (editors), An Introduction to Lifted Probabilistic Inference, MIT Press (2021), https://doi.org/10.7551/mitpress/10548.001.0001. [6] C. C. Chang and H. Jerome Keisler, Continuous Model Theory, Princeton University Press (1966). [7] H. Chernoff, A measure of the asymptotic efficiency for tests of a hypothesis based on the sum of observations, Annals of Mathematical Statistics, Vol. 23 (1952) 493–509. [8] Fabio G. Cozman, Denis D. Mauá, The finite model theory of Bayesian network specifications: Descriptive complexity and zero/one laws, International Journal of Approximate Reasoning, vol. 110 (2019), 107–126, https://doi.org/10.1016/j.ijar.2019.04.003. [9] Luc De Raedt, Kristian Kersting, Sriraam Natarajan, David Poole, Statistical Relational Artificial Intelligence: Logic, Probability, and Computation, Morgan & Claypool Publishers (2016), https://doi.org/10.2200/S00692ED1V01Y201601AIM032. [10] Heinz-Dieter Ebbinghaus, Jörg Flum, Finite Model Theory, second revised and enlarged edition, Springer, 1999, https://doi.org/10.1007/3-540-28788-4. [11] Lise Getoor, Ben Taskar (editors), Introduction to Statistical Relational Learning, MIT Press (2007), https://doi.org/10.7551/mitpress/7432.001.0001. [12] Grädel, Kolaitis, Libkin, Marx, Spencer, Vardi, Venema, Weinstein, Finite Model Theory and Its Applications, Springer (2007). [13] Wilfrid Hodges, Model Theory, Cambridge University Press (1993). [14] Manfred Jaeger, Convergence results for relational Bayesian networks, in Proceedings of the Thirteenth Annual IEEE Symposium on Logic in Computer Science (LICS 1998), IEEE Computer Society (1998), 44–55, https://doi.org/10.1109/LICS.1998.705642. [15] Angelika Kimmig, Lilyana Mihalkova, Lise Getoor, Lifted graphical models: A survey, Machine Learning, vol. 99 (2015), 1–45, https://doi.org/10.1007/s10994-014-5443-2. [16] Vera Koponen, Conditional probability logic, lifted Bayesian networks, and almost sure quantifier elimination, Theoretical Computer Science, vol. 848 (2020), 1–27, https://doi.org/10.1016/j.tcs.2020.08.006. [17] Vera Koponen, Random expansions of finite structures with bounded degree, Annals of Pure and Applied Logic, vol. 177, no. 3 (2026), 103665, https://doi.org/10.1016/j.apal.2025.103665. [18] Koponen, V., Karlsson, E., Convergence Laws for Expansions of Linear Preorders. In: Kozen, D., de Queiroz, R. (eds) Logic, Language, Information, and Computation. WoLLIC 2025, Lecture Notes in Computer Science, vol 15942. Springer (2026), https://doi.org/10.1007/978-3-031-99536-1_7 [19] Vera Koponen, Information and Computation, Vol. 310 (2026) 105441, https://doi.org/10.1016/j.ic.2026.105441. [20] Vera Koponen, Domain size asymptotics for Markov logic networks, arXiv:2509.04192v2 [cs.AI], 2026, https://doi.org/10.48550/arXiv.2509.04192. [21] Vera Koponen, Yasmin Tousinejad, Random expansions of trees with bounded height, Theoretical Computer Science, vol. 1040 (2025), 115201, https://doi.org/10.1016/j.tcs.2025.115201. [22] Vera Koponen, Felix Weitkämper, Asymptotic elimination of partially continuous aggregation functions in directed graphical models, Information and Computation, vol. 293 (2023), 105061, https://doi.org/10.1016/j.ic.2023.105061. [23] Vera Koponen, Felix Weitkämper, On the relative asymptotic expressivity of inference frameworks, Logical Methods in Computer Science, vol. 20, no. 4 (2024), 13:1–13:52, https://doi.org/10.46298/lmcs-20(4:13)2024. [24] Bernhard Mehlig, Machine Learning with Neural Networks: An Introduction for Scientists and Engineers, Cambridge University Press (2021). [25] Yasmin Tousinejad, Vera Koponen, Random coloured digraphs defined by a Markov logic network, https://arxiv.org/abs/2606.23715. [26] Felix Q. Weitkämper, An asymptotic analysis of probabilistic logic programming, with implications for expressing projective families of distributions, Theory and Practice of Logic Programming, vol. 21, no. 6 (2021), 802–817, https://doi.org/10.1017/S1471068421000314. [27] Felix Weitkämper, Probabilities of the Third Type: Statistical Relational Learning and Reasoning with Relative Frequencies, Journal of Artificial Intelligence Research, vol. 80 (2024), 1407–1436, https://doi.org/10.1613/JAIR.1.15679. [28] Yu Zhang, Peter Tino, Ales Leonardis, Ke Tang, A survey on neural network interpretability, IEEE Transactions on Emerging Topics in Computational Intelligence, Vol 5 (2021) 726–742.