Paper deep dive
Deep Learning is Singular, and That's Good
Daniel Murfet, Susan Wei, Mingming Gong, Hui Li, Jesse Gell-Redman, Thomas Quella
Models: two-layer ReLU networks, two-layer SiLU networks
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:37:22 PM
Summary
The paper argues that deep neural networks are singular statistical models, meaning they possess singularities in their parameter space that invalidate classical statistical inference methods like the Laplace approximation. The authors introduce singular learning theory as a framework for understanding deep learning, specifically highlighting the Real Log Canonical Threshold (RLCT) as a key metric for model complexity and generalization. Through theoretical analysis and experiments, they demonstrate that Bayesian predictive distributions outperform MAP and MLE estimators in singular models, providing a more robust foundation for deep learning theory.
Entities (5)
Relation Signals (3)
Deep Neural Networks → isa → Singular Statistical Model
confidence 100% · It has been understood for close to twenty years that neural networks are singular statistical models.
Real Log Canonical Threshold → measures → Effective Parameter Count
confidence 95% · The RLCT is the correct way to count the effective number of parameters in a deep neural network.
Singular Learning Theory → explains → Generalization Error
confidence 90% · The geometry of these singularities controls quantities of interest in statistical learning theory, e.g., the generalisation error.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.
Tags
Links
- Source: https://arxiv.org/abs/2010.11560
- Canonical: https://arxiv.org/abs/2010.11560
- Code: https://github.com/suswei/RLCT
Trouble viewing inline? Open PDF directly →
Full Text
56,782 characters extracted from source content.
Expand or collapse full text
Deep Learning is Singular, and That’s Good Daniel Murfet ∗ , Susan Wei ∗ , Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella School of Mathematics and Statistics University of Melbourne Melbourne, Australia October 23, 2020 Abstract In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus “dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice. 1 Introduction It has been understood for close to twenty years that neural networks are singular statistical models [Amari et al., 2003, Watanabe, 2007]. This means, in particular, that the set of network weights equivalent to the true model under the Kullback-Leibler divergence forms a real analytic variety which fails to be an analytic manifold due to the presence of singularities. It has been shown by Sumio Watanabe that the geometry of these singularities controls quantities of interest in statistical learning theory, e.g., the generalisation error. Singular learning theory [Watanabe, 2009] is the study of singular models and requires very different tools from the study of regular statistical models. The breadth of knowledge demanded by singular learning theory – Bayesian statistics, empirical processes and algebraic geometry – is rewarded with profound and surprising results which reveal that singular models are different from regular models in practically important ways. To illustrate the relevance of singular learning theory to deep learning, each section of this paper illustrates a key takeaway idea 1 . The real log canonical threshold (RLCT) is the correct way to count the effective number of parameters in a deep neural network (DNN) (Section 4). To every (model, truth, prior) triplet is associated a birational invariant known as the real log canonical threshold. The RLCT can be understood in simple cases as half the number of normal directions to the set of true parameters. We will explain why this matters more than the curvature of those directions (as measured for example by eigenvalues of the Hessian) laying bare some of the confusion over “flat” minima. ∗ Equal contribution 1 Source codeis available athttps://github.com/susanwe/RLCT. 1 arXiv:2010.11560v1 [cs.LG] 22 Oct 2020 For singular models, the Bayes predictive distribution is superior to MAP and MLE (Section 5).In regular statistical models, the 1) Bayes predictive distribution, 2) maximum a posteriori (MAP) estimator, and 3) maximum likelihood estimator (MLE) have asymptotically equivalent generalisation error (as measured by the Kullback-Leibler divergence). This is not so in singular models. We illustrate in our experiments that even “being Bayesian” in just the final layers improves generalisation over MAP. Our experiments further confirm that the Laplace approximation of the predictive distribution Smith and Le [2017], Zhang et al. [2018] is not only theoretically inappropriate but performs poorly. Simpler true distribution means lower RLCT (Section 6). In singular models the RLCT depends on the (model, truth, prior) triplet whereas in regular models it depends only on the (model, prior) pair. The RLCT increases as the complexity of the true distribution relative to the supposed model increases. We verify this experimentally with a simple family of ReLU and SiLU networks. 2 Related work In classical learning theory, generalisation is explained by measures of capacity such as the l 2 norm, Radamacher complexity, and VC dimension [Bousquet et al., 2003]. It has become clear however that these measures cannot capture the empirical success of DNNs [Zhang et al., 2017]. For instance, over-parameterised neural networks can easily fit random labels [Zhang et al., 2017, Du et al., 2018, Allen-Zhu et al., 2019a] indicating that complexity measures such as Rademacher complexity are very large. There is also a slate of work on generalisation bounds in deep learning. Uniform convergence bounds [Neyshabur et al., 2015, Bartlett et al., 2017, Neyshabur and Li, 2019, Arora et al., 2018] usually cannot provide non-vacuous bounds. Data-dependent bounds [Brutzkus et al., 2018, Li and Liang, 2018, Allen-Zhu et al., 2019b] consider the “classifiability” of the data distribution in generalisation analysis of neural networks. Algorithm-dependent bounds [Daniely, 2017, Arora et al., 2019, Yehudai and Shamir, 2019, Cao and Gu, 2019] consider the relation of Gaussian initialisation and the training dynamics of (stochastic) gradient descent to kernel methods [Jacot et al., 2018]. In contrast to many of the aforementioned works, we are interested in estimating the conditionaldistributionq(y|x). Specifically, we measure the generalisation error of some estimateˆq n (y|x)in terms of the Kullback-Leibler divergence betweenqandˆq n , see (5.1). The next section gives a crash course on singular learning theory. The rest of the paper illustrates the key ideas listed in the introduction. Since we cover much ground in this short note, we will review other relevant work along the way, in particular literature on “flatness", the Laplace approximation in deep learning, etc. 3 Singular Learning Theory To understand why classical measures of capacity fail to say anything meaningful about DNNs, it is important to distinguish between two different types of statistical models. Recall we are interested in estimating the true (and unknown) conditional distributionq(y|x)with a class of modelsp(y|x,w) :w∈WwhereW⊂R d is the parameter space. We say the model isidentifiableif the mappingw7→p(y|x,w)is one-to-one. Letq(x)be the distribution ofx. The Fisher information matrix associated with the modelp(y|x,w) :w∈Wis the matrix-valued function onWdefined by I(w) ij = ∫ ∫ ∂ ∂w i [logp(y|x,w)] ∂ ∂w j [logp(y|x,w)]q(y|x)q(x)dxdy, 2 if this integral is finite. Following the conventions in Watanabe [2009], we have the following bifurcation of statistical models. A statistical modelp(y|x,w)is calledregularif it is 1) identifiable and 2) has positive-definite Fisher information matrix. A statistical model is calledstrictly singularif it is not regular. Letφ(w)be a prior on the model parametersw. To every (model, truth, prior) triplet, we can associate the zeta function,ζ(z) = ∫ K(w) z φ(w)dw,z∈C,whereK(w)is the Kullback-Leibler (KL) divergence between the modelp(y|x,w)and the true distribution q(y|x): K(w) := ∫ ∫ q(y|x) log q(y|x) p(y|x,w) q(x)dxdy.(3.1) For a (model, truth, prior) triplet(p(y|x,w),q(y|x),φ), let−λbe the maximum pole of the corresponding zeta function. We callλthereal log canonical threshold(RLCT) [Watanabe, 2009] of the (model, truth, prior) triplet. The RLCT is the central quantity of singular learning theory. By Watanabe [2009, Theorem 6.4] the RLCT is equal tod/2in regular statistical models and bounded above byd/2in strictly singular models ifrealisabilityholds: let W 0 =w∈W:p(y|x,w) =q(y|x) be the set of true parameters, we sayq(y|x)isrealisableby the model class ifW 0 is non- empty. The condition of realisability is critical to standard results in singular learning theory. Modifications to the theory are needed in the case thatq(y|x)is not realisable, see the condition called relatively finite variance in Watanabe [2018]. Neural networks in singular learning theory.LetW⊆R d be the space of weights of a neural network of some fixed architecture, and letf(x,w) :R N ×W−→R M be the associated function. We shall focus on the regression task and study the model p(y|x,w) = 1 (2π) M/2 exp ( − 1 2 ‖y−f(x,w)‖ 2 ) (3.2) but singular learning theory can also apply to classification, for instance. It is routine to check (see Appendix A.1) that for feedforward ReLU networks not only is the model strictly singular but the matrixI(w)is degenerate for all nontrivial weight vectors and the Hessian of K(w)is degenerate at every point ofW 0 . RLCT plays an important role in model selection.One of the most accessible results in singular learning theory is the work related to the widely-applicable Bayesian information criterion (WBIC) Watanabe [2013], which we briefly review here for completeness. Let D n =(x i ,y i ) n i=1 be a dataset of input-output pairs. LetL n (w)be the negative log likelihood L n (w) =− 1 n n ∑ i=1 logp(y i |x i ,w)(3.3) andp(D n |w) =exp(−nL n (w)).The marginal likelihood of a modelp(y|x,w) :w∈Wis given byp(D n ) = ∫ W p(D n |w)φ(w)dwand can be loosely interpreted as the evidence for the model. Between two models, we should prefer the one with higher model evidence. However, since the marginal likelihood is an intractable integral over the parameter space of the model, one needs to consider some approximation. The well-known Bayesian Information Criterion (BIC) derives from an asymptotic approxi- mation of−logp(D n )using the Laplace approximation, leading toBIC=nL n (w MLE )+ d 2 logn. 3 Since we want the marginal likelihood of the data for some given model to be high one should almost never adopt a DNN according to the BIC, since in such modelsdmay be very large. However, this argument contains a serious mathematical error: the Laplace approximation used to derive BIC only applies toregularstatistical models, and DNNs are not regular. The correct criterion for both regular and strictly singular models was shown in Watanabe [2013] to benL n (w 0 ) +λlognwherew 0 ∈W 0 andλis the RLCT. Since DNNs are highly singular λmay be much smaller thand/2(Section 6) it is possible for DNNs to have high marginal likelihood – consistent with their empirical success. 4 Volume dimension, effective degrees of freedom, and flatness Volume codimension. The easiest way to understand the RLCT is as a volume codimension [Watanabe, 2009, Theorem 7.1]. Suppose thatW⊆R d andW 0 is nonempty, i.e., the true distribution is realisable. We consider a special case in which the KL divergence in a neighborhood of every pointv 0 ∈W 0 has an expression in local coordinates of the form K(w) = d ′ ∑ i=1 c i w 2 i ,(4.1) where the coefficientsc 1 ,...,c d ′ > 0may depend onv 0 andd ′ may be strictly less thand. If the model is regular then this is true withd=d ′ and if it holds ford ′ < dthen we say that the pair(p(y|x,w),q(y|x))isminimally singular. It follows that the setW 0 ⊆Wof true parameters is a regular submanifold of codimensiond ′ (that is,W 0 is a manifold of dimension d−d ′ whereWhas dimensiond). Under this hypothesis there are, near each true parameter v 0 ∈W 0 , exactlyd−d ′ directions in whichv 0 can be varied without changing the model p(y|x,w)andd ′ directions in which varying the parameters does change the model. In this sense, there ared ′ effective parametersnearv 0 . This number of effective parameters can be computed by an integral. Consider the volume of the set of almost true parametersV(t,v 0 ) = ∫ K(w)<t φ (w)dwwhere the integral is restricted to a small closed ball aroundv 0 . As long as the priorφ(w)is non-zero onW 0 it does not affect the relevant features of the volume, so we may assumeφis constant on the region of integration in the firstd ′ directions and normal in the remaining directions, so up to a constant depending only ond ′ we have V(t,v 0 )∝ t d ′ /2 √ c 1 ·c d ′ (4.2) and we can extract the exponent oftin this volume in the limit d ′ = 2 lim t→0 log V(at,v 0 )/V(t,v 0 ) log(a) (4.3) for anya >0,a6= 1. We refer to the right hand side of (4.3) as thevolume codimensionatv 0 . The functionK(w)has the special form (4.1) locally withd ′ =dif the statistical model is regular (and realisable) and withd ′ < din some singular models such as reduced rank regression (Appendix A.2). While such a local form does not exist for a singular model generally (in particular for neural networks) nonetheless under natural conditions [Watanabe, 2009, Theorem 7.1] we haveV(t,v 0 ) =ct λ +o(t λ )wherecis a constant. We assume that in a sufficiently small neighborhood ofv 0 the point RLCTλatv 0 [Watanabe, 2009, Definition 2.7] is less than or equal to the RLCT at every point in the neighborhood so that the multiplicity m= 1, see Section 7.6 of [Watanabe, 2009] for relevant discussion. It follows that the limit on 4 the right hand side of (4.3) exists and is equal toλ. In particularλ=d ′ /2in the minimally singular case. Note that for strictly singular models such as DNNs2λmay not be an integer. This may be disconcerting but the connection between the RLCT, generalisation error and volume dimension strongly suggests that2λis nonetheless the only geometrically meaningful “count” of the effective number of parameters nearv 0 . RLCT and likelihood vs temperature. Again working with the model in (3.2), consider the expectation over the posterior at temperatureTas defined in (A.7) of the negative log likelihood (3.3) E(T) =E 1/T w [ nL n (w) ] =E 1/T w [ 1 2 n ∑ i=1 ‖y i −f(x i ,w)‖ 2 ] + nM 2 log(2π). Note that whennis largeL n (v 0 )≈ M 2 log (2π)for anyv 0 ∈W 0 so forT≈0the posterior concentrates around the setW 0 of true parameters andE(T)≈ nM 2 log (2π). Consider the increase∆E=E(T+ ∆T)−E(T)corresponding to an increase in temperature∆T. It can be shown that∆E≈λ∆Twhere the reader should see [Watanabe, 2013, Corollary 3] for a precise statement. As the temperature increases, samples taken from the tempered posterior are more distant fromW 0 and the errorEwill increase. Ifλis smaller then for a given increase in temperature the quantityEincreases less: this is one way to understand intuitively why a model with smaller RLCT generalises better from the datasetD n to the true distribution. Flatness. It is folklore in the deep learning community that flatness of minima is related to generalisation [Hinton and Van Camp, 1993, Hochreiter and Schmidhuber, 1997] and this claim has been revisited in recent years [Chaudhari et al., 2017, Smith and Le, 2017, Jastrzebski et al., 2017, Zhang et al., 2018]. In regular models this can be justified using the lower order terms of the asymptotic expansion of the Bayes free energy [Balasubramanian, 1997, §3.1] but the argument breaks down in strictly singular models, since for example the Laplace approximation of Zhang et al. [2018] is invalid. The point can be understood via an analysis of the version of the idea in [Hochreiter and Schmidhuber, 1997]. Their measure of entropy compares the volume of the set of parameters with tolerable errort 0 (our almost true parameters) to a standard volume −log [ V(t 0 ,v 0 ) t d/2 0 ] = d−d ′ 2 log(t 0 ) + 1 2 d ∑ i=1 logc i .(4.4) Hence in the cased=d ′ the quantity− 1 2 ∑ i log(c i )is a measure of the entropy of the set of true parameters nearw 0 , a point made for example in Zhang et al. [2018]. However when d ′ < dthis conception of entropy is inappropriate because of thed−d ′ directions in which K(w)is flat nearv 0 , which introduce thet 0 dependence in (4.4). 5 Generalisation The generalisation puzzle [Poggio et al., 2018] is one of the central mysteries of deep learning. Theoretical investigations into the matter is an active area of research Neyshabur et al. [2017]. Many of the recent proposals of capacity measures for neural networks are based on the eigenspectrum of the (degenerate) Hessian, e.g., Thomas et al. [2019], Maddox et al. [2020]. But this is not appropriate for singular models, and hence for DNNs. 5 Since we are interested in learning thedistribution, our notion of generalisation is slightly different, being measured by the KL divergence. Precise statements regarding the generalisation behavior in singular models can be made using singular learning theory. Let the network weights be denotedθrather thanwfor reasons that will become clear. Recall in the Bayesian paradigm, prediction proceeds via the so-called Bayes predictive distribution,p(y|x,D n ) = ∫ p(y|x,θ)p(θ|D n )dθ.More commonly encountered in deep learning practice are the MAP and MLE point estimators. While in a regular statistical model, the three estimators 1) Bayes predictive distribution, 2) MAP, and 3) MLE have thesameleading term in their asymptotic generalisation behavior, the same is not true in singular models. More precisely, letˆq n (y|x) be some estimate of the true unknown conditional densityq(y|x)based on the datasetD n . The generalisation error of the predictorˆq n (y|x)is G(n) :=KL(q(y|x)||ˆq n (y|x)) = ∫ ∫ q(y|x) log q(y|x) ˆq n (y|x) q(x)dy dx.(5.1) To account for sampling variability, we will work with theaverage generalisation error,E n G(n), whereE n denotes expectation over the datasetD n . By Watanabe [2009, Theorem 1.2 and Theorem 7.2], we have E n G(n) =λ/n+o(1/n)ifˆq n is the Bayes predictive distribution,(5.2) whereλis the RLCT corresponding to the triplet(p(y|x,θ),q(y|x),φ(θ)). In contrast, we should note that Zhang et al. [2018] and Smith and Le [2017] rely on the Laplace approximation to explain the generalisation of the Bayes predictive distribution though both works acknowledge the Laplace approximate is inappropriate. For completeness, a quick sketch of the derivation of (5.2) is provided in Appendix A.4. Now by [Watanabe, 2009, Theorem 6.4] we have E n G(n) =C/n+o(1/n)ifˆq n is the MAP or MLE,(5.3) whereC(different for MAP and MLE) is the maximum of some Gaussian process. For regular models, the MAP, MLE, and the Bayes predictive distribution have the same leading term forE n G(n)sinceλ=C=d/2. However in singular models,Cis generally greater thanλ, meaning we should prefer the Bayes predictive distribution for singular models. That the RLCT has such a simple relationship to the Bayesian generalisation error is remarkable. On the other hand, the practical implications of (A.9) are limited since the Bayes predictive distribution is intractable. While approximations to the Bayesian predictive distribution, say via variational inference, might inherit a similar relationship between generalisation and the (variational) RLCT, serious theoretical developments will be required to rigorously establish this. The challenge comes from the fact that for approximate Bayesian predictive distributions, the free energy and generalisation error may have different learning coefficientsλ. This was well documented in the case of a neural network with one hidden layer [Nakajima and Watanabe, 2007]. We set out to investigate whether certain very simple approximations of the Bayes predictive distribution can already demonstrate superiority over point estimators. Suppose the input-target relationship is modeled as in (3.2) but we writeθinstead ofw. We set q(x) =N(0,I 3 ). For now consider the realisable case,q(y|x) =p(y|x,θ 0 )whereθ 0 is drawn randomly according to the default initialisation in PyTorch when model (3.2) is instantiated. We calculateE n G(n)using multiple datasetsD n and a large testing set, see Appendix A.5 for more details. Sincefis a hierarchical model, let’s write it asf θ (·) =h(g(·;v);w)with the dimension ofwbeing relatively small. Letθ MAP = (v MAP ,w MAP )be the MAP estimate forθusing batch gradient descent. The idea of our simple approximate Bayesian scheme is to freeze the 6 Figure 1:Realisable and full batch gradient descent for MAP.Average generalisation errorsE n G(n) are displayed for various approximations of the Bayes predictive distribution. The results of the Laplace approximations are reported in the Appendix and not displayed here because they are higher than other approximation schemes by at least an order of magnitude. Each subplot shows a different combination of hidden layers ing(1 or 5) and activation function inh(ReLU or identity). Note that the y-axis is not shared. network weights at the MAP estimate for early layers and perform approximate Bayesian inference for the final layers 2 . e.g., freeze the parameters ofgatv MAP and perform MCMC overw. Throughout the experiments,g:R 3 →R 3 is a feedforward ReLU block with each hidden layer having 5 hidden units andh:R 3 →R 3 is eitherBAxorBReLU(Ax)where A∈R 3×r ,B∈R r×3 . We setr= 3. We shall consider 1 or 5 hidden layers forg. To approximate the Bayes predictive distribution, we perform either the Laplace approx- imation or the NUTS variant of HMC [Hoffman and Gelman, 2014] in the last two layers, i.e., performing inference overA,Binh(g(·;v MAP );A,B).Note that MCMC is operating in a space of 18 dimensions in this case, which is small enough for us to expect MCMC to perform well. We also implemented the Laplace approximation and NUTS in the last layer only, i.e. performing inference overBinh 2 (h 1 (g(·;v MAP );A MAP );B).Further implementation details of these approximate Bayesian schemes are found in Appendix A.5. From the outset, we expect the Laplace approximation overw= (A,B)to be invalid since the model is singular. We do however expect the last-layer-only Laplace approximation over Bto be sound. Next, we expect the MCMC approximation in either the last layer or last two 2 This is similar in spirit to Kristiadi et al. [2020] who claim that even “being Bayesian a little bit" fixes overconfidence. They approach this via the Laplace approximation for the final layer of a ReLU network. It is also worth noting that Kristiadi et al. [2020] do not attempt to formalise what it means to "fix overconfidence"; the precise statement should be in terms ofG(n). 7 Table 1:Companion to Figure 1. The learning coefficient is the slope of the linear fit1/nversus E n G(n)(no intercept since realisable). TheR 2 value gives a sense of the goodness-of-fit. (a)1 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC9.7090270.966124 last layer only (B) MCMC6.4103800.988921 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace2154.9892660.801077 MAP10.7142160.951051 (b)5 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC1.2862900.985161 last layer only (B) MCMC1.2985040.982298 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace2038.6055890.803736 MAP1.4374730.983411 (c)1 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC3.1171870.977313 last layer only (B) MCMC3.1527100.980132 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace1120.6482980.742412 MAP5.3433110.972212 (d)5 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC0.8355930.957824 last layer only (B) MCMC1.4662730.920716 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace1416.2942880.808991 MAP1.9814830.889519 layers to be superior to the Laplace approximations and to the MAP. We further expect the last-two-layers MCMC to have better generalisation than the last-layer-only MCMC since the former is closer to the Bayes predictive distribution. In summary, we anticipate the following performance order for these five approximate Bayesian schemes (from worst to best): last-two-layers Laplace, last-layer-only Laplace, MAP, last-layer-only MCMC, last-two-layers MCMC. The results displayed in Figure 1 are in line with our stated expectations above,exceptfor the surprise that the last-layer-only MCMC approximation is often superior to the last-two- layers MCMC approximation. This may arise from the fact that MCMC finds the singular setting in the last-two-layers more challenging. In Figure 1, we clarify the effect of the network architecture by varying the following factors: 1) either 1 or 5 layers ing, and 2) ReLU or identity activation inh. Table 1 is a companion to Figure 1 and tabulates for each approximation scheme the slope of1/nversusE n G(n), also known as the learning coefficient. TheR 2 corresponding to the linear fit is also provided. In Appendix A.5, we also show the corresponding results when 1) the data-generating mechanism and the assumed model do not satisfy the condition of realisability and/or 2) the MAP estimate is obtained via minibatch stochastic gradient descent instead of batch gradient descent. 6 Simple functions and complex singularities In singular models the RLCT may vary with the true distribution (in contrast to regular models) and in this section we examine this phenomenon in a simple example. As the true distribution becomes more complicated relative to the supposed model, the singularities of the analytic variety of true parameters should become simpler and hence the RLCT should increase [Watanabe, 2009, §7.6]. Our experiments are inspired by [Watanabe, 2009, §7.2] where tanh(x)networks are considered and the true distribution (associated to the zero network) is held fixed while the number of hidden nodes is increased. Consider the modelp(y|x,w)in (3.2) wheref(x,w) =c+ ∑ H i=1 q i ReLU(〈w i ,x〉+b i )is a two-layer ReLU network with weight vectorw= (w i H i=1 ,b i H i=1 ,q i H i=1 ,c)∈R 4H+1 and w i ∈R 2 ,b i ∈R,q i ∈Rfor1≤i≤H. We letWbe some compact neighborhood of the origin. Given an integer3≤m≤Hwe define a networks m ∈Wandq m (y|x) :=p(y|x,s m )as follows. Letg∈SO(2)stand for rotation by2π/m, setw 1 = √ g(1,0) T . The components ofs m are the vectorsw i =g i−1 w 1 for1≤i≤mandw i = 0fori > m,b i =− 1 3 andq i = 1 for1≤i≤mandb i =q i = 0fori > m, and finallyc= 0. The factor of 1 3 ensures the relevant parts of the decision boundaries lie withinX= [−1,1] 2 . We letq(x)be the uniform 8 Figure 2:Increasingly complicated true distributionsq m (x,y)on[−1,1] 2 ×R. Table 2:RLCT estimates for ReLU and SiLU networks. We observe the RLCT increasing asmincreases, i.e., the true distribution becomes more “complicated” relative to the supposed model. m Nonlinearity RLCT StdR squared 3 ReLU0.526301 0.027181 0.983850 3 SiLU0.522393 0.026342 0.978770 4 ReLU0.539590 0.024774 0.991241 4 SiLU0.539387 0.020769 0.988495 5 ReLU0.555303 0.002344 0.993092 5 SiLU0.555630 0.021184 0.990971 distribution onXand defineq m (x,y) =q m (y|x)q(x). The functionsf(x,s m )are graphed in Figure 2. It is intuitively clear that the complexity of these true distributions increases with m. We letφbe a normal distributionN(0,50 2 )and estimate the RLCTs of the triples (p,q m ,φ). We conducted the experiments withH= 5,n= 1000. For eachm∈ 3,4,5, Table 2 shows the estimated RLCT. Algorithm 1 in Appendix A.3 details the estimation procedure which we base on [Watanabe, 2013, Theorem 4]. As predicted the RLCT increases withmverifying that in this case, the simpler true distributions give rise to more complex singularities. Note that the dimension ofWisd= 21and so if the model were regular the RLCT would be10.5. It can be shown that whenm=Hthe set of true parametersW 0 ⊆Wis a regular submanifold of dimensionm. If such a model were minimally singular its RLCT would be 1 2 ((4m+ 1)−m) = 1 2 (3m+ 1). In the casem= 5we observe an RLCT more than an order of magnitude less than the value8predicted by this formula. So the functionKdoes not behave like a quadratic form nearW 0 . Strictly speaking it is incorrect to speak of the RLCT of a ReLU network because the functionK(w)is not necessarily analytic (Example 4). However we observe empirically that the predicted linear relationship betweenE β w [nL n (w)]and1/βholds in our small ReLU networks (see theR 2 values in Table 2) and that the RLCT estimates are close to those for the two-layer SiLU network [Hendrycks and Gimpel, 2016] which is analytic (the SiLU or sigmoid weighted linear unit isσ(x) =x(1 +e −τx ) −1 which approaches the ReLU asτ→∞. We useτ= 100.0in our experiments). The competitive performance of SiLU on standard benchmarks [Ramachandran et al., 2017] shows that the non-analyticity of ReLU is probably not fundamental. 9 7 Future directions Deep neural networks are singular models, and that’s good: the presence of singularities is necessaryfor neural networks with large numbers of parameters to have low generalisation error. Singular learning theory clarifies how classical tools such as the Laplace approximation are not just inappropriate in deep learning on narrow technical grounds: the failure of this approximation and the existence of interesting phenomena like the generalisation puzzle have a common cause, namely the existence of degenerate critical points of the KL functionK(w). Singular learning theory is a promising foundation for a mathematical theory of deep learning. However, much remains to be done. The important open problems include: SGD vs the posterior. A number of works [ŞimŠekli, 2017, Mandt et al., 2017, Smith et al., 2018] suggest that mini-batch SGD may be governed by SDEs that have the posterior distribution as its stationary distribution and this may go towards understanding why SGD works so well for DNNs. RLCT estimation for large networks.Theoretical RLCTs have been cataloged for small neural networks, albeit at significant effort 3 [Aoyagi and Watanabe, 2005a,b]. We believe RLCT estimation in these small networks should be standard benchmarks for any method that purports to approximate the Bayesian posterior of a neural network. No theoretical RLCTs or estimation procedure are known for modern DNNs. Although MCMC provides the gold standard it does not scale to large networks. The intractability of RLCT estimation for DNNs is not necessarily an obstacle to reaping the insights offered by singular learning theory. For instance, used in the context of model selection, the exact value of the RLCT is not as important as model selection consistency. We also demonstrated the utility of singular learning results such as (5.2) and (5.3) which can be exploited even without knowledge of the exact value of the RLCT. Real-world distributions are unrealisable.The existence of power laws in neural lan- guage model training [Hestness et al., 2017, Kaplan et al., 2020] is one of the most remarkable experimental results in deep learning. These power laws may be a sign of interesting new phenomena in singular learning theory when the true distribution is unrealisable. References Shun-ichi Amari, Tomoko Ozeki, and Hyeyoung Park. Learning and inference in hierarchical models with singularities.Systems and Computers in Japan, 34(7):34–42, 2003. Sumio Watanabe. Almost All Learning Machines are Singular. In2007 IEEE Symposium on Foundations of Computational Intelligence, pages 383–388, 2007. Sumio Watanabe.Algebraic Geometry and Statistical Learning Theory. Cambridge University Press, USA, 2009. Samuel L Smith and Quoc V Le. A Bayesian perspective on generalization and stochastic gradient descent.arXiv preprint arXiv:1710.06451, 2017. Yao Zhang, Andrew M. Saxe, Madhu S. Advani, and Alpha A. Lee. Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning.Molecular Physics, 116(21-22):3214–3223, 2018. 3 Hironaka’s resolution of singularities guarantees existence. However it is difficult to do the required blowup transformations in high dimensions to obtain the standard form. 10 Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. InSummer School on Machine Learning, pages 169–207. Springer, 2003. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. InProceedings of the 5th International Conference on Learning Representations, 2017. arXiv: 1611.03530. Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. InInternational Conference on Learning Representations, 2018. Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. InInternational Conference on Machine Learning, pages 242–252, 2019a. Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. InConference on Learning Theory, pages 1376–1401, 2015. Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. InAdvances in Neural Information Processing Systems, pages 6240–6249, 2017. Behnam Neyshabur and Zhiyuan Li. Towards understanding the role of over-parametrization in generalization of neural networks. InInternational Conference on Learning Representations (ICLR), 2019. Sanjeev Arora, R Ge, B Neyshabur, and Y Zhang. Stronger generalization bounds for deep nets via a compression approach. In35th International Conference on Machine Learning, ICML 2018, 2018. Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns over- parameterized networks that provably generalize on linearly separable data. InInternational Conference on Learning Representations, 2018. Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. InAdvances in Neural Information Processing Systems, pages 8157–8166, 2018. Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overpa- rameterized neural networks, going beyond two layers. InAdvances in Neural Information Processing Systems, pages 6155–6166, 2019b. Amit Daniely. SGD learns the conjugate kernel class of the network. InAdvances in Neural Information Processing Systems, pages 2422–2430, 2017. Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322–332, 2019. Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. InAdvances in Neural Information Processing Systems, pages 6594–6604, 2019. Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. InAdvances in Neural Information Processing Systems, pages 10835–10845, 2019. 11 Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems, pages 8571–8580, 2018. Sumio Watanabe.Mathematical Theory of Bayesian Statistics. CRC Press, 2018. Sumio Watanabe. A Widely Applicable Bayesian Information Criterion.Journal of Machine Learning Research, 14:867–897, 2013. Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. InProceedings of the sixth annual conference on Computational learning theory, pages 5–13, 1993. Sepp Hochreiter and Jürgen Schmidhuber. Flat minima.Neural Computation, 9(1):1–42, 1997. Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. InInternational Conference on Learning Representations, 2017. Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD.arXiv preprint arXiv:1711.04623, 2017. Vijay Balasubramanian. Statistical inference, Occam’s razor and statistical mechanics on the space of probability distributions.Neural Computation, 9(2):349–368, 1997. Tomaso A. Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning I: explaining the non-overfitting puzzle.CoRR, abs/1801.00173, 2018. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. InAdvances in neural information processing systems, pages 5947–5956, 2017. Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Pierre-Antoine Mangazol, Yoshua Bengio, and Nicolas Le Roux. Information matrices and generalization.arXiv:1906.07774 [cs, stat], 2019. arXiv: 1906.07774. Wesley J Maddox, Gregory Benton, and Andrew Gordon Wilson. Rethinking parameter counting in deep models: Effective dimensionality revisited.arXiv preprint arXiv:2003.02139, 2020. Shinichi Nakajima and Sumio Watanabe. Variational Bayes Solution of Linear Neural Networks and Its Generalization Performance.Neural Computation, 19(4):1112–53, 2007. Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, even just a bit, fixes overconfidence in ReLU networks.arXiv preprint arXiv:2002.10118, 2020. Matthew D Hoffman and Andrew Gelman. The No-U-Turn sampler: adaptively setting path lengths in hamiltonian monte carlo.J. Mach. Learn. Res., 15(1):1593–1623, 2014. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs).arXiv preprint arXiv:1606.08415, 2016. 12 Prajit Ramachandran, Barret Zoph, and Quoc V Le. Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941, 2017. Umut ŞimŠekli. Fractional Langevin Monte Carlo: exploring Levy driven stochastic differential equations for Markov Chain Monte Carlo. InProceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3200–3209, 2017. Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate Bayesian inference.The Journal of Machine Learning Research, 18(1):4873– 4907, 2017. Samuel L Smith, Daniel Duckworth, Semon Rezchikov, Quoc V Le, and Jascha Sohl-Dickstein. Stochastic natural gradient descent draws posterior samples in function space.arXiv preprint arXiv:1806.09597, 2018. Miki Aoyagi and Sumio Watanabe. Stochastic complexities of reduced rank regression in Bayesian estimation.Neural Networks, 18(7):924–933, 2005a. Miki Aoyagi and Sumio Watanabe. Resolution of Singularities and the Generalization Error with Bayesian Estimation for Layered Neural Network. InIEICE Trans., pages 2112–2124, 2005b. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.CoRR, abs/1712.00409, 2017. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020. Levent Sagun, Léon Bottou, and Yann LeCun. Singularity of the Hessian in deep learning. CoRR, abs/1611.07476, 2016. Jeffrey Pennington and Pratik Worah. The spectrum of the Fisher information matrix of a single-hidden-layer neural network. InAdvances in Neural Information Processing Systems, pages 5410–5419, 2018. Mary Phuong and Christoph H. Lampert. Functional vs. parametric equivalence of ReLU networks. InInternational Conference on Learning Representations, 2020. William M Boothby.An introduction to differentiable manifolds and Riemannian geometry. Academic press, 1986. A Appendix A.1 Neural networks are strictly singular Many-layered neural networks are strictly singular [Watanabe, 2009, §7.2]. The degeneracy of the Hessian in deep learning has certainly been acknowledged in e.g., Sagun et al. [2016] which recognises the eigenspectrum is concentrated around zero and in Pennington and Worah [2018] which deliberately studies the Fisher information matrix of asingle-hidden-layer, rather than multilayer, neural network. 13 We first explain how to think about a neural network in the context of singular learning theory. A feedforward network of depthcparametrises a functionf:R N −→R M of the form f=A c ◦σ c−1 ◦A c−1 ·σ 1 ◦A 1 where theA l :R d l−1 −→R d l are affine functions andσ l :R d l −→R d l is coordinate-wise some fixed nonlinearityσ:R−→R. LetWbe a compact subspace ofR d containing the origin, whereR d is the space of sequences of affine functions(A l ) c l=1 with coordinates denotedw 1 ,...,w d so thatfmay be viewed as a functionf:R N ×W−→R M . We define p(y|x,w)as in (3.2). We assume the true distribution is realisable,q(y|x) =p(y|x,w 0 ) and that a distributionq(x)onR N is fixed with respect to whichp(x,y) =p(y|x)q(x)and q(x,y) =q(y|x)q(x). Given some priorφ(w)onWwe may apply singular learning theory to the triplet(p,q,φ). By straightforward calculations we obtain K(w) = 1 2 ∫ ‖f(x,w)−f(x,w 0 )‖ 2 q(x)dx(A.1) ∂ 2 ∂w i ∂w j K(w) = ∫ 〈 ∂ ∂w i f(x,w), ∂ ∂w j f(x,w) 〉 q(x)dx + ∫ 〈 f(x,w)−f(x,w 0 ), ∂ 2 ∂w i ∂w j f(x,w) 〉 q(x)dx(A.2) I(w) ij = 1 2 (M−3)/2 π (M−2)/2 ∫ 〈 ∂ ∂w i f(x,w), ∂ ∂w j f(x,w) 〉 q(x)dx(A.3) where〈−,−〉is the dot product. We assumeq(x)is such that these integrals exist. It will be convenient below to introduce another set of coordinates forW. Letw l jk denote the weight from thekth neuron in the(l−1)th layer to thejth neuron in thelth layer and letb l j denote the bias of thejth neuron in thelth layer. Here1≤l≤cand the input is layer zero. Letu l j anda l j denote the value of thejth neuron in thelth layer before and after activation, respectively. Letu l anda l denote the vectors with valuesu l j anda l j , respectively. Letd l denote the number of neurons in thelth layer. Then u l j = d l−1 ∑ k=1 w l jk a l−1 k +b l j ,1≤l≤c,1≤j≤d l a l j =σ(u l j )1≤l < c,1≤j≤d l with the convention thata 0 =xis the input andu c =yis the output. In the case whereσ=ReLUthe partial derivatives ∂ ∂w j fdo not exist on all ofR N . However givenw∈Wwe letD(w)denote the complement inR N of the union over all hidden nodes of the associated decision boundary, that is R N (w) = ⋃ 1≤l<c ⋃ 1≤j≤d l x∈R N :u l j (x) = 0. The partial derivative ∂ ∂w j fexists on the open subset(x,w) :x∈D(w)ofR N ×W. Lemma 1. Supposeσ=ReLUand there arec >1layers. For any hidden neuron1≤j≤d l in layerlwith1≤l < cthere is a differential equation d l−1 ∑ k=1 w l jk ∂ ∂w l jk +b l j ∂ ∂b l j − d l+1 ∑ i=1 w l+1 ij ∂ ∂w l+1 ij f= 0 14 which holds onD(w)for any fixedw∈W. Proof.Without loss of generality assumeM= 1, to simplify the notation. Lete i ∈R d l+1 denote a unit vector and letH(x) = d dx ReLU(x). Writing ∂f ∂u l+1 for a gradient vector ∂f ∂w l+1 ij = 〈 ∂f ∂u l+1 , ∂u l+1 ∂w l+1 ij 〉 = 〈 ∂f ∂u l+1 ,a l j e i 〉 = ∂f ∂u l+1 i u l j H(u l j ) ∂f ∂w l jk = 〈 ∂f ∂u l+1 , ∂u l+1 ∂w l jk 〉 = 〈 ∂f ∂u l+1 , d l+1 ∑ i=1 w l+1 ij a l−1 k H(u l j )e i 〉 = d l+1 ∑ i=1 ∂f ∂u l+1 i w l+1 ij a l−1 k H(u l j ) ∂f ∂b l j = 〈 ∂f ∂u l+1 , ∂u l+1 ∂b l j 〉 = 〈 ∂f ∂u l+1 , d l+1 ∑ i=1 w l+1 ij H(u l j )e i 〉 = d l+1 ∑ i=1 ∂f ∂u l+1 i w l+1 ij H(u l j ). The claim immediately follows. Lemma 2.Supposeσ=ReLU,c >1and thatw∈Whas at least one weight or bias at a hidden node nonzero. Then the matrixI(w)is degenerate and ifw∈W 0 then the Hessian of Katwis also degenerate. Proof.Letw∈Wbe given, and choose a hidden node where at least one of the incident weights (or bias) is nonzero. Then Lemma 1 gives a nontrivial linear dependence relation ∑ i λ i ∂ ∂w i f= 0as functions onD(w). The rows ofI(w)satisfy the same linear dependence relation. At a true parameter the second summand in (A.2) vanishes so by the same argument the Hessian is degenerate. Remark 3.Lemma 2 implies that every true parameter for a nontrivial ReLU network is a degenerate critical point ofK. Hence in the study of nontrivial ReLU networks it is never appropriate to divide by the determinant of the Hessian ofKat a true parameter, and in particular Laplace or saddle-point approximations at a true parameter are invalid. The well-known positive scale invariance of ReLU networks [Phuong and Lampert, 2020] is responsible for the linear dependence of Lemma 1, in the precise sense that the given differential operator is the infinitesimal generator [Boothby, 1986, §IV.3] of the scaling symmetry. However, this is only one source of degeneracy or singularity in ReLU networks. The degeneracy, as measured by the RLCT, is much lower than one would expect on the basis of this symmetry alone (see Section 6). Example 4.In general the KL functionK(w)for ReLU networks is not analytic. For the minimal counterexample, letq(x)be uniform on[−N,N]and zero outside and consider K(b) = ∫ q(x)(ReLU(x−b)−ReLU(x)) 2 dx. It is easy to check that up to a scalar factor K(b) = − 2 3 b 3 +b 2 N0≤b≤N − 1 3 b 3 +b 2 N−N≤b≤0 so thatKisC 2 but notC 3 let alone analytic. 15 A.2 Reduced rank regression For reduced rank regression, the model is p(y|x,w) = 1 (2πσ 2 ) N/2 exp ( − 1 2σ 2 |y−BAx| 2 ) , wherex∈R M ,y∈R N ,AanM×Hmatrix andBanH×Nmatrix; the parameterw denotes the entries ofAandB, i.e.w= (A,B), andσ >0is a parameter which for the moment is irrelevant. If the true distribution is realisable then there isw 0 = (A 0 ,B 0 )such thatq(y|x) = p(y|x,w 0 ). Without loss of generality assumeq(x)is the uniform density. In this case the KL divergence fromp(y|x,w)toq(y|x)is K(w) = ∫ q(y|x) log q(y|x) p(y|x,w) dxdy=‖BA−B 0 A 0 ‖ 2 (1 +E(w)) where the errorEis smooth andE(w) =O(‖BA−B 0 A 0 ‖ 2 )in any region where‖BA− B 0 A 0 ‖< C , soK(w)is equivalent to‖BA−B 0 A 0 ‖ 2 . We writeK(w) =‖BA−B 0 A 0 ‖ 2 for simplicity below. Now assume thatB 0 A 0 is symmetric and thatB 0 is square, i.e.N=H. Then the zero locus ofK(w)is explicitly given as follows W 0 =(A,B) : detB6= 0andA=B −1 B 0 A 0 . It follows thatW 0 is globally a graph overGL(H;R). Indeed, the set(B −1 B 0 A 0 ,B)withB∈ GL (H;R)is exactlyW 0 . ThusW 0 is a smoothH 2 -dimensional submanifold ofR H 2 ×R H×M . To prove thatW 0 is minimally singular in the sense of Section 4 it suffices to show that rank(D 2 A,B K )≥HMwhereD 2 A,B K denotes the Hessian, but as it is no more difficult to do so, we find explicit local coordinates(u,v)near an arbitrary point(A,B)∈W 0 for which v= 0=W 0 andK(u,v) =a(u,v)|u| 2 in this neighborhood, whereais aC ∞ function with a≥c >0for somec. Write A(v) = ( B+v) −1 B 0 A 0 . Thenu,v7→(A(v) +u,B+v)gives local coordinates onR H 2 ×R H×M near(A,B), and K(u,v) =|(B+v)((B+v) −1 B 0 A 0 +u)−B 0 A 0 | =|B 0 A 0 + (B+v)u−B 0 A 0 | 2 =|( B+v)u| 2 , so forvsufficiently small (and henceB+vinvertible) we can takea(u,v) =|(B+v)u| 2 /|u| 2 . A.3 RLCT estimation In this section we detail the estimation procedure for the RLCT used in Section 6. LetL n (w) be the negative log likelihood as in (3.3). Define the data likelihood at inverse temperature β >0to bep β (D n |w) = Π n i=1 p(y i |x i ,w) β which can also be written p β (D n |w) = exp(−βnL n (w)).(A.4) The posterior distribution, at inverse temperatureβ, is defined as p β (w|D n ) = Π n i=1 p(y i |x i ,w) β φ(w) ∫ W Π n i=1 p(y i |x i ,w) β φ(w) = p β (D n |w)φ(w) p β (D n ) (A.5) 16 Algorithm 1RLCT via Theorem 4 in Watanabe [2013] Input:range ofβ’s, set of training setsTeach of sizen, approximate samplesw 1 ,...,w R fromp β (w|D n )for each training setD n and eachβ fortraining setD n ∈Tdo forβin range ofβ’sdo ApproximateE β w [nL n (w)]with 1 R ∑ R i=1 nL n (w r )wherew 1 ,...,w R are approximate samples fromp β (w|D n ) end for Perform generalised least squares to fitλin (A.8), call result ˆ λ(D n ) end for Output: 1 |T| ∑ D n ∈T ˆ λ(D n ) whereφis the prior distribution on the network weightswand p β (D n ) = ∫ W p β (D n |w)φ(w)dw(A.6) is the marginal likelihood of the data at inverse temperatureβ. Finally, denote the expectation of a random variableR(w)with respect to the tempered posteriorp β (w|D n )as E β w [R(w)] = ∫ W R(w)p β (w|D n )dw(A.7) In the main text, we drop the superscript in the quantities (A.4), (A.5), (A.6), (A.7) when β= 1, e.g.,p(D n )rather thanp 1 (D n ). Assuming the conditions of Theorem 4 in Watanabe [2013] hold, we have E β w [nL n (w)] =nL n (w 0 ) + λ β +U n √ λ 2β +O p (1)(A.8) whereβ 0 is a positive constant andU n is a sequence of random variables satisfyingE n U n = 0. In Algorithm 1, we describe an estimation procedure for the RLCT based on the asymptotic result in (A.8). For the estimates in Table 2 thea posterioridistribution was approximated using the NUTS variant of Hamiltonian Monte Carlo [Hoffman and Gelman, 2014] where the first 1000 steps were omitted and20,000samples were collected. Each ˆ λ (D n )estimate in Algorithm 1 was performed by linear regression on the pairs(1/β i ,E β i w [nL n (w)]) 5 i=1 where the five inverse temperaturesβ i are centered on the inverse temperature1/log(20000). A.4 Connection between RLCT and generalisation For completeness, we sketch the derivation of (5.2) which gives the asymptotic expansion of the average generalisation errorE n G(n)of the Bayes prediction distribution in singular models. The exposition is an amalgamation of various works published by Sumio Watanabe, but is mostly based on the textbook [Watanabe, 2009]. To understand the connection between the RLCT andG(n), we first define the so-called Bayes free energyas F(n) =−logp(D n ) whose expectation admits the following asymptotic expansion [Watanabe, 2009]: E n F(n) =E n nS n +λlogn+o(logn) 17 whereS n =− 1 n ∑ n i=1 logq (y i |x i )is the entropy. The expected Bayesian generalisation error is related to the Bayes free energy as follows E n G(n) =EF(n+ 1)−EF(n) Then for the average generalisation error, we have E n G(n) =λ/n+o(1/n).(A.9) Since models with more complex singularities have smaller RLCTs, this would suggest that the more singular a model is, the better its generalisation (assuming one uses the Bayesian predictive distribution for prediction). In this connection it is interesting to note that simpler (relative to the model) true distributions lead to more singular models (Section 6). A.5 Details for generalisation error experiments Simulated data.The distribution ofx∈R 3 is set toq(x) =N(0,I 3 ). In the realisable case,y∈R 3 is drawn according toq(y|x) =p(y|x,θ 0 ). In the nonrealisable setting, we set q(y|x)∝exp−||y−h w 0 (x)|| 2 /2,wherew 0 = (A 0 ,B 0 )is drawn according to the PyTorch model initialisation ofh. MAP training.The MAP estimator is found via gradient descent using the mean-squared- error loss with either the full data set or minibatch set to 32. Training was set to 5000 epochs. No form of early stopping was employed. Calculating the generalisation error. Using a held-out-test setT n ′ =(x ′ i ,y ′ i ) n ′ i=1 , we calculate the average generalisation error as 1 n ′ n ′ ∑ i=1 logq(y ′ i |x ′ i )−E n 1 n ′ n ′ ∑ i=1 log ˆq n (y ′ i |x ′ i )(A.10) Assume the held-out test set is large enough so that the difference betweenE n G(n)and (A.10) is negligible. We will refer to them interchangeably as the average generalisation error. In our experiments we usen ′ = 10,000and30draws of the datasetD n to estimateE n . Last layer(s) inference.Without loss of generality, we discuss performing inference in thewparameters ofhwhile freezing the parameters ofgat the MAP estimate. The steps easily extend to performing inference over the final layer only off=h◦g. Let ̃x i =g v MAP (x i ). Define a new transformed dataset ̃ D n =( ̃x i ,y i ) n i=1 . We take the prior onwto be standard Gaussian. Define the posterior overwgiven ̃ D n as: p(w| ̃ D n )∝p( ̃ D n |w)φ(w) = Π n i=1 exp−||y i −h w ( ̃x i )|| 2 /2φ(w)(A.11) Define the following approximation to the Bayesian predictive distribution ̃p(y|x,D n ) = ∫ p(y|x,(v MAP ,w))p(w| ̃ D n )dw. Letw 1 ,...,w R be some approximate samples fromp(w| ̃ D n ). Then we approximate ̃p(y|x,D n ) with 1 R R ∑ r=1 p(y|x,(v MAP ,w r )) whereRis a large number, set to 1000 in our experiments. We consider the Laplace approximation and the NUTS variant of HMC for drawing samples fromp(w| ̃ D n ): 18 •Laplace in the last layer(s)Recallθ MAP = (v MAP ,w MAP )is the MAP estimate forf θ trained with the dataD n . With the Laplace approximation, we draww 1 ,...w R from the Gaussian N(w MAP ,Σ) whereΣ = (−∇ 2 logp(w| ̃ D n )| w MAP ) −1 is the inverse Hessian 4 of the negative log posterior evaluated at the MAP estimate of the mode. •MCMC in the last layer(s)We used the NUTS variant of HMC to draw samples from (A.11) with the first 1000 samples discarded.. Our implementation used thepyropackage inPyTorch. 4 Following Kristiadi et al. [2020], the code for the exact Hessian calculation is borrowed fromhttps: //github.com/f-dangel/hbp 19 Figure 3:Realisable and minibatch gradient descent for MAP training. Table 3:Companion to Figure 3. (a)1 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC36.7215940.992839 last layer only (B) MCMC20.6769200.983695 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace1768.6550880.838035 MAPinfNaN (b)5 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC13.7292780.924049 last layer only (B) MCMC9.1706420.945613 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace1943.7932360.794679 MAP14.1233080.917502 (c)1 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC22.1754480.975450 last layer only (B) MCMC10.6754550.968584 last two layers (A,B) LaplaceinfNaN last layer only (B) LaplaceinfNaN MAP35.6474640.983284 (d)5 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC4.6524830.922693 last layer only (B) MCMC3.5333660.862125 last two layers (A,B) LaplaceinfNaN last layer only (B) Laplace1004.8523670.901899 MAP6.2566960.940437 20 Figure 4:Nonrealisable and full batch gradient descent for MAP training. Table 4: Companion to Figure 4. The learning coefficient is the slope of the linear fit1/nversus E n G(n)(with intercept since nonrealisable). (a)1 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC11.0860230.969991 last layer only (B) MCMC7.3778710.957824 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace30.6929540.029238 MAP12.9479590.970173 (b)5 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC0.8086010.144260 last layer only (B) MCMC0.7991140.127686 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace-33.8174290.009074 MAP1.2047430.242671 (c)1 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC5.9871870.848490 last layer only (B) MCMC5.3846860.801313 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace38.6291670.059012 MAP8.5607220.816794 (d)5 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC0.7940550.088305 last layer only (B) MCMC1.1415800.162585 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace-5.6826020.000365 MAP1.6480730.284088 21 Figure 5:Nonrealisable and minibatch gradient descent for MAP training.Missing points on the MAP learning curve are due to estimated probabilities too close to 0. Table 5:Companion to Figure 5. The learning coefficient is the slope of the linear fit1/nversus E n G(n)(with intercept since nonrealisable). (a)1 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC11.0860230.969991 last layer only (B) MCMC7.3778710.957824 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace30.6929540.029238 MAP12.9479590.970173 (b)5 hidden layer(s) ing, identity activation inh methodlearning coefficient R squared last two layers (A,B) MCMC0.8086010.144260 last layer only (B) MCMC0.7991140.127686 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace-33.8174290.009074 MAP1.2047430.242671 (c)1 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC5.9871870.848490 last layer only (B) MCMC5.3846860.801313 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace38.6291670.059012 MAP8.5607220.816794 (d)5 hidden layer(s) ing, ReLU activation inh methodlearning coefficient R squared last two layers (A,B) MCMC0.7940550.088305 last layer only (B) MCMC1.1415800.162585 last two layers (A,B) LaplaceNaNNaN last layer only (B) Laplace-5.6826020.000365 MAP1.6480730.284088 22 Figure 6:Realisable and full batch gradient descent for MAP. average generalisation errors of Laplace approximations of the predictive distribution. The last-two-layers Laplace approximation results in numerical instabilities due to degenerate Hessian. Any missing points are due to estimated probabilities too close to 0. 23 Figure 7:Realisable and minibatch gradient descent for MAP training. Details are same as for Figure 6 24 Figure 8:Nonrealisable and full batch gradient descent for MAP training.Details are same as for Figure 6 25 Figure 9:Nonrealisable and minibatch gradient descent for MAP training.Details are same as for Figure 6 26