Paper deep dive
On the Statistical Mechanics of Attention
Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
Tags
Links
- Source: https://arxiv.org/abs/2610.17347v1
- Canonical: https://arxiv.org/abs/2610.17347v1
Trouble viewing inline? Open PDF directly →
Full Text
229,167 characters extracted from source content.
Expand or collapse full text
Last Update: October 12, 2026 On the Statistical Mechanics of Attention Jerry Yao-Chieh Hu†‡∗11 1 jhu@u.northwestern.edu Yu-An Chen§∗22 2 b09202059@ntu.edu.tw Maojiang Su†33 3 smj@u.northwestern.edu Han Liu†♮44 4 hanliu@northwestern.edu**footnotetext: These authors contributed equally to this work. † \;Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University ‡ \;Department of Computer Science, College of Connected Computing, Vanderbilt University § @sectionsign\;Department of Physics, National Taiwan University ♮ \;Department of Statistics and Data Science, Northwestern University We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms. Contents 1 Introduction We study the statistical-mechanics of Transformer attention through modern Hopfield models and characterize parameter regimes and critical threshold for robust attention inference. Our key contribution is a statistical-mechanical formulation in which Transformer attention emerges from the standard free-energy minimization principle naturally. Through this perspective, we give a clear characterization of first-order phase transitions of attention inference between glassy and robust-inference phases. Consequently, our framework provides a rigorous way to characterize reliable inference of Transformer attention, and further allows extension to the full attention family. During a Transformer forward pass, an input query selects and combines information from the entire context. Attention mechanism captures the core function of this operation. Concretely, for a query q∈ℝdkq ^d_k and stored key-value pairs kμ,vμ=1M\k_μ,v_μ\_μ=1^M, we denote K and V as the matrices with rows kμ∈ℝdkk_μ ^d_k and vμ∈ℝdvv_μ ^d_v, respectively. Then attention (Vaswani et al., 2017) computes Attn(q,K,V)=Softmax(βqK⊤dk)V=∑μ=1M(eβzμ∑ν=1Meβzν)vμ,withzμ:=qkμ⊤dk. Attn(q,K,V)= Softmax (β qK d_k )V= _μ=1^M( e^β z_μ _ν=1^Me^β z_ν)v_μ, z_μ:= qk_μ d_k. Here β is the inverse-temperature. Attn(q,K,V) Attn(q,K,V) specifies which key kμk_μ does the input query q retrieve and contribute to the output via the value projection V. Recent works link this inference of Transformer attention to the memory retrieval of modern Hopfield models (Ramsauer et al., 2020; Krotov and Hopfield, 2020; Hu et al., 2023), i.e., selecting or combining stored patterns based on the input query. This connection is decomposable into two aspects: • First, Hu et al. (2023); Wu et al. (2023) bridge between modern Hopfield models and attention family including softmax, sparsemax, and beyond (Martins and Astudillo, 2016). • On the other hand, Hu et al. (2023) reveal that softmax solves an entropy-regularized optimization problem via variation (Peters et al., 2019); see (Santos et al., 2025) for a review. Together, they view Transformer attentions as physical models and lead to useful methods (Widrich et al., 2020; Hoover et al., 2023; Auer et al., 2023; Hu et al., 2024a; Wu et al., 2024; Dehmamy et al., 2025). However, the prior work only investigate the retrieval rule itself. The parameter regimes for robust inference in these models remains unclear. This raises a natural question in physics, with beneficial implications for Transformer inference. In statistical mechanics, phase structure characterizes how a system’s behavior changes as its parameters vary. Amit et al. (1985); Amit et al. (1987); Nishimori (2001) analyze the phase structure of classical Hopfield models (Hopfield, 1982) via mean-field replica method. However, they consider i.i.d. and infinite-dimension assumptions. That is, how finite size and realistic structured patterns affect retrieval behavior remains open. Modern Hopfield models offer a natural setting for extending this study of phase transitions: connecting the physics of associative memory to Transformer attention. Yet their phase structure and the conditions governing these transitions remain poorly understood. To address those issues, we utilize the free-energy minimization principle from statistical mechanics (Landau and Lifshitz, 1980; Binder, 1987; Zinn-Justin, 2002) to study phase structures of modern Hopfield models and hence of Transformer attention. Within this framework, different entropy induces different sparsity and hence different attention mechanism (Martins and Astudillo, 2016; Peters et al., 2019; Ramsauer et al., 2020). Through this perspective, we explore first-order phase transitions and hence abrupt reorganization of attention weight beyond classical Hopfield models (Hopfield, 1982; Mezard et al., 1986). This leads to an parameter-dependent mechanism behind modern Hopfield models and hence the phase structure of Transformer attention. Contributions. We develop a statistical-mechanical framework for Transformer attention and use it to understand its inference behavior. Our main contributions are: • A Statistical-Mechanical Theory of Transformer Attention. We formulate Transformer attention as a statistical-mechanical model of associative memory through the free energy of modern Hopfield networks. In this formulation, attention emerges naturally from the equations of motion (Propositions 3.2 and D.7) and its forward pass describes how a query retrieves stored information. This framework allows the unification of attention’s temperature, entropy, sparsity, fluctuations, and retrieval dynamics into one coherent physical picture. • Phase Transitions in Attention Inference. We use the resulting free-energy landscape to study the reliability of information retrieval during a Transformer attention forward pass. Our analysis reveals two retrieval regimes: a glassy-like regime with competing metastable states and a robust-inference regime with stable, target-aligned retrieval. We characterize first-order phase transitions11 1 A first-order phase transition occurs when the equilibrium state changes discontinuously as a control parameter varies. In our case, varying attention temperature can induce such transitions in memory retrieval. Memory load and key correlations further shape the phase structure. between these regimes as attention parameters vary. • Critical Conditions for Robust Attention Inference. We further analyze what governs the stability of memory selection in attention inference. We characterize critical conditions for the coexistence of competing states (Proposition 4.1) and their spectral stability (Proposition 4.2) from correlations among learned keys. These conditions explain how learned key correlations and attention temperature determine whether target-memory selection remains stable or becomes vulnerable to interference from competing keys. Organization. Section 2 presents preliminaries. Section 3 presents physical formulation of attention through modern Hopfield models. Section 4 presents our phase transition results. Section 5 presents numerical validations. Section 6 presents concluding remarks and practical implications. We defer related work discussion and statistical-mechanics tutorial in Appendices A and B due to space limit. Appendix D extends to the stochastic version, and Appendix E analyzes the dynamical stability. Appendices F and G show experiment setup and additional experiments. Table 1: Roadmap from the technical terms to their role in attention. Concept Meaning for Attention Dictionary Fenchel Conjugate It transforms a microscopic quantity into a macroscopic one, and constructs the free-energy objective. Definitions C.6, 2.1 and 2.1 Tsallis Entropy Ψα _α Attention Family: softmax (α=1)(α=1), entmax (1<α<2)(1<α<2), and sparsemax (α=2)(α=2). Sections C.3, 2.1 and 3.3 Query-Key Similarity (Overlap z=1dξ⊤xz= 1 dξ x) It measures the alignment between queries and memories. A larger overpal indicates a stronger retrieval preference for that memory. Definition 3.2 Free-energy Γα _α It defines free-energy landscape and enables assigning energy value for each local minimum. It encodes equation of motion as attention rule. Definition 2.1, Propositions 3.2 and 3.3 Coexistence (Energy Crossing) It occurs when two distinct stable retrieval states have the same free energy Γα _α simultaneously. Definitions 4.2 and 4.1 Spinodal It occurs when the energy barrier of a local mimimum disappers. This drives a retrieval state to the other. Definitions 4.3 and 4.2 Exponential Memory Load κ=log(M)/dκ= (M)/d It quantifies the growth rate of the number of memories. This affects retrieval interference and reliability. Section 5.2 2 Preliminaries We first discuss convex analysis and geometric properties in statistical mechanics. This induces entropy and sparsity via the free energy framework. The purpose of this section is to identify the conjugate variables that characterize the retrieval dynamics and attention mechanism. 2.1 Statistical Mechanics and Energy Formalism We introduce partition function characterizing a system with all possible configurations, and adopt it to discuss free energy through the statistical mechanics (Zinn-Justin, 2002; Srednicki, 2007). Definition 2.1 (Partition Function). Let ℋ:→ℝH:X be a given function called an energy function. We define the partition function as the Gibbs average over all configurations. Z[J]≔∫exp−βℋ[x]+∑i=1dJixidμ(x), Z[J] _X \- [x]+ _i=1^dJ_ix_i\ \ μ(x), where β≔1/Tβ 1/T is the inverse temperature, J∈ℝdJ ^d is the generating source, and dμ(x) μ(x) denotes the Lebesgue measure in X. The partition function is positive. For simplicity, we consider =ℝdX=R^d. Remark 2.1. The generating source term represents the external magnetic field in a spin system. Furthermore, if the model is discrete, then we replace ∫dμ(x) _X \ μ(x) by ∑x∈ _x with =1,…,KX=\1,...,K\. Definition 2.2 (Stationary Equations and Stationary Points). The stationary equations are ∇xβℋ[x]−∑i=1dJixi|x=xJ=0,∇x2βℋ[x]−∑i=1dJixi|x=xJ≻0, _x\ [x]- _i=1^dJ_ix_i\|_x=x_J=0 \ , \ ∇^2_x\ [x]- _i=1^dJ_ix_i\|_x=x_J 0 \ , (2.1) where ≻ means positive definite. We say that the solution xJx_J is a stationary or (local) stable point. We obtain stable points in Definition 2.2 through the energy function ℋ[x]H[x]. However, it contains microscopic noise components. Hence, xJx_J is not a well-behaved stable point. To extract the aggregated stability from the original energy function ℋ[x]H[x], we should introduce the free energy (effective action) Γ[φ] [ ]. It later helps characterize the macroscopic averaged energy of the system. Definition 2.3 (Log-Partition Function and Free Energy). We have two different free energies that describe the partition function. To distinguish them, we define them carefully. • Log-Partition Function: It is defined as W[J]≔logZ[J]=log(∫exp−βℋ[x]+∑i=1dJixidμ(x)). W[J] Z[J]= ( _X \- [x]+ _i=1^dJ_ix_i\~ μ(x)). (2.2) • Free Energy (Effective Action): It is the Fenchel conjugate of the log-partition function. Γ[φ]≔supJ∈ℝd∑i=1dJiφi−W[J],with φ∈ℝd. [ ] _J ^d\ _i=1^dJ_i _i-W[J]\, \ ^d. (2.3) Remark 2.2. From Section C.1, W[J]W[J] and Γ[φ] [ ] are both convex, W[J]W[J] is smooth, ∇JW[J] _JW[J] is single-valued, and Γ[φ] [ ] is closed. However, we do not assume strict convexity. Consequently, the gradient maps ∇JW[J] _JW[J] and ∇φΓ[φ] _ [ ] are not guaranteed to be bijective according to Fact 1. In this way, distinct microscopic configurations are able to represent the same macroscopic state. The optimization problem becomes minimizing the free energy Γ[φ] [ ] with respect to φ . Next, we describe the physical interpretation/derivation of the Fenchel-Young duality. Proposition 2.1 (Fenchel-Young Duality Relation (Srednicki, 2007; Blondel et al., 2020)). Let pJp_J be the Gibbs measure with external source J. From the standard statistical field theory, the induced macroscopic field φ=pJ[x] =E_p_J[x] reveals the following source-field relation: φ=x∼pJ[x]∈∂W[J]⇔J∈∂Γ[φ], =E_x p_J[x]∈∂ W[J] \ \ J∈∂ [ ] \ , where ∂ is the subdifferential from Definition C.5. If W and Γ are differentiable, then ∂ reduce to gradients ∇. We call them equations of motion (EOMs) in the physics literature. It also refers to the Fenchel-Young duality relation. For φ and J, we call them the classical solutions or saddle points. Proof. Please see Section C.2 for a detailed proof. ∎ From Proposition 2.1, the Fenchel conjugate variable φ is macroscopic x∼pJ[x]E_x p_J[x] exactly. φ eliminates fluctuations and induces deterministic landscape Γ[φ] [ ]. Please see Figure 1 for illustration. Microscopic Configurationβℋ[x]−Jx [x]-JxxxxJx_JfluctuationspJ(x)∝e−βℋ[x]+Jxp_J(x) e^- [x]+JxGibbs AverageW[J]W[J]JJslope =φ= φ=x∼pJ[x]=∇JW[J] =E_x p_J[x]= _JW[J]Fenchel ConjugateΓ[φ] [ ]φ φ⋆ J=∇φΓ[φ]J= _ [ ] Figure 1: Illustration of the procedure from the microscopic to macroscopic energy landscapes. Left. “(Microscopic) Fluctuations” means x is random, has nonzero variance, and follows Gibbs distribution pJ(x)p_J(x). Built from microscopic x, the local stable point xJx_J exhibits randomness. Middle. W[J]=log(∫exp−βℋ[x]+Jx)W[J]= ( _X \- [x]+Jx\) encodes macroscopic responses, i.e., φ=x∼pJ[x]=∇JW[J] =E_x p_J[x]= _JW[J]. Hence, the slope of W[J]W[J] gives a deterministic average. Right. Γ[φ]=supJJφ−W[J] [ ]= _J\J -W[J]\ is the Fenchel conjugate of W[J]W[J]. Its response is J=∇φΓ[φ]J= _ [ ], so local stable macroscopic points occurs at J=0J=0. Γ[φ] [ ] describes the energy landscape built from macroscopic φ after integrating out microscopic fluctuations. 2.2 Connection between Free Energy and Tsallis Entropy In this section, we connect our free energy to Tsallis entropy. The value of α determines the sparsity structure of the retrieval dynamics. As we shall see in Section 3, this choice covers the general attention mechanisms, ranging from dense (α=1α=1) to the sparse case (∞>α>1∞>α>1). For preliminaries on Tsallis entropy, α-deformed exponential and logarithm family, please see Section C.3 for completeness. They generalize free energy to the α-deformed version. Theorem 2.1 (Tsallis Entropy and Deformed Free Energy). If we replace the exponential functions and logarithm functions by α-deformed functions, then we have the deformed log-partition function −Wα[J]=βx∼pJ(α)[ℋ[x]]−Ψα[pJ]Zα−1[J]∫pJα(x)dμ(x)−∑i=1dJix∼pJ(α)[xi], -W_α[J]= ^(α)_x p_J[H[x]]- _α[p_J]Z_α^α-1[J] _Xp_J^α(x) \ μ(x)- _i=1^dJ_iE^(α)_x p_J[x_i], where x∼pJ(α)[⋅]≔∫pJα(x)[⋅]dμ(x)/∫pJα(x)dμ(x)E^(α)_x p_J[·] _Xp^α_J(x)[·] μ(x)/ _Xp^α_J(x) μ(x) is the escort expectation value, and Γα=βx∼pJ⋆(α)[ℋ[x]]−Ψα[pJ⋆]Zα−1[J⋆]∫pJ⋆α(x)dμ(x),with φ=x∼pJ⋆(α)[x], _α= ^(α)_x p_J [H[x]]- _α[p_J ]Z_α^α-1[J ] _Xp_J ^α(x) \ μ(x), \ =E^(α)_x p_J [x], (2.4) where J⋆=argmaxJ∑i=1dJiφi−Wα[J]J = _J\ _i=1^dJ_i _i-W_α[J]\. We assume X is finite-volume to avoid divergence. Proof. Please see Section C.4 for a detailed proof. ∎ Remark 2.3. If α=1α=1, an escort expectation value becomes an ordinary expectation value. Hence, if we further consider the zero-temperature limit β→∞β→∞, then minimizing the free energy Γα _α is equivalent to minimizing the energy function ℋH. In the high-temperature limit, minimizing Γα _α is equivalent to maximizing the Tsallis entropy Ψα _α. The latter case indicates a disorder regime. Remark 2.4. Theorem 2.1 relates to the evidence lower bound (ELBO) in variational autoencoders (VAE), (Kingma and Welling, 2013). Specifically, with a posterior qϕ(z|x)q_φ(z|x), we have logpθ(x)≥z∼qϕ(z|x)[logpθ(x,z)]−z∼qϕ(z|x)[logqϕ(z|x)]. p_θ(x) _z q_φ(z|x)[ p_θ(x,z)]-E_z q_φ(z|x)[ q_φ(z|x)]. We recognize that the second term on the right hand side is Shannon entropy, (C.3). Then, ELBO≔z∼qϕ(z|x)[logpθ(x,z)]+S[qϕ(⋅|x)]. _z q_φ(z|x)[ p_θ(x,z)]+S[q_φ(·|x)]. It is an energy-entropy decomposition, where the expected log joint probability corresponds to a negative energy. Hence, maximizing ELBO is equivalent to minimizing a free energy (2.4). 3 Statistical Physics of Attention: Modern Hopfield Models In this section, we define an energy function of MHMs in (Hu et al., 2023; Wu et al., 2023). Then we express the free energy in terms of entropy and derive the emergent attention mechanism. 3.1 Modern Hopfield Model (MHM) Inspired by the optimization objectives in (Ramsauer et al., 2020; Krotov and Hopfield, 2020; Hu et al., 2023; Wu et al., 2023), we introduce the explicit joint energy function of MHM as follows. Definition 3.1 (Modern Hopfield Model (MHM)). The joint energy function of the MHM is ℋ~MHM[x,p,ξ]≔12∑i=1dxi2−∑μ=1Mzμpμ, H_MHM[x,p,ξ] 12 _i=1^dx_i^2- _μ=1^Mz_μp_μ, (3.1) where p∈ΔMp∈ ^M lives on a probability simplex, i.e., p is non-negative and satisfies ∑μ=1Mpμ=1 _μ=1^Mp_μ=1, and zμ≔∑i=1dξiμxi/dz_μ _i=1^d _i^μx_i/ d. Moreover, we denote ℋMHM[x,ξ]≔infp∈ΔMℋ~MHM[x,p,ξ]H_MHM[x,ξ] _p∈ ^M H_MHM[x,p,ξ]. In our setting, query states, the probability distribution, and the stored patterns are all fixed, i.e., the distributions of x and p collapse to their escort average. Specifically, a Taylor expansion of f(x)f(x) gives (x,p)∼ρJ(α)[f(x)]=f(φ)+O(Var(α)(x))E^(α)_(x,p) _J[f(x)]=f( )+O(Var^(α)(x)), where ρJ(x,p) _J(x,p) is the joint probability distribution, φ=(x,p)∼ρJ(α)[x] =E^(α)_(x,p) _J[x] is the expectation value, and Var(α)Var^(α) is the variance. Then the fixed-parameter setting corresponds to the zero variance. Hence, we are able to evaluate all quantities f(x)f(x) at φ under this fixed-parameter setting exactly. Thus, we identify φ with x, and similarly for p and z. 22 2 Here we restrict attention to the deterministic regime. This allows us to isolate the physics most directly relevant to the attention mechanism. In Appendix D, we consider the more general setting in which both x and p are random variables with nonzero variance. This leads to a richer and more complete statistical-mechanics structure, and allows us to analyze fluctuations and response properties. Definition 3.2 (Statistics-MHM Mapping). The mapping is: • Query State Vector. x∈ℝdx ^d: It represents continuous query states. Due to our fixed-parameter setting, it is macroscopic and parametrizes the free energy Γα[x]=Γα[φ] _α[x]= _α[ ]. • External Source. J∈ℝdJ ^d: It represents an external bias of the query states. In particular, J=0J=0 identifies local minima of Γα[x] _α[x] due to the Fenchel-Young relation (Proposition 2.1). • Probability Function. p∈ΔMp∈ ^M: It represents the retrieval distribution over memories. • Entropy. Ψα _α: It acts as a regularizer controlling how sparse the memory weights p are. • Score. z∈ℝMz ^M: It represents query-key similarity between the current x and each memory pattern ξμξ^μ via the relation: zμ≔∑i=1dξiμxi/dz_μ _i=1^d _i^μx_i/ d. Higher zμz_μ suggests higher similarity. In fact, the query state vector x in Definition 3.2 relates to the probability weight p as follows. Proposition 3.1 (Query-Probability Relation). A query state relates to a probability function via xi=1d∑μ=1Mξiμpμ+1βJi. x_i= 1 d _μ=1^M _i^μp_μ+ 1βJ_i. (3.2) Proof. Please see Section C.5 for a detailed proof. ∎ In Section 2.2, we relate free energy to Tsallis entropy via (2.4). Inspired by that relation, we are capable of expressing the free energy in terms of the variables of MHMs as follows. Theorem 3.1 (Free Energy of MHM). The joint free energy of the MHM is Γ~α[x,p,ξ]=β2∑i=1dxi2−β∑μ=1Mzμpμ. _α[x,p,ξ]= β2 _i=1^dx_i^2-β _μ=1^Mz_μp_μ. (3.3) Hence, after taking the infimum with respect to p, we obtain the free energy Γα[x,ξ]=β2∑i=1dxi2−βmaxμ∈[1,M]zμ. _α[x,ξ]= β2 _i=1^dx_i^2-β _μ∈[1,M] \ z_μ. (3.4) Proof. Please see Section C.6 for a detailed proof. ∎ This free energy selects the maximal component of the overlap maxμzμ _μz_μ. The hard selection mechanism arises from zero entropy due to the collapse of p. This captures a supremum over simplex vertices rather than the full simplex ΔM ^M. The relaxation from a hard selection to a distribution over multiple memories becomes a natural extension. Motivated by the sparsity of the entropy regularizer in (Hu et al., 2023), we consider the relaxation through Fenchel conjugate as follows. We introduce the entropy regularizer in (3.4). This yields a generalized (relaxed) model: Γα(mod)[x,ξ]=β2∑i=1dxi2−Ωα⋆[βz],Ωα≔−Ψα, ^(mod)_α[x,ξ]= β2 _i=1^dx_i^2- _α [β z], _α - _α, (3.5) where Ωα⋆[βz]=supp∈ΔM∑μ=1Mβzμpμ−Ωα[p] _α [β z]= _p∈ ^M\ _μ=1^Mβ z_μp_μ- _α[p]\ is the Fenchel conjugate of the negative Tsallis entropy (C.5). It recovers the energy form in (Hu et al., 2023; Wu et al., 2023) exactly. Remark 3.1. This generalization allows us to relax p to have a distribution over multiple memories on the full simplex ΔM ^M. This leads to several theoretical and practical advantages: 1. Equilibrium Selection vs. Averaging. The MHM performs a hard maximization, selecting a single dominant component of Γα(mod)[x,ξ] _α^(mod)[x,ξ]. In contrast, expectation-based rules compute averages over configurations. The generalization (3.5) interpolates between these regimes, i.e., it reduces to the original MHM in β→∞β→∞ limit. Please see Appendix D for details. 2. Robustness to Noise. As we will see in Proposition 3.3, Γα(mod) _α^(mod) emphasizes dominant components of z for α>1α>1. Hence, it suppresses contributions from noisy patterns. In contrast, expectation-based rules average all components, making them more sensitive to noise. 3. Sparsity. The entropy function Ψα _α controls the support of sparsity. In particular, for α>1α>1, the maximizer p⋆p is sparse, so some of the components are zero. This is also a consequence of robustness to noise, which does not appear in expectation-based formulations. 4. Computational Accessibility. Fenchel conjugation leads to an explicit variational object and is more accessible with gradient-based methods. In contrast, expectation-based formulations require sampling or approximation of Gibbs averages, which is more demanding. 3.2 Retrieval Dynamics and Attention Mechanism In Section 2, we introduce EOMs from Proposition 2.1. These equations describe the trajectory that the flow should follow. In this section, we investigate the physical meaning of the trajectories in associative memory systems. In particular, the attention emerges as an EOM naturally. Proposition 3.2 (EOMs of Free Energies in MHMs). Let ZαZ_α be the partition function with α-deformed Gibbs average. Then the EOM in the J-space corresponds to the Hopfield retrieval ∇JWα[J,ξ]=x,with Wα[J,ξ]≔logαZα[J,ξ]. _JW_α[J,ξ]=x, \ W_α[J,ξ] _αZ_α[J,ξ]. (3.6) The EOM of the joint log-partition function in the z-space corresponds to ∇zW~[z,p,J]=βp,with W~[z,p,J]≔−βℋ~MHM[x,p,ξ]+∑i=1dJixi. _z W[z,p,J]=β p, \ W[z,p,J] -β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i. (3.7) The EOMs of the joint free energy and the generalized one are ∇zΓ~α[x,p,ξ]=∇zΓ~α(mod)[x,p,ξ]=−βp. _z _α[x,p,ξ]= _z _α^(mod)[x,p,ξ]=-β p \ . (3.8) Proof. Under fixed x and p, the log-partition function is Wα[J,ξ]=−β/2∑i=1dxi2+β∑μ=1Mzμpμ+∑i=1dJixiW_α[J,ξ]=-β/2 _i=1^dx_i^2+β _μ=1^Mz_μp_μ+ _i=1^dJ_ix_i following from the proof of Theorem 3.1. After taking the derivative of Wα[J,ξ]W_α[J,ξ] with respect to J, we obtain the first equation. For the second and the last ones, we derive them by taking the derivative of (3.1), (3.4), and (3.5) with respect to zμz_μ directly. ∎ We observe the emergent attention mechanism from the EOMs in Proposition 3.2. Specifically, Proposition 3.3 (Emergent Attention Mechanism as EOMs in MHMs). Attention emerges as: • If α→∞α→∞, it corresponds to a hardmax function: pμ=hardmax(zμ)=1,for maximal zμ0,otherwise. p_μ=hardmax(z_μ)= cases1 \ , \ &for maximal z_μ\\ 0 \ , \ &otherwise cases. • If α=2α=2, it corresponds to a sparsemax function (Martins and Astudillo, 2016): pμ=Sparsemaxβ(zμ)=β2[zμ−θ]+,with β2∑μ=1M[zμ−θ]+=1. p_μ=Sparsemax_β(z_μ)= β2[z_μ-θ]_+, \ β2 _μ=1^M[z_μ-θ]_+=1. • If 1<α<21<α<2, it corresponds to an entmax family (Peters et al., 2019). • If α→1α→ 1, it corresponds to a softmax function (Blondel et al., 2020): pμ=Softmaxβ(zμ)=expβzμ∑ν=1Mexpβzν. p_μ= Softmax_β(z_μ)= \β z_μ\ _ν=1^M \β z_ν\. Proof. Please see Section C.7 for a detailed proof. ∎ Proposition 3.3 provides a re-derivation via EOMs of a free energy landscape, revealing the underlying physical interpretation of the attention mechanism. In Transformer language, (Vaswani et al., 2017), we denote Attn(Q,K,V)≔∑μ=1Mpμ(Q,K)VμAttn(Q,K,V) _μ=1^Mp_μ(Q,K)V_μ as the representation of x, where Q represents the microscopic states, and K and V represent the stored memory patterns. Q,K,VQ,K,V are all fixed inputs. This corresponds to our zero variance or deterministic setting. 4 Phase Transitions and Critical Points of Retrieval Success In this section, we discuss the spectral stability and thresholds for retrieval success. For the dynamical stability, please see Appendix E. β,M,dβ,M,d and κ≔exp(M)/dκ (M)/d determine the free energy and allow multiple local minima in some ranges. This leads to our study of the phase structure. In our setting, the stored memories ξμ\ξ^μ\ remain fixed, while the query state x evolves dynamically. Definition 4.1 (Branch). Choose an initial parameter θ0 _0 that determines Γα(mod)[x(θ0),ξ] _α^(mod)[x( _0),ξ]. As θ∈ℐθ0θ _ _0 varies from θ0 _0, we define a branch as a continuous family of the evolution from a specific local stable point xa⋆(θ,θ0)θ∈ℐθ0\x _a(θ; _0)\_θ _ _0 satisfying the stationary condition (Definition 2.2)ao68 ∇xΓα(mod)[xa⋆(θ,θ0),ξ]=0, and ∇x2Γα(mod)[xa⋆(θ,θ0),ξ]≻0. _x ^(mod)_α[x_a (θ; _0),ξ]=0 \ , \ and \ ∇^2_x ^(mod)_α[x _a(θ; _0),ξ] 0. Branches help us analyze the evolution of stable points and their loss of stability. Adopting this setup, we study the phase transitions of the generalized model (3.5) following stability analysis in statistical mechanics (Binder, 1987; Zinn-Justin, 2002). We start from defining two critical points. Definition 4.2 (Coexistence Point). Given two distinct branches, a,ba,b, with their stable points, xa⋆(θ,θ0)\x_a (θ; _0)\, xb⋆(θ,θ0)\x_b (θ; _0)\. A coexistence point is a parameter value θcoexist _coexist such that Γα(mod)[xa⋆(θcoexist,θ0),ξ]=Γα(mod)[xb⋆(θcoexist,θ0),ξ]. _α^(mod)[x _a( _coexist; _0),ξ]= _α^(mod)[x_b ( _coexist; _0),ξ]. (4.1) However, the coexistence condition is not a necessary condition for a real jump of local minima. Even if the equality holds, there is still an energy barrier that forbids the transition. Definition 4.3 (Spinodal Point). Given a smooth branch a, a spinodal point occurs when the branch loses its local stability. That is, the point where the stationary condition fails: λmin(∇x2Γα(mod)[xa⋆(θjump,θ0),ξ])=0. _min(∇^2_x _α^(mod)[x_a ( _jump; _0),ξ])=0. (4.2) This is also the point where the continuation along that branch breaks down. Remark 4.1. If a branch is non-smooth, e.g., there is a sudden change in the effective support size, then jumps also occur at that discontinuous point. In this case, (4.2) does not hold necessarily. We control the system by the parameters, θ=(β,M,d,ξ)θ=(β,M,d,ξ). For α=1α=1, the retrieval map is the softmax map (Proposition 3.3), so pμ⋆p_μ is positive for all memories and the wights vary with z⋆z smoothly. Hence, discontinuous change of retrieval only occurs if the Hessian vanishes, i.e., (4.2) holds. To investigate the transitions, we focus on the critical value of β in the following. Proposition 4.1 (Temperature Coexistence Point for Sparse Model). Let a,ba,b be two distinct branches and α=2α=2. Denote kak_a and kbk_b as the exact support sizes (see Lemma F.1) of branches a and b, respectively. Based on the statistical mechanics framework, the coexistence point satisfies ∑μ,ν∈a[zμ⋆−θ]+[zν⋆−θ]+Kμν(βcoexist,β0)=∑μ,ν∈b[zμ⋆−θ]+[zν⋆−θ]+Kμν(βcoexist,β0), _μ,ν∈ a[z _μ-θ]_+[z _ν-θ]_+K_μν( _coexist; _0)= _μ,ν∈ b[z _μ-θ]_+[z _ν-θ]_+K_μν( _coexist; _0), where Kμν(β)≔μ,ν−β2GμνK_μν(β) 1_\μ,ν\- β2G_μν, and Gμν≔∑i=1dξiμξiν/dG_μν _i=1^d _i^μ _i^ν/d is the Gram matrix. βcoexist _coexist does not admit a closed-form expression. However, if the distribution is uniform on the simplex and the system satisfies the well-separation condition, i.e., Gμν=1,μ=νϵ≪1,μ≠ν, G_μν= cases1 \ &, \ μ=ν\\ ε 1 \ &, \ μ≠ν cases, (4.3) then for ka=1k_a=1 and kb=2k_b=2, we have the analytic solution βcoexist=2/(1−ϵ) _coexist=2/(1-ε). At this point, β=βcoexistβ= _coexist, p⋆p is degenerate and every p⋆p is a global minimum of the joint free energy Γ~α=2(mod) _α=2^(mod). Proof. Please see Section C.8 for a detailed proof. ∎ Proposition 4.2 (Temperature Spinodal Point for Sparse Model). Consider the sparse case α=2α=2 and a branch a with exact support size k. Then the spinodal critical value is βjump=2λmax(PGP), _jump= 2 _max(PGP), where Pμν≔μ,ν−k−1P_μν 1_\μ,ν\-k^-1, and G is the Gram matrix. If the distribution is uniform and the system satisfies the well-separation condition, (4.3), then we have the analytic solution βjump=2/(1−ϵ) _jump=2/(1-ε). Proof. Please see Section C.9 for a detailed proof. ∎ Remark 4.2. Under the uniform distribution condition and the well-separation condition, we conclude that βcoexist=βjump=2/(1−ϵ) _coexist= _jump=2/(1-ε). However, we do not have these assumptions in general. Then the real jump and the coexistence point do not occur simultaneously, i.e., βjump≠βcoexist _jump≠ _coexist. 5 Numerical Results We provide numerical evidence supporting the ideas in Section 4. Specifically, we display the phase structure of MHMs under temperature and memory-load scans, including both α=1,2α=1,2. 5.1 Temperature Dependence We examine the temperature dependence of the phase structure in MHMs, including both the sparse (α=2α=2) and dense (α=1α=1) regimes. Specifically, we characterize the temperature effect on the attention map (Proposition 3.3) and the branches of free energy (3.5). For the experimental setup, please see Appendix F. We first start from the temperature analysis in the sparse case. Figure 2: Temperature analysis of the retrieval structure of MHM in the sparse regime (α=2α=2). (M=100,d=64)(M=100,d=64). Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆z_max under up- and down-sweeps of β exhibits hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution p), the inverse participation ratio (IPR), and the fixed score-threshold count (Section F.2). (c) Top branches. We track the top five stable branches ranked by zmax⋆z_max . (d) Branch energies. We show the corresponding free energies of those top five branches. Sparse case (α=2α=2). Here, we select four plots to represent the retrieval structure in Figure 2. (a) The up- and down-sweep branches do not coincide, which shows a hysteresis window in β. This indicates that the coexistence of multiple local stable retrieval states (Definitions 4.2 and 4.1). The selected branch depends on initialization, analogous to first-order transitions in freezing-melting hysteresis of water (Enninful et al., 2019). As β increases, the maximal overlap zmax⋆z_max grows stepwise and the shaded regions collapse at large β. This indicates a seed-independent saturation. Specifically, it saturates at zmax⋆≈0.0156z_max ≈ 0.0156. Indeed, this is the consequence of one pattern retrieval, i.e., zμ⋆=1d∑i=1dξiμxi⋆=1d×d×1d×1d=1d=164≈0.0156. z _μ= 1 d _i=1^d _i^μx _i= 1 d× d× 1 d× 1d= 1d= 164≈ 0.0156. (b) As β increases, the exact support size and IPR decreases (Section F.2). This indicates a transition from distributed to sparse retrieval. The discrepancy of |supp(p⋆)||supp(p )| and IPR arises since IPR counts the components non-uniformly. However, KΔK_ is zero in the high-temperature regime, where noise dominates and no overlap exceeds the threshold Δ=0.01 =0.01. This suggests a disordered regime. In the low-temperature regime, all support sizes approach to one, so the system reaches single-pattern retrieval with large overlap. The collapse of the shaded regions again indicates a perfect retrieval and seed-independence. (c) Initialization at β0=290 _0=290. Multi-branch continuation indicates several competing stable retrieval branches in the intermediate-β regime. The presence of these branches indicates multiple coexisting local minima. Their crossings and discontinuous jumps in zmax⋆z_max show that the sparse model admits branch switching between distinct local minima (Propositions 4.2 and 4.1). Hence, the selected state correspond to either stronger or weaker alignment depending on the branch we are tracking. (d) Initialization at β0=290 _0=290. The branch exhibits coexist point from the plot (Definitions 4.2 and 4.1). As β varies, the global minimizer switches between branches. However, under continuation (warm-start dynamics), the system follows a given local branch rather than the instantaneous global minimizer. Hence, the transition between branches occur before reaching the exact crossing point. This leads to discontinuous changes along a branch. Hence, we interpret this as a transition within the retrieval phase rather than a transition between retrieval and a disordered phase. This simulation provides a more precise phase structure than earlier analyses, (Amit et al., 1985; Amit et al., 1987; Nishimori, 2001). Figure 3: Temperature analysis of the retrieval structure of MHM in the dense regime (α=1α=1). (M=100,d=64)(M=100,d=64). Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆z_max under up- and down-sweeps of β shows no hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution p) and the inverse participation ratio (IPR) (Section F.2). (c) Top branches. We observe one single stable branch across the entire range of β. (d) Branch energies. We show the corresponding free energy of the branch. Dense case (α=1α=1). Here, we select four plots to represent the retrieval structure in Figure 3. (a) The up- and down-sweep branches coincide, which indicates the absence of hysteresis. Hence, the retrieval dynamics admits one stable solution for each β. There is no metastable branch or discontinuous transition. The shaded region becomes broader as β increases. Hence, the alignment with stored pattern increases, but the variance increases as well. (b) The support size remains full for all β (Section F.2), since pμ⋆∝exp(βzμ⋆)p_μ (β z_μ ) assigns positive weight to every component μ. As β increases, KeffIPRK_eff^IPR decreases, so the distribution becomes concentrated. However, this does not induce sparsity. The shaded region of KeffIPRK_eff^IPR becomes broader as β increases. This is due to the amplification of small differences from seeds. (c) Initialization at β0=140 _0=140. The maximal overlap zmax⋆z_max grows with β. This indicates a continuous sharpening of alignment with the target pattern. There is no discontinuity or spinodal instability due to the vanishing Hessian. Hence, it shows the absence of phase-transition-like behavior, based on our softmax experiment setup here. However, it does not indicate a unique local minimum necessarily. That is, other local minima may locate far away from our selected initial point so that the nearby search only displays one branch. (d) Initialization at β0=140 _0=140. The energy decreases along the branch, and the retrieval state becomes more concentrated. This reflects a continuous sharpening of the energy basin, rather than a restructuring of competing minima as in the sparse case. Remark 5.1. Koulischer et al. (2023) analyze temperature-dependent behavior in MHM under softmax attention (α=1α=1). They show that as β increases, the retrieval weights concentrate smoothly. Our results here are consistent with their findings. Besides, we further provide a characterization of this behavior by tracking the maximal overlap zmax⋆z_max and the support sizes. 5.2 Memory-Load Dependence Here we study the exponential memory load M=exp(κd)M= (κ d), where M is the pattern number and d is the dimension of each pattern. Specifically, we characterize the memory-load effect on the attention map (Proposition 3.3) and the branches of free energy (3.5). For the experimental setup, please see Appendix F. We start from the memory-load analysis of the sparse case. Figure 4: Memory-load analysis of the retrieval structure of MHM in the sparse regime (α=2α=2). (d=48,β=180d=48,β=180). Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆z_max under up- and down-sweeps of κ shows hysteresis. (b) Support size. We compare the exact support size |supp(p⋆)||supp(p )| (active components of the probability distribution p) and the inverse participation ratio (IPR) (Section F.2). (c) Top branches. We track the top five stable branches ranked by zmax⋆z _max. (d) Branch energies. We plot the corresponding free energy of the five branches. Sparse case (α=2α=2). Here, we select four plots to represent the retrieval structure in Figure 4. (a) The up- and down-sweep branches do not coincide, which shows a hysteresis window in κ. This indicates the coexistence of multiple local stable retrieval states and the presence of first-order phase transitions. As κ increases, the maximal overlap zmax⋆z_max decreases stepwise. This suggests the delocalization when the stored patterns are too many. (b) The system remains in a sparse retrieval regime: although additional overlaps enter the active set as κ increases, IPR remains close to one (Section F.2). This indicates that a single dominant memory continues to control the retrieval. Besides, the variance of fixed scored-threshold support size (Section F.2) grows with κ due to the delocalization. (c) Initialization at κ0=0.12 _0=0.12. Multiple branches coexist and represent distinct local stable retrieval states in this κ-window. The selected state depends on initialization and the continuation path. This reflects metastability in the energy landscape, i.e., multiple stable points coexist and the dynamics remains trapped in a given basin until stability is lost. (d) Initialization at κ0=0.12 _0=0.12. We plot the corresponding free energies of the five branches. As κ grows, all the branch energies decrease. However, it does not indicate that the retrieval is better. Instead, panel (a) suggests the delocalization (low score) at high κ. Figure 5: Memory-load analysis of the retrieval structure of MHM in the dense regime (α=1α=1). (M=exp(κd),d=48,β=180M= (κ d),d=48,β=180). Results are averaged over random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆z_max under up- and down-sweeps of κ shows no hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution p) and the inverse participation ratio (IPR) (Section F.2). (c) Top branches. We track and observe only a single stable branch across the entire range of κ. (d) Branch energies. We plot the corresponding free energy of the branch. Dense case (α=1α=1). Here, we select four plots to represent the retrieval structure in Figure 5. (a) The up- and down-sweep branches coincide, which shows the absence of hysteresis in κ. That is, there is no coexistence window of multiple local stable retrieval states. As κ increases, the maximal overlap zmax⋆z_max decreases. This suggests the delocalization at large κ. (b) Both the exact support size and IPR (Section F.2) grow with κ. This reflects the dense nature of softmax, i.e., pμ⋆=expβzμ⋆∑ν=1Mexpβzν⋆>0,∀μ,with overlap zμ⋆. p _μ= \β z _μ\ _ν=1^M \β z _ν\>0, ∀μ, overlap \ z _μ. All patterns have strictly positive weight. Hence, the support size |supp||supp| remains full, and the delocalization drives IPR to grow exponentially with κ as well. (c) Initialization at κ0=0.08 _0=0.08. Only a single continuation branch appear. The energy decreases with κ monotonically. This indicates a smooth deformation of the solution rather than transitions between competing states. Again, this does not indicate the uniqueness of this local minimum. Other local minima may locate far away from our initial point. (d) Initialization at κ0=0.08 _0=0.08. We plot the corresponding free energies of the branch. As κ grows, the branch energy decreases monotonically. However, it does not indicate that the retrieval is better. Instead, panel (a) suggests the delocalization (low κ) at high κ. Remark 5.2. The curves show the non-smoothness as functions of κ. This is the consequence of the fact that we require M to be an integer in our simulation. 6 Discussion and Conclusion We present a free energy framework for Transformer attention via modern Hopfield models. This viewpoint identifies attention as the equation of motion on an energy landscape. Our work shows the empirical reliable retrieval regime with respect to entropy, temperature, and exponential memory load (i.e., α/β/κα/β/κ). We study the temperature and exponential memory load effects on phase structure. In the sparse regime, discontinuous transitions occur due to support change in memories (keys). The accompanying hysteresis indicates the coexistence of multiple stable states. In contrast, the softmax regime exhibits smooth evolution without hysteresis. Consequently, the softmax attention is a mixture of all patterns, while the query-key alignment grows with inverse-temperature smoothly. These results reveal the entropy-dependent phase behavior of modern Hopfield models and hence in Transformers. Practical Implications. We sketch some design principles from our results: • First, the attention should match the task: sparse one is support-efficient, while dense one is more accurate when many relevant values contribute (Propositions 3.3, G.2 and G.3). • Second, the retrieval depends on path under temperature changes (Propositions 4.1, 2 and 3). Specifically, increasing versus decreasing β lead to different different memory selection in the sparse case. The outcome also depends on the initial query (i.e., input context or prompt) supplied to the retrieval update during inference. • Third, increasing the inverse-temperature strengthens the maximal query-key similarity zmax=max∑i=1dμxiξiμ/dz_ = _μ _i=1^dx_iξ^μ_i/ d (Figures 2 and 3). In particular, sparse attention reaches single-memory retrieval saturation, while dense one sharpens smoothly with full support. • Finally, increasing memory load reduces retrieval accuracy (Figures 4 and 5). This further reveals a capacity-reliability trade-off in realistic Transformer architectures (Section G.2). Together, these provide concrete guidance for selecting attention, temperature ane memory-load. Limitations. Our analysis relies on simplified assumptions, such as prescribed correlation and noise structures, and the fixed memory setting. In contrast, realistic Transformer architectures allow interactions across heads and layers. This thereby leads to complicated noise. Furthermore, the keys and queries co-evolve through learned projection matrices and upstream layers. Hence, the theoretical analysis on phase transitions and retrieval reliability of full Transformers remain less explored. Extending our framework along these directions is crucial for future work. We defer more experiments to Appendices F and G due to page limit. Impact Statement By the theoretical nature of this work, we do not anticipate any negative social impact. AI Usage Disclosure We used large language models (LLMs) to aid, polish writing and debug codes, such as improving clarity, grammar, and conciseness. We also used LLMs for retrieval and discovery, for example, exhausting literature to identify potential missing related work. Finally, we used LLMs to brainstorm ideas for selected experiments in Appendix G, including suggestions for the model architecture, the number of attention heads, and the hidden dimensions (Vaswani et al., 2017; Xiong et al., 2020). All technical content, proofs, and results are original contributions by the authors. The authors take full responsibility for the paper’s content and conclusions. Acknowledgments JH is partially supported by Northwestern University’s Walter P. Murphy Fellowship and Terminal Year Fellowship (Paul K. Richter Memorial Award). Han Liu is partially supported by NIH R01LM1372201, NSF AST-2421845, Simons Foundation MPS-AI-00010513, AbbVie , Dolby and Chan Zuckerberg Biohub Chicago Spoke Award. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. Appendix Appendix A Related Work Classical and Modern Hopfield Models. Hopfield-type models originate from the physical interpretation of associative memory in spin systems, including the Ising model (Onsager, 1944), the Sherrington-Kirkpatrick model (Sherrington and Kirkpatrick, 1975), and the classical Hopfield model (Hopfield, 1982). Modern Hopfield models modify the energy function so that stored associative memories become local minima on the energy landscape, leading to a higher memory capacity (Krotov and Hopfield, 2020). Attention mechanisms (Vaswani et al., 2017) have been highly successful in large-scale architectures. Ramsauer et al. (2020) establishes the connection between softmax modern Hopfield retrieval and Transformer attention. Hu et al. (2023) further extends the softmax to sparse retrieval and sparse attention by applying convex conjugate. Peters et al. (2019) introduce the entmax family of attention maps, which interpolates between dense and sparse attention. Complementarily, Millidge et al. (2022) develop universal Hopfield networks. It formulates a retrieval dynamics through three operations: similarity, separation and projection. Santos et al. (2025) systematize this line of work into a Hopfield-Fenchel-Young framework, covering Fenchel-Young losses and the variational formulation of retrieval. Free Energy. Our formulation adopts the variational principles used in physics. Specifically, this method appears in quantum field theory (Zinn-Justin, 2002), large-N mean-field theory (Moshe and Zinn-Justin, 2003), and low-temperature theory (Landau and Lifshitz, 1980). This viewpoint describes the retrieval process as a variational problem over macroscopic parameters, rather than microscopic states. Related optimization-based interpretations of Transformer architectures are also explored in (Yang et al., 2022), where Transformer layers are viewed as unfolding steps of an energy minimization process. Unlike traditional analyses of Hopfield networks that focus on energy minimization alone, the free energy framework incorporates entropy, leading to a richer landscape with multiple metastable states. Phase Transition. Physicists often characterize phase transitions in energy-based systems through spontaneous symmetry breaking (Coleman and Weinberg, 1973) and the emergence of metastable states. In classical Hopfield networks, Amit et al. (1985); Amit et al. (1987) and Mezard et al. (1986) analyze this behavior using replica methods. This reveals transitions between the retrieval and spin-glass phases, which contain numerous spurious states that lead to retrieval failure. Free energy provides a description of phase transitions through the deformation of energy landscapes and the emergence of local minima (Binder, 1987). Recent work from Koulischer et al. (2023) discusses the phase transition in the softmax case of modern Hopfield models. They identify a critical inverse temperature at which the mean minima are close to individual stored patterns, rather than one common averaged minimum. However, their analysis does not include sparse retrieval map and the entmax family. Dense Associative Memory. Dense associative memory models extend classical Hopfield models by introducing higher-order non-linear energy functions, which improved memory load storage in energy-based models (Demircigil et al., 2017; Krotov, 2023). These interactions further reshape the energy landscape and enhance robustness to noise (Krotov and Hopfield, 2018). Fuzzy Memory. Modern Hopfield models admit distributed retrieval over multiple memories. Recent work studies how such retrieval behavior relates to memory capacity. Specifically, Wu et al. (2024) introduce a learnable feature map that separates stored patterns in a kernel space. This makes the distribution of memory minima more uniform. On the other hand, Hu et al. (2024b) consider spherical codes, i.e., stored memories in the kernel feature space as points on a hypersphere. This induces a tight capacity bound by analyzing how many feature vectors maintain angular separation on a hypersphere. These works specify regimes where multiple patterns are activated simultaneously. Such distributed states interpolate between sparse (single-pattern) and dense (multi-pattern) retrieval regimes. Despite these prior works, a unified statistical framework of modern Hopfield models remains limited. Existing analyses focus on either retrieval capacity or dynamical behavior, without a systematic treatment of entropy, sparsity, and metastability within a single framework. Furthermore, replica analyses characterize transitions between retrieval, spin-glass, and disordered phases, but not capture the detailed finite-dimensional geometry for modern Hopfield retrieval. In this work, we introduce a free energy formulation that unifies these aspects and study the induced finite-dimensional retrieval landscape. In particular, we focus on phase behavior inside the retrieval regime, rather than only transitions between retrieval and disordered phases. This allows us to compare dense and sparse regime, analyze metastable states, and verify retrieval success. In addition, we analyze how exponential memory load κ changes empirical retrieval behaviors. This probes memory-dependent phase regimes which are not addressed by existing Fenchel-Young or sparse Hopfield analyses. Appendix B A Minimal Introduction of Statistical Mechanics This appendix gives a physical route from interacting spins to associative memory and then to attention. We set Boltzmann’s constant to one, so the inverse temperature is β=1/Tβ=1/T. B.1 From Interacting Spins to Associative Memory A Hamiltonian assigns an energy to each microscopic configuration; a choice of dynamics specifies how the system moves through that landscape. Definition B.1 (Classical Hopfield Model (Hopfield, 1982)). Consider binary spins xi∈−1,+1x_i∈\-1,+1\ with couplings Jij=d−1∑μ=1MξiμξjμJ_ij=d^-1 _μ=1^M _i^μ _j^μ and Jii=0J_i=0. This gives the classical Hopfield model ℋHop(x)=−12d∑i≠jd∑μ=1Mξiμξjμxixj, _Hop(x)=- 12d _i≠ j^d _μ=1^M _i^μ _j^μx_ix_j, (B.1) where memories are denoted as ξ1,…,ξM∈−1,+1dξ^1,…,ξ^M∈\-1,+1\^d. Thus, zero-temperature Hopfield dynamics is a dissipative descent toward a local energy minimum. At nonzero temperature the system instead fluctuates among configurations, as described in Section B.2. Definition B.2 (Overlap). The overlap between the current state and memory μ is zμ(x)≔1d∑i=1dξiμxi. z_μ(x) 1 d _i=1^d _i^μx_i. (B.2) Therefore, low energy means large overlap with one or more stored patterns. Definition B.3 (Frustration (Kirkpatrick, 1977)). A system is frustrated if it does not satisfy Jijxixj>0J_ijx_ix_j>0 for every pair with Jij≠0J_ij≠ 0. A one-pattern Hopfield model is equivalent to an ordinary ferromagnet and retrieves without frustration. However, with several patterns, the pairwise alignments become incompatible. This creates frustration and additional metastable minima. Hence, stored patterns are attractors, but not every local minimum must be a stored memory. This is the elementary mechanism of associative retrieval shown in Figure 6. (a) Associative retrievalstored memoryξ1ξ^1noisy queryx(t=0)x(t=0)retrieved statex⋆≃ξ1x ξ^1ℋ↓H ↑z (b) Frustrated bondsJ12>0J_12>0J23>0J_23>0J31<0J_31<0++++++x1x_1x3x_3x2x_2× Figure 6: Retrieval and frustration in the classical Hopfield picture. (a) From a noisy query set, we retrieve a stored pattern by minimizing ℋH. Equivalently, the overlap increases during this associative retrieval. (b) Two ferromagnetic bonds and one antiferromagnetic bond on a triangle is impossible to satisfy pairwise energy minimization simultaneously. This is the simplest illustration of frustration. B.2 A Concrete Hopfield Free Energy In this section, we take one-pattern Hopfield model for a self-contained example of the statistical mechanics construction. That is, we choose ℋ1pat(x)=−12z2(x)H_1pat(x)=- 12z^2(x). At temperature T=1/βT=1/β, the Gibbs distribution and partition function (Definition 2.1) are pJ(x)=e−βℋ1pat(x)+J⊤xZ(J),Z(J)=∑x∈−1,+1de−βℋ1pat(x)+J⊤x. p_J(x)= e^- _1pat(x)+J xZ(J), Z(J)= _x∈\-1,+1\^de^- _1pat(x)+J x. (B.3) By grouping states via Gibbs distribution and Fenchel conjugate (Definitions C.6 and 2.3), Γ(z)=supJJ⊤φ−logZ(J), (z)= _J \J - Z(J) \, we obtain the exact form of one-pattern Hopfield free energy. Proposition B.1 (Free Energy of One-Pattern Hopfield Model (Friedli and Velenik, 2017)). The free energy of a one-pattern Hopfield model is Γ(z)=−12z2−1βlog(d+dz2). (z)=- 12z^2- 1β pmatrixd\\ d+ dz2 pmatrix. In the large-d limit, it approaches to an explicit expression Γ(z)=β(U−TS)→d→∞Γ∞(q)=−β2q2−s(q), (z)=β (U-TS ) \ d→∞ \ _∞(q)=- β2q^2-s(q), (B.4) where U=−12pJ[z2]U=- 12E_p_J[z^2] is the internal energy, S is the entropy (Definition C.8) in the statistical mechanics language, q≔limd→∞pJ[z]q _d→∞E_p_J[z] and s(q)≔−d+qd2log1+q/d2−d−qd2log1−q/d2. s(q) - d+q d2 1+q/ d2- d-q d2 1-q/ d2. (B.5) Proof. Please see (Friedli and Velenik, 2017) for the detailed proof. ∎ Proposition B.2 (Stationary Equation of One-Pattern Hopfield Model (Friedli and Velenik, 2017)). The stationary condition of the large-d one-pattern Hopfield free energy is q⋆=dtanh(βq⋆d). q = d ( β q d ). (B.6) This is an implicit equation. The solutions depend on inverse-temperature β. Proof. Taking the derivative , we have 0=Γ∞′(q⋆)=−βq⋆+d2log1+q⋆/d1−q⋆/d. 0= _∞ (q )=-β q + d2 1+q / d1-q / d. This hence gives the stationary points q⋆q and proves the proposition. ∎ At the origin, Γ∞′(q⋆=0)=−β+1 _∞ (q =0)=-β+1. Therefore, as illustrated in Figure 7: • for β<1β<1 (high temperature), q⋆=0q =0 is the unique minimum. However, q⋆=limd→∞pJ[z]=0q = _d→∞E_p_J[z]=0 indicates disorderness, i.e., zero overlap. • at βc=1 _c=1, the Hessian at the origin vanishes. • for β>1β>1 (low temperature), q⋆=0q =0 becomes unstable and two minima q±⋆≠0q _±≠ 0 appear. Since the local minima are symmetric, the retrieval is still unique and stable. Γ∞(q)−Γ∞(0) _∞(q)- _∞(0)q⋆=0q =0(a) β=0.7β=0.7: high TTqqΓ∞′(0)=0 _∞ (0)=0(b) βc=1 _c=1: criticalqqq−⋆q_- q+⋆q_+ (c) β=1.5β=1.5: low T Figure 7: One-pattern free-energy landscape in the large-d limit. The curves show the one-pattern Hopfield model free energy and its local minima. Cooling the system turns the minimum at q⋆=0q =0 into two retrieval minima. The deformation of the energy landscape moves the local minimum. It leads to a phase transition. B.3 Landscapes, Metastability, and Retrieval Jumps The one-pattern transition above is continuous: the nonzero minima emerge from q⋆=0q =0 continuously. More complicated retrieval landscapes contain several separated local minima. Specifically, two minima coexist when they have equal free energy. If one is higher, then the minimum is metastable since a barrier prevents movement to the lower state. A system in one basin remains there until that basin loses local stability. It moves only when the energy barrier vanishes, which is called a spinodal (Definition 4.3). We illustrate this mechanism in Figure 8. qΓ (a) Metastabilityβ ⋆q βjump _jumpλmin→0 _ → 0at the fold(b) Spinodal Figure 8: Phase-transition picture. (a) A metastable minimum is protected by a barrier. (b) We plot two smooth branches. In one smooth branch, the minimal Hessian eigenvalue vanishes at a spinodal. This leads to a jump onto another branch. Appendix C Definitions and Proofs in the Main Text C.1 Basic Tools for Convex Analysis Here we define several fundamental terminologies from standard convex analysis (Rockafellar, 1997). Definition C.1 (Proper). A function f is proper if the support domain of f satisfies dom(f)≔x∈ℝd:f(x)<∞≠∅, and f≠+∞. (f) \x ^d:f(x)<∞\≠ , and \ f≠+∞. Definition C.2 (Lower Semi-Continuous (lsc)). A function f is lower semi-continuous (lsc) if f(x)≤liminfτ→∞f(xτ), f(x)≤ _τ→∞f(x_τ), for every sequence (xi)τ→xi(x_i)_τ→ x_i. Definition C.3 (Closed Convex). A function f is closed convex if f is proper, lsc and convex. Definition C.4 (Essentially Smooth). A function f is essentially smooth if f is differentiable on int dom(f)int \ dom(f), and ‖∇f(x)‖→∞\|∇ f(x)\|→∞ whenever xi→ @depthΔ @set@skewchar @nested@a111xi∈∂(dom(f))x_i→ @depth 1 @set@skewchar @nested@a 111x_i∈∂(dom(f)). Definition C.5 (Subgradient and Subdifferential). Let f:ℝd→[−∞,+∞)f:R^d→[-∞,+∞) be a proper convex function. A vector g∈ℝdg ^d is called a subgradient of f at x∈dom(f)x (f) if f(y)≥f(x)+∑i=1dgi(yi−xi)∀y∈ℝd. f(y)≥ f(x)+ _i=1^dg_i(y_i-x_i) ∀ \ y ^d. The set of all subgradients of f at x is called the subdifferential of f at x, and is denoted by ∂f(x)=g∈ℝd:f(y)≥f(x)+∑i=1dgi(yi−xi)∀y∈ℝd. ∂ f(x)=\g ^d:f(y)≥ f(x)+ _i=1^dg_i(y_i-x_i) \ ∀ \ y ^d\. If f is differentiable at x, then the subdifferential reduces to ∂f(x)=∇f(x)∂ f(x)=\∇ f(x)\. Definition C.6 (Fenchel Conjugation (Rockafellar, 1997)). Consider a function f:ℝd→ℝf:R^d and an input x∈ℝdx ^d. Then we define the Fenchel conjugate of f(x)f(x) by the following variational formula, f⋆(k)≔supx∈ℝd∑i=1dkixi−f(x)∈ℝ,k∈ℝd. f (k) _x ^d\ _i=1^dk_ix_i-f(x)\ \ , k ^d. Fact 1 (Fenchel Conjugation Properties (Rockafellar, 1997)). Definition C.6 has several properties: 1. Convexity. A Fenchel conjugate f⋆(x)f (x) is by definition a convex function. 2. Involutivity. If a function f is closed convex, then applying Fenchel conjugation twice yields the identity map, i.e., f⋆⋆=f =f for all x∈ℝdx ^d. This is the well-known Fenchel-Moreau Theorem. 3. Bijectiveness. If a function f is proper, lsc, strictly convex and essentially smooth, then ∇f⋆:int dom(f⋆)↔int dom(f)∇ f :int \ dom(f ) \ dom(f) is a bijection, where int and dom denote interior and domain. Definition C.7 (Legendre Transformation). A function f is called a Legendre function if it is proper, lsc, strictly convex and essentially smooth. In other words, Legendre functions satisfy involutivity and bijectiveness. In this case, we refer to the Fenchel conjugate f⋆f as the Legendre transformation. Remark C.1. A Legendre function f satisfies the Fenchel-Moreau involutivity f⋆⋆=f =f, and its gradient map ∇f∇ f is a bijection between the interior domains from Fact 1 automatically. Remark C.2. Here we adopt the definition of the Legendre transformation from (Rockafellar, 1997). However, in the physics and machine learning literature, for example, in (Krotov and Hopfield, 2020) and (Ramsauer et al., 2020), people denote the same conjugation without verifying the conditions. Specifically, in many cases, the domain is ℝdR^d, so ∂ℝd=∅ ^d= leads to the essential smoothness condition being vacuous. The real issue is the non-bijectiveness of gradient maps. We should understand the transformations in these works as the Fenchel conjugations rather than the true Legendre transforms. C.2 Proof of Proposition 2.1 Proposition C.1 (Proposition 2.1 Restated: Fenchel-Young Duality Relation (Srednicki, 2007; Blondel et al., 2020)). Let pJp_J be the Gibbs measure with external source J. From the standard statistical field theory, the induced macroscopic field φ=pJ[x] =E_p_J[x] reveals the following source-field relation: φ=x∼pJ[x]∈∂W[J]⇔J∈∂Γ[φ], =E_x p_J[x]∈∂ W[J] \ \ J∈∂ [ ] \ , where ∂ is the subdifferential from Definition C.5. If W and Γ are differentiable, then ∂ reduce to gradients ∇. We call them equations of motion (EOMs) in the physics literature. It also refers to the Fenchel-Young duality relation. For φ and J, we call them the classical solutions or saddle points. Proof. We repeat the proof from (Blondel et al., 2020; Rockafellar, 1997) for completeness. By definition, Γ[φ]=supJ∈ℝd∑i=1dJiφi−W[J] [ ]= _J ^d\ _i=1^dJ_i _i-W[J]\. Therefore, for any J, Γ[φ]≥∑i=1dJiφi−W[J] [ ]≥ _i=1^dJ_i _i-W[J]. By the definition of subgradient Definition C.5, we have W[J]≥W[J⋆]+∑i=1dφi(Ji−Ji⋆),∀J∈ℝd∀φ∈∂W[J⋆]. W[J]≥ W[J ]+ _i=1^d _i(J_i-J _i), ∀ J ^d \ ∀ ∈∂ W[J ]. Then ∑i=1dJi⋆φi−W[J⋆]≥∑i=1dJiφi−W[J]∀J∈ℝd. _i=1^dJ _i _i-W[J ]≥ _i=1^dJ_i _i-W[J] ∀ J ^d. Hence, J⋆J is a maximizer, and therefore J⋆∈∂Γ[φ]J ∈∂ [ ]. Conversely, if we assume that J⋆∈∂Γ[φ⋆]J ∈∂ [ ], then Γ[φ]≥Γ[φ⋆]+∑i=1dJi⋆(φi−φi⋆)∀φ∈ℝd. [ ]≥ [ ]+ _i=1^dJ _i( _i- _i ) ∀ ^d. In this way, ∑i=1dJi⋆φi⋆−Γ[φ⋆]≥∑i=1dJi⋆φi−Γ[φ]∀φ∈ℝd. _i=1^dJ _i _i - [ ]≥ _i=1^dJ _i _i- [ ] ∀ ^d. Note that W[J]W[J] is closed due to the form (2.2). Then by involutivity in Fact 1, we obtain W[J⋆]=W⋆⋆[J⋆]=supφ∈ℝd∑i=1dφiJi⋆−Γ[φ]≥∑i=1dφiJi⋆−Γ[φ]∀φ∈ℝd. W[J ]=W [J ]= _ ^d\ _i=1^d _iJ _i- [ ]\≥ _i=1^d _iJ _i- [ ] \ \ ∀ ^d. Hence, φ⋆ is a maximizer, and therefore φ⋆∈∂W[J⋆] ∈∂ W[J ]. After removing the redundant notation ⋆ , we arrive the Fenchel-Young duality relation. ∎ Remark C.3. The non-zero equation, ∇JW(J) _JW(J), reflects a symmetry breaking of a free theory due to the external input. For example, consider a spin system. Every spin has a random orientation, S∈−1,+1dS∈\-1,+1\^d, and there exists a ℤ2Z_2 symmetry that flips the entire system upside down. Then, ∇JW[J]|J=0=Sexp−βℋ[S]+∑i=1dJiSiZ[J]|J=0=S∼p(S)[S]=0, _JW[J]|_J=0= S \- [S]+ _i=1^dJ_iS_i\Z[J]|_J=0=E_S p(S)[S]=0 \ , (C.1) where p(S)∝exp−βℋ[S]p(S) \- [S]\ up to a normalization factor. This gives the macroscopic variable φ=0 =0. However, if we input the external source, J≠0J≠ 0, then the spin system favors one orientation. This corresponds to a response in a ferromagnet. After this, we approach J→0+J→ 0^+ as a stationary condition for Γ[φ] [ ]. The system stays at one specific local minima. See Figure 9 for the illustration. We shall see that in the sparse modern Hopfield models (Section 3), we always require this response, φ=∇JW[J]≠0 = _JW[J]≠ 0, to realize the retrieval states. Symmetric Landscape (J=0J=0)Γ[φ] [ ]φ −φ⋆- φ⋆ Two Stationary Points∇φΓ[φ]=0 _ [ ]=0Inputting External Field (J>0J>0)Γ[φ]−Jφ [ ]-J φ −φ⋆- φ⋆ FlowStationary State (J→0+J→ 0^+)Γ[φ] [ ]φ −φ⋆- φ⋆ Selected State Figure 9: Selection of a stationary state by an external source. This figure illustrates how a source J selects one state from multiple degenerate stationary states of the free energy Γ[φ] [ ]. Left. When J=0J=0, Γ[φ] [ ] is symmetric and has two degenerate local minima ±φ⋆± . Middle. Turning on a small external source J>0J>0 tilts the landscape to Γ[φ]−Jφ [ ]-J . This breaks the symmetry so that +φ⋆+ becomes preferred. Then ,the system flows toward the selected basin. Right. After the selection, we take the limit J→0+J→ 0^+. Although the explicit source is removed, the system remains in the chosen local minimum +φ⋆+ satisfying ∇φΓ[+φ⋆]=0 _ [+ ]=0. This source-selection procedure resolves the degeneracy between degenerate stationary states and specifies the retrieval state. Remark C.4. From the Fenchel-Young duality relation, Proposition 2.1, we know that a gradient descent trajectory in J-space has one-to-one mapping to a unique trajectory in φ -space. However, the induced unique trajectory is not a gradient flow. Specifically, suppose we define gradient descent as φ=J˙=−∇JW[J] = J=- _JW[J], where J˙≔dJ/dt J dJ/dt if we view update steps as infinitesimal. Then φ˙=∇J2W[J](−∇JW[J])=−φ∇J2W[J], =∇^2_JW[J](- _JW[J])=- ∇^2_JW[J] \ , which is not the gradient flow, φ˙=−∇φΓ[φ] =- _ [ ]. C.3 Tsallis Entropy In this section, we introduce the Tsallis entropy and its role in statistical mechanics. Definition C.8 (Tsallis Entropy (Tsallis, 1988)). We define the Tsallis entropy, (Tsallis, 1988), of the form Ψα[p]≔1α−1(1−∑μ=1Mpμα), _α[p] 1α-1(1- _μ=1^Mp_μ^α) \ , (C.2) where ∑μ=1Mpμ=1 _μ=1^Mp_μ=1, α∈ℝ+/1α _+/\1\. Remark C.5. The maximal Tsallis entropy occurs when the probability function is uniform, i.e., pμ=M−1p_μ=M^-1 for every μ. Hence, if the system is random, then the entropy becomes maximal. Proposition C.2 (Types of Entropy (Tsallis, 1988)). Specific values of α in Tsallis entropy reduce to other types of entropy. • If α→1α→ 1, it reduces to the Shannon entropy (Shannon, 1948). S[p]≔−∑μ=1Mpμlog(pμ). S[p] - _μ=1^Mp_μ (p_μ). (C.3) • If α→2α→ 2, it reduces to the Gini entropy (Gini, 1912). G[p]≔1−∑μ=1Mpμ2. G[p] 1- _μ=1^Mp_μ^2. (C.4) • If α→∞α→∞, it is zero. Proof. We repeat the proof from (Tsallis, 1988) for completeness. We prove them one-by-one as follows. • Applying the l’Hôpital’s rule for 0/00/0-form, we have Ψα→1[p]=−limα→1∑μ=1Mlog(pμ)pμα=−∑μ=1Mpμlog(pμ)≔S[p]. _α→ 1[p]=- _α→ 1 _μ=1^M (p_μ)p_μ^α=- _μ=1^Mp_μ (p_μ) S[p] \ . This is the form of Shannon entropy. • For α→2α→ 2, the Tsallis entropy function is by definition the Gini entropy. • Let pmax≔maxμpμp_max _μ \ p_μ and μmax _max be the corresponding maximal index. Then we separate the Tsallis entropy as Ψα[p]=1α−1[1−pmaxα(1+∑μ≠μmaxMpμαpmaxα)]. _α[p]= 1α-1[1-p_max^α(1+ _μ≠ _max^M p_μ^αp_max^α)] \ . After taking α→∞α→∞ and applying the l’Hôpital’s rule for ∞/∞/∞-form, we obtain Ψα→∞[p]=−limα→∞log(pmax)pmaxα, _α→∞[p]=- _α→∞ (p_max)p_max^α \ , where (pμ/pmax)α→0(p_μ/p_max)^α→ 0 as α→∞α→∞. If pmax<1p_max<1, then Ψα→∞=0 _α→∞=0. If pmax=1p_max=1, then Ψα→∞→0 _α→∞→ 0 due to the logarithm. Hence, Ψα→∞[p] _α→∞[p] for all p. This completes the proof. ∎ Remark C.6. If α>0α>0, then the Tsallis entropy is strictly concave since its second derivative respect to p is negative on the simplex interior. This includes Shannon entropy (α→1α→ 1), which is also strictly concave. Remark C.7. Although the Tsallis entropy is not defined at α=1α=1 from Definition C.8, we use the continuous extension Ψ1[p]≔limα→1Ψα[p] _1[p] _α→ 1 _α[p]. It is equal to the Shannon entropy according to Proposition C.2. Therefore, throughout the paper, the α=1α=1 refers to this Shannon/softmax limit. Proposition C.3 (Fenchel Conjugate of Entropy (Blondel et al., 2020)). The Fenchel conjugate of the negative Tsallis entropy is Ωα⋆[z]≔(−Ψα)⋆[z]=supp∈ΔM∑μ=1Mzμpμ+Ψα[p], _α [z] (- _α) [z]= _p∈ ^M\ _μ=1^Mz_μp_μ+ _α[p]\ \ , (C.5) where ΔM≔pμ∈ℝ+M|∑μ=1Mpμ=1 ^M \p_μ _+^M| _μ=1^Mp_μ=1\ is a (M−1)(M-1)-dimensional unit simplex. Then • In the Shannon case, α→1α→ 1, the negative entropy Ωα _α is a Legendre function. • For α>1α>1, Ωα _α satisfies involutivity, but neither Legendre condition nor bijectiveness. • For 1>α>01>α>0, Ωα _α is a Legendre function. Proof. First, we look at the Shannon case. • Due to (C.3) and Remark C.6, the Shannon entropy is closed and strictly concave. By Fact 1, it satisfies involutivity, S⋆⋆[p]=S[p]S [p]=S[p]. Furthermore, ‖∇pS[p]‖p→0=∞\| _pS[p]\|_p→ 0=∞. Thus, it is essentially smooth. The negative Shannon entropy Ωα _α is thereby a Legendre function and satisfies bijectiveness. • For α>1α>1, the Tsallis entropy is closed and strictly concave on the simplex. Therefore, Ωα=−Ψα _α=- _α satisfies involutivity condition Ωα>1⋆⋆[p]=Ωα>1[p] _α>1[p]= _α>1[p]. However, ‖∇pΩα>1[p]‖p→0\| _p _α>1[p]\|_p→ 0 does not diverge, so it is not essentially smooth. As a result, it is not a Legendre function nor satisfies bijectiveness. • For 1>α>01>α>0, the Tsallis entropy is strictly concave. Hence, Ωα _α is strictly convex. Furthermore, ∂Ωα/∂pμ→−∞∂ _α/∂ p_μ→-∞ near the boundary, so it satisfies the essential smoothness. Therefore, it is a Legendre function satisfying the invlolutivity and bijectiveness properties. This completes the proof. ∎ Remark C.8. We are able to express unit simplex condition in two different ways: 1=∑μ=1Mpμ=∫pμ(x)dμ(x)1= _μ=1^Mp_μ= _Xp_μ(x) \ μ(x). That is, we transform the x configuration space, X, onto a simplex, ΔM ^M. Definition C.9 (α-Deformed Exponential Family and Logarithm (Tsallis, 1988)). Let α>0α>0. We define the α-deformed exponential function as expα(x)≔exp(x),ifα=1[1+(1−α)x]+11−α,ifα≠1, _α(x) cases (x)&, \ if \ α=1\\ [1+(1-α)x]_+ 11-α&, \ if \ α≠ 1 cases \ , and its inverse as logα(y)≔log(y),ifα=1y1−α−11−α,ifα≠1, _α(y) cases (y)&, \ if \ α=1\\ y^1-α-11-α&, \ if \ α≠ 1 cases \ , where x∈ℝx , y∈ℝ+y _+ for well-definedness, and [z]+≔maxz,0[z]_+ \z,0\ for every z∈ℝz . This α-deformed family has several properties, and some of which differ from the ordinary ones. Lemma C.1 (Properties of α-Deformed Functions (Tsallis, 1988)). α-deformed functions satisfy • limα→1expα(x)=exp(x) _α→ 1 _α(x)= (x) and limα→1logα(x)=log(x) _α→ 1 _α(x)= (x). • expα(x)/logα(y) _α(x)/ _α(y) are monotonic and convex/concave. • α-deformed exponential/logarithm does not have multiplicity/additivity unless α=1α=1. • dxexpα(x)=[expα(x)]α ddx _α(x)=[ _α(x)]^α in the domain [1+(1−α)x]>0[1+(1-α)x]>0 and dylogα(y)=y−α ddy _α(y)=y^-α. Proof. We repeat the proof from (Tsallis, 1988) for completeness. We prove those properties one-by-one as follows. • Let u=1−αu=1-α. By Definition C.9, limα→1expα(x)=limα→1[1+(1−α)x]+11−α=limu→0[1+ux]1/u=exp(x). _α→ 1 _α(x)= _α→ 1[1+(1-α)x]_+ 11-α= _u→ 0[1+ux]^1/u= (x) \ . For the logarithm, we apply the l’Hôpital’s rule for 0/00/0-form. limα→1logα(y)=limα→1y1−α−11−α=−limα→1log(y)y1−α=log(y). _α→ 1 _α(y)= _α→ 1 y^1-α-11-α=- _α→ 1 (y)y^1-α= (y) \ . • If [1+(1−α)x]>0[1+(1-α)x]>0, by differentiating expα(x) _α(x) with respect to x, we have dxexpα(x)=[1+(1−α)x]α1−α>0,d2dx2expα(x)=α[1+(1−α)x]2α−11−α>0. ddx _α(x)=[1+(1-α)x] α1-α>0 \ , \ d^2dx^2 _α(x)=α[1+(1-α)x] 2α-11-α>0 \ . Hence, it is strictly increasing (monotonic) and strictly convex. For logα _α, dylogα(y)=y−α>0,d2dy2logα(y)=−αy−α−1<0. ddy _α(y)=y^-α>0 \ , \ d^2dy^2 _α(y)=-α y^-α-1<0 \ . Hence, it is strictly increasing (monotonic) and strictly concave. • Consider x1,x2∈ℝx_1,x_2 . expα(x1)expα(x2)= _α(x_1) _α(x_2)= [1+(1−α)(x1+x2)+(1−α)2x1x2]+11−α \ [1+(1-α)(x_1+x_2)+(1-α)^2x_1x_2]_+ 11-α = = expα(x1+x2)⋅[1+(1−α)2x1x21+(1−α)(x1+x2)]+11−α. \ _α(x_1+x_2)·[1+ (1-α)^2x_1x_21+(1-α)(x_1+x_2)]_+ 11-α \ . (C.6) If α→1α→ 1, then [⋯][·] in (C.6) is 11. Consider y1,y2∈ℝ+y_1,y_2 _+. logα(y1y2)= _α(y_1y_2)= y11−αy21−α−11−α=(y11−α−1)+(y21−α−1)+(y11−α−1)(y21−α−1)1−α \ y_1^1-αy_2^1-α-11-α= (y_1^1-α-1)+(y_2^1-α-1)+(y_1^1-α-1)(y_2^1-α-1)1-α = = logα(y1)+logα(y2)+(1−α)logα(y1)log(y2). \ _α(y_1)+ _α(y_2)+(1-α) _α(y_1) (y_2) \ . (C.7) If α→1α→ 1, the last term in (C.7) is zero. • For [1+(1−α)x]>0[1+(1-α)x]>0, we have dxexpα(x)=[1+(1−α)x]11−α−1=[expα(x)]α. ddx _α(x)=[1+(1-α)x] 11-α-1=[ _α(x)]^α \ . For logα(y) _α(y), we obtain the result by differentiation easily. For α→1α→ 1, they hold trivially. This completes the proof. ∎ Remark C.9. Since pμαlogα(pμ)=(1−α)−1(pμ−pμα)p_μ^α _α(p_μ)=(1-α)^-1(p_μ-p_μ^α), using ∑μ=1Mpμ=1 _μ=1^Mp_μ=1, we obtain Ψα[p]=−∑μ=1Mpμαlogα(pμ). _α[p]=- _μ=1^Mp_μ^α _α(p_μ). (C.8) C.4 Proof of Theorem 2.1 Theorem C.1 (Theorem 2.1 Restated: Tsallis Entropy and Deformed Free Energy). If we replace the exponential functions and logarithm functions by α-deformed functions, then we have the deformed log-partition function −Wα[J]=βx∼pJ(α)[ℋ[x]]−Ψα[pJ]Zα−1[J]∫pJα(x)dμ(x)−∑i=1dJix∼pJ(α)[xi], -W_α[J]= ^(α)_x p_J[H[x]]- _α[p_J]Z_α^α-1[J] _Xp_J^α(x) \ μ(x)- _i=1^dJ_iE^(α)_x p_J[x_i], where x∼pJ(α)[⋅]≔∫pJα(x)[⋅]dμ(x)/∫pJα(x)dμ(x)E^(α)_x p_J[·] _Xp^α_J(x)[·] μ(x)/ _Xp^α_J(x) μ(x) is the escort expectation value, and Γα=βx∼pJ⋆(α)[ℋ[x]]−Ψα[pJ⋆]Zα−1[J⋆]∫pJ⋆α(x)dμ(x),with φ=x∼pJ⋆(α)[x], _α= ^(α)_x p_J [H[x]]- _α[p_J ]Z_α^α-1[J ] _Xp_J ^α(x) \ μ(x), \ =E^(α)_x p_J [x], (C.9) where J⋆=argmaxJ∑i=1dJiφi−Wα[J]J = _J\ _i=1^dJ_i _i-W_α[J]\. We assume X is finite-volume to avoid divergence. Proof. By Definition C.9, we express Wα[J]=logαZα[J]W_α[J]= _αZ_α[J] and pJ(x)=expα−βℋ[x]+∑i=1dJixiZα[J],Zα[J]=∫expα−βℋ[x]+∑i=1dJixidμ(x). p_J(x)= _α\- [x]+ _i=1^dJ_ix_i\Z_α[J] \ , \ Z_α[J]= _X _α\- [x]+ _i=1^dJ_ix_i\ \ μ(x) \ . Notice that since X is finite-volume, Zα[J]Z_α[J] is finite for appropriate J and the following computation is reliable. Then we have logαpJ(x)= _αp_J(x)= logα(expα−βℋ[x]+∑i=1dJixiZα[J]) \ _α( _α\- [x]+ _i=1^dJ_ix_i\Z_α[J]) = = Zα−1[J]expα1−α−βℋ[x]+∑i=1dJixi−11−α \ Z_α^α-1[J] _α^1-α\- [x]+ _i=1^dJ_ix_i\-11-α (By Definition C.9) = = Zα−1[J]1+(1−α)(−βℋ[x]+∑i=1dJixi)−11−α \ Z_α^α-1[J]\1+(1-α)(- [x]+ _i=1^dJ_ix_i)\-11-α (By Definition C.9) = = Zα−1[J](−βℋ[x]+∑i=1dJixi)+Zα−1[J]−11−α \ Z^α-1_α[J](- [x]+ _i=1^dJ_ix_i)+ Z_α^α-1[J]-11-α = = Zα−1[J](−βℋ[x]+∑i=1dJixi)−Zα−1[J]⋅(Zα1−α[J]−11−α) \ Z^α-1_α[J](- [x]+ _i=1^dJ_ix_i)-Z_α^α-1[J]·( Z^1-α_α[J]-11-α) = = Zα−1[J](−βℋ[x]+∑i=1dJixi−Wα[J]). \ Z_α^α-1[J](- [x]+ _i=1^dJ_ix_i-W_α[J]). (By Definition C.9 and Wα[J]=logαZα[J]W_α[J]= _αZ_α[J]) By (C.8), we express the Tsaliis entropy as Ψα[pJ]= _α[p_J]= −∫pJα(x)logα(pJ(x))dμ(x) \ - _Xp_J^α(x) _α(p_J(x)) \ μ(x) = = −∫pJα(x)Zα−1[J](−βℋ[x]+∑i=1dJixi−Wα[J]). \ - _Xp_J^α(x)Z_α^α-1[J](- [x]+ _i=1^dJ_ix_i-W_α[J]). (C.10) This gives −Wα[J]= -W_α[J]= −Ψα[pJ]+β∫pJα(x)Zα−1[J]ℋ[x]dμ(x)−∫∑i=1dpJα(x)Zα−1[J]JixiZα−1[J]∫pJα(x)dμ(x) \ - _α[p_J]+β _Xp_J^α(x)Z_α^α-1[J]H[x] \ μ(x)- _X _i=1^dp_J^α(x)Z_α^α-1[J]J_ix_iZ^α-1_α[J] _Xp_J^α(x) \ μ(x) = = β∫pJα(x)ℋ[x]dμ(x)∫pJα(x)dμ(x)−Ψα[pJ]Zα−1[J]⋅∫pJα(x)dμ(x)−∑i=1dJi∫pJα(x)xidμ(x)∫pJα(x)dμ(x). \ β \ _Xp_J^α(x)H[x] \ μ(x) _Xp_J^α(x) \ μ(x)- _α[p_J]Z_α^α-1[J]· _Xp_J^α(x) \ μ(x)- _i=1^dJ_i \ _Xp_J^α(x)x_i \ μ(x) _Xp_J^α(x) \ μ(x). The first and last term are the escort expectation values x∼pJ(α)[⋅]E_x p_J^(α)[·]. Hence, it is the form of our first equation. After applying the Fenchel conjugation, we have Γα= _α= supJ∑i=1dJiφi−Wα[J] \ _J\ _i=1^dJ_i _i-W_α[J]\ = = supJ∑i=1dJi(φi−x∼pJ(α)[xi])+βx∼pJ(α)[ℋ[x]]−Ψα[pJ]Zα−1[J]⋅∫pJα(x)dμ(x) \ _J\ _i=1^dJ_i( _i-E_x p_J^(α)[x_i])+ ^(α)_x p_J[H[x]]- _α[p_J]Z_α^α-1[J]· _Xp_J^α(x) \ μ(x)\ = = βx∼pJ⋆(α)[ℋ[x]]−Ψα[pJ⋆]Zα−1[J⋆]⋅∫pJ⋆α(x)dμ(x), \ ^(α)_x p_J [H[x]]- _α[p_J ]Z_α^α-1[J ]· _Xp_J ^α(x) \ μ(x), where φ=x∼pJ⋆(α)[x] =E_x p_J ^(α)[x]. This proves the theorem. ∎ Remark C.10. The finite-volume assumption is only to ensure that the α-deformed partition function is well-defined. Specifically, if we consider whole ℝdR^d with α=2α=2 and a>0a>0, then ∫ℝdexp2−a‖x‖22dμ(x)=∫ℝdμ(x)1+a‖x‖22=Sd−1∫0∞rd−11+ar2dr, _R^d _2\-a\|x\|_2^2\ \ μ(x)= _R^d μ(x)1+a\|x\|_2^2=S_d-1 _0^∞ r^d-11+ar^2 \ r, where Sd−1S_d-1 is the surface area of a unity sphere Sd−1S^d-1. Hence, only if d<2d<2 is the integral finite. This leads to the consideration of a finite-volume domain X. Passing to a finite-volume domain ⊂ℝdX ^d introduces a boundary. It removes the divergence but does not guarantee strict convexity of Wα[J]W_α[J] nor the essential smoothness at the boundary of domWαdomW_α. Hence, according to Definition C.7, the transformation here should be understood as a Fencehl conjugate on domWαdomW_α, not as a Legendre transform necessarily. C.5 Proof of Proposition 3.1 Proposition C.4 (Proposition 3.1 Restated: Query-Probability Relation). A query state vector relates to a probability function via xi=1d∑μ=1Mξiμpμ+1βJi. x_i= 1 d _μ=1^M _i^μp_μ+ 1βJ_i. Proof. In the deterministic limit, we have Zα[J]=∫expα−βℋ~MHM[x,p,ξ]+∑i=1dJixidμ(x)=expα−βℋ~MHM[x,p,ξ]+∑i=1dJixi, Z_α[J]= _X _α\-β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i\ \ μ(x)= _α\-β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i\, since we identify x with the mean. The query state vector that maximizes Zα[J]Z_α[J] is the deterministic x itself. Hence, we have 0= 0= ∂xiβℋ~MHM[x,p,ξ]−∑j=1dJjxj \ ∂ x_i\β H_MHM[x,p,ξ]- _j=1^dJ_jx_j\ = = βxi−βd∑μ=1Mξiμpμ−Ji. \ β x_i- β d _μ=1^M _i^μp_μ-J_i. This proves the proposition. ∎ C.6 Proof of Theorem 3.1 Proposition C.5 (Theorem 3.1 Restated: Free Energy of MHM). The joint free energy of the MHM is Γ~α[x,p,ξ]=β2∑i=1dxi2−β∑μ=1Mzμpμ. _α[x,p,ξ]= β2 _i=1^dx_i^2-β _μ=1^Mz_μp_μ. Hence, Γα[x,ξ]=β2∑i=1dxi2−βmaxμ∈[1,M]zμ. _α[x,ξ]= β2 _i=1^dx_i^2-β _μ∈[1,M] \ z_μ. Proof. The form of the partition function is Zα[J,ξ]=expαWα[J,ξ]=∫ΔMexpα−β2∑i=1dxi2+β∑μ=1Mzμpμ+∑i=1dJixidμ(p)dμ(x). Z_α[J,ξ]= _α\W_α[J,ξ]\= _X _ ^M _α\- β2 _i=1^dx_i^2+β _μ=1^Mz_μp_μ+ _i=1^dJ_ix_i\ \ μ(p) μ(x) \ . According to the form of the Tsallis entropy (C.8) and (C.10), we have Ψα[ρJ]= _α[ _J]= −∫ΔMρJα(x,p)logα[ρJ(x,p)]dμ(p)dμ(x) \ - _X _ ^M _J^α(x,p) _α[ _J(x,p)] \ μ(p) μ(x) = = β2Zα−1[J,ξ]∑i=1d∫ΔMρJα(x,p)xi2dμ(p)dμ(x) \ β2Z_α^α-1[J,ξ] _i=1^d _X _ ^M _J^α(x,p)x_i^2 \ μ(p) μ(x) −βZα−1[J,ξ]∑μ=1M∫ΔMρJα(x,p)pμzμ(x)dμ(p)dμ(x) \ -β Z_α^α-1[J,ξ] _μ=1^M _X _ ^M _J^α(x,p)p_μz_μ(x) \ μ(p) μ(x) −Zα−1[J,ξ]∑i=1dJi∫ΔMρJα(x,p)xidμ(p)dμ(x) \ -Z_α^α-1[J,ξ] _i=1^dJ_i _X _ ^M _J^α(x,p)x_i \ μ(p) μ(x) +Zα−1[J,ξ]Wα[J,ξ]∫ΔMρJα(x,p)dμ(p)dμ(x). \ +Z_α^α-1[J,ξ]W_α[J,ξ] _X _ ^M _J^α(x,p) \ μ(p) μ(x). It yields −Wα[J,ξ]= -W_α[J,ξ]= β2∑i=1d(x,p)∼ρJ(α)[xi2]−β∑μ=1M(x,p)∼ρJ(α)[zμpμ] \ β2 _i=1^dE^(α)_(x,p) _J[x_i^2]-β _μ=1^ME^(α)_(x,p) _J[z_μp_μ] −Ψα[ρJ]Zα−1[J,ξ]∫ΔMρJα(x,p)dμ(p)dμ(x)−∑i=1dJi(x,p)∼ρJ(α)[xi]. \ - _α[ _J]Z_α^α-1[J,ξ] _X _ ^M _J^α(x,p) \ μ(p) μ(x)- _i=1^dJ_iE^(α)_(x,p) _J[x_i]. (C.11) Since x and p are fixed, so the probability density function concentrates to an indicator, i.e., ρJ(x,p)→x,~x∼pJ(α)[x]p,~p∼pΔ,J(α)[p], _J(x,p)→ 1_\x, E^(α)_x p_J[x]\ 1_\p, E^(α)_p p_ ,J[p]\, where ~x∼pJ(α)[x]≔(x,p)∼ρJ(α)[x]=∫C(x)xpJα(x)dμ(x)∫C(x)pJα(x)dμ(x),with C(x)≔∫ΔMρJα(p|x)dμ(p), E^(α)_x p_J[x] ^(α)_(x,p) _J[x]= _XC(x)xp^α_J(x) \ μ(x) _XC(x)p^α_J(x) \ μ(x), \ C(x) _ ^M _J^α(p|x) \ μ(p), (C.12) and ~p∼pΔ,J(α)[pμ]≔(x,p)∼ρJ(α)[pμ]=∫ΔMC(p)pμpΔ,Jα(p)dμ(p)∫ΔMC(p)pΔ,Jα(p)dμ(p),with C(p)≔∫ρJα(x|p)dμ(x). E_p p_ ,J^(α)[p_μ] ^(α)_(x,p) _J[p_μ]= _ ^MC(p)p_μp^α_ ,J(p) \ μ(p) _ ^MC(p)p^α_ ,J(p) \ μ(p), \ C(p) _X _J^α(x|p) \ μ(x). (C.13) Then Zα[J,ξ]→expα−βℋ~MHM[~x∼pJ(α)[x],~p∼pΔ,J(α)[p],ξ]+∑i=1dJi~x∼pJ(α)[xi]. Z_α[J,ξ]→ _α\-β H_MHM[ E^(α)_x p_J[x], E^(α)_p p_ ,J[p],ξ]+ _i=1^dJ_i E^(α)_x p_J[x_i]\. Hence, by the Fenchel conjugate, we obtain Γ~α= _α= supJ∈domWα∑i=1dJiφi−Wα[J,ξ] \ _J _α\ _i=1^dJ_i _i-W_α[J,ξ]\ = = supJ∈domWα∑i=1dJi(φi−~x∼pJ(α)[xi])+β2∑i=1d(~x∼pJ(α)[xi2])−β∑μ=1M~x∼pJ(α)[zμ]~p∼pΔ,J(α)[pμ] \ _J _α \ _i=1^dJ_i( _i- E_x p_J^(α)[x_i])+ β2 _i=1^d( E_x p_J^(α)[x_i^2])-β _μ=1^M E_x p_J^(α)[z_μ] E_p p_ ,J^(α)[p_μ] −Ψα[]expα−1−βℋ~MHM[~x∼pJ(α)[x],~p∼pΔ,J(α)[p],ξ]+∑i=1dJi~x∼pJ(α)[xi] \ - _α[ 1] ^α-1_α\-β H_MHM[ E^(α)_x p_J[x], E^(α)_p p_ ,J[p],ξ]+ _i=1^dJ_i E^(α)_x p_J[x_i]\ \ = = β2∑i=1d(~x∼pJ⋆(α)[xi2])−β∑μ=1M~x∼pJ⋆(α)[zμ]~p∼pΔ,J⋆(α)[pμ], \ β2 _i=1^d( E_x p_J ^(α)[x_i^2])-β _μ=1^M E_x p_J ^(α)[z_μ] E_p p_ ,J ^(α)[p_μ], (By Ψα[]=0 _α[ 1]=0) where J⋆J is the maximizing source. Since we identify ~x∼pJ⋆(α)[x] E_x p_J ^(α)[x] with x, ~x∼pJ⋆(α)[x2] E_x p_J ^(α)[x^2] with x2x^2, and ~p∼pΔ,J⋆(α)[p] E_p p_ ,J ^(α)[p] with p, we have Γ~α[x,p,ξ]=β2∑i=1dxi2−β∑μ=1Mzμpμ. _α[x,p,ξ]= β2 _i=1^dx_i^2-β _μ=1^Mz_μp_μ. Notice that infp∈ΔM−∑μ=1Mzμpμ=−maxμ∈[1,M]zμ. _p∈ ^M\- _μ=1^Mz_μp_μ\=- _μ∈[1,M] \ z_μ. After substituting into the joint free energy to eliminate the dependence of p, we prove the second equation in this proposition. ∎ C.7 Proof of Proposition 3.3 Proposition C.6 (Proposition 3.3 Revisited: Emergent Attention Mechanism as EOMs in MHMs). Attention emerges as follows. • If α→∞α→∞, it corresponds to a hardmax function. pμ=hardmax(zμ)=1,for maximal zμ0,otherwise. p_μ=hardmax(z_μ)= cases1 \ , \ &for maximal z_μ\\ 0 \ , \ &otherwise cases. • If α=2α=2, it corresponds to a sparsemax function (Martins and Astudillo, 2016). pμ=Sparsemaxβ(zμ)=β2[zμ−θ]+,with β2∑μ=1M[zμ−θ]+=1. p_μ=Sparsemax_β(z_μ)= β2[z_μ-θ]_+, \ β2 _μ=1^M[z_μ-θ]_+=1. • If 1<α<21<α<2, it corresponds to an entmax family (Peters et al., 2019). • If α→1α→ 1, it corresponds to a softmax function (Blondel et al., 2020). pμ=Softmaxβ(zμ)=expβzμ∑ν=1Mexpβzν. p_μ= Softmax_β(z_μ)= \β z_μ\ _ν=1^M \β z_ν\. Proof. We include the proof for completeness, adapting the standard argument from (Martins and Astudillo, 2016; Peters et al., 2019; Blondel et al., 2020) to our notation. The training trajectory follows pμ=argmaxp∈ΔMβ∑μ=1Mzμpμ+Ψα[p]p_μ= _p∈ ^M\β _μ=1^Mz_μp_μ+ _α[p]\. We derive the physical meanings of this optimization problem via the Lagrangian function: ∂pμ∑ν=1Mzνpν+1β⋅1α−1(1−∑ν=1Mpνα)+θ(1−∑ν=1Mpν)+∑ν=1Mλνpν=0, ∂ p_μ\ _ν=1^Mz_νp_ν+ 1β· 1α-1(1- _ν=1^Mp_ν^α)+θ(1- _ν=1^Mp_ν)+ _ν=1^M _νp_ν\=0, where we use the form of Tsallis entropy, (C.2), and the Lagrangian multipliers, λ and λν≥0 _ν≥ 0, to include the simplex constraints. This gives the complementary slackness, λμpμ=0 _μp_μ=0, and the stationary condition, zμ−αβ(α−1)pμα−1−θ+λμ=0⟹pμ=β(1−1α)[zμ−θ]+1α−1,ifλμ=0pμ=0,ifλμ>0, z_μ- αβ(α-1)p_μ^α-1-θ+ _μ=0 \ \ casesp_μ=\β(1- 1α)[z_μ-θ]_+\ 1α-1&, \ if \ _μ=0\\ p_μ=0&, \ if \ _μ>0 cases \ , where [u]+≔max(u,0)[u]_+ (u,0). For α→∞α→∞ and α=2α=2, we obtain the forms of pμp_μ by inserting α directly. For α=1α=1, we use the Shannon form, (C.3). Hence, for non-zero pμp_μ, we have zμ+1β∂pμ(−∑ν=1Mpνlogpν)−θ=0. z_μ+ 1β ∂ p_μ(- _ν=1^Mp_ν p_ν)-θ=0. This gives pμ=expβ(zμ−θ)−1. p_μ= \β(z_μ-θ)-1\. The normalization condition gives 1=∑μ=1Mpμ=exp−βθ∑μ=1Mexpβzμ−11= _μ=1^Mp_μ= \-βθ\ _μ=1^M \β z_μ-1\. This yields θ=β−1log∑μ=1Mexpβzμ−1θ=β^-1 _μ=1^M \β z_μ-1\. After plugging this into pμp_μ, we obtain the softmax function. For 2>α>12>α>1, (α−1)−1>1(α-1)^-1>1. Hence, pμp_μ is the entmax family interpolating between softmax and sparsemax. ∎ C.8 Proof of Proposition 4.1 Proposition C.7 (Proposition 4.1 Restated: Temperature Coexistence Point for Sparse Model). Let a,ba,b be two distinct branches and α=2α=2. Denote kak_a and kbk_b as the exact support sizes (see Lemma F.1) of branches a and b, respectively. Based on the statistical mechanics framework, the coexistence point satisfies ∑μ,ν∈a[zμ⋆−θ]+[zν⋆−θ]+Kμν(βcoexist,β0)=∑μ,ν∈b[zμ⋆−θ]+[zν⋆−θ]+Kμν(βcoexist,β0), _μ,ν∈ a[z _μ-θ]_+[z _ν-θ]_+K_μν( _coexist; _0)= _μ,ν∈ b[z _μ-θ]_+[z _ν-θ]_+K_μν( _coexist; _0), where Kμν(β)≔μ,ν−β2GμνK_μν(β) 1_\μ,ν\- β2G_μν, and Gμν≔∑i=1dξiμξiν/dG_μν _i=1^d _i^μ _i^ν/d is the Gram matrix. βcoexist _coexist does not admit a closed-form expression. However, if the distribution is uniform on the simplex and the system satisfies the well-separation condition, i.e., Gμν=1,μ=νϵ≪1,μ≠ν, G_μν= cases1 \ &, \ μ=ν\\ ε 1 \ &, \ μ≠ν cases, then for ka=1k_a=1 and kb=2k_b=2, we have the analytic solution βcoexist=2/(1−ϵ) _coexist=2/(1-ε). At this point, β=βcoexistβ= _coexist, p⋆p is degenerate and every p⋆p is a global minimum of the joint free energy Γ~α=2(mod) _α=2^(mod). Proof. Consider a system with the minimizer, pμ⋆p _μ. Using that at a stable point, Ji=0J_i=0 for all i=1,…,di=1,...,d, we have zμ⋆=1d∑i=1dξiμxi⋆=1d∑i=1dξiμ(1d∑ν=1Mξiνpν⋆+1βJi)=∑ν=1MGμνpν⋆. z _μ= 1 d _i=1^d _i^μx _i= 1 d _i=1^d _i^μ( 1 d _ν=1^M _i^νp _ν+ 1βJ_i)= _ν=1^MG_μνp _ν. Moreover, ∑i=1d(xi⋆)2=1d∑i=1d∑μ,ν=1Mξiμξiνpμ⋆pν⋆=∑μ,ν=1MGμνpμ⋆pν⋆. _i=1^d(x _i)^2= 1d _i=1^d _μ,ν=1^M _i^μ _i^νp _μp _ν= _μ,ν=1^MG_μνp _μp _ν. Suppose the support is k, we denote the indices of the non-zero components as μ~=1,…,k μ=1,...,k. Then we have Γ~α(mod)[x⋆,p⋆,ξ]= ^(mod)_α[x ,p ,ξ]= β2∑i=1d(xi⋆)2−β∑μ~=1kzμ~⋆pμ~⋆−(1−∑μ~=1k(pμ~⋆)2) \ β2 _i=1^d(x _i)^2-β _ μ=1^kz _ μp _ μ-(1- _ μ=1^k(p _ μ)^2) = = −1−∑μ~,ν~=1kpμ~⋆(β2Gμ~ν~−μ~,ν~)pν~⋆ \ -1- _ μ, ν=1^kp _ μ( β2G_ μ ν- 1_\ μ, ν\)p _ ν From the uniform and well-separation condition, we write Gμ~ν~=(1−ϵ)μ~,ν~+ϵG_ μ ν=(1-ε) 1_\ μ, ν\+ε. Then this gives ∑μ~,ν~=1kpμ~⋆Gμ~ν~pν~⋆=(1−ϵ)‖p⋆‖22+ϵ _ μ, ν=1^kp _ μG_ μ νp _ ν=(1-ε)\|p \|_2^2+ε. In this way, we have Γ~α(mod)[x⋆,p⋆,ξ]=(1−β2(1−ϵ))‖p⋆‖2−1−β2ϵ. _α^(mod)[x ,p ,ξ]= (1- β2(1-ε) )\|p \|^2-1- β2ε. (C.14) By Lemma F.1, we express the minimizer as pμ~⋆=β2(zμ~⋆−θ)p _ μ= β2(z _ μ-θ) for α=2α=2. By matching the free energy on branch a and branch b, we prove the first equation. For the well-separation and uniform distribution conditions, we have Γ~α(mod)[xa⋆,p⋆,ξ]=−1−βcoexist2×1+1=−βcoexist2. ^(mod)_α[x _a,p ,ξ]=-1- _coexist2× 1+1=- _coexist2. For branch b, Γ~α(mod)[xb⋆,p⋆,ξ]=−1−(1212)(βcoexist2−1βcoexistϵ2βcoexistϵ2βcoexist2−1)(1212)=−12−βcoexist4(1+ϵ). ^(mod)_α[x _b,p ,ξ]=-1- pmatrix 12& 12 pmatrix pmatrix _coexist2-1& _coexistε2\\ _coexistε2& _coexist2-1 pmatrix pmatrix 12\\ 12 pmatrix=- 12- _coexist4(1+ε). After equating these two energies, we obtain βcoexist=2/(1−ϵ) _coexist=2/(1-ε). Notice that when β=βcoexistβ= _coexist, all p⋆p are degenerate, i.e., they are all global minima in free energy (C.14). This proves the proposition. ∎ C.9 Proof of Proposition 4.2 Proposition C.8 (Proposition 4.2 Restated: Temperature Spinodal Point for Sparse Model). Consider the sparse case α=2α=2 and a branch a with exact support size k. Then the spinodal critical value is βjump=2λmax(PGP), _jump= 2 _max(PGP), where Pμν≔μ,ν−k−1P_μν 1_\μ,ν\-k^-1, and G is the Gram matrix. If the distribution is uniform and the system satisfies the well-separation condition, (4.3), then we have the analytic solution βjump=2/(1−ϵ) _jump=2/(1-ε). Proof. Since for α=2α=2, the minimizer satisfies p⋆=β2(zμ⋆−θ)p = β2(z _μ-θ), where μ=1,…,kμ=1,...,k, we have ∂pμ⋆∂zν⋆=β2μ,ν−β2∂zν⋆(∑ρ=1kzρ⋆−1k)=β2(μ,ν−1k)=β2Pμν. ∂ p _μ∂ z _ν= β2 1_\μ,ν\- β2 ∂ z _ν( _ρ=1^kz _ρ-1k)= β2( 1_\μ,ν\- 1k)= β2P_μν. Then the gradient of the free energy at the local stable point, x⋆x , is ∂xiΓα(mod)[x⋆,ξ]= ∂ x_i _α^(mod)[x ,ξ]= βxi⋆−∂xisupp∈ΔMβ∑μ=1kzμpμ+1−∑μ=1kpμ2|x=x⋆ \ β x _i- ∂ x_i _p∈ ^M\β _μ=1^kz_μp_μ+1- _μ=1^kp_μ^2\|_x=x = = βxi⋆−βd∑μ=1kξiμpμ⋆. \ β x _i- β d _μ=1^k _i^μp _μ. The Hessian is ∂2∂xi∂xjΓα(mod)[x⋆,ξ]= ∂^2∂ x_i∂ x_j _α^(mod)[x ,ξ]= βi,j−βd∑μ,ν=1kξiμ∂pμ⋆∂zν⋆∂zν⋆∂xj|x=x⋆ \ β 1_\i,j\- β d _μ,ν=1^k _i^μ ∂ p _μ∂ z _ν ∂ z _ν∂ x_j|_x=x = = βi,j−β22d∑μ,ν=1kξiμξjνPμν. \ β 1_\i,j\- β^22d _μ,ν=1^k _i^μ _j^νP_μν. (C.15) The smallest Hessian eigenvalue vanishes when βjump2dλmax(∑μ,ν=1kξiμPμνξjν)=1 _jump2d _max( _μ,ν=1^k _i^μP_μν _j^ν)=1. Notice that PμνP_μν is a projector, i.e., ∑ν=1kPμνPνρ=Pμρ. _ν=1^kP_μνP_νρ=P_μρ. Suppose u is the eigenvector such that ∑j=1d∑μ,ν=1kξiμPμνξjνuj=λui _j=1^d _μ,ν=1^k _i^μP_μν _j^νu_j=λ u_i. We construct Cμν≔∑ρ,σ=1kPμρ(∑i=1dξiρξiσ)Pσν=d∑ρ,σ=1kPμρGρσPσν. C_μν _ρ,σ=1^kP_μρ( _i=1^d _i^ρ _i^σ)P_σν=d _ρ,σ=1^kP_μρG_ρσP_σν. In fact, its eigenvector is vμ=∑ν=1k∑i=1dPμνξiνuiv_μ= _ν=1^k _i=1^dP_μν _i^νu_i. In this way, (Cv)μ= (Cv)_μ= d∑ν,ρ,σ=1k∑κ=1k∑i=1dPμρGρσPσνPνκξiκui \ d _ν,ρ,σ=1^k _κ=1^k _i=1^dP_μρG_ρσP_σνP_νκ _i^κu_i = = ∑ρ,σ=1k∑κ=1k∑i,j=1dPμρξjρξjσPσκξiκui \ _ρ,σ=1^k _κ=1^k _i,j=1^dP_μρ _j^ρ _j^σP_σκ _i^κu_i (By P being a projector) = = ∑ρ=1k∑j=1dPμρξjρλuj \ _ρ=1^k _j=1^dP_μρ _j^ρλ u_j (By ∑j=1d∑μ,ν=1kξiμPμνξjνuj=λui _j=1^d _μ,ν=1^k _i^μP_μν _j^νu_j=λ u_i) = = λvμ. \ λ v_μ. Hence, its eigenvalue is the same as which of ∑μ,ν=1kξiμPμνξjν _μ,ν=1^k _i^μP_μν _j^ν. That is, λmax(∑μ,ν=1kξiμPμνξjν)=d⋅λmax(∑ρ,σ=1kPμρGρσPσν). _max( _μ,ν=1^k _i^μP_μν _j^ν)=d· _max( _ρ,σ=1^kP_μρG_ρσP_σν). Finally, we obtain βjump=2dλmax(∑μ,ν=1kξiμPμνξjν)=2λmax(∑ρ,σ=1kPμρGρσPσν). _jump= 2d _max( _μ,ν=1^k _i^μP_μν _j^ν)= 2 _max( _ρ,σ=1^kP_μρG_ρσP_σν). If the system satisfies the well-separation condition, Gμν=(1−ϵ)μ,ν+ϵG_μν=(1-ε) 1_\μ,ν\+ε, with ϵ≪1ε 1. Then ∑ρ,σ=1kPμρGρσPσν= _ρ,σ=1^kP_μρG_ρσP_σν= (1−ϵ)∑ρ,σ=1kPμρρ,σPσν+ϵ∑ρ,σ=1kPμρPσν \ (1-ε) _ρ,σ=1^kP_μρ 1_\ρ,σ\P_σν+ε _ρ,σ=1^kP_μρP_σν = = (1−ϵ)Pμν \ (1-ε)P_μν (By P being a projector and P(1 1⋯ 1)⊤=0P(1 \ 1 \ · \ 1) =0) Let the vector space be I⊕I⟂I I , where I≔span(1,…,1)⊤I \(1,...,1) \. Since the eigenvalue is (1−ϵ)(1-ε) on I⟂I and zero on I, the largest eigenvalue is (1−ϵ)(1-ε). After substituting into the expression of βjump _jump, we prove the proposition. ∎ Appendix D Fluctuated Version and Thermal Attention In this section, we revisit the modern Hopfield models but with the fluctuated variables, x,p,x,p, and z. First, we study how the temperature influences the fluctuation property. Then we move on to discuss the modern Hopfield models. D.1 Physics of Temperature In this section, we discuss the relation of φ and xJx_J first. Then we introduce the role that temperature plays in the perturbative analysis. Proposition D.1 (Saddle Points as Stationary Points in the Zero-Temperature Limit (Landau and Lifshitz, 1980)). Consider the zero-temperature limit, β→∞β→∞. If (−βℋ[x]+∑i=1dJixi)(- [x]+ _i=1^dJ_ix_i) has a unique maximizer, then the saddle points approach to the stationary points, φi=(xJ)i _i=(x_J)_i. Proof. We repeat the proof from (Landau and Lifshitz, 1980; Dembo and Zeitouni, 2010) for completeness. We denote the maximizer as xJx_J. Since it is unique, for any xi≠(xJ)ix_i≠(x_J)_i, we have pJ(x)pJ(xJ)=exp−β(ℋ[x]−ℋ[xJ])+∑i=1dJi[xi−(xJ)i]≔e−βΔ+C, p_J(x)p_J(x_J)= \-β(H[x]-H[x_J])+ _i=1^dJ_i[x_i-(x_J)_i]\ e^-β +C \ , where Δ>0 >0, and C is a finite value. Then in the zero-temperature limit, pJ(x)→pJ(xJ)p_J(x)→ p_J(x_J). Thus, φi=pJ[xi]→(xJ)i _i=E_p_J[x_i]→(x_J)_i. ∎ Proposition D.2 (Fluctuation and Loop Expansion (Thouless et al., 1977)). The free energy captures noise and fluctuations. The fluctuation term behaves like a data-dependent regularizer. Specifically, if we expand x around φ , then Γ[φ]= [ ]= βℋ[φ]−d2log(2π)+12logdet(β∇x2ℋ[φ])+O(β−1) \ [ ]- d2 (2π)+ 12 (β∇^2_xH[ ])+O(β^-1) (D.1) ≔ Γtree[φ]+Γ1-loop[φ]+Γ2-loop[φ]+⋯. \ _tree[ ]+ _1-loop[ ]+ _2-loop[ ]+· \ . (D.2) This free energy admits higher-order (higher-loop) corrections arising from fluctuations, which we denote as O(β−1)O(β^-1). Specifically, the two-loop is of order β−1β^-1, three-loop is of order β−2β^-2, and so forth. Conventionally, we ignore the constant term, dlog(2π)/2d (2π)/2, since it is independent of φ and does not affect the dynamics. Proof. We include the proof for completeness, adapting the standard argument from (Thouless et al., 1977; Zinn-Justin, 2002). By Definition 2.3, we write exp−Γ[φ] \- [ ]\ =expW[J]−∑i=1dJiφi = \ \W[J]- _i=1^dJ_i _i\ =∫exp−βℋ[x]+∑i=1dJi(xi−φi)dμ(x). = \ _X \- [x]+ _i=1^dJ_i(x_i- _i)\ \ μ(x) \ . Then we expand the right hand side around the stationary point, φ , perturbatively: exp−Γ[φ]= \- [ ]\= ∫exp−βℋ[φ]−β∑i=1d∂ℋ∂xi|x=φ(xi−φi) \ _X \- [ ]-β _i=1^d ∂ x_i|_x= (x_i- _i) −β2∑i,j=1d∂2ℋ∂xi∂xj|x=φ(xi−φi)(xj−φj) \ - β2 _i,j=1^d ∂^2H∂ x_i∂ x_j|_x= (x_i- _i)(x_j- _j) −β3!∑i,j,k=1d∂3ℋ∂xi∂xj∂xk|x=φ(xi−φi)(xj−φj)(xk−φk) \ - β3! _i,j,k=1^d ∂^3H∂ x_i∂ x_j∂ x_k|_x= (x_i- _i)(x_j- _j)(x_k- _k) +O(∥x−φ∥24)exp∑i=1dJi(xi−φi)dμ() \ +O(\|x- \|_2^4)\ \ _i=1^dJ_i(x_i- _i)\ \ μ(x) = = exp−βℋ[φ]∫exp−β2∑i,j=1d(xi−φi)(∇x2ℋ[φ])ij(xj−φj)⋅(1−β3!× \ \- [ ]\ _X \- β2 _i,j=1^d(x_i- _i)(∇^2_xH[ ])_ij(x_j- _j)\·(1- β3!× OPEN∑k,ℓ,m=1d∂3ℋ∂xk∂xℓ∂xm|x=φ(xk−φk)(xℓ−φℓ)(xm−φm)+O(β‖x−φ‖24))dμ(x) \ _k, ,m=1^d ∂^3H∂ x_k∂ x_ ∂ x_m|_x= (x_k- _k)(x_ - _ )(x_m- _m)+O(β\|x- \|_2^4)) \ μ(x) (By Definition 2.2) where ‖x′−φ‖22≔∑i=1d(xi′−φi)2\|x - \|_2^2 _i=1^d(x _i- _i)^2. Firstly, ∫exp−β2∑i,j=1d(xi−φi)(∇x2ℋ[φ])ij(xj−φj)dμ(x)=(2π)d/2(det(β∇x2ℋ[φ]))−1/2. _X \- β2 _i,j=1^d(x_i- _i)(∇^2_xH[ ])_ij(x_j- _j)\ \ μ(x)=(2π)^d/2( (β∇^2_xH[ ]))^-1/2 \ . (By Gaussian Formula) Since the odd function vanishes after the integration, the first leading correction is of O(‖x−φ‖24)O(\|x- \|^4_2). We focus on this integration: ∫exp−β2∑i,j=1d(xi−φi)(∇x2ℋ[φ])ij(xj−φj)O(β∥x−φ∥24)dμ(x) \ _X \- β2 _i,j=1^d(x_i- _i)(∇^2_xH[ ])_ij(x_j- _j)\O(β\|x- \|_2^4) \ μ(x) = = ∫exp−12∑i,j=1d(∇x2ℋ[φ])i,jyiyjO(β∥y∥24)⋅(β−1/2)4dμ(y) \ _Y \- 12 _i,j=1^d(∇^2_xH[ ])_i,jy_iy_j\O(β\|y\|_2^4)·(β^-1/2)^4 \ μ(y) (By x→y=β(x−φ)x→ y= β(x- )) = = O(β−1). \ O(β^-1) \ . Putting everything together, we obtain exp−Γ[φ]=exp−βℋ[φ]−12logdet(β∇x2ℋ[φ])+d2log(2π)+O(β−1). \- [ ]\= \- [ ]- 12 (β∇^2_xH[ ])+ d2 (2π)+O(β^-1)\ \ . This proves the proposition. ∎ Proposition D.3 (Low-Temperature Limit and Laplace Approximation (Zinn-Justin, 2002; Dembo and Zeitouni, 2010)). If (−βℋ[x]+∑i=1dJixi)(- [x]+ _i=1^dJ_ix_i) has unique maximizer xJx_J with positive-definite Hessian, then for continuous cases, 1βlog∫exp−βℋ[x]+∑i=1dJixidμ(x)=supx∈−ℋ[x]+1β∑i=1dJixi+O(logβ). 1β _X \- [x]+ _i=1^dJ_ix_i\ \ μ(x)= _x \ \-H[x]+ 1β _i=1^dJ_ix_i\+O( β) \ . For discrete cases, 1βlog∑x∈exp−βℋ[x]+∑i=1Jixi=supx∈−ℋ[x]+1β∑i=1dJixi+O(log||β). 1β _x \- [x]+ _i=1J_ix_i\= _x \ \-H[x]+ 1β _i=1^dJ_ix_i\+O( |X|β) \ . (D.3) Proof. We repeat the proof from (Zinn-Justin, 2002; Dembo and Zeitouni, 2010) for completeness. We start from the continuous case. Since the maximizer is unique, we Taylor expand up to the first leading order as −βℋ[x]+∑i=1dJixi= - [x]+ _i=1^dJ_ix_i= −βℋ[φ]+∑i=1dJiφi−12β[xi−φi](∇x2ℋ[φ])ij[xj−φj]. - [ ]+ _i=1^dJ_i _i- 12β[x_i- _i](∇^2_xH[ ])_ij[x_j- _j] \ . (By Definition 2.2) This yields 1βlog∫=ℝdexp−βℋ[x]+∑i=1dJixidμ(x) \ 1β _X=R^d \- [x]+ _i=1^dJ_ix_i\ \ μ(x) = = 1βlog(exp−βℋ[φ]+∑i=1dJiφi)+1βlog((2πβ)d/2(det(β∇x2ℋ[φ]))−1/2) \ 1β ( \- [ ]+ _i=1^dJ_i _i\)+ 1β (( 2πβ)^d/2( (β∇^2_xH[ ]))^-1/2) = = −ℋ[φ]+1β∑i=1dJiφi+(logβ). \ -H[ ]+ 1β _i=1^dJ_i _i+O( β) \ . This proves the first equation. For discrete cases, maxx∈(−βℋ[x]+∑i=1dJixi)≤ _x \ (- [x]+ _i=1^dJ_ix_i)≤ log∑x∈exp−βℋ[x]+∑i=1dJixi \ _x \- [x]+ _i=1^dJ_ix_i\ ≤ ≤ maxx∈(−βℋ[x]+∑i=1dJixi)+log||. \ max_x \ (- [x]+ _i=1^dJ_ix_i)+ |X| \ . After plugging this into (D.3), we complete the proof. ∎ Remark D.1. In the zero-temperature limit, Proposition D.3 shows that W[J]/βW[J]/β is a Fenchel conjugate of the Hamiltonian by Definition C.6. Remark D.2. From the proof in Proposition D.2 and Proposition D.3, we notice that the perturbation only works in the low-temperature regime. When the temperature increases, the strength of fluctuations increases and the higher-loop corrections starts to grow. There exists a level where the perturbative method breaks down. D.2 Fluctuated Variables in Modern Hopfield Model In the following, we start to consider the fluctuated version of MHMs. Definition D.1 (Statistics-MHM (Fluctuated) Mapping). The mapping between the variables in statistical mechanics and which in the fluctuated version of MHMs is: • Microscopic State x∈ℝdx ^d: It represents continuous query. It is also the dynamical variable whose energy landscape defines retrieval basins. • External Source J∈ℝdJ ^d: It represents an external bias of the query. • Classical Solution φ∈ℝd ^d: It represents macroscopic state. Retrieval corresponds to φi _i aligning with some pattern ξiμξ^μ_i. • Probability Function p∈ΔMp∈ ^M: It represents the retrieval distribution over memories. • Tsallis Entropy Ψα _α: It represents a regularizer controlling how sparse the memory weights p are. • Conjugate Field z∈ℝMz ^M: It represents the query-key similarity between current state and each memory, i.e., zμ≔∑i=1dξiμxi/dz_μ _i=1^d _i^μx_i/ d. Notice that we distinguish the microscopic state, x, from the macroscopic one, φ , and so are p and z in this case. Proposition D.4 (Expectation Values and Macroscopic States). Let X be a finite-volume domain in ℝdR^d and J∈domWαJ _α. The macroscopic states are determined by the mean of microscopic states. Specifically, x∼ρJ(⋅|p)(α)[xi|p]=1d∑μ=1Mξiμpμ+1βJi+Bi(p,J)≔mi(p,J)+Bi(p,J), \ E^(α)_x _J(·|p)[x_i|p]= 1 d _μ=1^M _i^μp_μ+ 1βJ_i+B^X_i(p,J) m_i(p,J)+B^X_i(p,J) \ , (D.4) (x,p)∼ρJ(α)[xi]=φi=1d∑μ=1Mξiμ(x,p)∼ρJ(α)[pμ]+1βJi+(x,p)∼ρJ(α)[Bi(p,J)], \ E^(α)_(x,p) _J[x_i]= _i= 1 d _μ=1^M _i^μE^(α)_(x,p) _J[p_μ]+ 1βJ_i+E^(α)_(x,p) _J[B_i^X(p,J)] \ , (D.5) where ρJ(x,p) _J(x,p) is the probability density function in ×ΔMX× ^M, pJ(x)p_J(x) is which in X, and pΔ,J(p)p_ ,J(p) is which in ΔM ^M. Furthermore, there is a boundary correction Bi(p,J)≔∫yiexpα−β2(‖y‖22−‖m‖22)dμ(y)∫expα−β2(‖y‖22−‖m‖22)dμ(y),with =−m. B_i^X(p,J) _Yy_i ^α_α\- β2(\|y\|_2^2-\|m\|^2_2)\ \ μ(y) _Y ^α_α\- β2(\|y\|_2^2-\|m\|^2_2)\ \ μ(y), \ Y=X-m. Proof. We consider x and p follow the distribution ρJ(x,p) _J(x,p) with the joint energy function, (3.1). Then x∼ρJ(⋅|p)(α)[xi|p]= ^(α)_x _J(·|p)[x_i|p]= ∫xiρJα(x|p)dμ(x)∫ρJα(x|p)dμ(x) \ _Xx_iρ^α_J(x|p) \ μ(x) _X _J^α(x|p) \ μ(x) = = ∫xiexpα−β2∑j=1dxj2+βd∑μ=1M∑j=1dξjμxjpμ+∑j=1dJjxjdμ(x)∫expα−β2∑j=1dxj2+βd∑μ=1M∑j=1dξjμxjpμ+∑j=1dJjxjdμ(x) \ _Xx_i _α^α\- β2 _j=1^dx_j^2+ β d _μ=1^M _j=1^d _j^μx_jp_μ+ _j=1^dJ_jx_j\ \ μ(x) _X _α^α\- β2 _j=1^dx_j^2+ β d _μ=1^M _j=1^d _j^μx_jp_μ+ _j=1^dJ_jx_j\ \ μ(x) = = ∫(yi+mi)expα−β2∑j=1dyj2+β2∑j=1dmj2dμ(y)∫expα−β2∑j=1dyj2+β2∑j=1dmj2dμ(y) \ _Y(y_i+m_i) ^α_α\ -β2 _j=1^dy_j^2+ β2 _j=1^dm_j^2\ \ μ(y) _Y ^α_α\ -β2 _j=1^dy_j^2+ β2 _j=1^dm_j^2\ \ μ(y) = = mi(p,J)+Bi(p,J). \ m_i(p,J)+B_i^X(p,J). This proves the first equation. In this way, (x,p)∼ρJ(α)[xi]= ^(α)_(x,p) _J[x_i]= ∫ΔM∫xiρJα(x,p)dμ(x)dμ(p)∫ΔM∫ρJα(x,p)dμ(x)dμ(p) \ _ ^M _Xx_iρ^α_J(x,p) \ μ(x) μ(p) _ ^M _X _J^α(x,p) \ μ(x) μ(p) = = ∫ΔM(∫xiρJα(x|p)dμ(x))pΔ,Jα(p)dμ(p)∫ΔM(∫ρJα(x|p)dμ(x))pΔ,Jα(p)dμ(p) \ _ ^M( _Xx_iρ^α_J(x|p) \ μ(x))p^α_ ,J(p) \ μ(p) _ ^M( _Xρ^α_J(x|p) \ μ(x))p^α_ ,J(p) \ μ(p) (By pΔ,J(p)ρJ(x|p)≔ρJ(x,p)p_ ,J(p) _J(x|p) _J(x,p)) = = ∫ΔM(∫ρJα(x′|p)dμ(x′)x∼ρJ(⋅|p)(α)[xi|p])pΔ,Jα(p)dμ(p)∫ΔM(∫ρJα(x|p)dμ(x))pΔ,Jα(p)dμ(p) \ _ ^M( _X _J^α(x |p) \ μ(x )E_x _J(·|p)^(α)[x_i|p])p_ ,J^α(p) \ μ(p) _ ^M( _Xρ^α_J(x|p) \ μ(x))p^α_ ,J(p) \ μ(p) = = 1d∑μ=1Mξiμ~p∼pΔ,J(α)[pμ]+1βJi+~p∼pΔ,J(α)[Bi(p,J)], \ 1 d _μ=1^M _i^μ E_p p_ ,J^(α)[p_μ]+ 1βJ_i+ E_p p_ ,J^(α)[B_i^X(p,J)], where C(p)≔∫ρJα(x|p)dμ(x),and ~p∼pΔ,J(α)[pμ]≔(x,p)∼ρJ(α)[pμ]=∫ΔMC(p)pμpΔ,Jα(p)dμ(p)∫ΔMC(p)pΔ,Jα(p)dμ(p). C(p) _X _J^α(x|p) \ μ(x), \ E_p p_ ,J^(α)[p_μ] ^(α)_(x,p) _J[p_μ]= _ ^MC(p)p_μp^α_ ,J(p) \ μ(p) _ ^MC(p)p^α_ ,J(p) \ μ(p). This completes the proof. ∎ Remark D.3. The correction BiB_i^X vanishes on translation-invariant domains (e.g., ℝdR^d if the integral is normalizable). Furthermore, it is negligible in the large-volume or low-temperature regime when the dominant integrand is localized far from the boundary of X. D.3 Connection between Free Energy and Tsallis Entropy in Fluctuated Case In Section 2.2, we relate the log-partition function to the Tsallis entropy via (2.4). Here, we express the log-partition function using the variables in fluctuated version of MHMs, Definition D.1. To make Zα[J]Z_α[J] finite, we assume that the integration region of x is finite volume, i.e., ⊂ℝdX ^d such that ||<+∞|X|<+∞. Proposition D.5. Let ρJ(x,p) _J(x,p) and pJ(x)p_J(x) be the probability density function in ×ΔMX× ^M and X, respectively. Inspired by Theorem 2.1, we have −Wα[J,ξ]= -W_α[J,ξ]= β2∑i=1d(x,p)∼ρJ(α)[xi2]−β∑μ=1M(x,p)∼ρJ(α)[zμpμ] \ β2 _i=1^dE^(α)_(x,p) _J[x_i^2]-β _μ=1^ME^(α)_(x,p) _J[z_μp_μ] −Ψα[ρJ]Zα−1[J,ξ]∫ΔMρJα(x,p)dμ(p)dμ(x)−∑i=1dJi(x,p)∼ρJ(α)[xi]. \ - _α[ _J]Z_α^α-1[J,ξ] _X _ ^M _J^α(x,p) \ μ(p) μ(x)- _i=1^dJ_iE^(α)_(x,p) _J[x_i]. (D.6) Then the free energy is Γ~α= _α= β2∑i=1d(x,p)∼ρJ⋆(α)[xi2]−β∑μ=1M(x,p)∼ρJ⋆(α)[zμpμ] \ β2 _i=1^dE_(x,p) _J ^(α)[x_i^2]-β _μ=1^ME_(x,p) _J ^(α)[z_μp_μ] −Ψα[ρJ⋆]Zα−1[J⋆,ξ]∫ΔMρJ⋆α(x,p)dμ(p)dμ(x), \ - _α[ _J ]Z_α^α-1[J ,ξ] _X _ ^M _J ^α(x,p) \ μ(p) μ(x), (D.7) where J⋆J is the maximizing source, i.e., J⋆=argmaxJ∑i=1dJiφi−Wα[J,ξ]J = _J\ _i=1^dJ_i _i-W_α[J,ξ]\, and φ=(x,p)∼ρJ(α)[x] =E^(α)_(x,p) _J[x]. Proof. Following the proof of Theorem 3.1, we have (C.11). This is exactly the first equation we would like to prove. For the free energy, we consider the Fenchel conjugation Definition C.6 of the log-partition function. Γ~α= _α= supJ∈domWα∑i=1dJiφi−Wα[J,ξ] \ _J _α\ _i=1^dJ_i _i-W_α[J,ξ]\ = = supJ∈domWα∑i=1dJi(φi−(x,p)∼ρJ(α)[xi])+β2∑i=1d(x,p)∼ρJ(α)[xi2]−β∑μ=1M(x,p)∼ρJ(α)[zμpμ] \ _J _α \ _i=1^dJ_i( _i-E_(x,p) _J^(α)[x_i])+ β2 _i=1^dE_(x,p) _J^(α)[x_i^2]-β _μ=1^ME_(x,p) _J^(α)[z_μp_μ] −Ψα[ρJ]Zα−1[J,ξ]∫ΔMρJα(x,p)dμ(p)dμ(x) \ - _α[ _J]Z_α^α-1[J,ξ] _X _ ^M _J^α(x,p) \ μ(p) μ(x) \ = = β2∑i=1d(x,p)∼ρJ⋆(α)[xi2]−β∑μ=1M(x,p)∼ρJ⋆(α)[zμpμ] \ β2 _i=1^dE_(x,p) _J ^(α)[x_i^2]-β _μ=1^ME_(x,p) _J ^(α)[z_μp_μ] −Ψα[ρJ⋆]Zα−1[J⋆,ξ]∫ΔMρJ⋆α(x,p)dμ(p)dμ(x), \ - _α[ _J ]Z_α^α-1[J ,ξ] _X _ ^M _J ^α(x,p) \ μ(p) μ(x), where φ=(x,p)∼ρJ(α)[x] =E^(α)_(x,p) _J[x]. This completes the proof. ∎ Proposition D.6 (MHMs in the Zero-Temperature Limit). In the zero-temperature limit, β→∞β→∞, minimizing the free energy becomes minimizing the following quantity: ℱ[φ,ξ]=12∑i=1dφi2−maxμ @depthΔ @set@skewchar @nested@a111zμ, [ ,ξ]= 12 _i=1^d _i^2- _μ @depth 1 @set@skewchar @nested@a 111z_μ \ , (D.8) where @depthΔ @set@skewchar @nested@a111zμ≔∑i=1dξiμφi/d @depth 1 @set@skewchar @nested@a 111z_μ _i=1^d _i^μ _i/ d. Notably, this is the free energy in the fixed case, (3.4), up to a scaling , β. Proof. In the zero-temperature limit, the Gibbs probability density function concentrates to an indicator, i.e., ρJ(x,p)→x,(x,p)∼ρJ(α)[x]p,(x,p)∼ρJ(α)[p], _J(x,p)→ 1_\x,E^(α)_(x,p) _J[x]\ 1_\p,E^(α)_(x,p) _J[p]\ \ , as β→∞β→∞. Hence, (x,p)∼ρJ(α)[x2]→((x,p)∼ρJ(α)[x])2=(~x∼pJ(α)[x])2=φ2, ^(α)_(x,p) _J[x^2]→(E^(α)_(x,p) _J[x])^2=( E_x p_J^(α)[x])^2= ^2, and (x,p)∼ρJ[zμpμ]→1d∑i=1dξiμ(x,p)∼ρJ(α)[xi](x,p)∼ρJ(α)[pμ]=1d∑i=1dξiμφi~p∼pΔ,J(α)[pμ]≔ @depthΔ @set@skewchar @nested@a111zμ @depthΔ @set@skewchar @nested@a111pμ, _(x,p) _J[z_μp_μ]→ 1 d _i=1^d _i^μE^(α)_(x,p) _J[x_i]E_(x,p) _J^(α)[p_μ]= 1 d _i=1^d _i^μ _i E^(α)_p p_ ,J[p_μ] @depth 1 @set@skewchar @nested@a 111z_μ @depth 1 @set@skewchar @nested@a 111p_μ, where ~x∼pJ(α)[x] E^(α)_x p_J[x] and ~p∼pΔ,J(α)[p] E_p p_ ,J^(α)[p] are defined in (C.12) and (C.13), respectively. The free energy (D.7) becomes Γα→β2∑i=1dφi2−β∑μ=1M @depthΔ @set@skewchar @nested@a111zμ @depthΔ @set@skewchar @nested@a111pμ−Ψα[]. _α→ β2 _i=1^d _i^2-β _μ=1^M @depth 1 @set@skewchar @nested@a 111z_μ @depth 1 @set@skewchar @nested@a 111p_μ- _α[ 1] \ . Since Ψα[]=0 _α[ 1]=0, we have inf @depthΔ @set@skewchar @nested@a111p∈ΔM−∑μ=1M @depthΔ @set@skewchar @nested@a111zμ @depthΔ @set@skewchar @nested@a111pμ=−maxμ @depthΔ @set@skewchar @nested@a111zμ. _ @depth 1 @set@skewchar @nested@a 111p∈ ^M\- _μ=1^M @depth 1 @set@skewchar @nested@a 111z_μ @depth 1 @set@skewchar @nested@a 111p_μ\=- _μ @depth 1 @set@skewchar @nested@a 111z_μ \ . Substituting it into the free energy to eliminate the dependence of @depthΔ @set@skewchar @nested@a111pμ @depth 1 @set@skewchar @nested@a 111p_μ, we prove this proposition. ∎ Remark D.4. Proposition D.6 is consistent with Proposition D.2 in the zero-temperature limit. Specifically, Γα[φ]→(βℋ[φ]+constant) _α[ ]→( [ ]+constant) as β→∞β→∞. Following the proof of Theorem 3.1, we conclude the same result. D.4 Retrieval Dynamics in Fluctuated Case and Thermal Attention In Section 2, we introduce the equation of motions according to Proposition 2.1 and investigate the attention mechanism from the trajectories in associative memory systems. In this section, we consider the fluctuated version, which gives an alternative thermal attention mechanism. Let W~α≔log∫ΔMαexpα−βℋ~MHM+∑i=1dJixidμ(p) W_α _α _ ^M _α\-β H_MHM+ _i=1^dJ_ix_i\ \ μ(p). Notice that in the fixed setting, it reduces to the simpler form, (3.7). Then, Proposition D.7 (EOMs of Free Energies in MHMs). The EOM of the log-partition function in the x-sector corresponds to the Hopfield retrieval ∇JWα[J,ξ]=∫ΔMxiρJα(x,p)dμ(p)dμ(x). _JW_α[J,ξ]= _X _ ^Mx_i _J^α(x,p) \ μ(p) μ(x) \ . (D.9) The EOM of the log-partition function in the p-sector corresponds to the unnormalized escort moment (Naudts, 2004) ∇zW~α[z,J,ξ]=∫ΔMβpρJα(p|x)dμ(p)≔βp(esc). _z W_α[z,J,ξ]= _ ^Mβ p _J^α(p|x) \ μ(p) β p^(esc) \ . (D.10) Proof. By Definition 2.3, Wα[J,ξ]=logαZα[J,ξ]W_α[J,ξ]= _αZ_α[J,ξ]. After taking the derivative with respect to J, we obtain ∂JiWα[J,ξ]= ∂ J_iW_α[J,ξ]= ∂JilogαZα[J,ξ] \ ∂ J_i _αZ_α[J,ξ] = = Zα−α[J,ξ]∂Zα[J,ξ]∂Ji \ Z_α^-α[J,ξ] ∂ Z_α[J,ξ]∂ J_i (By Lemma C.1) = = Zα−α[J,ξ]∂Ji∫ΔMexpα−βℋ~MHM[x,p,ξ]+∑j=1dJjxjdμ(p)dμ(x) \ Z_α^-α[J,ξ] ∂ J_i _X _ ^M _α\-β H_MHM[x,p,ξ]+ _j=1^dJ_jx_j\ \ μ(p) μ(x) = = Zα−α[J,ξ]∫ΔMexpα−βℋ~MHM[x,p,ξ]+∑j=1dJjxj \ Z_α^-α[J,ξ] _X _ ^M ^α_α\-β H_MHM[x,p,ξ]+ _j=1^dJ_jx_j\ (By Lemma C.1) ×∂Ji(−βℋ~MHM[x,p,ξ]+∑k=1dJkxk)dμ(p)dμ(x) \ × ∂ J_i(-β H_MHM[x,p,ξ]+ _k=1^dJ_kx_k) \ μ(p) μ(x) = = ∫ΔMxiexpα−βℋ~MHM[x,p,ξ]+∑j=1dJjxjZα[J,ξ]dμ(p)dμ(x) \ _X _ ^Mx_i \ _α^α\-β H_MHM[x,p,ξ]+ _j=1^dJ_jx_j\Z_α^α[J,ξ] \ μ(p) μ(x) = = ∫ΔMxiρJα(x,p)dμ(p)dμ(x). \ _X _ ^Mx_i _J^α(x,p) \ μ(p) μ(x). For the second equation, we substitute the definition of W~α W_α directly: ∂zμW~α[z,J,ξ]= ∂ z_μ W_α[z,J,ξ]= ∂zμlogα∫ΔMexpα−β2∑i=1dxi2+β∑μ=1Mzμpμ+∑i=1dJixidμ(p) \ ∂ z_μ _α _ ^M _α\- β2 _i=1^dx_i^2+β _μ=1^Mz_μp_μ+ _i=1^dJ_ix_i\ \ μ(p) = = (∫ΔMexpα−β2∑i=1dxi2+β∑μ=1Mzμpμ+∑i=1dJixidμ(p))−α \ ( _ ^M _α\- β2 _i=1^dx_i^2+β _μ=1^Mz_μp_μ+ _i=1^dJ_ix_i\ \ μ(p) )^-α ×∫ΔMβpμexpα−β2∑i=1dxi2+β∑μ=1Mzμpμ+∑i=1dJixidμ(p) \ × _ ^Mβ p_μ ^α_α\- β2 _i=1^dx_i^2+β _μ=1^Mz_μp_μ+ _i=1^dJ_ix_i\ \ μ(p) (By Lemma C.1) = = ∫ΔMβpμρJα(p|x)dμ(p). \ _ ^Mβ p_μ _J^α(p|x) \ μ(p) \ . This completes the proof. ∎ Remark D.5. If α→1α→ 1, then everything simplifies to the ordinary Gibbs case. In this way, the Hopfield retrieval becomes ∇JWα→1[J,ξ]= _JW_α→ 1[J,ξ]= ∫ΔMxρJ(x,p)dμ(p)dμ(x) \ _X _ ^Mx _J(x,p) \ μ(p) μ(x) = = (x,p)∼ρJ[x] \ E_(x,p) _J[x] = = ∫xpJ(x)dμ(x) \ _Xxp_J(x) \ μ(x) = = x∼pJ[x]. \ E_x p_J[x]. Furthermore, the escort moment becomes the conditional expectation value of the probability function for general β. ∇zW~α→1[z,J,ξ]=βp∼ρJ(⋅|x)[p]. _z W_α→ 1[z,J,ξ]= _p _J(·|x)[p] \ . Remark D.6. If we do the similar analysis of Proposition 3.3 for the fluctuated case, then we have the thermal attention mechanism. In particular, if α→1α→ 1, then the learning trajectory follows the ordinary Gibbs expectation of the conditional distribution function, i.e., ∇zW~α→1=βp∼ρJ(⋅|x)[p] _z W_α→ 1= _p _J(·|x)[p], due to Remark D.5. If α>1α>1, then the system amplifies high-density regions. This leads to a sharper attention mechanism. If 1>α>01>α>0, then it corresponds to a softer attention mechanism. Remark D.7. From (D.9), we obtain a macroscopic retrieval state depending on the external field at microscopic queries, xi\x_i\. It is similar to a spin system in an external magnetic field. After turning on the magnetic field, the probability distribution of spins starts to deform and yields an expectation value [Si]E[S_i], where SiS_i is the notation of a spin at site i. We call it a Landau response. Hence, the Hopfield retrieval is a Landau response in the x-sector (J-space). For the attention mechanism, it is an inner Landau response in the p-sector (z-space). In this way, we view the query-to-key score, zμ=∑i=1dξiμxi/dz_μ= _i=1^d _i^μx_i/ d, as an external field at pμ\p_μ\ in ΔM ^M space. This formalism gives a physical interpretation of the retrieval dynamics. Lemma D.1 (Linear Function and its Minimizer on Simplex (Rockafellar, 1997)). Consider a function h(p)h(p) that is linear in p, i.e., h(p)=∑μ=1Maμpμ+b, h(p)= _μ=1^Ma_μp_μ+b, for some constant aμ∈ℝMa_μ ^M. Then h(p)h(p) cannot have a unique minimizer p⋆p in the interior of the simplex. Proof. This is a standard convex analysis (Rockafellar, 1997). We repeat the proof for the completeness. Assume that a minimizer lies in the interior of ΔM ^M. Then p⋆>0p >0. Since p⋆p is a minimizer, we have dεh(p⋆+εv)|ε=0=0, dd h(p + v)|_ =0=0, for all tangent vectors v∈Tp⋆ΔM=v∈ℝM|∑μ=1Mvμ=0≃ℝM−1v∈ T_p ^M=\v ^M| _μ=1^Mv_μ=0\ ^M-1. Then, after expanding the linear function h, it yields 0=dε(∑μ=1Maμpμ⋆+ε∑μ=1Maμvμ+b)|ε=0=∑μ=1Maμvμ. 0= dd ( _μ=1^Ma_μp _μ+ _μ=1^Ma_μv_μ+b)|_ =0= _μ=1^Ma_μv_μ. Since the tangent vector satisfies ∑μ=1Mvμ=0 _μ=1^Mv_μ=0, we conclude that aμ=λa_μ=λ for all μ and some constant λ. Hence, h(p)=λ∑μ=1Mpμ+b=λ+b, h(p)=λ _μ=1^Mp_μ+b=λ+b, which is constant on ΔM ^M. This contradicts the assumption that the minimizer p⋆p is unique. This proves the lemma. ∎ Lemma D.2 (Normalization Factor in the Zero-Temperature Limit). Consider the zero-temperature limit β→∞β→∞ and α>1α>1. Assume that ℋ~MHM≥0 H_MHM≥ 0, and ρJ(p|x) _J(p|x) has a unique maximizer p⋆p such that ℋ~MHM[x,p⋆,ξ]=0 H_MHM[x,p ,ξ]=0. If the p is continuous on the simplex ΔM ^M, where M>1M>1, then the normalization function has the following limit limβ→∞C(x)=limβ→∞∫ΔMρJα(p|x)dμ(p)=∞. _β→∞C(x)= _β→∞ _ ^M _J^α(p|x) \ μ(p)=∞. If α→1α→ 1 for general β or if p is discrete on the simplex ΔM ^M for general α, then limα→1C(x)=limβ→∞Cdis(x)=limβ→∞∑p∈ΔM(ρJdis)α(p|x)=1. _α→ 1C(x)= _β→∞C^dis(x)= _β→∞ _p∈ ^M( _J^dis)^α(p|x)=1. Proof. We first consider the continuous case. Since ℋ~MHM H_MHM is linear in p, its unique minimizer p⋆p must be at a vertex of ΔM ^M according to Lemma D.1. Hence, we denote pμ⋆=1,if μ=μ⋆0,if μ≠μ⋆. p _μ= cases1, &if \ μ=μ \\ 0, &if \ μ≠μ cases. (D.11) In this way, for p≠p⋆p≠ p , we have ℋ~MHM[x,p,ξ]= H_MHM[x,p,ξ]= ℋ~MHM[x,p,ξ]−ℋ~MHM[x,p⋆,ξ] \ H_MHM[x,p,ξ]- H_MHM[x,p ,ξ] (By ℋ~MHM[x,p⋆,ξ]=0 H_MHM[x,p ,ξ]=0) = = 12∑i=1dxi2+∑μ=1Mzμpμ−(12∑i=1dxi2+∑μ=1Mzμpμ⋆) \ 12 _i=1^dx_i^2+ _μ=1^Mz_μp_μ-( 12 _i=1^dx_i^2+ _μ=1^Mz_μp _μ) = = ∑μ=1Mzμ(pμ−pμ⋆) \ _μ=1^Mz_μ(p_μ-p _μ) = = ∑μ=μ⋆zμ(pμ−pμ⋆)+∑μ=1μ≠μ⋆Mzμ(pμ−pμ⋆) \ _μ=μ z_μ(p_μ-p _μ)+ _ subarraycμ=1\\ μ≠μ subarray^Mz_μ(p_μ-p _μ) = = zμ⋆pμ⋆−zμ⋆pμ⋆+∑μ=1μ≠μ⋆Mzμ(pμ−pμ⋆) \ z_μ p_μ -z_μ p _μ + _ subarraycμ=1\\ μ≠μ subarray^Mz_μ(p_μ-p _μ) = = zμ⋆pμ⋆−zμ⋆+∑μ=1μ≠μ⋆Mzμpμ \ z_μ p_μ -z_μ + _ subarraycμ=1\\ μ≠μ subarray^Mz_μp_μ (By (D.11)) = = ∑μ=1μ≠μ⋆M(zμ−zμ⋆)pμ. \ _ subarraycμ=1\\ μ≠μ subarray^M(z_μ-z_μ )p_μ. (By 1=pμ⋆+∑μ≠μ⋆Mpμ1=p_μ + _μ≠μ ^Mp_μ) Since p⋆p is a unique minimizer of ℋ~MHM H_MHM, (zμ−zμ⋆)>0(z_μ-z_μ )>0 for all μ≠μ⋆μ≠μ . From Definition C.9, we obtain expα−βℋ~MHM[x,p,ξ]+∑i=1dJixi= _α\-β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i\= 1+(α−1)(βℋ~MHM[x,p,ξ]−∑i=1dJixi)1/(1−α) \ \1+(α-1)(β H_MHM[x,p,ξ]- _i=1^dJ_ix_i) \^1/(1-α) = = 1+(α−1)β∑μ≠μ⋆M(zμ−zμ⋆)pμ−(α−1)∑i=1dJixi1/(1−α) \ \1+(α-1)β _μ≠μ ^M(z_μ-z_μ )p_μ-(α-1) _i=1^dJ_ix_i \^1/(1-α) for α>1α>1. Then C(x)= C(x)= ∫ΔMexpα−βℋ~MHM[x,p,ξ]+∑i=1dJixidμ(p)(∫ΔMexpα−βℋ~MHM[x,p′,ξ]+∑i=1dJixidμ(p′))α \ _ ^M ^α_α\-β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i\ \ μ(p)( _ ^M _α\-β H_MHM[x,p ,ξ]+ _i=1^dJ_ix_i\ \ μ(p ))^α = = ∫ΔM[1−(α−1)∑i=1dJixi]+(α−1)β∑μ≠μ⋆M(zμ−zμ⋆)α/(1−α)dμ(p)(∫ΔM[1−(α−1)∑i=1dJixi]+(α−1)β∑μ≠μ⋆M(zμ−zμ⋆)1/(1−α)dμ(p′))α \ _ ^M \[1-(α-1) _i=1^dJ_ix_i]+(α-1)β _μ≠μ ^M(z_μ-z_μ ) \^α/(1-α) \ μ(p) ( _ ^M \[1-(α-1) _i=1^dJ_ix_i]+(α-1)β _μ≠μ ^M(z_μ-z_μ ) \^1/(1-α) \ μ(p ) )^α Since the integrands only depend on μ≠μ⋆μ≠μ , we change the variable to uμ=βqμu_μ=β q_μ, where qμ≔pμq_μ p_μ with μ≠μ⋆μ≠μ . There are (M−1)(M-1) such coordinates, so dμ(p)∼β−(M−1)dμ(u) μ(p) β^-(M-1) μ(u). Then, for α>1α>1 and M>1M>1, we have C(x)∼β−(M−1)(β−(M−1))α=β(α−1)(M−1)→∞ C(x) β^-(M-1) (β^-(M-1) )^α=β^(α-1)(M-1)→∞ as β→∞β→∞. For α→1α→ 1, it is simply C(x)=∫ΔMρJ(p|x)dμ(p)=1C(x)= _ ^M _J(p|x) \ μ(p)=1 for any β. For the discrete case, the conditional distribution is ρJdis(p|x)=expα−βℋ~MHM[x,p,ξ]+∑i=1dJixi∑p′∈ΔMexpα−βℋ~MHM[x,p′,ξ]+∑i=1dJixi, ρ^dis_J(p|x)= _α\-β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i\ _p ∈ ^M _α\-β H_MHM[x,p ,ξ]+ _i=1^dJ_ix_i\, with discrete p. Notice that as β→∞β→∞, expα−βℋ~MHM[x,p,ξ] _α\-β H_MHM[x,p,ξ]\ is suppressed to zero except for the maximizer p⋆p , i.e., expα−βℋ~MHM[x,p⋆,ξ]=1 _α\-β H_MHM[x,p ,ξ]\=1. Hence, as β→∞β→∞, the denominator of ρJdis(p|x)ρ^dis_J(p|x) approaches to limβ→∞∑p′∈ΔMexpα−βℋ~MHM[x,p′,ξ]+∑i=1dJixi=expα∑i=1dJixi. _β→∞ _p ∈ ^M _α\-β H_MHM[x,p ,ξ]+ _i=1^dJ_ix_i\= _α\ _i=1^dJ_ix_i\. Then limβ→∞Cdis(x)= _β→∞C^dis(x)= limβ→∞∑p∈ΔM(ρJdis)α(p|x) \ _β→∞ _p∈ ^M( _J^dis)^α(p|x) = = limβ→∞∑p=p⋆(ρJdis)α(p|x)+∑p∈ΔMp≠p⋆(ρJdis)α(p|x) \ _β→∞ \ _p=p ( _J^dis)^α(p|x)+ _ subarraycp∈ ^M\\ p≠ p subarray( _J^dis)^α(p|x) \ = = limβ→∞expα∑i=1dJixi(expα∑i=1dJixi)α \ _β→∞ \ ^α_α\ _i=1^dJ_ix_i\( _α\ _i=1^dJ_ix_i\)^α (By ℋ~MHM[x,p⋆,ξ]=0 H_MHM[x,p ,ξ]=0) +∑p∈ΔMp≠p⋆expα−βℋ~MHM[x,p,ξ]+∑i=1dJixi(expα∑i=1dJixi)α \ + _ subarraycp∈ ^M\\ p≠ p subarray ^α_α\-β H_MHM[x,p,ξ]+ _i=1^dJ_ix_i\( _α\ _i=1^dJ_ix_i\)^α \ = = limβ→∞1+0 \ _β→∞ \1+0 \ = = 1. \ 1. This completes the proof. ∎ Remark D.8. In practical attention mechanisms, the retrieved memories are a finite set. The attention weights are finite-dimensional vectors. Therefore, the discrete normalization behavior in Lemma D.2 occurs in practice. In contrast, the divergence of C(x)C(x) in the continuous case is mainly a theoretical issue. Proposition D.8 (Emergence of Hard Argmax from Escort Moments). Assume that ℋ~MHM≥0 H_MHM≥ 0 has a unique minimizer p⋆∈ΔMp ∈ ^M. In the zero-temperature limit, β→∞β→∞, the following escort variables become deterministic hard argmax selections: α→1:p(esc)α>1,Continuous:p(esc)/C(x)α>1,Discrete:p(esc)⟶β→∞p⋆=argmaxp∈ΔM∑μ=1Mzμpμ. casesα→ 1&:p^(esc)\\ α>1, \ Continuous&:p^(esc)/C(x)\\ α>1, \ Discrete&:p^(esc) cases β→∞ p = _p∈ ^M\ _μ=1^Mz_μp_μ\. (D.12) Proof. For α→1α→ 1, the escort moment reduces to the ordinary Gibbs expectation, p(esc)=∫ΔMpρJ(p|x)dμ(p). p^(esc)= _ ^Mp _J(p|x) \ μ(p). By Proposition D.3, the Gibbs distribution collapses at the unique minimizer in the zero-temperature limit, ρJ(p|x)→p,p⋆, _J(p|x)→ 1_\p,p \, as β→∞β→∞. Hence, p(esc)→p⋆p^(esc)→ p as β→∞β→∞. Next, we consider α>1α>1 with continuous p. By Lemma D.2, limβ→∞C(x) _β→∞C(x) diverges. However, since the escort measure still concentrates at the unique minimizer p⋆p , we obtain limβ→∞p(esc)C(x)=limβ→∞∫ΔMpρJα(p|x)dμ(p)∫ΔMρJα(p′|x)dμ(p′)=p⋆. _β→∞ p^(esc)C(x)= _β→∞ _ ^Mp _J^α(p|x) \ μ(p) _ ^Mρ^α_J(p |x) \ μ(p )=p . Finally, for α>1α>1 with discrete p, we have limβ→∞Cdis(x)=1 _β→∞C^dis(x)=1 according to Lemma D.2. Hence, limβ→∞p(esc)Cdis(x)=limβ→∞p(esc)=p⋆. _β→∞ p^(esc)C^dis(x)= _β→∞p^(esc)=p . This proves the proposition. ∎ Remark D.9. Proposition D.8 shows that escort-weighted retrieval (D.10) reduces to a deterministic hard argmax selection rule in the zero-temperature limit. This statistical-mechanics framework demonstrates how sparse retrieval behavior is able to emerge from escort concentration in the general fluctuated (stochastic) setting. Appendix E Dynamical Retrieval Stability Analysis In this section, we investigate the dynamical retrieval stability. Then we discuss how the convergences of x and p relate to each other. To do this, we apply gradient-based optimization and perturb around the fixed point. The stability analysis verifies whether the training trajectory flows back to the fixed-point or not. Definition E.1 (Retrieval Dynamics). The retrieval update rules are dtxi(t)=−(∇xΓα(mod)[x(t),ξ])i ddtx_i(t)=-( _x ^(mod)_α[x(t),ξ])_i \ (Continuous), \ \ \ (Continuous) \ , (E.1) xi(t+1)=xi(t)−η(∇xΓα(mod)[x(t),ξ])i x_i(t+1)=x_i(t)-η( _x ^(mod)_α[x(t),ξ])_i \ (Discrete), \ \ \ (Discrete) \ , (E.2) where η is the learning rate. In Section 2, we note that ∇φΓα[φ] _ _α[ ] is not bijective. Hence, it converges to a set rather than a point in general. In our setting, we consider retrieval dynamics of the generalized model Γα(mod) ^(mod)_α in the query space (x-space), while the memory patterns remain fixed. The projection mapping zμ=∑i=1dξiμxi/dz_μ= _i=1^d _i^μx_i/ d is not injective generally, so distinct x⋆x correspond to one specific fixed-point z⋆z . However, due to β2∑i=1dxi2 β2 _i=1^dx_i^2 term, such projection degeneracy does not imply the degeneracy of the full Γα(mod)[x,ξ] _α^(mod)[x,ξ]. By the proposition below, we are capable of studying the stability and refer the stability to the stability modulo the full degeneracy. This interpretation is consistent with the analysis of gradient-based optimization developed in (Lee et al., 2016), where the local Hessian structure governs the stability of gradient flows near critical points. Furthermore, we only apply the gradient-based optimization in x-space. Hence, the stable fixed-point in x-space must correspond to one specific stable fixed-point in z-space. Proposition E.1 (State-Retrieval Stability Modulo Degeneracy). Let the macroscopic state space be ℳxM_x, (x)≔ker(∇x2Γα(mod)[x,ξ])K(x) (∇^2_x ^(mod)_α[x,ξ]) be the kernel space, and ⟂(x)K (x) be the orthogonal complement of (x)K(x). Assume there exists a local stable point x⋆∈ℝdx ^d such that ∇xΓα(mod)[x⋆,ξ]=0, _x ^(mod)_α[x ,ξ]=0 \ , (E.3) and Hessian∇x2Γα(mod)[x⋆,ξ]is positive definite on⟂(x⋆). \ \ _x^2 _α^(mod)[x ,ξ] \ \ is positive definite on \ \ K (x ). (E.4) Let x⋆⊂ℳφN_x _ be the degeneracy space through x⋆x such that Γα(mod)[x,ξ] ^(mod)_α[x,ξ] is constant on it, and Tx⋆x⋆=(x⋆)T_x N_x =K(x ). Then x⋆x is a stable local minimizer modulo degeneracy. That is, the gradient flow in the x-space converges to (x⋆+v)(x +v) for some v∈(x⋆)v (x ). Additionally, for discrete gradient descent, we require 0<η<2λmax(∇x2Γα(mod)[x⋆,ξ])|⟂(x⋆), 0<η< 2 _max( _x^2 ^(mod)_α[x ,ξ])|_K (x ) \ , (E.5) where λmax(⋅)|⟂(x⋆) _max(·)|_K (x ) is the maximum eigenvalue with the Hessian being on ⟂(x⋆)K (x ). Proof. Let xi−xi⋆≔δi≔δi∥+δi⟂x_i-x _i _i _i^\|+δ _i be infinitesimal, where δ∥∈(x⋆)δ^\| (x ) and δ⟂∈⟂(x⋆)δ (x ). Then for non-zero δ⟂δ , Γα(mod)[x,ξ] ^(mod)_α[x,ξ] =Γα(mod)[x⋆,ξ]+∑i=1d(∇xΓα(mod)[x⋆,ξ])iδi+12∑i,j=1d(∇x2Γα(mod)[x⋆,ξ])ijδiδj+O(δ3) = \ ^(mod)_α[x ,ξ]+ _i=1^d( _x ^(mod)_α[x ,ξ])_i _i+ 12 _i,j=1^d( _x^2 ^(mod)_α[x ,ξ])_ij _i _j+O(δ^3) =Γα(mod)[x⋆,ξ]+12∑i,j=1d(∇x2Γα(mod)[x⋆,ξ])ijδi⟂δj⟂+O(δ3). = \ ^(mod)_α[x ,ξ]+ 12 _i,j=1^d( _x^2 ^(mod)_α[x ,ξ])_ijδ _iδ _j+O(δ^3) \ . (By (E.3) and (E.4)) Since the Hessian is positive definite on ⟂(x⋆)K (x ), we obtain Γα(mod)[x,ξ]=Γα(mod)[x⋆+δ,ξ]>Γα(mod)[x⋆,ξ]. ^(mod)_α[x,ξ]= ^(mod)_α[x +δ,ξ]> ^(mod)_α[x ,ξ] \ . Hence, the system flows back to the fixed-point, x⋆x . If the infinitesimal change does not contain the orthogonal complementary component, i.e., δi=δi∥ _i=δ^\|_i, then x⋆+δ∥∈x⋆x +δ^\| _x . By the assumption of x⋆N_x , Γα(mod)[x,ξ] ^(mod)_α[x,ξ] remains unchanged on x⋆N_x . Hence, the higher-order correction vanishes, i.e., O(δ3)=0O(δ^3)=0. Consider the gradient flow: dtΓα(mod)[x(t),ξ]−Γα(mod)[x⋆,ξ]= ddt\ ^(mod)_α[x(t),ξ]- ^(mod)_α[x ,ξ]\= ∑i=1d(∇xΓα(mod)[x(t),ξ])idxi(t)dt \ _i=1^d( _x ^(mod)_α[x(t),ξ])_i dx_i(t)dt = = −∑i=1d(∇xΓα(mod)[x(t),ξ])i2≤0, \ - _i=1^d( _x ^(mod)_α[x(t),ξ])^2_i≤ 0 \ , where we use (E.1). The gradient flow approaches to (x⋆+v)(x +v) for some v∈(x⋆)v (x ). For gradient descent, δi(t+1)= _i(t+1)= xi(t+1)−xi⋆ \ x_i(t+1)-x _i = = δi(t)−η(∇xΓα[x(t),ξ])i \ _i(t)-η( _x _α[x(t),ξ])_i = = ∑j=1i,j−η(∇x2Γα(mod)[x⋆,ξ])ijδj(t). \ _j=1\ 1_\i,j\-η(∇^2_x ^(mod)_α[x ,ξ])_ij\ _j(t) \ . (E.6) To guarantee that the gradient descent converges for ⟂(x⋆)K (x ), we require that the maximal absolute eigenvalue of ⋯\·\ above should be smaller than one. According to (E.4), the eigenvalues are all positive along ⟂(x⋆)K (x ). Hence, |1−ηλmax(∇x2Γα(mod)[x⋆,ξ])|⟂(x⋆)|<1⟹ 0<η<2λmax(∇x2Γα(mod)[x⋆,ξ])|⟂(x⋆). |1-η _max(∇^2_x ^(mod)_α[x ,ξ])|_K (x )|<1 \ \ 0<η< 2 _max( _x^2 ^(mod)_α[x ,ξ])|_K (x ) \ . If we do not restrict on ⟂(x⋆)K (x ), then the λmax(⋅)=0 _max(·)=0 along (x⋆)K(x ). Hence, the gradient descent approaches to (x⋆+v)(x +v) for some v∈(x⋆)v (x ). This proves (E.5). ∎ Remark E.1. For symmetry and projection degeneracy, Proposition E.1 holds since symmetries are always global, i.e., the group element does not differ for different φ , and the projection is affine. However, for constrained systems, we require the condition that Γα _α remains the same on the degeneracy manifold of the constrained space. This does not always hold. Proposition E.2 (State-Retrieval Stability as Query-Key-Retrieval Stability). The dynamical retrieval stability in the macroscopic state space, ℳxM_x, corresponds to which in the memory score z-space, ℳzM_z. Specifically, let x⋆x be the stable fixed-point in ℳxM_x, and ≔ker(ξiμ)⊂ℝdD ( _i^μ) ^d. x(t)→x⋆⟹z(t)→z⋆≔1d∑i=1dξiμxi⋆. x(t)→ x \ \ z(t)→ z 1 d _i=1^d _i^μx _i \ . (E.7) Conversely, z(t)→z⋆⟹Π⟂x(t)→Π⟂x⋆, z(t)→ z \ \ _D x(t)→ _D x \ , (E.8) where Π⟂ _D is the projection map to ⟂D . Moreover, the convergence rates in z imply convergence rates in x via ‖δx‖2≤dσmin‖δz‖2,∀δx∈⟂, \|δ x\|_2≤ d _min\|δ z\|_2 \ , \ ∀ \ δ x \ , (E.9) where σmin>0 _min>0 denotes the smallest non-zero singular value of (ξiμ)( _i^μ). Proof. Since ℝd=⊕⟂R^d=D , we decompose xi=xi∥+xi⟂x_i=x^\|_i+x _i with ∑i=1dξiμxi∥=0 _i=1^d _i^μx_i^\|=0. For (E.7), it is trivial since @depthΔ @set@skewchar @nested@a111z @depth 1 @set@skewchar @nested@a 111z is deterministic due to zμ=∑i=1dξiμxi/dz_μ= _i=1^d _i^μx_i/ d. For (E.8), we consider δx(t)=Π⟂(x(t)−x⋆)∈⟂δ x(t)= _D (x(t)-x ) . Then 1d∑i=1dξiμδxi(t)=1d∑i=1dξiμ(xi(t)−xi⋆)=zμ(t)−z⋆→0. 1 d _i=1^d _i^μδ x_i(t)= 1 d _i=1^d _i^μ(x_i(t)-x _i)=z_μ(t)-z → 0. If δx∈δ x , then ∑i=1dξiμδxi(t)=∑i=1dξiμΠ(xi(t)−xi⋆)=0 _i=1^d _i^μδ x_i(t)= _i=1^d _i^μ _D(x_i(t)-x_i )=0 does not give any information of x(t)x(t) and x⋆x . We restrict to ⟂D now. Since on ⟂D , (ξiμ _i^μ) is injective and δx∈⟂δ x satisfies ∑μ=1M(zμ(t)−zμ⋆)2=1d∑μ=1M(∑i=1dξiμδxi(t))2≥σmin2d∑i=1dδxi2(t), _μ=1^M(z_μ(t)-z _μ)^2= 1d _μ=1^M( _i=1^d _i^μδ x_i(t))^2≥ _min^2d _i=1^dδ x_i^2(t), we thereby prove (E.9). ∎ Proposition E.3 (Score-space Attention-Retrieval Stability). Consider the fixed-point in the z-space be zμ⋆z _μ. Then the Jacobian of the update map is μν(z⋆)=−1β∑ρ=1MGμρ(∇z2Γ~α(mod)[x⋆,p,ξ])ρν, _μν(z )=- 1β _ρ=1^MG_μρ(∇^2_z ^(mod)_α[x ,p,ξ])_ρν \ , (E.10) where zμ⋆=∑i=1dxi⋆ξiμ/dz _μ= _i=1^dx _iξ^μ_i/ d. The dynamical retrieval update is stable at zμ⋆z _μ if ρ((z⋆))<1, ρ(J(z ))<1 \ , where ρ(⋅)ρ(·) is the spectral radius, i.e., the maximal absolute value of eigenvalues. Proof. If we perturb around the fixed-point, δzμ(t)≔zμ(t)−zμ⋆δ z_μ(t) z_μ(t)-z _μ gives δzμ(t+1)= δ z_μ(t+1)= ∑ν=1MGμνpν(z(t))+bμ(J)−∑ν=1MGμνpν(z⋆)−bμ(J) \ _ν=1^MG_μνp_ν(z(t))+b_μ(J)- _ν=1^MG_μνp_ν(z )-b_μ(J) = = ∑ν,ρ=1MGμν∂pν∂zρ|z=z⋆δzρ(t). \ _ν,ρ=1^MG_μν ∂ p_ν∂ z_ρ|_z=z \ δ z_ρ(t) \ . According to (3.8), we express p in terms of the gradient of Γ~α(mod) _α^(mod). In this way, δzμ(t+1)=−1β∑ν,ρ=1MGμν(∇z2Γ~α(mod)[x⋆,p,ξ])νρδzρ(t) δ z_μ(t+1)=- 1β _ν,ρ=1^MG_μν(∇^2_z _α^(mod)[x ,p,ξ])_νρδ z_ρ(t) By δzμ(t+1)=∑ν=1Mμνδzν(t)δ z_μ(t+1)= _ν=1^MJ_μνδ z_ν(t), we obtain the Jacobian, (E.10). If ρ()<1ρ(J)<1, then t→0J^t→ 0 as t→∞t→∞. Hence, δzμ(t)→0δ z_μ(t)→ 0, and the system converges to the fixed-point, zμ⋆z _μ. ∎ Remark E.2. The update map is a non-linear discrete map. Hence, it is neither a gradient flow nor a gradient descent in general. This fixed-point perturbation analysis is consistent to the correspondence between the gradient-based optimization in ℳxM_x. Now we define the Tsallis divergence which helps us characterizes the training dynamics and the convergences. It generalizes KL-divergence and induces dual geometric structures on the probability simplex. (Amari, 1985) Definition E.2 (Tsallis Divergence (Furuichi et al., 2004)). The Tsallis divergence generalizes the Kullback-Leibler (KL) divergence (Kullback and Leibler, 1951). It has the following form: Dα(p∥q)≔−∑μ=1Mpμlogα(qμpμ)=1α−1(∑μ=1Mpμαqμ1−α−1). D_α(p\|q) - _μ=1^Mp_μ _α( q_μp_μ)= 1α-1( _μ=1^Mp^α_μq_μ^1-α-1) \ . Remark E.3. If α→1α→ 1, the Tsallis divergence reduces to the KL divergence. It follows from Lemma C.1 in which logα→log _α→ as α→1α→ 1. Hence, Dα→1(p∥q)=−limα→1∑μ=1Mpμlogα(qμpμ)=∑μ=1Mpμlog(pμqμ)≔DKL(p∥q). D_α→ 1(p\|q)=- _α→ 1 _μ=1^Mp_μ _α( q_μp_μ)= _μ=1^Mp_μ ( p_μq_μ) D_KL(p\|q) \ . The dynamical stability of p relates to the attention-retrieval stability via the Tsallis divergence. Proposition E.4 (Tsallis Divergence and Attention-Retrieval Stability). Let α>1α>1, p⋆∈ΔMp ∈ ^M be the fixed maximizer in the simplex interior satisfying zμ⋆=(Gp)μ⋆+bμ(J)z_μ =(Gp)_μ +b_μ(J). Then there exist constants, c1,c2>0c_1,c_2>0 such that c1∥z(t)−z⋆∥22≤Dα(p(t)∥p⋆)≤c2∥p(t)−p⋆∥22. c_1\|z(t)-z \|^2_2≤ D_α(p(t)\|p )≤ c_2\|p(t)-p \|^2_2 \ . Proof. Since ∂pμDα(p∥q)=α−1qμ1−αpμα−1,∂2∂pμ∂pνDα(p∥q)=αqμ1−αpμα−2μ,ν, ∂ p_μD_α(p\|q)= α-1q_μ^1-αp_μ^α-1 \ , \ ∂^2∂ p_μ∂ p_νD_α(p\|q)=α q_μ^1-αp_μ^α-2 1_\μ,ν\ \ , we have the Taylor expansion Dα(p(t)∥p⋆)= D_α(p(t)\|p )= Dα(p⋆∥p⋆)+α−1∑μ=1M(pμ(t)−pμ⋆)+α2∑μ=1M1pμ⋆(pμ(t)−pμ⋆)2+O(∥p(t)−p⋆∥23) \ D_α(p \|p )+ α-1 _μ=1^M(p_μ(t)-p _μ)+ α2 _μ=1^M 1p_μ (p_μ(t)-p _μ)^2+O(\|p(t)-p \|_2^3) = = α2∑μ=1M1pμ⋆(pμ(t)−pμ⋆)2+O(‖p(t)−p⋆‖23). \ α2 _μ=1^M 1p_μ (p_μ(t)-p _μ)^2+O(\|p(t)-p \|_2^3) \ . Since p⋆p is in the simplex interior, 1>pμ⋆>01>p _μ>0 for all μ. Hence, there exist c1,c2>0c_1,c_2>0 such that c1∥p(t)−p⋆∥22≤Dα(p(t)∥p⋆)≤c2∥p(t)−p⋆∥22. c_1\|p(t)-p \|_2^2≤ D_α(p(t)\|p )≤ c_2\|p(t)-p \|^2_2 \ . Moreover, zμ(t)−zμ⋆=[G(p(t)−p⋆)]μ⟹‖z(t)−z⋆‖2≤‖G‖op⋅‖p(t)−p⋆‖2. z_μ(t)-z _μ=[G(p(t)-p )]_μ \ \ \|z(t)-z \|_2≤\|G\|_op·\|p(t)-p \|_2 \ . Combining these two inequalities, we prove this proposition. ∎ Remark E.4. From Proposition E.4, we notice that the convergence in ℳzM_z does not control the convergence of the Tsallis divergence and the convergence in p-space. If the Gram matrix is invertible, then ‖p(t)−p⋆‖2≤‖G−1‖op⋅‖z(t)−z⋆‖2\|p(t)-p \|_2≤\|G^-1\|_op·\|z(t)-z \|_2. In this way, the convergence in ℳxM_x controls the convergence in ℳzM_z and the p-space. However, in the real world, the high correlations between memory patterns make GμνG_μν non-invertible. Hence, we do not guarantee the convergence of p⋆p . Appendix F Experiment Setup In our simulation, Section 5 shows the phase structure of modern Hopfield models by scanning the inverse temperature β, and the exponential memory load κ=log(M)/dκ= (M)/d in both the sparse (α=2)(α=2) and dense (α=1)(α=1) regimes. Here we introduce the support size to characterize the usage of memory weights. Then we provide the experimental setup and further details in Section F.2. F.1 Effective Support Size and Sparsity To relate entropy functions to the size and the randomness of information concretely, we introduce the effective support size and the sparsity here. Definition F.1 (Effective Support Size (Hill, 1973)). We define the (Tsallis-Rényi) effective support size as Keff(α)[p]≔(∑μ=1Mpμα)11−α, K_eff^(α)[p] ( _μ=1^Mp^α_μ) 11-α \ , where p∈ΔMp∈ ^M satisfying ∑μ=1Mpμ=1 _μ=1^Mp_μ=1 and α∈ℝ+/1α _+/\1\. Proposition F.1 (Range of Effective Support Size (Hill, 1973)). The effective support size satisfies the inequality: 1≤Keff(α)[p]≤M 1≤ K_eff^(α)[p]≤ M If Keff(α)[p]=1K_eff^(α)[p]=1, then p is a peak. If Keff(α)[p]=MK_eff^(α)[p]=M, then p is uniform. Proof. We repeat the proof from (Hill, 1973) for completeness. We decompose the range of α into α>1α>1 and 1>α>01>α>0 and focus on the former case first. For α>1α>1, we have ∑μ=1Mpμα≤1 _μ=1^Mp_μ^α≤ 1 and 1≤ 1≤ (∑μ=1Mpμα)11−α=Keff(α)[p] \ ( _μ=1^Mp_μ^α) 11-α=K_eff^(α)[p] = = M11−α⋅(1M∑μ=1Mpμα)11−α \ M 11-α·( 1M _μ=1^Mp_μ^α) 11-α ≤ ≤ M11−α⋅[(1M∑μ=1Mpμ)α]11−α \ M 11-α·[( 1M _μ=1^Mp_μ)^α] 11-α (By the Jensen’s Inequality) = = M. \ M \ . For 1>α>01>α>0, then 1≤∑μ=1Mpμα1≤ _μ=1^Mp_μ^α and the similar derivation holds. We note that it saturates 11 when p is a vertex on the simplex, i.e., pμmax=1p_ _max=1 and pμ≠μmax=0p_μ≠ _max=0, where μmax≔argmaxμpμ _max _μ \ p_μ is non-degenerate. It saturates the Jensen’s inequality when pμ=M−1p_μ=M^-1 for every μ. ∎ Remark F.1. By Definition C.8, Keff(α)[p]=(1−(α−1)Ψα[p])11−α, K_eff^(α)[p]=(1-(α-1) _α[p]) 11-α \ , we differentiate it with respect to Ψα _α. It yields (Keff(α)[p])α/(1−α)>0(K_eff^(α)[p])^α/(1-α)>0. Hence, the effective support size is strictly increasing (monotonic) with respect to the Tsallis entropy. The higher the entropy is, the larger the effective support size is. Lemma F.1 (Threshold of Probability Function (Martins and Astudillo, 2016; Peters et al., 2019)). The minimizer, p∈ΔMp∈ ^M, of Γα(mod)[x,p,ξ] ^(mod)_α[x,p,ξ] satisfies pμ=(α−1αβ[zμ−θ]+)1α−1, p_μ=( α-1αβ[z_μ-θ]_+) 1α-1 \ , (F.1) where [u]+≔maxu,0[u]_+ \u,0\. We determine θ by the normalization condition, ∑μ=1Mpμ=1 _μ=1^Mp_μ=1. This truncation only appears for α>1α>1. The support of the minimizers is thereby supp(p)=μ∈[1,M]:zμ>θ. (p)=\μ∈[1,M]:z_μ>θ\ \ . We define |supp(p)||supp(p)| as the exact support size. Proof. We include the proof for completeness, adapting the standard argument from (Martins and Astudillo, 2016; Peters et al., 2019) to our notation. Since 0≤pμ≤10≤ p_μ≤ 1 and ∑μ=1Mpμ=1 _μ=1^Mp_μ=1, we impose these constriants as Lagrangian multiplier terms: ℒ(p,z,θ,λ)=β∑μ=1Mzμpμ+1α−1(1−∑μ=1Mpμα)+βθ(∑μ=1Mpμ−1)+∑μ=1Mλμpμ. (p,z,θ,λ)=β _μ=1^Mz_μp_μ+ 1α-1(1- _μ=1^Mp_μ^α)+βθ( _μ=1^Mp_μ-1)+ _μ=1^M _μp_μ \ . Let the optimal saddles that maximizes the Lagrangian function be @depthΔ @set@skewchar @nested@a111p @depth 1 @set@skewchar @nested@a 111p, @depthΔ @set@skewchar @nested@a111z @depth 1 @set@skewchar @nested@a 111z, @depthΔ @set@skewchar @nested@a111 @depth 1 @set@skewchar @nested@a 111, and @depthΔ @set@skewchar @nested@a111 @depth 1 @set@skewchar @nested@a 111. Then the Karush-Kuhn-Tucker (KKT) conditions are @depthΔ @set@skewchar @nested@a111pμ≥0,∑μ=1M @depthΔ @set@skewchar @nested@a111pμ=1 @depth 1 @set@skewchar @nested@a 111p_μ≥ 0 \ , \ _μ=1^M @depth 1 @set@skewchar @nested@a 111p_μ=1 , (Primal) @depthΔ @set@skewchar @nested@a111μ≥0 @depth 1 @set@skewchar @nested@a 111_μ≥ 0 , (Dual) @depthΔ @set@skewchar @nested@a111 @depthμΔ @set@skewchar @nested@a111pμ=0 @depth 1 @set@skewchar @nested@a 111_μ @depth 1 @set@skewchar @nested@a 111p_μ=0 , (Complementary) ∂ℒ∂pμ|p= @depthΔ @set@skewchar @nested@a111p,z= @depthΔ @set@skewchar @nested@a111z,θ= @depthΔ @set@skewchar @nested@a111,λ= @depthΔ @set@skewchar @nested@a111=β @depthΔ @set@skewchar @nested@a111zμ−α−1( @depthΔ @set@skewchar @nested@a111pμ)α−1+β @depthΔ @set@skewchar @nested@a111+ @depthΔ @set@skewchar @nested@a111=μ0 ∂ p_μ|_p= @depth 1 @set@skewchar @nested@a 111p,z= @depth 1 @set@skewchar @nested@a 111z,θ= @depth 1 @set@skewchar @nested@a 111,λ= @depth 1 @set@skewchar @nested@a 111=β @depth 1 @set@skewchar @nested@a 111z_μ- α-1( @depth 1 @set@skewchar @nested@a 111p_μ)^α-1+β @depth 1 @set@skewchar @nested@a 111+ @depth 1 @set@skewchar @nested@a 111_μ=0 , (Stationary) for all μ∈[1,M]μ∈[1,M]. By solving the stationary equation, for @depthΔ @set@skewchar @nested@a111pμ>0 @depth 1 @set@skewchar @nested@a 111p_μ>0 and α>1α>1, @depthΔ @set@skewchar @nested@a111pμ=(α−1αβ[ @depthΔ @set@skewchar @nested@a111zμ+ @depthΔ @set@skewchar @nested@a111])1α−1. @depth 1 @set@skewchar @nested@a 111p_μ=( α-1αβ[ @depth 1 @set@skewchar @nested@a 111z_μ+ @depth 1 @set@skewchar @nested@a 111]) 1α-1 \ . For @depthΔ @set@skewchar @nested@a111pμ=0 @depth 1 @set@skewchar @nested@a 111p_μ=0 and α>1α>1, @depthΔ @set@skewchar @nested@a111zμ+ @depthΔ @set@skewchar @nested@a111+ @depthΔ @set@skewchar @nested@a111μ=0⟹ @depthΔ @set@skewchar @nested@a111zμ+ @depthΔ @set@skewchar @nested@a111≤0. @depth 1 @set@skewchar @nested@a 111z_μ+ @depth 1 @set@skewchar @nested@a 111+ @depth 1 @set@skewchar @nested@a 111_μ=0 \ \ @depth 1 @set@skewchar @nested@a 111z_μ+ @depth 1 @set@skewchar @nested@a 111≤ 0 \ . Hence, after replacing @depthΔ @set@skewchar @nested@a111→− @depthΔ @set@skewchar @nested@a111 @depth 1 @set@skewchar @nested@a 111→- @depth 1 @set@skewchar @nested@a 111, we have pμ=(α−1αβ[zμ−θ]+)1α−1, p_μ=( α-1αβ[z_μ-θ]_+) 1α-1 \ , where we neglect bars to avoid redundant notations. ∎ Remark F.2. For 0<α<10<α<1, the exponent of (F.1) is negative, so the base should remain positive. In this way, there is no truncation and all minimizers pμ>0p_μ>0. For α→1α→ 1, the Shannon entropy has the stationary equation, βzμ−log(pμ)−1+βθ+λμ=0. β z_μ- (p_μ)-1+βθ+ _μ=0 \ . The minimizer, pμp_μ, is positive everywhere and does not truncate for finite (θ,zμ,λμ)(θ,z_μ, _μ). Definition F.2 (Sparsity (Martins and Astudillo, 2016; Peters et al., 2019)). We say p∈ΔMp∈ ^M is k-sparse if the exact support size satisfies |supp(p)|=k, |supp(p)|=k \ , where 0≤k≤M0≤ k≤ M. The large the k is, the less the sparsity is. In other words, if more points satisfy p=0p=0, then the system is sparser. Remark F.3. Higher entropy suggests larger effective support size but not higher sparsity. The sparsity depends on the boundary geometry of the entropy function. Lemma F.1 states that only for α>1α>1, there is a truncation and several pμp_μ are zero. It is consistent with the argument in Proposition C.3. Due to Definition C.4, the Tsallis entropy is not essentially smooth only for α>1α>1. Therefore, for α>1α>1, the optimizer sit on the boundaries of the simplex when the corresponding zμz_μ is below the threshold. This introduces the sparsity. Otherwise, the divergence of gradient on the boundary forbids the sparsity. F.2 Settings Setup. We consider a system of dimension d with M=eκdM=e^κ d normalized stored patterns, ξμ=1M\ξ^μ\_μ=1^M. To probe the energy landscape, Γα(mod) _α^(mod), we perform continuous scans by varying β or κ. For each scan, we use both up and down sweeps to detect hysteresis and metastability. 1. Up Sweep: Initialize from low β (or low κ) and increase. 2. Down Sweep: Initialize from high β (or high κ) and decrease. At each parameter value (β or |κ|κ), we initialize the optimization from the converged solution at the previous step (warm start). Then we minimize the free energy with respect to x by gradient descent, x(t+1)=x(t)−η(t)∇xΓα(mod)(x(t)),η(t)=η02−m(t), x(t+1)=x(t)-η(t) _x _α^(mod)(x(t)), η(t)= _02^-m(t), where η0=0.5 _0=0.5 and m(t)≔minm∈ℕ0∣Γα(mod)(x(t+1))≤Γα(mod)(x(t))m(t) \m _0 _α^(mod)(x(t+1))≤ _α^(mod)(x(t))\. This procedure tracks continuous branches of local minima of Γα(mod) ^(mod)_α. To explore competing solutions, we perform multi-branch continuation. Data. In our experiment, we consider clustered normalized patterns. ξiμ=1−δηic(μ)+δζiμ, ξ^μ_i= 1-δη^c(μ)_i+ δζ^μ_i, where c(μ)∈1,…,Cc(μ)∈\1,...,C\ assigns pattern μ to a cluster, ηic(μ) _i^c(μ) are cluster centers sampled as ηc∼(0,d)η^c (0, 1_d), ζμ∼(0,d)ζ^μ (0, 1_d) are i.i.d. normalized noise vectors, and δ∈[0,1]δ∈[0,1] indicates the noise strength. This construction introduces off-diagonal correlations in the Gram matrix, which makes the simulation more realistic than the i.i.d. setting. In our simulation, we choose C=4C=4 and δ=0.04δ=0.04. Observables. To characterize the coexistence regimes and retrieval success, we measure the following quantities. 1. Maximum Overlap: To verify whether a stable point aligns with the stored patterns, we measure the maximum overlap. zmax⋆=maxμ|zμ⋆|z _max= _μ|z _μ|. 2. Exact Support Size: Each stable point has its own support and exact support size. For example, zμ⋆=∑i=1dxi⋆ξiμ/dz _μ= _i=1^dx_i _i^μ/ d gives a specific value of pμ⋆p _μ via Lemma F.1, which determines Keff(α>1)K_eff^(α>1). For α=2α=2, we measure |supp(p⋆)|=#μ∈[1,M]:zμ⋆>θ|supp(p )|=\#\μ∈[1,M]:z _μ>θ\ according to Lemma F.1, where #\# indicates the cardinality. For α=1α=1, we measure μ∈[1,M]:pμ⋆>ε\μ∈[1,M]:p _μ> \, where we choose ε=10−10 =10^-10 in our simulation. Hence, we denote the measured exact support size as |supp(p⋆)|≔#μ∈[1,M]:pμ⋆>10−10,α=1#μ∈[1,M]:zμ⋆>θ,α=2. |supp(p )| cases\#\μ∈[1,M]:p_μ >10^-10\ \ &, \ α=1\\ \#\μ∈[1,M]:z_μ >θ\ \ &, \ α=2 cases. 3. Fixed Scored-Threshold Support Size: In temperature analysis in the sparse case, we further define KΔ≔μ∈[1,M]:zμ⋆>Δ, K_ \μ∈[1,M]:z_μ > \, and we choose Δ=0.01 =0.01. Hence, it serves as a diagnostic for counting how many memory components achieve large query-key alignment. 4. Inverse Participation Ratio (IPR): It is defined as the effective support size with α=2α=2, i.e., KeffIPR≔Keff(2)=(∑μ(pμ⋆)2)−1K_eff^IPR K_eff^(2)=( _μ(p_μ )^2)^-1. If IPR is large, then there are more components of pμ⋆p _μ that are non-zero. 5. Free Energy: Distinct stable points have distinct free energy in general. To see which is more preferable, i.e., which one has the lowest free energy, we measure Γα(mod)[x,ξ] ^(mod)_α[x,ξ]. Results. We display the results in Figures 2, 3, 4 and 5. Please see Section 5 for detailed illustrations. Appendix G Additional Experiments In this section, we first show the experiment of dynamical stability in modern Hopfield models. Then we analyze the empirical retrieval behaviors in realistic end-to-end Transformer-architectures. Finally, we validate the special coexistence point formula Proposition 4.1 and spinodal formula Proposition 4.2 numerically. G.1 Dynamical Stability Analysis for Temperature In this subsection, we provide the numerical validation for the theoretical dynamical retrieval stability condition; please see Appendix E for a detailed discussion and derivations. Specifically, we only analyze the ordinary gradient descent (discrete) algorithm and verify the bound in Proposition E.1. Setup. We consider the number of pattern be M=100M=100, pattern dimension be d=64d=64, and inverse-temperature be β=300β=300 as an example. For simplicity, we apply fixed-step gradient descent x(t+1)=x(t)−η∇xΓα(mod)(x(t)), x(t+1)=x(t)-η _x ^(mod)_α(x(t)), where η is a constant. Let ηcrit=2/λmax(∇2Γα(mod)(x⋆)) _crit=2/ _max(∇^2 _α^(mod)(x )) be the theoretical critical learning rate threshold, λmax(⋅) _max(·) be the maximal eigenvalue, and x⋆x be a fixed local stable point. We investigate the convergence through iteration numbers, t. We define the initial condition as x(t=0)=x⋆+Ausoftx(t=0)=x +Au_soft, where A=10−2A=10^-2 is a small amplitude and usoftu_soft is the eigen-direction of the Hessian with the smallest eigenvalue. Notice that we derive usoftu_soft numerically, so it is not the exact eigenvector but exhibits tiny component in other directions. Data. We use the same pattern set ξμ=1M\ξ^μ\_μ=1^M as in the previous experiments, where each pattern ξμ∈ℝdξ^μ ^d is normalized. We generate this collection through the method in Section F.2. Observables. We track two quantities along the gradient-descent trajectory: the distance to the fixed point, ‖x(t)−x⋆‖2\|x(t)-x \|_2 and the maximal overlap, zmax(x(t))≔maxμzμ(x(t))z_max(x(t)) _μz_μ(x(t)). Results. In the sparse regime (α=2α=2), we demonstrate the result in Figure 10. Figure 10: Dynamical retrieval stability of MHM gradient descent in the sparse regime (α=2α=2). (M=100,d=64M=100,d=64). We study the gradient descent update x(t+1)=x(t)−η∇Γα(mod)(x(t))x(t+1)=x(t)-η∇ _α^(mod)(x(t)). We denote the critical learning rate as ηcrit=2/λmax(∇2Γα(mod)(x⋆)) _crit=2/ _max(∇^2 _α^(mod)(x )) with λmax _max the largest eigenvalue at a fixed stable point x⋆x . (a) Distance to the fixed point. For η<ηcritη< _crit, the training converges to the fixed point. Otherwise, the dynamics is unstable or even diverges. Maximal overlap. For η<ηcritη< _crit, the training converges to the fixed value of overlap zmax≈0.0156z_max≈ 0.0156. It is the saturation value of one-pattern perfect retrieval in Figure 2 (a). Otherwise, the dynamics is unstable or even diverges. (a) For η<ηcritη< _crit, the distance (blue curve) approaches to zero asymptotically, indicating that the training x(t)x(t) converges to the fixed stable point, x⋆x . At η≈ηcritη≈ _crit, the convergence becomes marginal. For η>ηcritη> _crit, the training x(t)x(t) deviates from the local point as iteration increases. This indicates a dynamical instability. Therefore, this plot provides the evidence that ηcrit _crit separates the convergent regime. (b) The maximal overlap zmax(x(t))z_max(x(t)) exhibits the same behavior. For η<ηcritη< _crit, zmax(x(t))z_ (x(t)) converges to the fixed value zmax(x⋆)z_ (x ). This indicates stable retrieval of the dominant memory pattern. Notably, the convergent value is zmax(x⋆)≈1/d≈0.0156z_max(x )≈ 1/d≈ 0.0156, which is the one-pattern retrieval in Figure 2 (a). At η≈ηcritη≈ _crit, it exhibits persistent oscillations around zmax(x⋆)z_max(x ). For η>ηcritη> _crit, the oscillations grow, indicating a dynamical instability in retrieval. In the dense regime (α=1α=1), we demonstrate the result in Figure 11. Figure 11: Dynamical retrieval stability of MHM gradient descent in the dense regime (α=1α=1). (M=100,d=64M=100,d=64). We study the gradient descent update x(t+1)=x(t)−η∇Γα(mod)(x(t))x(t+1)=x(t)-η∇ _α^(mod)(x(t)). We denote the critical learning rate as ηcrit=2/λmax(∇2Γα(mod)(x⋆)) _crit=2/ _max(∇^2 _α^(mod)(x )) with λmax _max the largest eigenvalue at a fixed stable point x⋆x . (a) Distance to the fixed point. For η<ηcritη< _crit, the training converges to the fixed point. For η>ηcritη> _crit, a divergence follows after a decay. This reflects that the iterate close to x⋆x initially. After this, the unstable eigen-directions start to dominate. We provide the precise explanation in this section. (b) Maximal overlap. For η<ηcritη< _crit, the training converges to the fixed value of overlap zmaxz_max. For η>ηcritη> _crit, a growing oscillations follow after an initial convergence. This is similar for the behavior in the distance plot. (a) For η<ηcritη< _crit, the iteration converges to the stable fixed point x⋆x rapidly. Larger the learning rate is, more rapid the convergence is. For η>ηcritη> _crit, the distance declines first, and then grows up after some number of iteration. This phenomenon arises from unstable modes being small due to our initialization x(t=0)x(t=0). In particular, from (E.6), we have δi(t+1)=(1−ηλi)δi(t) _i(t+1)=(1-η _i) _i(t), where δi(t)≡xi(t)−xi⋆ _i(t)≡ x_i(t)-x_i , and λi _i is the i-th eigenvalue of the Hessian ∇x2Γα(mod)∇^2_x _α^(mod). The convergence condition is |1−ηλi|<1,or 0<λi<2/η. |1-η _i|<1, \ 0< _i<2/η. Hence, even if η>ηcritη> _crit, some modes xi(t)x_i(t) still converge to the fixed point xi⋆x_i if the eigenvalues satisfy λi<2/η _i<2/η. Initially, we consider the dominant mode be which with λmin _min. Hence, it converges to the fixed point in the early stage. However, for λi>2/η _i>2/η, δi(t) _i(t) grows with t. Hence, the unstable modes start to dominate, and the trajectory diverges. (b) For η<ηcritη< _crit, the iteration converges to the stable fixed value zmax(x⋆)z_max(x ). Larger the learning rate is, more rapid the convergence is. For η>ηcritη> _crit, it exhibits the same behavior as in distance analysis. The dynamics converges first, then it diverges after certain iteration. This arises from unstable modes being small due to our initialization x(t=0)x(t=0) as well. G.2 Support and Capacity-Reliability Trade-off We study how increasing the number of memories affects retrieval reliability and effective support (IPR) in a trained Transformer. We compare three attention maps, α∈1,1.5,2α∈\1,1.5,2\ (softmax, entmax-1.5, and sparsemax), respectively. Setup. The data generation pipeline is uQ→fixedAQqin,and uμ→fixedAKkμin. u_Q [fixed]A_Qq^in, \ u_μ [fixed]A_Kk^in_μ. We generate latent query and key variables and map them to observed inputs. After that, we input qinq^in and kμink^in_μ into a multi-head cross-attention Transformer, yielding the output y^∈ℝdv y ^d_v. Data. Each example contains a query qin∈ℝdinq^in ^d_in, a set of memory keys kμin∈ℝdink_μ^in ^d_in, and independent values vμ∼(0,dv)v_μ (0, 1_d_v) with dv=32d_v=32. The latent dimension is dlat=16d_lat=16, the input dimension is din=24d_in=24, and the number of latent clusters is C=4C=4. For each example, we draw normalized cluster centers independently cr=c~r‖c~r‖2.c~r∼(0,dlat),r∈1,⋯,C. c_r= c_r\| c_r\|_2. c_r (0, 1_d_lat), r∈\1,·,C\. Memory μ is assigned a cluster index a(μ)a(μ), and its latent key is uμ=1−δca(μ)+δϵμ‖1−δca(μ)+δϵμ‖2,ϵμ∼(0,dlat). u_μ= 1-δ\,c_a(μ)+ δ\, _μ\| 1-δ\,c_a(μ)+ δ\, _μ\|_2, _μ (0, 1_d_lat). We choose δ=0.12δ=0.12 to control the latent key variance. The observed query and keys are then qin=AQuQ+σobsηQ,kμin=AKuμ+σKημ, q^in=A_Qu_Q+ _obs _Q, k^in_μ=A_Ku_μ+ _K _μ, where AQ,AK∈ℝdin×dlatA_Q,A_K ^d_in× d_lat with the orthogonalization AQ⊤AQ=AK⊤AK=latA_Q A_Q=A_K A_K= 1_lat, ηQ,ημ∼(0,in) _Q, _μ (0, 1_in), σobs=0.03 _obs=0.03, and σK=0.05 _K=0.05. Here we divide uQu_Q generation into two cases, the selective retrieval case and the uniform-aggregation case. 1. Selective retrieval. In this case, uQu_Q retrieves one particular memory. Consider that we sample a target memory index t from 1,…,M\1,...,M\. Therefore, we denote utu_t as the latent key located at the sampled target slot t. We then generate the latent query by adding noise uQ=ut+σQϵQ‖ut+σQϵQ‖2,ϵQ∼(0,dlat). u_Q= u_t+ _Q _Q\|u_t+ _Q _Q\|_2, _Q (0, 1_d_lat). During training, the query-noise level cycles through σQ∈0.05,0.10,0.15 _Q∈\0.05,0.10,0.15\ and the number of memories cycles through M=8,16,32,64M=\8,16,32,64\ simultaneously. During evaluation, the query noise is fixed at σQ=0.10 _Q=0.10 while M is increasing. 2. Uniform-aggregation. In this case, we sample a target cluster r⋆r . Then the latent query is uQ=cr⋆+σQϵQ‖cr⋆+σQϵQ‖2,ϵQ∼(0,dlat). u_Q= c_r + _Q _Q\|c_r + _Q _Q\|_2, _Q (0, 1_d_lat). We denote the relevant set as R=μ:a(μ)=r⋆R=\μ:a(μ)=r \. Since there are C=4C=4 clusters, we choose |R|=M/4|R|=M/4 in this case. Transformer. Inside the Transformer, we apply one-layer multi-head cross-attention. Specifically, we conduct the first step via qh=WhQLNQ(WQinqin+rQ),kh,μ=WhKLNK(WKinkμin+rK), q_h=W^Q_hLN_Q (W_Q^inq^in+r_Q ), k_h,μ=W^K_hLN_K (W_K^ink^in_μ+r_K ), vh,μ=WhVLNV(WVinvμ+rV), v_h,μ=W^V_hLN_V (W^in_Vv_μ+r_V ), where WQin,WKin∈ℝdmod×dinW_Q^in,W_K^in ^d_mod× d_in, WVin∈ℝdmod×dvW_V^in ^d_mod× d_v, WhQ,WhK,WhV∈ℝdh×dmodW_h^Q,W_h^K,W_h^V ^d_h× d_mod, LN denotes the LayerNorm, and h=1,⋯,Hh=\1,·,H\. Then the multi-head output is followed by two residual updates x1=WQinqin+rQ+WOConcat(∑μ=1Mp1,μv1,μ,⋯,∑μ=1MpH,μvH,μ)∈ℝdmod, x_1=W_Q^inq^in+r_Q+W^OConcat ( _μ=1^Mp_1,μv_1,μ,·, _μ=1^Mp_H,μv_H,μ ) ^d_mod, and x2=x1+FFN(LNFFN(x1))∈ℝdmod, x_2=x_1+FFN (LN_FFN(x_1) ) ^d_mod, where WO∈ℝdmod×dmodW^O ^d_mod× d_mod, ph,μp_h,μ is the attention with the form Proposition 3.3, and FFN(x)≔W2FFNGELU(W1FFNx)(x) W_2^FFNGELU(W_1^FFNx). Finally, the final predicted output is y^=WoutLNfinal(x2)∈ℝdv, y=W^outLN_final(x_2) ^d_v, where Wout∈ℝdv×dmodW^out ^d_v× d_mod. We choose the architectural hyperparameters as • Model dimension dmod=dh×H=96d_mod=d_h× H=96. • 4 attention heads (H=4H=4) with head dimension dh=24d_h=24. • FFN with hidden dimension 192, i.e., W1FFN∈ℝ192×96W_1^FFN ^192× 96, and W2FFN∈ℝ96×192W_2^FFN ^96× 192. In the Training stage, we choose 2500 AdamW steps, batch size B=64B=64, learning rate η=3×10−4η=3× 10^-4, weight decay 10−410^-4, MSE loss, β=1β=1, M∈8,16,32,64M∈\8,16,32,64\. Specifically, the MSE we evaluate is MSE=1B∑b=1B‖y^(b)−ytarget(b)‖22, = 1B _b=1^B \| y^(b)-y_target^(b) \|^2_2, where ytarget(b)=ytby^(b)_target=y_t_b for the selective retrieval case, and ytarget(b)=∑μ∈Rbyμ(b)/|Rb|y^(b)_target= _μ∈ R_by_μ^(b)/|R_b| for the uniform-aggregation case. In inference stage, we fix β=1β=1 and the learned parameters, e.g., WhQ,WO,rQ,⋯W_h^Q,W^O,r_Q,·. Then we increase the memory load from M=8M=8 to M=4096M=4096 to measure how good the prediction y y is. Observables. We measure • Selective retrieval: Value top-1 accuracy. We define the value top-1 accuracy as Avalue≔1B∑b=1Bargminμ∥y^(b)−yμ(b)∥22=tb. A_value 1B _b=1^B 1 \ _μ \| y^(b)-y^(b)_μ \|_2^2=t_b \. This measures end-to-end success, rather than simply measures whether attention puts its largest weight on the correct key. • Selective retrieval: Support size (IPR). As introduced in Section F.2, we measure the support via the inverse partition ratio (IPR). However, since we consider batch size and multi-head here, we further average over them, i.e., KeffIPR≔1HB∑b=1B∑h=1H1∑μ=1Mpb,h,μ2. K^IPR_eff 1HB _b=1^B _h=1^H 1 _μ=1^Mp_b,h,μ^2. • Uniform-aggregation: Error/zero-output baseline. The target is ytarget=∑μ∈Ryμ/|R|y_target= _μ∈ Ry_μ/|R|, where yμ∼(0,dv)y_μ (0, 1_d_v). Hence, the zero-output baseline is E0≔‖ytarget‖22=dv/|R|E_0 \|y_target\|^2_2=d_v/|R|. In this way, we define the error/zero-output baseline ErelE_rel as Erel≔MSEE0=|R|Bdv∑b=1B‖y^(b)−1|Rb|∑μ∈Rbyμ(b)‖22. E_rel MSEE_0= |R|Bd_v _b=1^B \| y^(b)- 1|R_b| _μ∈ R_by^(b)_μ \|^2_2. • Uniform-Aggregation: IPR / |R| |R|. We measure the normalized IPR with the size of the target set ρIPR≔KeffIPR|R|=1HB|R|∑b=1B∑h=1H1∑μ=1Mpb,h,μ2. _IPR K_eff^IPR|R|= 1HB|R| _b=1^B _h=1^H 1 _μ=1^Mp_b,h,μ^2. Results. We present the experiments as follows. Figure 12: Capacity-reliability trade-off for selective retrieval. We evaluate the trained Transformer at fixed βtrain=1 _train=1 while increasing M. (a). We measure the value top-1 accuracy to characterize the retrieval reliability. Sparsemax retain the highest accuracy. Furthermore, their accuracy decreases as M grows, indicating the capacity-reliability trade-off. (b). Softmax support remains full, while sparsemax use the least supports and is the most support-efficient. From Figure 12 (a), the sparsemax arrives the best value top-1 accuracy, while the softmax arrives the worst one. Furthermore, both of their accuracy decrease as M increases. This indicates that high memory load leads to lower reliability, and reveals the capacity-reliability trade-off. From Figure 12 (b), the support of softmax remains full, while which of sparsemax is low. This suggests that sparsemax is support-efficient. Those plots conclude that for a selective task, sparsemax has not only high-accuracy but also high support-efficiency. Figure 13: Capacity-reliability trade-off for uniform-aggregated retrieval. We evaluate the trained Transformer at fixed βtrain=1 _train=1 while increasing M. (a). We measure the ratio of the output error and the zero-output baseline to characterize the retrieval reliability. Sparsemax exhibits the worst reliability. Furthermore, their retrieval reliabilities become worse as M grows, indicating the capacity-reliability trade-off. (b). Softmax support remains full, while sparsemax use the least supports and is the most support-efficient. Compared to Figure 12, Figure 13 (a) shows that softmax arrives the highest reliability among three attentions and the capacity-reliability trade-off as well. From Figure 13 (b), we still observe that the sparsemax uses the least aggregated support size, and hence the most support-efficient. In conclusion, although a large α leads to support-efficiency, the reliability depends on the task: sparsemax predicts a selective retrieval the best, while softmax predicts a uniform-aggregation retrieval the best. G.3 Support, Temperature, and Reliability In this section, we investigate the temperature effect on effective support size (IPR) and retrieval reliability in a realistic Transformer architecture numerically. We compare three attention maps, α∈1,1.5,2α∈\1,1.5,2\ (softmax, entmax-1.5, and sparsemax), respectively. Setup. We adopt the same setup and Transformer architecture as in Section G.2, i.e., one-layer four-head cross-attention Transformer with MSE training objective. The only different setup is that we isolate β by fixing M at M=128M=128 with query-noise level σQ=0.10 _Q=0.10 in the inference stage. We vary the inverse-temperature over β∈2−2,2−1,20,⋯,29β∈\2^-2,2^-1,2^0,·,2^9\. Furthermore, we compare two protocals: • Frozen-β β. In this case, we first train each model at βtrain=1 _train=1. After training, we freeze every learned parameter (WhQ,WO,rQ,W_h^Q,W^O,r_Q, and so forth). At inference, we observe how reliable the output performance is at each β∈2−2,2−1,20,⋯,29β∈\2^-2,2^-1,2^0,·,2^9\. • Matched-β β. In this case, we train a separate Transformer for every β. That is, for different β, we have different trained parameters WhQ,WO,rQ,W_h^Q,W^O,r_Q, and so forth. Observables. We adopt the same observables as which in Section G.2. Results. We present the experiments as follows. Figure 14: Temperature effect on frozen-parameter Transformer. We train the Transformer parameters at βtrain=1 _train=1, then we fix those parameters, set M=128M=128 and vary β during inference. (a). The three attention families both exhibit high accuracy for a selective retrieval task. (b). For selective retrieval, the sparsemax uses the least effective support size (IPR), which shows the support-efficiency and reliability of high-β sparsemax. (c). For a uniform-aggregation task, the three attention families have worse error relative to the zero-output baseline as β increases. (d). The IPR relative to target-set size decline as β grows, which indicates that the retrieval becomes selective. This leads to a mismatch for multi-memory aggregation. From Figure 14 (a) and (b), we observe that sparsemax has highest accuracy and support-efficiency for frozen-β case in a selective retrieval task. Furthermore, both of the accuracy and IPR saturates at high β, arriving a single-pattern reliable retrieval for sparsemax. For softmax, the support is full, but the IPR shows that its retrieval becomes strongly concentrated. For the uniform-aggregation task, Figure 14 (c) and (d) show that all three frozen models perform far worse than the zero-output baseline in the high-β regime. That is, since we train the parameters only at β=1β=1, the Transformer exhibits greater selectivity, but creates a mismatch for multi-memory aggregation. Figure 15: Temperature effect on matched-parameter Transformer. We train the Transformer parameters at each β, then we evaluate with M=128M=128 during inference. (a). All three attention families maintain high accuracy in a selective retrieval task. (b). Sparsemax and entmax-1.5 are support-efficient in a selective retrieval task. (c). Softmax has the highest reliability among the three in a uniform-aggregation task. (d). Softmax uses numerous memories, which indicates a poor support-efficiency. From Figure 15 (a) and (b), all three attention families maintain perfect accuracy in a selective retrieval task. Furthermore, sparsemax and entmax-1.5 are more support-efficient that softmax. For the uniform-aggregation task. Figure 15 (c) and (d) show that softmax achieves the highest reliability but its effective support remains broader than the relevant set. Again, this indicates that high β increases the selectivity, but achieves poor performance for an aggregation task. G.4 Numerical Validation of Coexistence and Spinodal Formula In this section, we validate the coexistence formula in Proposition 4.1 and the spinodal formula in Proposition 4.2 for α=2α=2 numerically. Setup. We test the special two-key formulas under the uniform distribution and well-separation assumptions of Proposition 4.1 and Proposition 4.2. Then we construct a second dataset to test the full spinodal formula on larger supports. Data. We construct two experiments as follows. • Two-key systems. We consider two unit-norm keys k~1 k_1 and k~2 k_2 satisfying k~1⊤k~2=ϵ k_1 k_2=ε. Therefore, we set k~1=e1,k~2=ϵe1+1−ϵ2e2 k_1=e_1, k_2=ε e_1+ 1-ε^2\,e_2 Then we apply a random orthogonal transformation QrQ_r to both keys k1=Qrk~1,k2=Qrk~2, k_1=Q_r k_1, k_2=Q_r k_2, where the Gram matrix is preserved under this rotation, but removes the coordinate dependence. We choose the state dimension as d=16d=16, correlation as ϵ∈0,0.05,0.10⋯,0.85ε∈\0,0.05,0.10·,0.85\ and use 5 independent rotation. Hence, this gives 5×18=905× 18=90 two-key systems. • General k k-support systems. In this experiment, we use k∈3,4,5,6,8k∈\3,4,5,6,8\ each with 10 independently generated systems and ambient dimension d=20d=20. For each key index μ∈0,⋯,k−1μ∈\0,·,k-1\, we construct the feature vector k~μ=(b,aℓcos(2πℓμk),aℓsin(2πℓμk)ℓ=1m),m=⌊k−12⌋, k_μ= (b, \a_ ( 2π μk ),a_ ( 2π μk ) \^m_ =1 ), m= k-12 , where we sample b∼Unif(1.5,3.0)b (1.5,3.0), aℓ∼Unif(0.15,0.80)a_ (0.15,0.80) independently. Then we normalize very vector and apply a random orthogonal rotation to kμk_μ. Observables. We measure • Two-key: Special coexistence formula. Under a uniform distribution and well-separation condition, the coexistence point occurs at βcoexist=21−ϵ. _coexist= 21-ε. This is the special colsed-form result of Proposition 4.1. • Two-key: Numerical energy crossing. We define the free energy difference ΔΓ(β)=Γα=2(mod)(xloc,β)−Γα=2(mod)(xmix,β), (β)= ^(mod)_α=2(x_loc;β)- ^(mod)_α=2(x_mix;β), where xloc≔k1x_loc k_1 and xmix≔(k1+k2)/2x_mix (k_1+k_2)/2. Then we compute the coexistence point from ΔΓ(β^coexist)=0 ( β_coexist)=0 numerically. • Two-key: Relative error. We define the coexistence relative error as Ecoexist≔|β^coexist−βcoexist|βcoexist. E_coexist | β_coexist- _coexist | _coexist. For the spinodal formula, we have βsp=2/(1−ϵ) _sp=2/(1-ε) in the two-key systems. We obtain the empirical spinodal point by solving c^h(β^sp)=0 c_h( β_sp)=0 numerically, where c^h(β)=Γα=2(mod)(x⋆+hu,β)+Γα=2(mod)(x⋆−hu,β)−2Γα=2(mod)(x⋆,β)h2, c_h(β)= ^(mod)_α=2(x +hu;β)+ ^(mod)_α=2(x -hu;β)-2 ^(mod)_α=2(x ;β)h^2, h=2×10−5h=2× 10^-5, and u≔(k1−k2)/‖k1−k2‖2u (k_1-k_2)/\|k_1-k_2\|_2. Then the spinodal relative error is defined as Esp≔|β^sp−βsp|βsp. E_sp | β_sp- _sp | _sp. • General k k-support: Spinodal formula. From Proposition 4.2, the predicted critical spinodal point is βsp=2λmax(PGP), _sp= 2 _max(PGP), where Pμν=μ,ν−k−1P_μν= 1_\μ,ν\-k^-1, and Gμν=kμ⊤kν/dG_μν=k_μ k_ν/d. We compute the empirical spinodal point β^sp β_sp from the equation c^h(β^sp)=0 c_h( β_sp)=0 numerically. Results. We present the experiments as follows. Figure 16: Two-key coexistence and spinodal validation. We consider two unit keys with correlation ϵε. (a). The solid blue curve presents βcoexist=2/(1−ϵ) _coexist=2/(1-ε). The numerical points are the root of the free energy difference ΔΓ(β^coexist)=0 ( β_coexist)=0. The numerical energy crossings coincide with the theoretical curve over the full correlation range. (b). We plot the relative errors of energy crossing (coexistence) and curvature zero (spinodal). The low values validate the theoretical fromulas in Proposition 4.1 and Proposition 4.2. From Figure 16 (a), the blue curve represents the theoretical formula βcoexist=2/(1−ϵ) _coexist=2/(1-ε). The numerical energy crossing points lie on the theoretical curve. This validates the special coexistence formula. For Figure 16 (b), we plot the relative errors of energy crossing (coexistence) and curvature zero (spinodal). The maximal relative error of the coexistence formula is 5.73×10−155.73× 10^-15, and which of the spinodal formula is 1.72×10−61.72× 10^-6. They both validate the theoretical formulas in Proposition 4.1 and Proposition 4.2 under the uniform distribution and well-separation conditions. Figure 17: General k k-support spinodal validation. We test the general k-support spinodal formula βjump=2/λmax(PGP) _jump=2/ _max(PGP). We consider 50 systems with unit keys. The numerical points agree with the theoretical values, which validates Proposition 4.2. In Figure 17, we compare the theoretical spinodal formula Proposition 4.2 and the empirical spinodal values. Across all 50 general k-support systems, the predicted βsp _sp aligns with the empirical one closely, with the maximal relative error 1.92×10−61.92× 10^-6. Hence, the agreement supports not only the qualitative existence of the transitions, but also the quantitative location of the predicted critical inverse-temperatures. References Amari (1985) S. Amari α-Divergence and α-projection in statistical manifold. In Differential-Geometrical Methods in Statistics, p. 66–103. External Links: ISBN 978-1-4612-5056-2, Document Cited by: Appendix E. Amit et al. (1987) D. J. Amit, H. Gutfreund, and H. Sompolinsky Statistical mechanics of neural networks near saturation. Annals of Physics 173 (1), p. 30–67. External Links: ISSN 0003-4916, Document Cited by: Appendix A, §1, item (d). Amit et al. (1985) D. J. Amit, H. Gutfreund, and H. Sompolinsky Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Lett. 55, p. 1530–1533. External Links: Document Cited by: Appendix A, §1, item (d). Auer et al. (2023) A. Auer, M. Gauch, D. Klotz, and S. Hochreiter Conformal prediction for time series with modern hopfield networks. Advances in neural information processing systems 36, p. 56027–56074. Cited by: §1. Binder (1987) K. Binder Theory of first-order phase transitions. Reports on Progress in Physics 50 (7), p. 783. External Links: Document Cited by: Appendix A, §1, §4. Blondel et al. (2020) M. Blondel, A. F. Martins, and V. Niculae Learning with fenchel-young losses. Journal of Machine Learning Research 21 (35), p. 1–69. Cited by: 4th item, §C.2, §C.7, Proposition C.1, Proposition C.3, Proposition 2.1, 4th item. Coleman and Weinberg (1973) S. Coleman and E. Weinberg Radiative corrections as the origin of spontaneous symmetry breaking. Phys. Rev. D 7, p. 1888–1910. External Links: Document Cited by: Appendix A. Dehmamy et al. (2025) N. Dehmamy, B. Hoover, B. Saha, L. Kozachkov, J. Slotine, and D. Krotov NRGPT: an energy-based alternative for gpt. arXiv preprint arXiv:2512.16762. Cited by: §1. Dembo and Zeitouni (2010) Amir. Dembo and Ofer. Zeitouni Large deviations techniques and applications. 2nd ed. edition, Stochastic modelling and applied probability, 38, Springer, Berlin (eng). External Links: ISBN 9783642033117 Cited by: §D.1, §D.1, Proposition D.3. Demircigil et al. (2017) M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and F. Vermet On a model of associative memory with huge storage capacity. Journal of Statistical Physics 168 (2), p. 288–299. External Links: ISSN 1572-9613, Document Cited by: Appendix A. Enninful et al. (2019) H. R. N. B. Enninful, D. Schneider, A. Hoppe, S. König, M. Fröba, D. Enke, and R. Valiullin Comparative gas sorption and cryoporometry study of mesoporous glass structure: application of the serially connected pore model. Frontiers in Chemistry Volume 7 - 2019. External Links: Document, ISSN 2296-2646 Cited by: item (a). Friedli and Velenik (2017) S. Friedli and Y. Velenik Statistical mechanics of lattice systems: a concrete mathematical introduction. Cambridge University Press. Cited by: §B.2, Proposition B.1, Proposition B.2. Furuichi et al. (2004) S. Furuichi, K. Yanagi, and K. Kuriyama Fundamental properties of tsallis relative entropy. Journal of Mathematical Physics 45 (12), p. 4868–4877. External Links: ISSN 0022-2488, Document, https://pubs.aip.org/aip/jmp/article-pdf/45/12/4868/19300986/4868_1_online.pdf Cited by: Definition E.2. Gini (1912) C. Gini Variabilità e mutabilità. Tipografia di Paolo Cuppini, Bologna. Cited by: 2nd item. Hill (1973) M. O. Hill Diversity and evenness: a unifying notation and its consequences. Ecology 54 (2), p. 427–432. External Links: Document, https://esajournals.onlinelibrary.wiley.com/doi/pdf/10.2307/1934352 Cited by: §F.1, Definition F.1, Proposition F.1. Hoover et al. (2023) B. Hoover, Y. Liang, B. Pham, R. Panda, H. Strobelt, D. H. Chau, M. Zaki, and D. Krotov Energy transformer. Advances in neural information processing systems 36, p. 27532–27559. Cited by: §1. Hopfield (1982) J. J. Hopfield Neural networks and physical systems with emergent collective computational abilities.. Proceedings of the National Academy of Sciences 79 (8), p. 2554–2558. External Links: Document, https://w.pnas.org/doi/pdf/10.1073/pnas.79.8.2554 Cited by: Appendix A, Definition B.1, §1, §1. Hu et al. (2024a) J. Y. Hu, P. Chang, H. Luo, H. Chen, W. Li, W. Wang, and H. Liu Outlier-efficient hopfield layers for large transformer-based models. In International Conference on Machine Learning, Cited by: §1. Hu et al. (2024b) J. Y. Hu, D. Wu, and H. Liu Provably optimal memory capacity for modern hopfield models: transformer-compatible dense associative memories as spherical codes. Advances in Neural Information Processing Systems 37, p. 70693–70729. Cited by: Appendix A. Hu et al. (2023) J. Y. Hu, D. Yang, D. Wu, C. Xu, B. Chen, and H. Liu On sparse modern hopfield model. Advances in neural information processing systems 36, p. 27594–27608. Cited by: Appendix A, 1st item, 2nd item, §1, §3.1, §3.1, §3.1, §3. Kingma and Welling (2013) D. P. Kingma and M. Welling Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: Remark 2.4. Kirkpatrick (1977) S. Kirkpatrick Frustration and ground-state degeneracy in spin glasses. Phys. Rev. B 16, p. 4630–4641. External Links: Document Cited by: Definition B.3. Koulischer et al. (2023) F. Koulischer, C. Goemaere, T. Van Der Meersch, J. Deleu, and T. Demeester Exploring the temperature-dependent phase transition in modern hopfield networks. arXiv preprint arXiv:2311.18434. Cited by: Appendix A, Remark 5.1. Krotov and Hopfield (2018) D. Krotov and J. Hopfield Dense associative memory is robust to adversarial inputs. Neural Computation 30 (12), p. 3151–3167. External Links: ISSN 0899-7667, Document, https://direct.mit.edu/neco/article-pdf/30/12/3151/1048104/neco_a_01143.pdf Cited by: Appendix A. Krotov and Hopfield (2020) D. Krotov and J. Hopfield Large associative memory problem in neurobiology and machine learning. arXiv preprint arXiv:2008.06996. Cited by: Appendix A, Remark C.2, §1, §3.1. Krotov (2023) D. Krotov A new frontier for hopfield networks. Nature Reviews Physics 5, p. . External Links: Document Cited by: Appendix A. Kullback and Leibler (1951) S. Kullback and R. A. Leibler On information and sufficiency. The annals of mathematical statistics 22 (1), p. 79–86. Cited by: Definition E.2. Landau and Lifshitz (1980) L. D. Landau and E. M. Lifshitz Statistical Physics, Part 1. Course of Theoretical Physics, Vol. 5, Butterworth-Heinemann, Oxford. External Links: Document, ISBN 978-0-7506-3372-7 Cited by: Appendix A, §D.1, Proposition D.1, §1. Lee et al. (2016) J. D. Lee, M. Simchowitz, M. I. Jordan, and B. Recht Gradient descent converges to minimizers. arXiv preprint arXiv:1602.04915. Cited by: Appendix E. Martins and Astudillo (2016) A. Martins and R. Astudillo From softmax to sparsemax: a sparse model of attention and multi-label classification. In International conference on machine learning, p. 1614–1623. Cited by: 2nd item, §C.7, §F.1, Definition F.2, Lemma F.1, 1st item, §1, 2nd item. Mezard et al. (1986) M. Mezard, G. Parisi, and M. Virasoro Spin glass theory and beyond. edition, WORLD SCIENTIFIC, . External Links: Document, https://w.worldscientific.com/doi/pdf/10.1142/0271 Cited by: Appendix A, §1. Millidge et al. (2022) B. Millidge, T. Salvatori, Y. Song, T. Lukasiewicz, and R. Bogacz Universal hopfield networks: a general framework for single-shot associative memory models. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 15561–15583. Cited by: Appendix A. Moshe and Zinn-Justin (2003) M. Moshe and J. Zinn-Justin Quantum field theory in the large n limit: a review. Physics Reports 385 (3–6), p. 69–228. External Links: ISSN 0370-1573, Document Cited by: Appendix A. Naudts (2004) J. Naudts Generalized thermostatistics and mean-field theory. Physica A: Statistical Mechanics and its Applications 332, p. 279–300. External Links: ISSN 0378-4371, Document Cited by: Proposition D.7. Nishimori (2001) H. Nishimori In Statistical Physics of Spin Glasses and Information Processing: An Introduction, External Links: ISBN 9780198509417, Document, https://academic.oup.com/book/0/chapter/147881172/chapter-ag-pdf/56732709/book_5185_section_147881172.ag.pdf Cited by: §1, item (d). Onsager (1944) L. Onsager Crystal statistics. i. a two-dimensional model with an order-disorder transition. Phys. Rev. 65, p. 117–149. External Links: Document Cited by: Appendix A. Peters et al. (2019) B. Peters, V. Niculae, and A. F. Martins Sparse sequence-to-sequence models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 1504–1519. Cited by: Appendix A, 3rd item, §C.7, §F.1, Definition F.2, Lemma F.1, 2nd item, §1, 3rd item. Ramsauer et al. (2020) H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217. Cited by: Appendix A, Remark C.2, §1, §1, §3.1. Rockafellar (1997) R. T. Rockafellar Convex analysis. Vol. 28, Princeton university press. Cited by: §C.1, §C.2, Definition C.6, Remark C.2, §D.4, Lemma D.1, Fact 1. Santos et al. (2025) S. Santos, V. Niculae, D. McNamee, and A. F. Martins Hopfield-fenchel-young networks: a unified framework for associative memory retrieval. Journal of Machine Learning Research 26 (265), p. 1–51. Cited by: Appendix A, 2nd item. Shannon (1948) C. E. Shannon A mathematical theory of communication. Bell System Technical Journal 27 (3), p. 379–423. Cited by: 1st item. Sherrington and Kirkpatrick (1975) D. Sherrington and S. Kirkpatrick Solvable model of a spin-glass. Phys. Rev. Lett. 35, p. 1792–1796. External Links: Document Cited by: Appendix A. Srednicki (2007) M. Srednicki Quantum field theory. Cambridge University Press. External Links: Document, ISBN 978-0-521-86449-7, 978-0-511-26720-8 Cited by: Proposition C.1, §2.1, Proposition 2.1. Thouless et al. (1977) D. J. Thouless, P. W. Anderson, and R. G. Palmer Solution of ’solvable model of a spin glass’. The Philosophical Magazine: A Journal of Theoretical Experimental and Applied Physics 35 (3), p. 593–601. External Links: Document, https://doi.org/10.1080/14786437708235992 Cited by: §D.1, Proposition D.2. Tsallis (1988) C. Tsallis Possible Generalization of Boltzmann-Gibbs Statistics. J. Statist. Phys. 52, p. 479–487. External Links: Document Cited by: §C.3, §C.3, Definition C.8, Definition C.8, Definition C.9, Lemma C.1, Proposition C.2. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. Cited by: Appendix A, §1, §3.2, AI Usage Disclosure. Widrich et al. (2020) M. Widrich, B. Schäfl, M. Pavlović, H. Ramsauer, L. Gruber, M. Holzleitner, J. Brandstetter, G. K. Sandve, V. Greiff, S. Hochreiter, et al. Modern hopfield networks and attention for immune repertoire classification. Advances in neural information processing systems 33, p. 18832–18845. Cited by: §1. Wu et al. (2024) D. Wu, J. Y. Hu, T. Hsiao, and H. Liu Uniform memory retrieval with larger capacity for modern hopfield models. In International Conference on Machine Learning, p. 53471–53514. Cited by: Appendix A, §1. Wu et al. (2023) D. Wu, J. Y. Hu, W. Li, B. Chen, and H. Liu STanhop: sparse tandem hopfield model for memory-enhanced time series prediction. arXiv preprint arXiv:2312.17346. Cited by: 1st item, §3.1, §3.1, §3. Xiong et al. (2020) R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu On layer normalization in the transformer architecture. In International conference on machine learning, p. 10524–10533. Cited by: AI Usage Disclosure. Yang et al. (2022) Y. Yang, z. Huang, and D. P. Wipf Transformers from an optimization perspective. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 36958–36971. Cited by: Appendix A. Zinn-Justin (2002) J. Zinn-Justin Quantum field theory and critical phenomena. Int. Ser. Monogr. Phys. 113, p. 1–1054. Cited by: Appendix A, §D.1, §D.1, Proposition D.3, §1, §2.1, §4.