Paper deep dive
Logit Distance Bounds Representational Similarity
Beatrix M. G. Nielsen, Emanuele Marconato, Luigi Gresele, Andrea Dittadi, Simon Buchholz
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 3:39:03 AM
Summary
The paper investigates the relationship between distributional distance and representational similarity in discriminative models, including autoregressive language models. It introduces a 'logit distance' metric based on logit differences and proves that closeness in this distance guarantees linear representational similarity. In contrast, it demonstrates that KL divergence, while upper-bounding logit distance under strict probability bounds, often fails to provide meaningful control over representational properties in practice. Empirical results on distillation tasks show that minimizing logit distance preserves the teacher's linearly recoverable concepts better than standard KL-based distillation.
Entities (8)
Relation Signals (5)
Logit Distance → guarantees → Representational Similarity
confidence 95% · closeness in this distance does yield linear similarity guarantees
Logit Distance → bounds → Representational Dissimilarity
confidence 93% · prove that it is bounded by the logit distance
KL Divergence → usedin → Knowledge Distillation
confidence 92% · Standard approaches train a student to match a teacher’s predictions by minimizing a KL divergence between output distributions
KL Divergence → failstoguarantee → Representational Similarity
confidence 90% · closeness in KL divergence need not imply high linear representational similarity
Logit Distance → improves → Knowledge Distillation
confidence 88% · logit-distance distillation yields students with higher linear representational similarity
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:For a broad family of discriminative models that includes autoregressive language models, identifiability results imply that if two models induce the same conditional distributions, then their internal representations are equal up to an invertible linear transformation. We ask whether an analogous conclusion holds approximately when the distributions are close instead of equal. Building on the observation of Nielsen et al. (2025) that closeness in KL divergence need not imply high linear representational similarity, we study a distributional distance based on logit differences and show that closeness in this distance does yield linear similarity guarantees. Specifically, we define a representational dissimilarity measure based on the models' identifiability class and prove that it is bounded by the logit distance. We further show that, when model probabilities are bounded away from zero, KL divergence upper-bounds logit distance; yet the resulting bound fails to provide nontrivial control in practice. As a consequence, KL-based distillation can match a teacher's predictions while failing to preserve linear representational properties, such as linear-probe recoverability of human-interpretable concepts. In distillation experiments on synthetic and image datasets, logit-distance distillation yields students with higher linear representational similarity and better preservation of the teacher's linearly recoverable concepts.
Tags
Links
- Source: https://arxiv.org/abs/2602.15438v3
- Canonical: https://arxiv.org/abs/2602.15438v3
Trouble viewing inline? Open PDF directly →
Full Text
250,425 characters extracted from source content.
Expand or collapse full text
Logit Distance Bounds Representational Similarity Beatrix M. G. Nielsen IT University of Copenhagen beat@itu.dk Equal contribution. Emanuele Marconato11footnotemark: 1 University of Trento emanuele.marconato@unitn.it Luigi Gresele University of Copenhagen lg@math.ku.dk Shared last author. Andrea Dittadi22footnotemark: 2 Helmholtz Munich & TUM andrea.dittadi@gmail.com Simon Buchholz22footnotemark: 2 MPI IS, Tübingen sbuchholz@tue.mpg.de Abstract For a broad family of discriminative models that includes autoregressive language models, identifiability results imply that if two models induce the same conditional distributions, then their internal representations agree up to an invertible linear transformation. We ask whether an analogous conclusion holds approximately when the distributions are close instead of equal. Building on the observation of Nielsen et al. (2025) that closeness in KL divergence need not imply high linear representational similarity, we study a distributional distance based on logit differences and show that closeness in this distance does yield linear similarity guarantees. Specifically, we define a representational dissimilarity measure based on the models’ identifiability class and prove that it is bounded by the logit distance. We further show that, when model probabilities are bounded away from zero, KL divergence upper-bounds logit distance; yet the resulting bound fails to provide nontrivial control in practice. As a consequence, KL-based distillation can match a teacher’s predictions while failing to preserve linear representational properties, such as linear-probe recoverability of human-interpretable concepts. In distillation experiments on synthetic and image datasets, logit-distance distillation yields students with higher linear representational similarity and better preservation of the teacher’s linearly recoverable concepts. Figure 1: In the center: The intuition of bounding representational similarity using distributional distance. ΘP_ is the set of probability distributions parametrized by models in Θ (Eq. (1)) which satisfy Asm. 2.1. These distributions are one-to-one with identifiability classes [(,)][( f, g)] in the quotient space Θ/∼L / _L (Khemakhem et al., 2020a). The colored areas in ΘP_ contain the distributions which are ϵε-close to a reference p,p_ f, g, as measured by dlogitd_logit (Def. 3.1, blue area) or by dKLd_KL (Eq. (9), pink area). Our Thm. 3.4 lower-bounds representational similarity (in terms of mCCAm_CCA, blue arrow) using the logit distance dlogitd_logit; similarly, Thm. 3.9 upper-bounds dissimilarity in terms of our drepd_rep (Def. 3.7). In Thm. 3.3 (pink arrow) we prove that dKLd_KL yields weak bounds on dlogitd_logit. We illustrate this with representations of two student models distilled from a teacher on the SUB dataset (Bader et al., 2025), see § 5.2 for details. On the left, a student model trained to minimize a variant of dlogitd_logit (Eq. 19) to the teacher distribution p,p_ f, g preserves linearly encoded concepts (Thm. 4.3): for 66 attributes, we visualize their linear separability in the embeddings by projecting them to two dimensions through LDA (Bishop and Nasrabadi, 2006, Ch. 4.1). Distinct concept attributes can be well separated linearly in this 22d subspace. On the right, for a model trained to minimize dKLd_KL the LDA reduction shows that different concept attributes are not linearly separable, as reflected by the extremely low accuracy in Tab. 2. 1 Introduction It is widely believed that the success of deep learning models depends on the data representation they learn Bengio et al. (2013); yet it is unclear what properties “good” representations have in common Bansal et al. (2021). Accordingly, prior work studies when models with comparable performance exhibit similar internal representations Morcos et al. (2018); Kornblith et al. (2019); Klabunde et al. (2025) and how human-interpretable concepts are encoded within them Bricken et al. (2023); Gurnee and Tegmark (2023). Empirically, many such concepts can be predicted from internal representations by simple linear probes Alain and Bengio (2016); Kim et al. (2018), suggesting substantial linear structure in the representations of successful models Mikolov et al. (2013); Park et al. (2024). However, this regularity is not universal Engels et al. (2024); Li et al. (2025a), and it remains unclear to what extent linear representational properties are consistently shared across models that perform comparably well on the same task. We study this question for a broad family of discriminative models, including autoregressive next-token prediction. Prior identifiability results show that, under a suitable diversity assumption, if two such models induce the same conditional distribution, then their representations are equal up to an invertible linear transformation Khemakhem et al. (2020b); Roeder et al. (2021); Lachapelle et al. (2023), and one can characterize which linear properties are shared across the equivalence class (Marconato et al., 2025). A key question is whether these conclusions hold approximately when two models induce distributions that are close but not equal (Buchholz and Schölkopf, 2024). Nielsen et al. (2025) show that the answer depends on how distributional closeness is measured: in particular, models can be arbitrarily close in KL divergence while their representations remain far from linearly related. On the other hand, robustness can be recovered under suitable divergences for which distributional closeness does imply representational similarity. In this work, we ask: when two models in this family induce similar conditional distributions, to what extent do their representations agree up to an invertible linear map? We introduce a distributional distance based on logit differences and prove quantitative representation-level guarantees: small logit distance (i) implies high linear representational similarity, yielding an explicit lower bound on mCCA between representation spaces Raghu et al. (2017); Morcos et al. (2018) and (i) upper-bounds a representational dissimilarity measure that we design to respect the model family’s equivalence class. Finally, we clarify the (limited) extent to which KL control can recover such guarantees: if model probabilities are bounded away from zero, then the KL divergence upper-bounds the logit distance, but the resulting tight robustness bounds are insufficient for practical settings. Our results have an immediate implication for knowledge distillation. Standard approaches train a student to match a teacher’s predictions by minimizing a KL divergence between output distributions Hinton et al. (2015); yet our theory implies that a student can be very close to its teacher in KL while still learning representations far from being linearly aligned. This motivates alternative objectives, and our theory suggests that minimizing logit differences—an approach considered in prior distillation work (Ba and Caruana, 2014; Menon et al., 2021; Kim et al., 2021)— should yield representations that are closer to being linearly equivalent. Empirically, we compare vanilla (KL-based) distillation to logit-distance distillation and find that the latter preserves the teacher’s representational structure substantially better, both in terms of representational similarity measured by mCCA and our metric (on a synthetic dataset and CIFAR 100100 (Krizhevsky et al., 2009)) and in terms of downstream linear properties recoverable by probing (on the SUB dataset (Bader et al., 2025)). Overall, our work clarifies when linear properties are approximately shared across models whose distributions are close, and suggests a practical distillation takeaway: if representational similarity matters, KL is the wrong loss; a logit-distance objective is a better one. Our contributions can be summarized as follows: • We analyze a logit-difference distance between conditional distributions and show it is a proper metric for the model family we consider (§ 3). • We show KL divergence can upper-bound logit distance under strong assumptions (§ 3.1), but the resulting bounds do not yield meaningful representation guarantees in typical regimes. • We propose a representation dissimilarity measure which is zero if and only if models are in the same linear equivalence class (§ 3.3). • We prove that small logit distance implies high linear representational similarity, giving quantitative guarantees both for mCCA (§ 3.2), our dissimilarity measure (§ 3.4) and for linearly encoded concepts (§ 4). • In distillation experiments, we show empirically that logit matching preserves the teacher’s representational structure better than KL-based distillation (§ 5). 2 Preliminaries In this section, we introduce the notation and model family, recall its linear identifiability under a diversity assumption, and formulate the question of approximate identifiability that motivates our analysis. Let X denote the input space, either continuous or discrete. We assume a data distribution p_ x supported on supp(p)⊆supp(p_ x) , and a finite label set Y of size k, such as tokens in a language model or class labels in a classification problem. We focus on a broad model family which encompasses autoregressive language models (Radford et al., 2019), methods for self-supervised pretraining for image classification (Oord et al., 2018), and several supervised classifiers (Khemakhem et al., 2020b; Ibrahim et al., 2024). We denote this model family by Θ . Each element (,)∈Θ( f, g)∈ induces a conditional distribution p,(y∣)p_ f, g(y x), where :→ℝm f:X ^m is the embedding function and :→ℝm g:Y ^m the unembedding function. The conditional distribution takes the form p,(y∣)=exp(()⊤(y))∑y′∈exp(()⊤(y′)). p_ f, g(y x)= ( f( x) g(y)) _y ( f( x) g(y ))\;. (1) We also indicate the logits generated by the model ():=(()⊤(y1),…,()⊤(yk))⊤, u( x):= ( f( x) g(y_1),\,…,\, f( x) g(y_k) ) \,, (2) so the probability distribution can be equivalently described as a categorical distribution with probability vector softmax(())softmax( u( x)). Without loss of generality, we assume the unembeddings to be centered with zero mean,111Applying a fixed translation to all unembeddings does not change the probability distribution p,p_ f, g in Eq. 1. which also results in the logits being centered: ∑y∈(y)=⟹∑i=1ku()i=0. _y g(y)=0 _i=1^ku( x)_i=0\,. (3) When studying this model family, it is convenient to introduce the shifted unembeddings of a model (,)∈Θ( f, g)∈ . For any pivot label y~∈ y , these are given by defining ~(y)≔(y)−(y~) g(y) g(y)- g( y). We then introduce the matrix of shifted unembeddings constructed from ~ g only considering a subset of m labels =y1,…,ym⊆∖y~J=\y_1,…,y_m\ \ y\: ~:=(~(y1)⋯~(ym))∈ℝm×m. L_J:= pmatrix g(y_1)&·s& g(y_m) pmatrix ^m× m\,. (4) Identifiability of the model class. Several works (Khemakhem et al., 2020b; Roeder et al., 2021; Lachapelle et al., 2023; Marconato et al., 2025; Reizinger et al., 2025; Nielsen et al., 2025) have investigated the symmetries belonging to the model class for which, two different choices of embeddings and unembeddings, say (,),(′,′)∈Θ( f, g),( f , g )∈ , generate the same conditional probability distribution. In our setting, similar to (Nielsen et al., 2025), we study the case where labels are abundant and their number exceeds that of the representations, i.e., k>mk>m. Below, we report the linear identifiability result from (Roeder et al., 2021). This requires an extra condition, known as diversity, which can be interpreted as a restriction to models that span the whole representation space (Marconato et al., 2025). Assumption 2.1 (Diversity). A model (,)∈Θ( f, g)∈ satisfies the diversity condition if dim(span():∈supp(p))=m.dim (span\ u( x): x (p_ x)\ )=m\,. (5) In particular, there exists a choice of y~∈ y and ⊆∖y~J \ y\, for which ~ L_J (Eq. 4) is invertible. This condition gives the following result. Theorem 2.2 (Linear Identifiability). For two models (,),(′,′)∈Θ( f, g),( f , g )∈ that satisfy the diversity condition (Asm. 2.1), let y~∈ y and ⊆∖y~J \ y\ be a choice of pivot point and m labels such that ~ L_J is invertible. Let ~≔~−⊤~′⊤∈ℝm×m A_J L_J^- L_J ^m× m. Let ∼L _L be the equivalence relation in Θ , such that (,)∼L(′,′)( f, g) _L( f , g ) if and only if there exists a matrix A such that ()=′(),~(y)=−⊤~′(y). f( x)= A f ( x), g(y)= A^- g (y)\,. (6) Then we have that p,(y∣)=p′,′(y∣),∀(,y)∈×⇔(,)∼L(′,′). split&p_ f, g(y x)=p_ f , g (y x),\;∀( x,y) ×Y\\ & ( f, g) _L( f , g )\,. split (7) In particular, we can set =~ A= A_J. This result shows that when the distributions generated by two models are equal and satisfy the diversity condition, their representations are related by an invertible linear transformation given by ~∈ℝm×m A_J ^m× m. We show in Proposition C.1 that the form of ~ A_J does not depend on the choice of y~ y and ⊆∖y~J \ y\ as long as the matrix ~ L_J is invertible. Closeness in distribution and representational similarity. The result in Thm. 2.2 assumes exact equality of conditional distributions. In practice, however, independently trained models will almost never satisfy this condition exactly, even when trained on the same data. This raises the question of whether the identifiability conclusion is robust: namely, whether models whose distributions are merely close, rather than equal, must also have similar internal representations up to the model class’s intrinsic symmetry (an invertible linear change of basis). This question was first systematically investigated by Nielsen et al. (2025). Addressing it requires specifying (i) a notion of distance between conditional distributions and (i) a notion of dissimilarity between representations. For the latter, we require the dissimilarity to vanish for ∼L _L-linearly equivalent models. This is because, for any p,(y∣)p_ f, g(y x), different reparametrizations of (,)( f, g) exists that do not change the probability distribution (by Thm. 2.2). With this requirement in place, our goal is to identify distributional distances for which small discrepancies in conditional distributions provably imply small representational dissimilarity. 3 The Logit Distance and Connection to Representational Similarity We now define a metric between distributions from our model family. Definition 3.1. Let p,,p′,′p_ f, g,p_ f , g be two sets of conditional distributions arising from two models from our model class. We denote by dlogitd_logit the following logit distance dlogit2(p,,p′,′)=∼p‖()−′()‖22. d_logit^2(p_ f, g,p_ f , g )=E_ x p_ x \| u( x)- u ( x) \|_2^2\;. (8) Notice that dlogitd_logit coincides with the square root of the mean squared error between models’ logits. This can be written in terms of log-probabilities (see Appendix D.1 for details), and the expression ∥()−′()∥2 u( x)- u ( x) _2 corresponds to the Aitchison distance (see § D.2) of the conditional distributions p,(⋅|)p_ f, g(·| x) and p′,′(⋅|)p_ f , g (·| x) which is a metric in compositional data analysis (Aitchison et al., 2000; Pawlowsky-Glahn et al., 2007). We show in § D.3 that Def. 3.1 is a metric (potentially with infinite value) between distributions from models in our model class. A sufficient condition for dlogit(p,p′)<∞d_logit(p,p )<∞, is that the probabilities are lower bounded by some τ>0τ>0 for all y∈y and ∈ x . 3.1 KL Divergence Bounds Logit Distance We consider the relationship between the logit distance and the Kullback–Leibler (KL) divergence which both measure the closeness between two distributions. Following Nielsen et al. (2025), given (,),(′,′)∈Θ( f, g),( f , g )∈ , we consider dKL(p,,p,′)≔∼p[KL(p,(⋅∣)||p′,′(⋅∣))],d_KL(p_ f, g,p_ f, g ) _ x p_ x[KL(p_ f, g(· x)||p_ f , g (· x))]\,, (9) which is related to the cross-entropy loss ℒp(′,′)=∼p,y∼p(y∣)[−logp′,′(y|)] _p( f , g )=E_ x p_ x,y p(y x)[- p_ f , g (y| x)] (10) for a well-specified model through dKL(p,,p,′)=ℒp,(′,′)−ℒp,(,). d_KL(p_ f, g,p_ f, g )=L_p_ f, g( f , g )-L_p_ f, g( f, g)\;. (11) This means that learning a model (′,′)∈Θ( f , g )∈ on data sampled from a distribution p,p_ f, g in the model class, will be equivalent in expectation to minimizing dKL(p,,p′,′)d_KL(p_ f, g,p_ f , g ). However, Nielsen et al. (2025, Theorem 3.1) show that the equivalence relation ∼L _L is not robust to small changes in the probability distribution as measured by dKLd_KL. Specifically, they construct pairs (,)( f, g) and (′,′)( f , g ) with arbitrarily small KL divergence, yet whose representations cannot be made close by any invertible linear map (low mCCAm_CCA). This can happen because, without additional regularity assumptions on Θ , models may assign extremely small probabilities to some events. Changes concentrated on such near-zero-probability events can correspond to large (potentially nonlinear) changes in representation geometry while incurring only a negligible increase in dKLd_KL. To rule out these corner cases, we restrict attention to models (,)∈Θ( f, g)∈ whose probability assignments are uniformly lower bounded by some constant τ>0τ>0. Assumption 3.2 (τ-lower bounded). For τ>0τ>0, we say that a model (,)∈Θ( f, g)∈ is τ-lower bounded if, for p_ x-almost every ∈ x , we have miny∈p,(y∣)≥τ. _y p_ f, g(y x)≥τ\,. (12) We also show in § E.1 that this is equivalent to having bounded embeddings and unembeddings. We can now prove the next result (full proof in § E.2). Theorem 3.3. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ , we have dlogit2(p,,p′,′)≥2dKL(p,,p′,′).d_logit^2(p_ f, g,p_ f , g )≥ 2d_KL(p_ f, g,p_ f , g )\,. (13) If there exists 0<τ<1/30<τ<1/3 such that both (,)( f, g) and (′,′)( f , g ) are τ-lower bounded (Asm. 3.2), then dlogit2(p,,p′,′)≤4log(τ)2τdKL(p,,p′,′).d_logit^2(p_ f, g,p_ f , g )≤ 4 (τ)^2τd_KL(p_ f, g,p_ f , g )\,. (14) In contrast to the general setting discussed above—where arbitrarily small dKLd_KL may still correspond to highly dissimilar representations—Thm. 3.3 shows that, under the additional τ-boundedness assumption, the KL divergence does bound the logit distance. As we show below, this in turn implies that dKLd_KL also bounds representation dissimilarity. However, the upper bound in (14) is only quantitatively meaningful when dKL=O(τ)d_KL=O(τ), since the bound scales as τ−1τ^-1 up to logarithmic factors. As τ≤1/kτ≤ 1/k in a k-class setting and is typically much smaller in practice, this condition can be restrictive. We remark that a similar but weaker bound holds when only one of the models (e.g., a constrained student) is τ-bounded (see Lemma E.4). Moreover, the dependence on τ is tight up to logarithmic terms. 3.2 Logit Distance Bounds Mean CCA We next show that the representations are approximately linearly related when the distance between the logits of two models is small. Canonical Correlation Analysis (CCA) (Hotelling, 1936) is a common way of measuring whether sets of vectors are linearly related (see § B.1). We denote by mCCAm_CCA the mean canonical correlation which is 1 when two vector-valued variables are almost surely linear transformations of each other. In our setting, the pair of variables will be either a pair of embeddings or unembeddings from two different models. Theorem 3.4. Let m be the dimension of the representations. Let μm _m be the m-th eigenvalue of the matrix ,≔∼p[()()⊤]−∼p[()]∼p[()⊤] _ u, u _ x p_ x [ u( x) u( x) ]-E_ x p_ x [ u( x)]E_ x p_ x [ u( x) ]. Assume μm>0 _m>0. Then we have mCCA((),′())≥1−dlogit2(p,,p′,′)mμm m_CCA( f( x), f ( x))≥ 1- d_logit^2(p_ f, g,p_ f , g )m _m (15) and the same bound holds for mCCA((y),′(y))m_CCA( g(y), g (y)). Full proof in § F.4. We emphasize that this result is invariant to linear transformations (,)↦(,−⊤)( f, g) ( A f, A^- g). The theorem shows that, with a small difference in distributions in terms of dlogitd_logit of two models, we get similar representations in terms of mCCAm_CCA. In particular, if the distributions of p,p_ f, g and p′,′p_ f , g agree, we conclude that mCCA((),′())=1m_CCA( f( x), f ( x))=1, which means that there exists a matrix B such that ()=′() f( x)= B f ( x) for all x and similarly (y)=~(y) g(y)= B g(y) for another matrix ~ B. Therefore this result generalizes the linear identifiability of f and g in Thm. 2.2 to models that are similar in terms of dlogitd_logit or dKLd_KL (via Theorem 3.3). However, having mCCA((),′())=1m_CCA( f( x), f ( x))=1 and small KL divergence is not sufficient to conclude that the joint representations (,)( f, g) and (′,′)( f , g ) are similar (see App. G for an example), and while we can conclude that =~−⊤ B= B^- holds for dlogit=0d_logit=0 and thus recover the identifiability result, no rates for the error can be obtained when dlogitd_logit is positive (see Appendix F.3). This motivates the study of further dissimilarity measures which naturally incorporate the relation of embeddings and unembeddings and therefore better match the linear identifiability result. 3.3 Linear Identifiability Dissimilarity In this section we define a dissimilarity measure, drepd_rep, between representations inspired by Thm. 2.2, in the sense that it is zero if and only if the representations are linearly equivalent. We first notice that linear equivalence in Thm. 2.2, mediated by ~ A_J, necessitates a choice of pivot point and m other labels so that ~ L_J is invertible. Since we do not want the invertibility to depend on the choice, we introduce the following assumption: Assumption 3.5. A model (,)∈Θ( f, g)∈ is in general position if, for every y~∈ y and ⊆∖y~J \ y\, the matrix ~∈ℝm×m L_J ^m× m (Eq. 4) is invertible, and the embeddings span the representation space: span():∈supp(p)=ℝmspan\ f( x): x (p_ x)\=R^m. This assumption naturally requires that for any choice of pivot point, y~ y, the shifted unembeddings ~(y) g(y) are in general position (Cover, 1965).222We say n≥mn≥ m vectors are in general position in ℝmR^m if, for any choice of m or fewer vectors, these are linearly independent. It also ensures that the diversity condition (Asm. 2.1) is satisfied. For models satisfying Asm. 3.5, Thm. 2.2 gives us the following corollary: Corollary 3.6. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ that satisfy Asm. 3.5. If p,(y∣)=p′,′(y∣),∀(,y)∈×p_ f, g(y x)=p_ f , g (y x),\;∀( x,y) ×Y then for every choice of pivot point and m labels, we have ()=~′() and ~(y)=~−⊤~′(y). f( x)= A_J f ( x) and g(y)= A_J^- g (y)\,. (16) Equivalently, if there exists a choice of pivot and labels such that the linear relation does not hold, then ∃,y∃ x,y s.t. p,(y∣)≠p′,′(y∣)p_ f, g(y x)≠ p_ f , g (y x). Therefore, if we want to make a dissimilarity measure which measures how far the representations are from being in the same equivalence class —independent of the choice of pivot points and labels— we have to consider the error for all possible choices of labels which can be used to construct the ~ A_J matrix. We capture this in the definition below. Definition 3.7. Let J=(k−1m)J= k-1m. For any two choices of models (,),(′,′)∈Θ( f, g),( f , g )∈ satisfying Asm. 3.5, the squared linear identifiability dissimilarity is given by drep2((,),(′,′))≔1kJ∑y~∈∑⊆∖y~∼p‖()−~′()‖22. d_rep^2(( f, g),( f , g )) 1kJ _ y _J \ y\E_ x p_ x \| f( x)- A_J f ( x) \|_2^2\;. The linear identifiability dissimilarity in Def. 3.7 averages the squared distance between () f( x) and ~′() A_J f ( x) over every choice of pivot and labels and every input. We see that by Corollary 3.6, if the distributions of (,)( f, g) and (′,′)( f , g ) are equal, then drepd_rep is zero. Next, we will show that using the logit distance between distributions, this can be bounded. We will also show that if drep((,),(′,′))=0d_rep(( f, g),( f , g ))=0 then distributions are also equal. 3.4 Logit Distance Bounds Representational Distance In this section, we show that, for our model family, the logit distance is zero if and only if the linear identifiability dissimilarity vanishes and that the former bounds the latter. As a corollary, we show that this bound extends also to KL. Full proofs for this section are in App. H. We note that, for two models (,),(′,′)∈Θ( f, g),( f , g )∈ that satisfy Asm. 3.5, the squared logit distance can be written as a sum of terms of the form ‖−⊤(()−~′())‖22 \| B^- ( f( x)- A_J f ( x) ) \|_2^2 for a matrix B related to the SVD decomposition of ~ L_J (see Thm. H.1 for the precise statement). This shows that the logit difference is naturally connected to the linear identifiability dissimilarity and gives us the following corollary: Corollary 3.8. For (,),(′,′)∈Θ( f, g),( f , g )∈ satisfying Asm. 3.5, we have drep((,),(′,′))=0⇔dlogit(p,,p′,′)=0. d_rep(( f, g),( f , g ))=0 d_logit(p_ f, g,p_ f , g )=0\,. Moreover, we can upper (and lower, see § H.2) bound drepd_rep using dlogitd_logit and the singular values of ~ L_J. Theorem 3.9. Let p,,p′,′p_ f, g,p_ f , g be as before, and m the dimension of the representations. Let C=2mk−1C= 2mk-1. Let σmin _ be the smallest singular value of all ~ L_J. Then dlogitd_logit, bounds the linear identifiability dissimilarity drepd_rep as follows: drep((,),(′,′))≤Cdlogit(p,,p′,′)σmin. d_rep(( f, g),( f , g ))≤ C\; d_logit(p_ f, g,p_ f , g ) _ \;. (17) Interestingly, this result highlights that the connection between distributional distance and representational dissimilarity is tightly connected to both embeddings and unembeddings. In fact, the bound depends on the minimum singular value among the unembeddings matrices ~ L_J, i.e., σmin _ , accounting for the direction in embedding space of minimum variation of the shifted unembeddings. In particular, small values of dlogitd_logit will ensure that the embeddings of the two models are close to being a linear invertible transformation of each other, where the transformation is mediated by the unembeddings. This also links to a bound on drepd_rep through dKLd_KL when models have lower bounded probabilities: Corollary 3.10. Let (,),(′,′)∈Θ( f, g),( f , g )∈ be two models from our model class (1) which satisfy Asm. 3.2 and Asm. 3.5 . Let C=2mk−1C= 2mk-1 and CKL=2C|log(τ)|τC_KL= 2C| (τ)| τ. Then drep((,),(′,′))≤CKLdKL(p,,p′,′)σmin. d_rep(( f, g),( f , g ))≤ C_KL d_KL(p_ f, g,p_ f , g ) _ \,. (18) 4 Applications: Distillation and Interpretability Knowledge distillation (KD) aims to train a compact student model to match the predictive distribution of a larger teacher. In its original formulation (Hinton et al., 2015), KD combines a supervised cross-entropy loss, ℒpL_p_D, on the data distribution p(y∣)p_D(y x) with a KL divergence term that aligns the student distribution with that of a fixed teacher (,)( f, g) with distribution p,p_ f, g. The student (′,′)( f , g ) with distribution p′′p_ f g is obtained by solving (′,′)∈argmin(∗,∗)∈Θ(ℒp(∗,∗)+dKL(p,,p∗,∗))( f , g )∈ *argmin_( f^*, g^*)∈ (L_p_D( f^*, g^*)+d_KL(p_ f, g,p_ f^*, g^*) ) typically using only a small subset of training samples. A large body of work explores refinements of this objective to better match the teacher distribution (Mansourian et al., 2025), including alternative formulations of the KL term (Zhao et al., 2022) or by considering an equivalent loss to dlogitd_logit (Navaneet et al., 2021; Kim et al., 2021). When available, the teacher’s intermediate activations can further guide training (Romero et al., 2015; Huang et al., 2023), with regression on embeddings or unembeddings encouraging closer representational alignment. Our analysis highlights that, when only teacher probabilities are available, the original dKLd_KL minimization does not significantly bound representational similarity (as measured with drepd_rep) between teacher and student. When only teacher probabilities or logits are available, a better learning target is to minimize an equivalent loss to dlogitd_logit. Apart from using dlogitd_logit as a loss directly, we consider training with a L1L_1 variant of the logit distance between teacher logits u and student ′ u , that is ℒlogit1(p,,p′,′)≔∼p‖()−′()‖1, _logit^1(p_ f, g,p_ f , g ) _ x p_ x \| u( x)- u ( x) \|_1\,, (19) Using ℒlogit1L^1_logit improves training stability (Qi et al., 2020) and, importantly, still leads to decreasing dlogit(p,,p′,′)d_logit(p_ f, g,p_ f , g ): Proposition 4.1. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ which both satisfy Assumption 3.2 for some τ>0τ>0, then dlogit2(p,,p′,′)≤2|log(τ)|ℒlogit1(p,,p′,′). d_logit^2(p_ f, g,p_ f , g )≤ 2| (τ)|L^1_logit(p_ f, g,p_ f , g )\,. (20) In particular, with this result we have that minimizing ℒlogit1L_logit^1 will naturally reduce dlogitd_logit between the teacher and student distributions, independently of the representation dimensionality of the two. Full proof in App. J. Synth Teacher KLKL-student L1L_1-student Figure 2: On the left, input data of the Synth dataset (§ 5), where inputs are colored based on their labels. The remaining plots display the model embeddings of the teacher and student models. We notice that embeddings of points belonging to class 1 are nearest neighbors to those of class 6 and class 2 for the teacher and similarly for the L1L_1-student, but not for the KL student. Here, the KL-student has low linear similarity to the teacher (drep≈6.9d_rep≈ 6.9 and mCCA≈0.62m_CCA≈ 0.62), while the L1L_1 student has higher similarity (drep≈1.3d_rep≈ 1.3 and mCCA≈0.99m_CCA≈ 0.99). Interpretable properties. Prior works have investigated whether model embeddings linearly encode latent, interpretable concepts (Mikolov et al., 2013; Kim et al., 2018; Engels et al., 2024). For example, Marks and Tegmark (2023) showed that the truth value of a statement x can be approximated by a linear function of its language model embedding () f( x). Similar forms of linear separability have also been studied in pretrained neural networks for visual concepts in image classification (Alain and Bengio, 2016). Formally, we define a categorical concept as a function :→ΔC−1 h:X→ ^C-1, where ΔC−1 ^C-1 is the simplex over C≥2C≥ 2 values, giving a distribution over concept values p(c∣):=()c,∀c∈1,…,C.p_ h(c x):= h( x)_c,\;∀ c∈\1,…,C\\,. (21) Inspired by previous works (Alain and Bengio, 2016; Rajendran et al., 2024), we propose a definition that captures linear separability of concepts from model embeddings: Definition 4.2 (Linearly encoded concept h in f). For a concept :→ΔC−1 h:X→ ^C-1, with C≥2C≥ 2, giving a distribution p_ h as in Eq. 21, we say that that h is linearly encoded in f if there exist C linear weights c∈ℝm w_c ^m and biases bc∈ℝb_c such that, for all c∈1,…,Cc∈\1,…,C\, exp(c⊤()+bc)∑j=1Cexp(j⊤()+bj)=p(c∣). ( w_c f( x)+b_c ) _j=1^C ( w_j f( x)+b_j )=p_ h(c x)\,. (22) Then, we indicate with ∈ℝm×C W ^m× C the matrix of weights c w_c and with ∈ℝC b ^C the vector of biases bcb_c. In words, Def. 4.2 states that the probability assignments over concept values defined by p_ h can be recovered through a linear function of the model embeddings. This also implies that the embedding space is partitioned linearly into convex regions, each corresponding to the most likely concept label (Snell et al., 2017), as illustrated in Fig. 1 (left). Crucially, we show that dlogitd_logit bounds the degree to which concepts are shared between models that are close: Theorem 4.3. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ and a concept :→ΔC−1 h:X→ ^C-1 with C≥2C≥ 2, if h is linearly encoded in f, then min′,′dKL(p _ W , b d_KL (p_ h ,softmax(⊤′(⋅)+′))≤12∥op2dlogit2(p,,p′,′), ,softmax ( W^ f (·)+ b ) )≤ 12 A _op^2d_logit^2(p_ f, g,p_ f , g )\,, (23) where ∥⋅∥op · _op denotes the operator norm, A is determined by = W= L A, L is the unembedding matrix, and W, b are obtained from Def. 4.2. This result (proof in App. I) also shows that equality between model distributions preserves linearly encoded concepts between models that are linearly equivalent, mirroring previous results (Park et al., 2023; Marconato et al., 2025). 5 Experiments Datasets. We evaluate all methods on three datasets: (1) Synth is a synthetically generated dataset with two-dimensional inputs and 77 labels. The inputs are designed so that the labels are not linearly separable (see Fig. 2 for an illustration). We train models with (m=2)(m=2)-dimensional representations. (2) CIFAR 100100 (Krizhevsky et al., 2009) consists of natural images from 100100 classes. We use representations of dimension m=50m=50. (3) SUB (Bader et al., 2025) is a high-quality synthetic variant of CUB200 (Wah et al., 2011) containing bird images from 3333 classes. Each instance is also annotated with a human-interpretable attribute used during generation (e.g., crest color), yielding 3333 distinct values. We train the teacher using both class labels and concept annotations, and use m=10m=10. Methods. We fix the gauge on unembeddings by centering them as in Eq. 3. In all experiments, we first train a reference teacher model by minimizing the cross-entropy loss on labeled data. We then compare three types of student models: KLKL-students, trained to minimize dKLd_KL to the teacher distribution, L2L_2-students and L1L_1-students, trained to minimize dlogit2d_logit^2 and ℒlogit1L^1_logit, respectively. We use ResNet (He et al., 2016) and DINOv2 (Oquab et al., 2023) as feature extractors for CIFAR and SUB, respectively. All architectural details, hyperparameter choices, and train–validation–test splits are reported in App. K. Teachers and students share the same architectures and representational dimensionality, except on CIFAR, where we use different ResNet sizes. Metrics. We measure the representational similarity between students and teachers using mCCAm_CCA and drepd_rep computed on their embeddings, and distributional distance with dKLd_KL and dlogitd_logit. We report classification accuracy on ground-truth labels, denoted Acc(Y)(Y). On SUB, we additionally evaluate how well linear classifiers recover human-annotated concepts from the embeddings, reported as Acc(C) Acc(C). We evaluate all metrics on the test set and average the results over 55 teacher seeds and 55 student seeds per teacher, reporting means and standard deviations. More details and results are provided in App. K. 5.1 Distillation with ℒlogit1L^1_logit or dlogit2d^2_logit yields more similar representations than dKLd_KL All teachers excel in label prediction on Synth, attaining the maximum Acc(Y) Acc(Y) (Tab. 2). On SUB, they exhibit high accuracy for both labels and concepts (Tab. 2). Performance on CIFAR is moderate. The students, compatibly, score close to teachers in terms of Acc(Y)(Y) on all datasets, with students improving over the teachers on SUB. For Synth and SUB, we observe that L1L_1 and L2L_2 students display smaller dKLd_KL than the KL student, and for all datasets L1L_1 and L2L_2 students have lower dlogitd_logit (even one order of magnitude below KL students in Synth and SUB, see § K.3). In all datasets, we observe a drastic increase in similarity when minimizing ℒlogit1L^1_logit and dlogitd_logit rather than dKLd_KL. L1L_1 and L2L_2 students reduce drepd_rep by an order of magnitude compared to KLKL students in Synth, while L2L_2 reduces by two orders in CIFAR, and three orders of magnitude on SUB. Also, mCCAm_CCA is always higher for L1L_1 and L2L_2, scoring (almost) perfectly on Synth and SUB and improving by more than 2020 p on CIFAR. This results in more similar representations, as qualitatively displayed in Fig. 2 for the Synth dataset. 5.2 Distillation with ℒlogit1L^1_logit or dlogit2d^2_logit recovers linear concepts of the teacher, while dKLd_KL does not We experiment on SUB by training teachers to be (linearly) predictive of both labels and concepts from the embeddings. We observe that KLKL students, while achieving relatively high values of mCCAm_CCA, have worse drepd_rep than L1L_1 and L2L_2 students and fare extremely poorly on concept classification, scoring slightly above a random classifier (1/33≈0.031/33≈ 0.03). On the other hand, L1L_1 and L2L_2 students, while showing lower Acc(Y) Acc(Y), improve over all other metrics. Optimal mCCAm_CCA and lower drepd_rep reflect also in higher Acc(C) Acc(C) (below teacher by 1717 p.p. for L1L_1). This is also qualitatively visible in Fig. 1, where the 2d2d LDA projection on 66 concept attributes shows high linear separability for the L1L_1 student (on the left) but not for KLKL student (on the right). Table 1: Results on Synth and CIFAR. Best in bold. Acc(Y)(↑)( ) drep(↓)d_rep( ) mCCAm_CCA(↑)( ) Synth !30 teach !30 0.999±0.0010.999± 0.001 !30 −- !30 −- 2!30 KLKL 2!30 .999±.001.999±.001 2!30 49.5±2.949.5± 2.9 2!30 .580±.048.580±.048 2!30 L1L_1 2!30 .999±.001.999±.001 2!30 1.44±0.041.44± 0.04 2!30 .999±.001.999±.001 !30 L2L_2 !30 .999±.001.999±.001 !30 0.20±0.010.20± 0.01 !30 .999±.001.999±.001 CIFAR !30 teach !30 .540±.010.540±.010 !30 −- !30 −- 2!30 KLKL 2!30 .480±.001.480±.001 2!30 87.6±13.787.6± 13.7 2!30 .515±.001.515±.001 2!30 L1L_1 2!30 .492±.001.492±.001 2!30 4.56±0.604.56± 0.60 2!30 .767±.001.767±.001 !30 L2L_2 !30 .493±.001.493±.001 !30 0.66±0.170.66± 0.17 !30 .763±.001.763±.001 Table 2: Results on SUB. Teachers are in gray, KL-students in pink, L1L_1-students in light blue, and L2L_2-students in blue. Acc(Y)(↑)( ) Acc(C)(↑)( ) drepd_rep(↓)( ) mCCAm_CCA(↑)( ) !30 0.91±0.010.91± 0.01 !30 0.92±0.010.92± 0.01 !30 −- !30 −- 2!30 0.93±0.010.93± 0.01 2!30 0.06±0.010.06± 0.01 2!30 3100±1703100± 170 2!30 0.79±0.010.79± 0.01 2!30 0.92±0.010.92± 0.01 2!30 0.75±0.010.75± 0.01 2!30 10.0±0.910.0± 0.9 2!30 0.99±0.010.99± 0.01 !30 0.92±0.010.92± 0.01 !30 0.72±0.010.72± 0.01 !30 1.6±0.21.6± 0.2 !30 0.99±0.010.99± 0.01 6 Discussion and Conclusion Limitations. (i) As in prior works (Khemakhem et al., 2020b; Roeder et al., 2021; Nielsen et al., 2025), we assume that the number of labels exceeds the representation dimension plus one. This is satisfied for language models with large vocabularies, but may be restrictive in standard image-classification settings with few classes. In future work, we would thus like to do experiments on language models, where our assumptions are likely to be met. Extending our results to regimes where the number of labels is equal to, or smaller than, the representation dimension (Marconato et al., 2025) is another key next step. (i) In our Def. 3.7, we assumed our models to be in general position (Asm. 3.5). This resembles the diversity assumption (Asm. 2.1), but it is stronger. Similar to diversity, it is satisfied with high probability for random vectors in high dimension (m large) under mild conditions. Notions of representational similarity. Our results show that having models whose distributions are close in KL divergence does not provide strong guarantees for linear representational similarity. If closeness in KL induces a robust notion of representational similarity, it is therefore likely to be of a different kind. Relatedly, Huh et al. (2024) report that standard linear similarity metrics reveal limited convergence, whereas a mutual k-nearest-neighbor metric on representation space uncovers strong alignment across models. Other similarity notions are discussed in App. A. Drivers of representational similarity. Our Thm. 3.3 implies weak linear representational similarity guarantees for KL, which is connected to the cross entropy loss (11); nonetheless, some works report empirical observations of linear representational similarity under the cross entropy loss—e.g., (Reizinger et al., 2025). There may thus be additional factors that drive the observed similarity, for example, learning dynamics in early phases of training (e.g., Frankle et al. (2020); Kapoor et al. (2025)), see also App. A. Distillation and representational similarity. There is also a line of work on similarity-based knowledge distillation that explicitly matches hidden states between teacher and student, for example by aligning embedding spaces or internal features with auxiliary losses (Singh and Wang, 2024; Wang et al., 2023). Our analysis is complementary: rather than adding representational regularizers, we study how purely distributional objectives—KL divergence and our logit-based distance—control (or fail to control) linear representational similarity. Conclusion. We showed that the logit distance is a metric between model distributions and, under a lower bound on conditional probabilities, can be controlled by the KL divergence. Building on this, we introduced the linear identifiability dissimilarity, tightly linked to the identifiability of the model family, and proved that logit distance upper-bounds both this dissimilarity and the mCCA between representations. Empirically, minimizing the logit distance yields student models whose representations are substantially more linearly similar to their teachers’ than when training with standard KL-based distillation. Acknowledgments We thank Anton Rask Lundborg for pointers to the compositional data analysis literature and David Klindt for interesting discussions. B. M. G. N. was supported by the Danish Pioneer Centre for AI, DNRF grant number P1 and partially by the Novo Nordisk Foundation grant NNF24OC0092612. E.M. acknowledges support from TANGO, Grant Agreement No. 101120763, funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. S.B. was supported by the Tübingen AI Center. L.G. was supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (NNF21SA0069429), and by the Pioneer Centre for AI, DNRF grant number P1. References J. Aitchison, C. Barceló-Vidal, J. A. Martín-Fernández, and V. Pawlowsky-Glahn (2000) Logratio analysis and compositional distance. Mathematical geology 32 (3), p. 271–275. Cited by: §3. G. Alain and Y. Bengio (2016) Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. Cited by: §1, §4, §4. J. Ba and R. Caruana (2014) Do deep nets really need to be deep?. Advances in neural information processing systems 27. Cited by: §1. J. Bader, L. Girrbach, S. Alaniz, and Z. Akata (2025) SUB: benchmarking cbm generalization via synthetic attribute substitutions. Proceedings of the IEEE/CVF International Conference on Computer Vision. Cited by: Figure 1, §1, §5. Y. Bansal, P. Nakkiran, and B. Barak (2021) Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems (NeurIPS) 34, p. 225–236. Cited by: Appendix A, §1. Y. Bengio, A. Courville, and P. Vincent (2013) Representation learning: a review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35 (8), p. 1798–1828. Cited by: §1. C. M. Bishop and N. M. Nasrabadi (2006) Pattern recognition and machine learning. Vol. 4, Springer. Cited by: Figure 1. S. Bortolotti, E. Marconato, P. Morettin, A. Passerini, and S. Teso (2025) Shortcuts and identifiability in concept-based models from a neuro-symbolic lens. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Appendix A. L. Braun, E. Grant, and A. M. Saxe (2025) Not all solutions are created equal: an analytical dissociation of functional and representational similarity in deep linear neural networks. In Forty-second International Conference on Machine Learning, Cited by: Appendix A. T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah (2023) Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread. Cited by: §1. S. Buchholz and B. Schölkopf (2024) Robustness of nonlinear representation learning. In International Conference on Machine Learning (ICML), p. 4785–4821. Cited by: §1. L. Ciernik, L. Linhardt, M. Morik, J. Dippel, S. Kornblith, and L. Muttenthaler (2024) Objective drives the consistency of representational similarity across datasets. In Forty-second International Conference on Machine Learning, Cited by: Appendix A. T. M. Cover (1965) Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers 14 (14), p. 326–334. Cited by: §3.3. J. Engels, I. Liao, E. J. Michaud, W. Gurnee, and M. Tegmark (2024) Not all language model features are linear. arXiv preprint arXiv:2405.14860. Cited by: §1, §4. J. Frankle, D. J. Schwab, and A. S. Morcos (2020) The early phase of neural network training. arXiv preprint arXiv:2002.10365. Cited by: Appendix A, §6. N. Goyal, H. Daumé I, A. Drouin, and D. Sridhar (2025) Causal differentiating concepts: interpreting lm behavior via causal representation learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Appendix A. W. Gurnee and M. Tegmark (2023) Language models represent space and time. arXiv preprint arXiv:2310.02207. Cited by: §1. P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun (2023) Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems 36, p. 17643–17668. Cited by: Appendix A. K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §K.1, §5. E. Hernandez, A. S. Sharma, T. Haklay, K. Meng, M. Wattenberg, J. Andreas, Y. Belinkov, and D. Bau (2023) Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124. Cited by: Appendix A. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §1, §4. H. Hotelling (1936) Relations between two sets of variates. Biometrika 28 (3-4), p. 321–377. Cited by: §3.2. T. Huang, Y. Zhang, M. Zheng, S. You, F. Wang, C. Qian, and C. Xu (2023) Knowledge diffusion for distillation. Advances in Neural Information Processing Systems 36, p. 65299–65316. Cited by: §4. M. Huh, B. Cheung, T. Wang, and P. Isola (2024) The platonic representation hypothesis. arXiv preprint arXiv:2405.07987. Cited by: §6. M. Ibrahim, D. Klindt, and R. Balestriero (2024) Occam’s razor for self supervised learning: what is sufficient to learn good representations?. In NeurIPS 2024 Workshop: Self-Supervised Learning-Theory and Practice, Cited by: §2. C. Kapoor, S. Srivastava, and M. Khosla (2025) Bridging critical gaps in convergent learning: how representational alignment evolves across layers, training, and distribution shifts. arXiv preprint arXiv:2502.18710. Cited by: §6. I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen (2020a) Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics (AISTATS), p. 2207–2217. Cited by: Figure 1. I. Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen (2020b) Ice-beem: identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems (NeurIPS) 33, p. 12768–12778. Cited by: §1, §2, §2, §6. B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. (2018) Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International Conference on Machine Learning (ICML), p. 2668–2677. Cited by: §1, §4. T. Kim, J. Oh, N. Kim, S. Cho, and S. Yun (2021) Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. arXiv preprint arXiv:2105.08919. Cited by: §1, §4. M. Klabunde, T. Schumacher, M. Strohmaier, and F. Lemmerich (2025) Similarity of neural network models: a survey of functional and representational measures. ACM Computing Surveys 57 (9), p. 1–52. Cited by: Appendix A, §1. S. Kornblith, M. Norouzi, H. Lee, and G. Hinton (2019) Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), p. 3519–3529. Cited by: Appendix A, §1. A. Krizhevsky, G. Hinton, et al. (2009) Learning multiple layers of features from tiny images. Cited by: Appendix K, §1, §5. S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y. Bengio, S. Lacoste-Julien, and Q. Bertrand (2023) Synergies between disentanglement and sparsity: generalization and identifiability in multi-task learning. In International Conference on Machine Learning (ICML), p. 18171–18206. Cited by: §1, §2. Y. Li, S. Salehi, L. Ungar, and K. P. Kording (2025a) Does object binding naturally emerge in large pretrained vision transformers?. arXiv preprint arXiv:2510.24709. Cited by: §1. Z. M. Li, H. A. Vu, D. Awofisayo, and E. Wenger (2025b) Exploring causes of representational similarity in machine learning models. arXiv preprint arXiv:2505.13899. Cited by: Appendix A. A. M. Mansourian, R. Ahmadi, M. Ghafouri, A. M. Babaei, E. B. Golezani, Z. Y. Ghamchi, V. Ramezanian, A. Taherian, K. Dinashi, A. Miri, et al. (2025) A comprehensive survey on knowledge distillation. arXiv preprint arXiv:2503.12067. Cited by: §4. E. Marconato, S. Lachapelle, S. Weichwald, and L. Gresele (2025) All or none: identifiable linear properties of next-token predictors in language modeling. International Conference on Artificial Intelligence and Statistics (AISTATS). Cited by: Appendix A, Appendix F, §1, §2, §4, §6. E. Marconato, S. Teso, A. Vergari, and A. Passerini (2023) Not all neuro-symbolic concepts are created equal: analysis and mitigation of reasoning shortcuts. Advances in Neural Information Processing Systems 36, p. 72507–72539. Cited by: Appendix A. S. Marks and M. Tegmark (2023) The geometry of truth: emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. Cited by: §4. A. K. Menon, A. S. Rawat, S. Reddi, S. Kim, and S. Kumar (2021) A statistical perspective on distillation. In International Conference on Machine Learning, p. 7632–7642. Cited by: §1. T. Mikolov, W. Yih, and G. Zweig (2013) Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies, p. 746–751. Cited by: Appendix A, §1, §4. A. Morcos, M. Raghu, and S. Bengio (2018) Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems (NeurIPS) 31. Cited by: Appendix A, §1, §1. K. Navaneet, A. Tejankar, S. A. Koohpayegani, and H. Pirsiavash (2021) Regression as a simple yet effective tool for self-supervised knowledge distillation. In British Machine Vision Conference (BMVC) 2021., Cited by: §4. Y. Nesterov (2013) Introductory lectures on convex optimization: a basic course. Vol. 87, Springer Science & Business Media. Cited by: §E.2. B. M. Nielsen, E. Marconato, A. Dittadi, and L. Gresele (2025) When does closeness in distribution imply representational similarity? an identifiability perspective. arXiv preprint arXiv:2506.03784. Cited by: Appendix A, §E.2, §1, §2, §2, §3.1, §3.1, §6. A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §2. M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §K.1, §5. K. Park, Y. J. Choe, and V. Veitch (2023) The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. Cited by: Appendix A, §4. K. Park, Y. J. Choe, and V. Veitch (2024) The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning (ICML), p. 39643–39666. Cited by: §1. V. Pawlowsky-Glahn, J. J. Egozcue, and R. Tolosana-Delgado (2007) Lecture notes on compositional data analysis. Universitat de Girona, Girona. Cited by: §3. J. Qi, J. Du, S. M. Siniscalchi, X. Ma, and C. Lee (2020) On mean absolute error for deep neural network based vector-to-vector regression. IEEE Signal Processing Letters 27, p. 1485–1489. Cited by: §4. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019) Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: §2. M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein (2017) Svcca: singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in Neural Information Processing Systems (NeurIPS) 30. Cited by: Appendix A, §1. G. Rajendran, S. Buchholz, B. Aragam, B. Schölkopf, and P. K. Ravikumar (2024) From causal to concept-based representation learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: Appendix A, §4. P. Reizinger, A. Bizeul, A. Juhos, J. E. Vogt, R. Balestriero, W. Brendel, and D. Klindt (2025) Cross-entropy is all you need to invert the data generating process. In International Conference on Learning Representations (ICLR), Cited by: §2, §6. G. Roeder, L. Metz, and D. Kingma (2021) On linear identifiability of learned representations. In International Conference on Machine Learning (ICML), p. 9030–9039. Cited by: §1, §2, §6. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio (2015) FitNets: hints for thin deep nets. External Links: 1412.6550, Link Cited by: §4. A. Singh and H. Wang (2024) Simple unsupervised knowledge distillation with space similarity. In European Conference on Computer Vision, p. 147–164. Cited by: §6. J. Snell, K. Swersky, and R. Zemel (2017) Prototypical networks for few-shot learning. Advances in neural information processing systems 30. Cited by: §4. A. Stolfo, V. Balachandran, S. Yousefi, E. Horvitz, and B. Nushi (2024) Improving instruction-following in language models through activation steering. arXiv preprint arXiv:2410.12877. Cited by: Appendix A. A. Taeb, N. Ruggeri, C. Schnuck, and F. Yang (2022) Provable concept learning for interpretable predictions using variational autoencoders. arXiv preprint arXiv:2204.00492. Cited by: Appendix A. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie (2011) The caltech-ucsd birds-200-2011 dataset. Cited by: Appendix K, §5. Y. Wang, L. Cheng, M. Duan, Y. Wang, Z. Feng, and S. Kong (2023) Improving knowledge distillation via regularizing feature norm and direction. arXiv preprint arXiv:2305.17007. Cited by: §6. B. Zhao, Q. Cui, R. Song, Y. Qiu, and J. Liang (2022) Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, p. 11953–11962. Cited by: §4. Y. Zheng, S. Xie, and K. Zhang (2025) Nonparametric identification of latent concepts. arXiv preprint arXiv:2510.00136. Cited by: Appendix A. Appendix A Additional Related Work In this section we reference additional related work. Measures of representational similarity and dissimilarity. Similar to our use of mCCAm_CCA, several prior works on representational similarity employ CCA-based measures [Raghu et al., 2017, Morcos et al., 2018] and only measure (dis)similarity between embeddings. Kornblith et al. [2019] introduce the CKA measure, invariant only to linear orthogonal transformations. Nielsen et al. [2025] also introduce a dissimilarity measure for our model class, d,d_ f, g. What constitutes the most appropriate measure of representational similarity remains an open question [Bansal et al., 2021], and existing approaches reflect different objectives and associated trade-offs Klabunde et al. [2025]. Other drivers of representational similarity. Ciernik et al. [2024] show that the training objective strongly impacts how consistently representational similarity generalizes across datasets. Li et al. [2025b] find that dataset and task overlap both correlate strongly with increased representational similarity, especially in combination. The role of early stages of training in shaping shared representational structure was investigated by [Frankle et al., 2020]. Braun et al. [2025] showed that enforcing robustness to parameter noise during training can force more aligned representations across runs. Interpretable properties in model representations. A wealth of works studies learning concepts in weakly-supervised settings [Taeb et al., 2022, Marconato et al., 2023, Rajendran et al., 2024, Zheng et al., 2025, Bortolotti et al., 2025, Goyal et al., 2025]. Our work differs in spirit from these, as we show which distillation schemes preserve already existing linearly encoded concepts in the teacher representations, especially when distributional equality cannot be guaranteed. The notion we propose of linearly encoded concepts shares some similarities to that of Rajendran et al. [2024], but we focus on probability distributions over categorical values instead of conceptual subspaces. Other interpretable properties can be linearly encoded in model representations, such as parallel directions in model unembeddings [Mikolov et al., 2013, Park et al., 2023] and relational linearity in model embeddings of text sentences [Hernandez et al., 2023, Marconato et al., 2025]. Both have also been shown to allowing steering model behavior [Hase et al., 2023, Stolfo et al., 2024]. We expect small values of dlogitd_logit to preserve these properties as well, but we leave an in-depth analysis to future work. Appendix B Definitions and background material In this section, we recall useful mathematical definitions and results for later proofs of main paper claims. B.1 Background on Canonical Correlation Analysis (CCA) Consider two random vectors ∈ℝm X ^m and ∈ℝm′ Y ^m . The first canonical variables are given by (1,1)=argmax∈ℝm,∈ℝm′corr(⋅,⋅) ( a_1, b_1)= *argmax_ a ^m, b ^m corr( a· X, b· Y) (24) where corrcorr denotes the correlation coefficient. And we denote by ρ1=max∈ℝm,∈ℝm′corr(⋅,⋅) _1= _ a ^m, b ^m corr( a· X, b· Y) (25) the first canonical correlation. The second and all further canonical variables and canonical correlations are defined similarly except that we restrict the maximum to the orthogonal complements of all previous canonical variables. There are a total of min(m,m′) (m,m ) canonical correlations. It can be shown that these correspond to the left and right singular vectors and the singular values of the matrix −1/2−1/2 _ X X^-1/2 _ X Y _ Y Y^-1/2 (26) where _ X X and _ Y Y are the covariance matrices of X and Y and =((−())(−())⊤) _ X Y=E (( X-E( X))( Y-E( Y)) ) (27) the cross correlation. Clearly, this is invariant to linear transformations of the variables. For m=m′m=m We consider the mean CCA mCCA=1m∑i=1mρi. m_CCA= 1m _i=1^m _i. (28) Variables X and Y are linearly related iff mCCA=1m_CCA=1. Sums of canonical correlations can be related to matrix norms through (26), e.g., by ∑i=1min(m,m′)ρi2=Tr(−1/2−1−1/2). _i=1 (m,m ) _i^2=Tr ( _ X X^-1/2 _ X Y _ Y Y^-1 _ Y X _ X X^-1/2 ). (29) B.2 Von Neumann’s trace inequality For reference we state von Neumann’s trace inequality which is a bound for the trace of the product of two positive semi-definite matrices. Theorem B.1 (Von Neumann’s trace inequality). For positive semi-definite symmetric (or hermitian) matrices ,∈ℝm×m A, B ^m× m with eigenvalues a1≥a2≥…≥ama_1≥ a_2≥…≥ a_m and b1≥b2≥…≥bmb_1≥ b_2≥…≥ b_m the following bound holds ∑i=1maibm+1−i≤Tr()≤∑i=1maibi. _i=1^ma_ib_m+1-i ( A B)≤ _i=1^ma_ib_i. (30) Appendix C Theoretical Material for Section 2 Proposition C.1. For any two models (,),(′,′)∈Θ( f, g),( f , g )∈ that satisfy the diversity condition (Asm. 2.1) and such that p,(y∣)=p′,′(y∣),∀(,y)∈×,p_ f, g(y x)=p_ f , g (y x), ∀( x,y) ×Y\,, (31) any choice of y~ y and ~⊆∖y~ J \ y\ such that ~ L_J is invertible, gives the same matrix =~=~−⊤~⊤′ A= A_J= L_J^- L_J^ . Proof. Consider two difference choices of pivots y~,y^∈ y, y and corresponding m-dimensional subsets =y~1,…,y~m⊆∖y~J=\ y_1,…, y_m\ \ y\ and =y^1,…,y^m⊆∖y^K=\ y_1,…, y_m\ \ y\ for which, both ~=((y~1)−(y~)⋯(y~m)−(y~)),^=((y^1)−(y^)⋯(y^m)−(y^)) L_J= pmatrix g( y_1)- g( y)&·s& g( y_m)- g( y) pmatrix, L_K= pmatrix g( y_1)- g( y)&·s& g( y_m)- g( y) pmatrix (32) are m×m× m invertible matrices. Because they are invertible, there exists an invertible matrix ∈ℝm×m O ^m× m such that ^=~. L_K= L_J O\,. (33) We now make use of Eq. 31 to rewrite f in terms of ′ f by considering shifted unembeddings ~(y):=(y)−(y~) g(y):= g(y)- g( y) logp,(y∣)p,(y~∣) p_ f, g(y x)p_ f, g( y x) =logp′,′(y∣)p′,′(y~∣) = p_ f , g (y x)p_ f , g ( y x) (34) ⟹()⊤~(y) f( x) g(y) =′()⊤~′(y). = f ( x) g (y)\,. (35) In particular, this is true also for ^(y):=(y)−(y^) g(y):= g(y)- g( y), giving ()⊤^(y)=′()⊤^′(y). f( x) g(y)= f ( x) g (y)\,. (36) Considering the subsets ,J,K of labels, we can write ~⊤() L_J f( x) =~⊤′′() = L_J^ f ( x) (37) ^⊤() L_K f( x) =^⊤′′(), = L_K^ f ( x)\,, (38) and taking the inverse of ~⊤ L_J and of ^⊤ L_K , we have () f( x) =~−⊤~⊤′′(), = L_J^- L_J^ f ( x)\,, (39) () f( x) =^−⊤^⊤′′(). = L_K^- L_K^ f ( x)\,. (40) We subtract the two expressions and obtain (~−⊤~⊤′−^−⊤^⊤′)′() ( L_J^- L_J^ - L_K^- L_K^ ) f ( x) = =0 (41) (~−⊤⊤~⊤′−^−⊤^⊤′)′() ( L_K^- O L_J^ - L_K^- L_K^ ) f ( x) = =0 (42) ~−⊤(⊤~⊤′−^⊤′)′() L_K^- ( O L_J^ - L_K^ ) f ( x) =. =0\,. (43) Now, the diversity condition for the two models gives that L_K is an invertible matrix and that ′ f spans the whole ℝmR^m. This means that ⊤~⊤′−^⊤′= O L_J^ - L_K^ =0 (44) and so we get ^⊤′=⊤~⊤′. L_K^ = O L_J^ \,. (45) Combining Eq. 33 and Eq. 45, we get: A =^−⊤^⊤′ = L_K^- L_K^ (46) =~−⊤−⊤⊤~⊤′ = L_J^- O^- O L_J^ (47) =~−⊤~⊤′ = L_J^- L_J^ (48) which shows the claim. ∎ Appendix D Theoretical Material for first part of Section 3: How Logit Distance is Connected to Log Probabilities and Proof that it is a Metric In this section we first show how the logit distance, dlogit(p,p′)d_logit(p,p ), from Def. 3.1 can be written as a difference between log probabilities of sets of distributions with only non-zero probabilities, § D.1. We then show that logit distance is a proper metric, § D.3. D.1 Logit Distance in Terms of log Probabilities We recall that we defined the logits of models from our model class as: ()=((y1)⊤()⋮(yk)⊤()) u( x)= pmatrix g(y_1) f( x)\\ \\ g(y_k) f( x)\\ pmatrix (49) We will show that the logit distance is equal to a difference of log probabilities as in the following result: Lemma D.1. Let p,p′p,p , be distributions generated by the model class Eq. 1, where m is the dimension of the representations. Let N=(k−2m−1)N= k-2m-1. When evaluating dlogit(p,p′)d_logit(p,p ), we have that ‖()−′()‖22=12kN∑y~∈∑⊆∖y~‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j∈‖22. split&\| u( x)- u ( x)\|_2^2= 12kN _ y _J \ y\ \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|_2^2\;. split (50) where on the right we have a vector indexed with j running over the set J. We show the result in two steps: First we show that the logit distance can be written in terms of shifted logits, defined in the following way: i()=((y1)⊤()⋮(yk)⊤())−(yi)⊤()(1⋮1) u_i( x)= pmatrix g(y_1) f( x)\\ \\ g(y_k) f( x)\\ pmatrix- g(y_i) f( x) pmatrix1\\ \\ 1 pmatrix (51) Then we will show that the shifted logits are the same as a difference of log probabilities. Proposition D.2. The following relation holds ‖()−′()‖22=12k∑yi∈‖i()−i′()‖22=12k(k−2m−1)∑yi∈∑⊆∖yi,||=m∑j∈(ui()j−ui′()j)2. || u( x)- u ( x)||^2_2= 12k _y_i || u_i( x)- u _i( x)||^2_2= 12k k-2m-1 _y_i _J \y_i\,\,|J|=m _j (u_i( x)_j-u _i( x)_j )^2. (52) Remark D.3. The first identity is essentially a version of the well-known identity (−′)2=2Var() ( X- X )^2=2Var( X) (53) holding true for two independent copies X and ′ X of a random variable. Proof. We observe that since we assume that the unembeddings (yi) g(y_i) and ′(yi) g (y_i) are centred we get (()−′())⋅(1⋮1)=∑i=1k()⊤(yi)−′()⊤′(yi)=()⊤(∑i=1k(yi))−′()⊤(∑i=1k′(yi))=0. ( u( x)- u ( x) )· pmatrix1\\ \\ 1 pmatrix= _i=1^k f( x) g(y_i)- f ( x) g (y_i)= f( x) ( _i=1^k g(y_i) )- f ( x) ( _i=1^k g (y_i) )=0. (54) We then note that using the definition of i u_i and i′ u_i and then expand the squared norm we get ∥i()−i′()∥2=∥()−′()∥2−2(()−′())⋅(1⋮1)((yi)⊤()−′(yi)⊤′())+((yi)⊤()−′(yi)⊤′())2‖(1⋮1)‖2=∥()−′()∥2+k((u())i−(u′())i)2. split u_i( x)- u_i ( x) ^2&= u( x)- u ( x) ^2-2( u( x)- u ( x))· pmatrix1\\ \\ 1 pmatrix( g(y_i) f( x)- g (y_i) f ( x))\\ & +( g(y_i) f( x)- g (y_i) f ( x))^2 pmatrix1\\ \\ 1 pmatrix ^2\\ &= u( x)- u ( x) ^2+k((u( x))_i-(u ( x))_i)^2. split (55) Summing this over i we get ∑i=1k∥i()−i′()∥2=k∥()−′()∥2+k∑i=1k((u())i−(u′())i)2=2k∥()−′()∥2 _i=1^k u_i( x)- u_i ( x) ^2=k u( x)- u ( x) ^2+k _i=1^k((u( x))_i-(u ( x))_i)^2=2k u( x)- u ( x) ^2 (56) and therefore the first relation. For the second relation we first note that by definition ui()i−ui′()i=(()⊤(yi)−()⊤(yi))−(′()⊤′(yi)−′()⊤′(yi))=0. u_i( x)_i-u _i( x)_i=( f( x) g(y_i)- f( x) g(y_i))-( f ( x) g (y_i)- f ( x) g (y_i))=0. (57) Then we use that each index j≠ij≠ i appears in (k−2m−1) k-2m-1 subsets of size m not containing index i. Indeed, we have to select m−1m-1 remaining points of the set of size m from k−2k-2 options (i and j are removed from the k options). Therefore, for each pivot yiy_i ∑⊆∖yi,||=m∑j∈(ui()j−ui′()j)2=(k−2m−1)∑j≠i(ui()j−ui′()j)2=(k−2m−1)∑j=1k(ui()j−ui′()j)2=(k−2m−1)∥i()−i′()∥2. split _J \y_i\,\,|J|=m _j (u_i( x)_j-u _i( x)_j )^2&= k-2m-1 _j≠ i (u_i( x)_j-u _i( x)_j )^2\\ &= k-2m-1 _j=1^k (u_i( x)_j-u _i( x)_j )^2\\ &= k-2m-1 u_i( x)- u _i( x) ^2. split (58) This implies the second part of (52) by summing over yiy_i. ∎ We recall that the matrix of shifted unembeddings constructed from ~ g using a subset of m labels =y1,…,ym⊆∖y~J=\y_1,…,y_m\ \ y\: ~(y):=(y)−(y~),~:=(~(y1)⋯~(ym))∈ℝm×m. g(y):= g(y)- g( y), L_J:= pmatrix g(y_1)&·s& g(y_m) pmatrix ^m× m\,. (59) With this definition we can also write the following result. Lemma D.4. Let N=(k−2m−1)N= k-2m-1. When evaluating dlogit(p,p′)d_logit(p,p ) on distributions, p,p′p,p , generated by the model class Eq. 1, where m is the dimension of the representations, we have that: ‖()−′()‖2=12kN∑y~∈∑⊆∖y~‖~⊤()−~′⊤′()‖22 || u( x)- u ( x)||^2= 12kN _ y _J \ y\\| L_J f( x)- L_J f ( x)\|_2^2 (60) Proof. This is since taking the entries of the shifted logits i() u_i( x) corresponding to the m labels from J, is the same as ~⊤() L_J f( x). ∎ Lemma D.5. When evaluating dlogit(p,p′)d_logit(p,p ) on distributions, p,p′p,p , generated by the model class Eq. 1, where m is the dimension of the representations, we have that: ‖~⊤()−~′⊤′()‖2=‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j∈‖2 split&\| L_J f( x)- L_J f ( x)\|_2= \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|_2 split (61) where on the right we have a vector with entries indexed by j running over the set J. Proof. We consider first the definition of the norm: ‖~⊤()−~′⊤′()‖22 \| L_J f( x)- L_J f ( x)\|_2^2 =∑j=1m(()⊤(yj)−′()⊤′(yj)−()⊤(y~)+′()⊤′(y~))2 = _j=1^m( f( x) g(y_j)- f ( x) g (y_j)- f( x) g( y)+ f ( x) g ( y))^2 (62) We then consider each term of the sum, and see that if we both add and subtract a normalization term for each model, then the value will be unchanged. That is, we have ()⊤(yj)−()⊤(y~)=log(exp(()⊤(yj)))−log(exp(()⊤(y~)))−log(∑ynexp(()⊤(yn)))+log(∑ynexp(()⊤(yn)))=log(p(yj|))−log(p(y~|)) split f( x) g(y_j)- f( x) g( y)&= ( ( f( x) g(y_j)))- ( ( f( x) g( y)))\\ & - ( _y_n ( f( x) g(y_n)) )+ ( _y_n ( f( x) g(y_n)) )\\ &= (p(y_j| x))- (p( y| x)) split (63) And thus we see that ‖~⊤()−~′⊤′()‖22 \| L_J f( x)- L_J f ( x)\|_2^2 =∑j=1m(log(p(yj|))−log(p(y~|))−log(p′(yj|))+log(p(y~|)))2 = _j=1^m( (p(y_j| x))- (p( y| x))- (p (y_j| x))+ (p( y| x)))^2 (64) =∑j=1m(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))2 = _j=1^m ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )^2 (65) =‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j∈‖22. = \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|_2^2. (66) ∎ D.2 Connection between the Logit Distance and the Aitchison Distance In this section we show how the Logit Distance is related to the Aitchison distance. The Aitchison distance takes inputs from the hull of the simplex for C values defined as, ΔC−1==z1,…,zC|zi>0,i=1,…,C,∑i=1Czi=1, ^C-1= \ z=\z_1,...,z_C\|z_i>0,i=1,...,C, _i=1^Cz_i=1 \\;, (67) and the distance is defined for ,∈ΔC−1 z, w∈ ^C-1 as da(,)=12C∑i=1C∑j=1C(log(zjzi)−log(wjwi))2. d_a( z, w)= 12C _i=1^C _j=1^C ( ( z_jz_i )- ( w_jw_i ) )^2\;. (68) Since our models always assign non-zero probabilities and sum to 11, the probabilities are in the hull of the simplex. This means that the squared distance between logits is the same as the Aitchison distance between the probabilities for a given input, x. Proposition D.6. Let p,p′p,p be sets of probabilties from our model class and let (),′() u( x), u ( x) be the logits. Then for each x we have ∥()−′()∥22=da2(p(⋅|),p′(⋅|)). \| u( x)- u ( x)\|_2^2=d_a^2(p(·| x),p (·| x))\;. (69) Proof. We see that (i()−i′())j=()⊤(yj)−()⊤(yi)−′()⊤′(yj)+′()⊤′(yi)=log(p(yj|))log(p(yi|))−log(p′(yj|))log(p′(yi|)). ( u_i( x)- u _i( x) )_j= f( x) g(y_j)- f( x) g(y_i)- f ( x) g (y_j)+ f ( x) g (y_i)= (p(y_j| x)) (p(y_i| x))- (p (y_j| x)) (p (y_i| x))\;. (70) Which means that ‖()−′()‖22 \| u( x)- u ( x)\|_2^2 =12k∑yi∈‖i()−i′()‖22 = 12k _y_i || u_i( x)- u _i( x)||^2_2 (71) =12k∑i=1k∑j=1k(log(p(yj|))log(p(yi|))−log(p′(yj|))log(p′(yi|)))2 = 12k _i=1^k _j=1^k ( (p(y_j| x)) (p(y_i| x))- (p (y_j| x)) (p (y_i| x)) )^2 (72) =da2(p(⋅|),p′(⋅|)) =d_a^2(p(·| x),p (·| x)) (73) ∎ D.3 The Logit Distance is a Metric In this section we provide a full proof that the logit distance is a metric. We will begin by defining a distance between sets of probability distributions which are not necessarily from our model class: Definition D.7. Let p,p′p,p be two sets of conditional distributions with k labels and only non-zero probabilities. Let m∈ℕm , N=(k−2m−1)N= k-2m-1. The shifted log-likelihood distance for p,p′p,p is defined as: dLLDm(p,p′)=12kN∑y~∈∑⊆∖y~∼p‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j‖2 d^m_LLD(p,p )= 12kN _ y _J \ y\E_ x p_ x \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|_2 (74) Theorem D.8. For any m∈ℕm , the function from Def. D.7 is a metric (which can have infinite value) between sets of conditional distributions, p,p′p,p , over k labels with non-zero probabilities. Proof. We will show that Def. D.7 is a metric. That is, i it is non-negative, dLLDm(p,p′)≥0d^m_LLD(p,p )≥ 0, and zero only when all distributions are equal, dLLDm(p,p′)=0⇔p=p′d^m_LLD(p,p )=0 p=p . i It is symmetric, dLLDm(p,p′)=dLLDm(p′,p)d^m_LLD(p,p )=d^m_LLD(p ,p). i It satisfies the triangle inequality, dLLDm(p1,p2)≤dLLDm(p1,p3)+dLLDm(p3,p2)d^m_LLD(p_1,p_2)≤ d^m_LLD(p_1,p_3)+d^m_LLD(p_3,p_2). W.r.t. i) we see that Eq. 74 is non-negative, since it is a norm. We will show that if p=p′p=p , then dLLDm(p,p′)=0d^m_LLD(p,p )=0: When p=p′p=p , p(yj|)=p′(yj|)p(y_j| x)=p (y_j| x) for any choice of yjj=0m\y_j\_j=0^m, therefore dLLDm(p,p′)=0d^m_LLD(p,p )=0. We will now show that if dLLDm(p,p′)=0d^m_LLD(p,p )=0 then p=p′p=p : Assume dLLDm(p,p′)=0d^m_LLD(p,p )=0, that is: 12kN∑y~∈∑⊆∖y~∼p‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j‖=0 12kN _ y _J \ y\E_ x p_ x \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|=0 (75) Since the norm is always non-negative, this means that ∼p‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j‖=0,∀yjj=0m⊆ _ x p_ x \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|=0,\;∀\y_j\_j=0^m (76) Which means that for all choices of yjj=0m⊆\y_j\_j=0^m , we have that each entry in the vector (log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j is zero almost everywhere333Almost everywhere, also abbreviated a.e.: Everywhere except on some null-set, that is except on a set with measure zero.. That is, we have log(p(yj|))−log(p(y~|)) (p(y_j| x))- (p( y| x)) −log(p′(yj|))+log(p′(y~|))=0 - (p (y_j| x))+ (p ( y| x))=0 (77) log(p(yj|))−log(p(y~|)) (p(y_j| x))- (p( y| x)) =log(p′(yj|))−log(p′(y~|)) = (p (y_j| x))- (p ( y| x)) (78) log(p(yj|)p(y~|)) ( p(y_j| x)p( y| x) ) =log(p′(yj|)p′(y~|)) = ( p (y_j| x)p ( y| x) ) (79) p(yj|)p(y~|) p(y_j| x)p( y| x) =p′(yj|)p′(y~|) = p (y_j| x)p ( y| x) (80) for all yj,y~∈y_j, y . Now since p(yi|)p(y_i| x) and p′(yi|)p (y_i| x) are probability distributions, we have that 1p(y~|)=∑j=0cp(yj|)p(y~|)=∑j=0cp′(yj|)p′(y~|)=1p′(y~|) 1p( y| x)= _j=0^c p(y_j| x)p( y| x)= _j=0^c p (y_j| x)p ( y| x)= 1p ( y| x) (81) which means that Eq. 80 gives us p(yj|)=p′(yj|),∀yj∈ p(y_j| x)=p (y_j| x),∀ y_j (82) and almost everywhere w.r.t. x thus, p=p′p=p a.e.. W.r.t. i) Eq. 74 is symmetric, since the norm is symmetric. W.r.t. i): Assume we have three sets of distributions, p1,p2,p3p_1,p_2,p_3 and let us consider dLLDm(p1,p2)d^m_LLD(p_1,p_2). 12kN∑y~∈∑⊆∖y~∼p‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j‖ 12kN _ y _J \ y\E_ x p_ x \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \| (83) =12kN∑y~∈∑⊆∖y~∼p∥(log(p1(yj|′))−log(p2(yj|′))−log(p1(y~|′))+log(p2(y~|′)))j∥ = 12kN _ y _J \ y\E_ x p_ x \| ( (p_1(y_j| x ) )- (p_2(y_j| x ))- (p_1( y| x ) )+ (p_2( y| x )) )_j \| (84) =12kN∑y~∈∑⊆∖y~∼p∥(log(p1(yj|′))−log(p3(yj|′))+log(p3(yj|′))−log(p2(yj|′)) = 12kN _ y _J \ y\E_ x p_ x\|( (p_1(y_j| x ))- (p_3(y_j| x ))+ (p_3(y_j| x ))- (p_2(y_j| x )) −log(p1(y~|′))+log(p3(y~|′))−log(p3(y~|′))+log(p2(y~|′)))j∥ ==- (p_1( y| x ))+ (p_3( y| x ))- (p_3( y| x ))+ (p_2( y| x )))_j\| (85) =12kN∑y~∈∑⊆∖y~∼p∥(log(p1(yj|′)p3(yj|′))−log(p1(y~|′)p3(y~|′))+log(p3(yj|′)p2(yj|′)−log(p3(y~|′)p2(y~|′)))j∥ = 12kN _ y _J \ y\E_ x p_ x \|( ( p_1(y_j| x )p_3(y_j| x ))- ( p_1( y| x )p_3( y| x ))+ ( p_3(y_j| x )p_2(y_j| x )- ( p_3( y| x )p_2( y| x )))_j \| (86) ≤12kN∑y~∈∑⊆∖y~∼p(∥(log(p1(yj|′)p3(yj|′))−log(p1(y~|′)p3(y~|′)))j∥+∥(log(p3(yj|′)p2(yj|′)−log(p3(y~|′)p2(y~|′)))j∥) ≤ 12kN _ y _J \ y\E_ x p_ x ( \|( ( p_1(y_j| x )p_3(y_j| x ))- ( p_1( y| x )p_3( y| x )))_j \|+ \|( ( p_3(y_j| x )p_2(y_j| x )- ( p_3( y| x )p_2( y| x )))_j \| ) (87) =12kN∑y~∈∑⊆∖y~∼p‖(log(p1(yj|′)p3(yj|′))−log(p1(y~|′)p3(y~|′)))j‖ = 12kN _ y _J \ y\E_ x p_ x \|( ( p_1(y_j| x )p_3(y_j| x ))- ( p_1( y| x )p_3( y| x )))_j \| (88) +12kN∑y~∈∑⊆∖y~∼p∥(log(p3(yj|′)p2(yj|′)−log(p3(y~|′)p2(y~|′)))j∥ + 12kN _ y _J \ y\E_ x p_ x \|( ( p_3(y_j| x )p_2(y_j| x )- ( p_3( y| x )p_2( y| x )))_j \| (89) In other words: dLLDm(p1,p2)≤dLLDm(p1,p3)+dLLDm(p3,p2)d^m_LLD(p_1,p_2)≤ d^m_LLD(p_1,p_3)+d^m_LLD(p_3,p_2). ∎ We are now ready to prove the theorem: Theorem D.9. For any two models from our model class, the logit distance from Def. 3.1 is a metric between their sets of distributions, p,p′p,p . Proof. We will show that Def. 3.1 is a metric. That is, i it is non-negative, dlogit(p,p′)≥0d_logit(p,p )≥ 0, and zero only when all distributions are equal, dlogit(p,p′)=0⇔p=p′d_logit(p,p )=0 p=p . i It is symmetric, dlogit(p,p′)=dlogit(p′,p)d_logit(p,p )=d_logit(p ,p). i It satisfies the triangle inequality, dlogit(p1,p2)≤dlogit(p1,p3)+dlogit(p3,p2)d_logit(p_1,p_2)≤ d_logit(p_1,p_3)+d_logit(p_3,p_2). W.r.t. i) we see that Eq. 8 is non-negative, since it is a norm. We will now show that if p=p′p=p , then dlogit(p,p′)=0d_logit(p,p )=0. Assume p=p′p=p . Then, p(yj|)=p′(yj|)p(y_j| x)=p (y_j| x) for all j∈1,…,kj∈\1,...,k\ and x a.e.. Since we saw in Lemma D.1 that ‖()−′()‖2=12kN∑y~∈∑⊆∖y~‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j‖22 || u( x)- u ( x)||^2= 12kN _ y _J \ y\ \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|_2^2 (90) We have that for each x, p(yj|)=p′(yj|),∀j∈1,…,k⟹‖()−′()‖2=0p(y_j| x)=p (y_j| x),∀ j∈\1,...,k\ || u( x)- u ( x)||^2=0, which in turn gives us that ‖()−′()‖=0|| u( x)- u ( x)||=0. So when p=p′p=p , dlogit(p,p′)=0d_logit(p,p )=0 for x a.e.. We will now show that if dlogit(p,p′)=0d_logit(p,p )=0 then p=p′p=p a.e.. Assume dlogit(p,p′)=0d_logit(p,p )=0. Then ‖()−′()‖=0|| u( x)- u ( x)||=0 for x a.e. and Lemma D.1 gives us that ‖(log(p(yj|)p′(yj|))−log(p(y~|)p′(y~|)))j‖2=0,∀yjj=1m⊆,y~∈ \| ( ( p(y_j| x)p (y_j| x) )- ( p( y| x)p ( y| x) ) )_j \|_2=0,\;∀\y_j\_j=1^m , y (91) for x a.e. which is the same as in Eq. 76. By the same steps as in the other proof we get that p=p′p=p a.e.. W.r.t. i) Eq. 8 is symmetric, since the norm is symmetric. W.r.t. i): Assume we have three models with sets of distributions, p,p′,p′p,p ,p and logits (),′(),′() u( x), u ( x), u ( x) and let us consider dlogit(p,p′)d_logit(p,p ). We see that dlogit2(p,p′)=∼p∥()−′()∥22=∼p((()−′())+(′()−′()))2=∼p(()−′())2+2∼p(()−′())⋅(′()−′())+∼p((′()−′())2. splitd^2_logit(p,p )&=E_ x p_ x u( x)- u ( x) ^2_2=E_ x p_ x (( u( x)- u ( x))+( u ( x)- u ( x)) )^2\\ &=E_ x p_ x ( u( x)- u ( x) )^2+2E_ x p_ x ( u( x)- u ( x) )· ( u ( x)- u ( x) )+E_ x p_ x (( u ( x)- u ( x) )^2. split (92) Considering the mixed term, we see that by applying the Cauchy Schwarz inequality first to the inner product on ℝkR^k and then to ∼p(⋅)E_ x p_ x(·) we find ∼p(()−′())⋅(′()−′())≤∼p(∥()−′()∥2⋅∥′()−′()∥2)≤(∼p∥()−′()∥22)1/2⋅(∼p∥′()−′()∥22)1/2. splitE_ x p_ x ( u( x)- u ( x) )· ( u ( x)- u ( x) )& _ x p_ x ( u( x)- u ( x) _2· u ( x)- u ( x) _2 )\\ &≤ (E_ x p_ x u( x)- u ( x) _2^2 )^1/2· (E_ x p_ x u ( x)- u ( x) _2^2 )^1/2. split (93) Combining Eq. 92 and Eq. 93, we get dlogit2(p,p′) d^2_logit(p,p ) ≤∼p(()−′())2+∼p((′()−′())2 _ x p_ x ( u( x)- u ( x) )^2+E_ x p_ x (( u ( x)- u ( x) )^2 (94) +(∼p∥()−′()∥22)1/2⋅(∼p∥′()−′()∥22)1/2 + (E_ x p_ x u( x)- u ( x) _2^2 )^1/2· (E_ x p_ x u ( x)- u ( x) _2^2 )^1/2 (95) =dlogit2(p,p′)+dlogit2(p′,p′)+2dlogit(p,p′)dlogit(p′,p′) =d^2_logit(p,p )+d^2_logit(p ,p )+2d_logit(p,p )d_logit(p ,p ) (96) =(dlogit(p,p′)+dlogit(p′,p′))2 =(d_logit(p,p )+d_logit(p ,p ))^2 (97) Thus, the triangle inequality is satisfied. ∎ Appendix E Theoretical Material for Section 3.1: Bounding logit difference by the KL divergence E.1 Assumption 3.2 guarantees that embeddings and unembeddings are bounded We notice that requiring that a model (,)∈Θ( f, g)∈ is τ-lower bounded (Asm. 3.2) is equivalent to ask that embeddings and unembeddings are bounded. We remark that the unembeddings are always bounded when the number of labels is finite. We first show that bounded f and g implies lower bounded probability. For this, notice that: p,(y∣) p_ f, g(y x) =exp(()⊤(y))∑y′∈exp(()⊤(y′)) = ( f( x) g(y)) _y ( f( x) g(y )) (98) ≥exp(−‖()‖⋅maxy∈‖(y)‖)kexp(‖()‖⋅maxy∈‖(y)‖) ≥ (-|| f( x)||· _y || g(y)|| )k (|| f( x)||· _y || g(y)|| ) (99) =miny∈exp(−2‖()‖⋅‖(y)‖)k = _y (-2|| f( x)||·|| g(y)|| )k (100) ≥essinf∈supp(p)miny∈exp(−2‖()‖⋅‖(y)‖)k. ≥ *ess\,inf_ x (p_ x) _y (-2|| f( x)||·|| g(y)|| )k. (101) The last quantity is finite for bounded f and g and Asm. 3.2 holds for τ=essinf∈supp(p)miny∈p(y∣).τ= *ess\,inf_ x (p_ x) _y p(y x). (102) On the contrary, if embeddings or unembeddings are not bounded, Eq. 12 cannot be satisfied for τ>0τ>0. Since we only consider a finite number of labels we focus on unbounded embeddings. Assume the embeddings are unbounded. Then there is a sequence i x_i so that ∥(i)∥→∞ f( x_i) →∞. By the diversity assumption we find that inf∈Sm−1maxy,y¯∈((y)−(y¯))≥α>0 _ v∈ S^m-1 _y, y ( g(y)- g( y) ) v≥α>0 (103) for some α. Indeed, the left hand side is a continuous function on a compact domain and pointwise positive. Thus we find a sequence yiy_i, y¯i y_i such that (i)⋅((yi)−(y¯i))=(i)⋅maxy,y¯∈((y)−(y¯))≥α∥(i)∥. f( x_i)· ( g(y_i)- g( y_i) )= f( x_i)· _y, y ( g(y)- g( y) )≥α f( x_i) . (104) This implies that p,(y¯i|i)=p,(yi|i)e−(i)⊤((yi)−(y¯i))≤1⋅e−α∥(i)∥→0 p_ f, g( y_i| x_i)=p_ f, g(y_i| x_i)e^- f( x_i) ( g(y_i)- g( y_i) )≤ 1· e^-α f( x_i) → 0 (105) as i→∞i→∞. Therefore, probabilities cannot be τ-bounded for any τ>0τ>0. E.2 Bounding logit distance with KL divergence In this section, we start by considering two generic probability distributions =(p1,…,pd) p=(p_1,…,p_d) and =(q1,…,qd) q=(q_1,…,q_d) with d outcomes, such that pi,qi≥0p_i,q_i≥ 0. We show how to provide bounds for the logit differences ∑i(log(pi)−log(qi))2 _i( (p_i)- (q_i))^2 in terms of the KL-divergence ∑ipilog(pi/qi) _ip_i (p_i/q_i) While these bounds are generally standard, we could not locate the required bounds in the literature and therefore provide complete proofs here. Let us first clarify that this is not generally possible. We consider the two probability distributions with two outcomes (d=2d=2) whose probabilities are given by (p1,p2)=(τ,1−τ)(p_1,p_2)=(τ,1-τ) and (q1,q2)=(τ2,1−τ2)(q_1,q_2)=(τ^2,1-τ^2) for some 0≤τ<10≤τ<1. Then, on one hand, we find ∑i(log(pi)−log(qi))2≥log(τ)2 _i( (p_i)- (q_i))^2≥ (τ)^2 (106) and on the other hand KL(||)=τlog(τ/τ2)+(1−τ)log((1−τ)/(1−τ2))≤τ|log(τ)|. ( p|| q)=τ (τ/τ^2)+(1-τ) ((1-τ)/(1-τ^2))≤τ| (τ)|. (107) Since the former diverges as τ→0τ→ 0 while the latter converges to 0 it is generally not possible to bound the squared logit difference by a function of the KL-divergence only. This is also the root of the observation that small KL-divergence between models in general does not imply similar representations [Nielsen et al., 2025]. We therefore consider assuming a lower bound on pip_i for such a bound. We first assume in addition that qiq_i is also lower bounded. Before stating the bound in this case we provide one auxiliary lemma that we will need in the proof. Lemma E.1. For τ≤1/3τ≤ 1/3 the following bound holds for all x≥τx≥τ log(x)2≤4log(τ)2(xlog(x)−x+1). (x)^2≤ 4 (τ)^2(x (x)-x+1). (108) Proof. We define the function Φ:ℝ+∖1→ℝ :R_+ \1\ given by Φ(x)=log(x)2xlog(x)−x+1. (x)= (x)^2x (x)-x+1. (109) We observe that the function at thet denominator x→xlog(x)−x+1x→ x (x)-x+1 is convex, minimized at x=1x=1, and non-negative on ℝ+R_+ and thus Φ is well defined. We notice that Φ can be extended to a continuous function at 11 by setting Φ(1)=2 (1)=2. To prove the lemma we need to show that Φ(x)≤4log(τ)2 (x)≤ 4 (τ)^2 for x≥τx≥τ. We will proceed in two steps: First we show that Φ is non-increasing and then we bound Φ(τ) (τ). To show that Φ is non-increasing we will need the following auxiliary fact: The function Ψ:ℝ+→ℝ :R_+ given by Ψ(x)=log(x)−1+1x−log(x)22 (x)= (x)-1+ 1x- (x)^22 (110) is decreasing on (0,∞)(0,∞) and satisfies Ψ(1)=0 (1)=0, so in particular Ψ(x)≥0 (x)≥ 0 for x≤1x≤ 1 and Ψ(x)≤0 (x)≤ 0 for x≥1x≥ 1. To show this we note that Ψ′(x)=1x−1x2−log(x)x=−1x2(xlog(x)−x+1)≤0. (x)= 1x- 1x^2- (x)x=- 1x^2 (x (x)-x+1 )≤ 0. (111) Now we calculate Φ′ using (log(x)x−x+1)′=log(x)( (x)x-x+1) = (x) and find Φ′(x)=2log(x)x(xlog(x)−x+1)−log(x)2⋅log(x)(xlog(x)−x+1)2=2(xlog(x)−x+1)2log(x)(log(x)−1+1x−log(x)22)=2(xlog(x)−x+1)2log(x)Ψ(x)≤0 split (x)= 2 (x)x (x (x)-x+1 )- (x)^2· (x) (x (x)-x+1 )^2&= 2 (x (x)-x+1 )^2 (x) ( (x)-1+ 1x- (x)^22 )\\ &= 2 (x (x)-x+1 )^2 (x) (x)≤ 0 split (112) where we used that log(x) (x) and Ψ(x) (x) both change their sign at x=1x=1 so the product is always non-positive. Therefore we conclude that supx∈[τ,τ−1]∖1Φ(x)=Φ(τ)=log(τ)2τlog(τ)−τ+1≤4log(τ)2 _x∈[τ,τ^-1] \1\ (x)= (τ)= (τ)^2τ (τ)-τ+1≤ 4 (τ)^2 (113) where we used τlog(τ)−τ+1≥1/3log(1/3)−1/3+1≥1/4τ (τ)-τ+1≥ 1/3 (1/3)-1/3+1≥ 1/4 for τ≤1/3τ≤ 1/3 in the last step. ∎ We can now state and prove the upper bound. Lemma E.2. Let p and q be two probability distributions satisfying pi≥τp_i≥τ and qi≥τq_i≥τ for all i and some 0<τ≤1/30<τ≤ 1/3. Then ∑i(log(pi)−log(qi))2≤4log(τ)2τKL(||). _i( (p_i)- (q_i))^2≤ 4 (τ)^2τKL( p|| q). (114) Remark E.3. The bound is tight up to the logarithmic terms. Indeed, consider =(2τ,1−2τ) p=(2τ,1-2τ) and =(τ,1−τ) q=(τ,1-τ). Then the left hand side is bigger than log(2)2 (2)^2 while the KL-divergence of these two distributions is bounded by log(2)τ (2)τ. Proof. We write pi=xiqip_i=x_iq_i. By assumption, pi=xiqi≥τp_i=x_iq_i≥τ, which means that xi≥τqi≥τx_i≥ τq_i≥τ. Also, xi=piqi≤1qi≤1τx_i= p_iq_i≤ 1q_i≤ 1τ. So we have xi∈[τ,τ−1]x_i∈[τ,τ^-1]. The starting point is to rewrite the KL-divergence as follows KL(||)=∑ipilog(pi/qi)=∑iqi(xilog(xi)−xi+1). ( p|| q)= _ip_i (p_i/q_i)= _iq_i(x_i (x_i)-x_i+1). (115) Here we used in the last step that ∑iqixi=∑ipi=1=∑iqi _iq_ix_i= _ip_i=1= _iq_i. Recall that the function x→xlog(x)−x+1x→ x (x)-x+1 is convex, minimized at x=1x=1, and non-negative. We apply (108) from Lemma E.1 to control ∑i(log(pi)−log(qi))2=∑ilog(xi)2≤4log(τ)2∑i(xilog(xi)−xi+1)≤4log(τ)2∑iqiτ(xilog(xi)−xi+1)=4log(τ)2τKL(||). split _i( (p_i)- (q_i))^2&= _i (x_i)^2≤ 4 (τ)^2 _i(x_i (x_i)-x_i+1)\\ &≤ 4 (τ)^2 _i q_iτ(x_i (x_i)-x_i+1)= 4 (τ)^2τKL( p|| q). split (116) ∎ We can similarly show a slightly weaker bound without assuming that q is lower bounded. Lemma E.4. Let p and q be two probability distributions satisfying pi≥τp_i≥τ for all i and some 0<τ≤1/30<τ≤ 1/3. Then ∑i(log(pi)−log(qi))2≤12log(τ)2τKL(||)+9τ2KL(||)2. _i( (p_i)- (q_i))^2≤ 12 (τ)^2τKL( p|| q)+ 9τ^2KL( p|| q)^2. (117) Remark E.5. The second term can not be avoided. Note that for =(τ,1−τ) p=(τ,1-τ) and =(ε,1−ε) q=( ,1- ) we find limε→0∑i(log(pi)−log(qi))2log(ε)2=1 _ → 0 _i( (p_i)- (q_i))^2 ( )^2=1 (118) while limε→0KL(,)|log(ε)|=τ. _ → 0 KL( p, q)| ( )|=τ. (119) Proof. As before we write pi=xiqip_i=x_iq_i but in contrast to before xix_i is not upper bounded but only satisfies xi≥pi≥τx_i≥ p_i≥τ. The main idea is to control log(xi)2 (x_i)^2 separately, when x is large we use a new bound based on a similar approach while we can apply the approach from before when x is small because this lower bounds qiq_i. We start with the former. Note that for x≥3x≥ 3 we find log(x)x3≤xlog(x)−x+1. (x)x3≤ x (x)-x+1. (120) Indeed, note that for x=3x=3 23xlog(x)−x+1=23⋅3⋅log(3)−3+1>0 23x (x)-x+1= 23· 3· (3)-3+1>0 (121) and (23xlog(x)−x+1)′=23log(x)+23−1≥13 ( 23x (x)-x+1 ) = 23 (x)+ 23-1≥ 13 (122) for x≥3x≥ 3. This implies that ∑i:xi>3log(xi)i2≤∑i:xi>3(3xixilog(xi)3)2≤∑i:xi>3(3piτxi(xilog(xi)−xi+1))2≤9τ2∑i(qi(xilog(xi)−xi+1))2≤9τ2(∑iqi(xilog(xi)−xi+1))2=9τ2KL(||)2. split _i:\,x_i>3 (x_i)^2_i&≤ _i:x_i>3 ( 3x_i x_i (x_i)3 )^2\\ &≤ _i:x_i>3 ( 3p_iτ x_i(x_i (x_i)-x_i+1) )^2\\ &≤ 9τ^2 _i (q_i(x_i (x_i)-x_i+1) )^2\\ &≤ 9τ^2 ( _iq_i(x_i (x_i)-x_i+1) )^2= 9τ^2KL( p|| q)^2. split (123) Here we used in the last step that all terms in the sum are non-negative which implies that all cross terms in the expansion of the square of the sum are non-negative. We now consider the indices i so that xi<3x_i<3. Note that qi=pi/xi≥τ/3q_i=p_i/x_i≥τ/3. Therefore we obtain using xi≥τx_i≥τ and (108) ∑i:xi≤3log()i2≤4log(τ)2∑i:xi≤3(xilog(xi)−xi+1)≤4log(τ)2∑i:xi≤3qiτ/3(xilog(xi)−xi+1)≤12log(τ)2τ∑iqi(xilog(xi)−xi+1)=12log(τ)2τKL(||). split _i:x_i≤ 3 ( x)^2_i&≤ 4 (τ)^2 _i:x_i≤ 3 (x_i (x_i)-x_i+1 )\\ &≤ 4 (τ)^2 _i:x_i≤ 3 q_iτ/3 (x_i (x_i)-x_i+1 )\\ &≤ 12 (τ)^2τ _iq_i (x_i (x_i)-x_i+1 )= 12 (τ)^2τKL( p|| q). split (124) Combining (123) and (124) ends the proof. ∎ From these results it is straightforward to relate ()⊤(yi) f( x) g(y_i) and ′()⊤′(yi) f ( x) g (y_i). Theorem E.6. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ , denote with () u( x) and ′() u ( x) the model logits i()=()⊤(yi) u_i( x)= f( x) g(y_i) and (′())i=′()⊤′(yi)( u ( x))_i= f ( x) g (y_i). Fix a point x. Assume that miny∈p,(y|)≥τ _y p_ f, g(y| x)≥τ. Then ‖()−′()‖2 \| u( x)- u ( x)\|^2 ≤12log(τ)2τKL(p,(⋅|)||p′,′(⋅|))+9τ2KL(p,(⋅|)||p′,′(⋅|))2. ≤ 12 (τ)^2τKL(p_ f, g(·| x)||p_ f , g (·| x))+ 9τ^2KL(p_ f, g(·| x)||p_ f , g (·| x))^2. (125) If in addition miny∈p′,′(y|)≥τ _y p_ f , g (y| x)≥τ then ‖()−′()‖2 \| u( x)- u ( x)\|^2 ≤4log(τ)2τKL(p,(⋅|)||p′,′(⋅|)). ≤ 4 (τ)^2τKL(p_ f, g(·| x)||p_ f , g (·| x)). (126) Proof. We note that by (1) ()⊤(y)=log(p,(y∣))+log(∑y′∈e()⊤(y′)) f( x) g(y)= (p_ f, g(y x) )+ ( _y e f( x) g(y ) ) (127) Using that g is centred we find 0=1k∑y′∈()⊤(y)=1k∑y∈(log(p,(y∣))+log(∑y′∈e()⊤(y′)))=log(p,(⋅∣))¯+log(∑y′∈e()⊤(y′)). 0= 1k _y f( x) g(y)= 1k _y ( (p_ f, g(y x) )+ ( _y e f( x) g(y ) ) )= (p_ f, g(· x) )+ ( _y e f( x) g(y ) ). (128) Where log(p,(⋅∣))¯ (p_ f, g(· x) ) denote the mean over y∈y . We infer that ()⊤(y)=log(p,(y∣))−log(p,(⋅∣))¯. f( x) g(y)= (p_ f, g(y x) )- (p_ f, g(· x) ). (129) Now we can conclude that ‖()−′()‖2=∑y∈(()⊤(y)−′()⊤′(y))2=∑y∈(log(p,(y∣))−log(p,(⋅∣))¯−log(p′,′(y∣))+log(p′,′(⋅∣))¯)2=∑y∈(log(p,(y∣))−log(p′,′(y∣)))2+||(log(p′,′(⋅∣))¯−log(p,(⋅∣))¯)2+2(log(p′,′(⋅∣))¯−log(p,(⋅∣))¯)∑y∈(log(p,(y∣))−log(p′,′(y∣)))=∑y∈(log(p,(y∣))−log(p′,′(y∣)))2+||(log(p′,′(⋅∣))¯−log(p,(⋅∣))¯)2−2||(log(p′,′(⋅∣))¯−log(p,(⋅∣))¯)2≤∑y∈(log(p,(y∣))−log(p′,′(y∣)))2. split\| u( x)- u ( x)\|^2&= _y ( f( x) g(y)- f ( x) g (y))^2\\ &= _y ( (p_ f, g(y x) )- (p_ f, g(· x) )- (p_ f , g (y x) )+ (p_ f , g (· x) ) )^2\\ &= _y ( (p_ f, g(y x) )- (p_ f , g (y x) ) )^2+|Y| ( (p_ f , g (· x) )- (p_ f, g(· x) ) )^2\\ & +2 ( (p_ f , g (· x) )- (p_ f, g(· x) ) ) _y ( (p_ f, g(y x) )- (p_ f , g (y x) ) )\\ &= _y ( (p_ f, g(y x) )- (p_ f , g (y x) ) )^2+|Y| ( (p_ f , g (· x) )- (p_ f, g(· x) ) )^2\\ & -2|Y| ( (p_ f , g (· x) )- (p_ f, g(· x) ) )^2\\ &≤ _y ( (p_ f, g(y x) )- (p_ f , g (y x) ) )^2. split (130) The last steps mirror the computation for showing that the squared norm is larger than the variance of a random variable. Now we can apply Lemma E.4 to conclude (125) and Lemma E.2 to get (126). ∎ We can integrate the previous result over x to relate the mean logit difference to the cross entropy loss. Corollary E.7. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ , denote with () u( x) and ′() u ( x) the model logits u()i=()⊤(yi)u( x)_i= f( x) g(y_i) and u′()i=′()⊤′(yi)u ( x)_i= f ( x) g (y_i). Assume that for p_ x almost every ∈ x miny∈p,(y|)≥τ,miny∈p′,′(y|)≥τ. _y p_ f, g(y| x)≥τ, _y p_ f , g (y| x)≥τ. (131) Then ∼p‖()−′()‖2 _ x p_ x\| u( x)- u ( x)\|^2 ≤4log(τ)2τdKL(p,,p′,′). ≤ 4 (τ)^2τd_KL(p_ f, g,p_ f , g ). (132) Proof. The result follows by integrating (126) using relation (9). ∎ Remark E.8. Without assuming a lower bound on p′,′(⋅|)p_ f , g (·| x) we cannot derive a similar bound as (KL(p,(⋅|)||p′,′(⋅|))2)E(KL(p_ f, g(·| x)||p_ f , g (·| x))^2) cannot be bounded even when KL(p,(⋅|)||p′,′(⋅|))KL(p_ f, g(·| x)||p_ f , g (·| x)) is small. Lemma E.9. For two distributions p,p′p,p , with (finite) logits (),′()∈ℝk u( x), u ( x) ^k, respectively, we have ∥()−′()∥2≥2KL(p(⋅∣),p′(⋅∣)). u( x)- u ( x) ^2≥ 2KL(p(· x),p (· x))\,. (133) Proof. For any distribution p(y∣)p(y x), the logits are given by considering p(yi∣)=expu()iZ()p(y_i x)= \,u( x)_iZ( x) (134) where Z(()):=∑i=1kexp(u()i)Z( u( x)):= _i=1^k (u( x)_i). The KL divergence for each element ∈ x can be written as: KL(p(⋅∣)||p′(⋅∣)) KL(p(· x)||p (· x)) =∑i=1kp(yi∣)logp(yi∣)p′(yi∣) = _i=1^kp(y_i x) p(y_i x)p (y_i x) (135) =∑i=1keui()Z(())(ui()−ui′())+logZ(′())Z(()) = _i=1^k e^u_i( x)Z( u( x))(u_i( x)-u _i( x))+ Z( u ( x))Z( u( x)) (136) =(∇logZ(()))⊤(()−′())+logZ(′())−logZ(()). =( _ u Z( u( x))) ( u( x)- u ( x))+ Z( u ( x))- Z( u( x))\,. (137) where ∇ _ u is the gradient over the logits. Notice that, by setting F(⋅)=logZ(⋅)F(·)= Z(·), the KL divergence coincides with the Bregman divergence DF(′(),())D_F( u ( x), u( x)). Denote with =softmax() p=softmax( u). Now, considering the Hessian of logZ Z, which is the Fisher information matrix. We get that: ∇2logZ()=diag()−⊤.∇^2_ u Z( u)=diag( p)- p p \,. (138) From this, we observe that ∇2logZ()⪯∇^2 Z( u) I.444Recall that a matrix ⪯ A B if and only if − B- A is positive semi-definite, i.e., ⊤(−)≥0 x ( B- A) x≥ 0 for all x. Equivalently, this implies that the F is 1-smooth [Nesterov, 2013, Theorem 2.1.6] . This means that, for any ,′∈ℝk u, u ^k, we can write ‖∇logZ()−∇logZ(′)‖2≤1‖−′‖2,|| _ u Z( u)- _ u Z( u )||^2≤ 1|| u- u ||^2\,, (139) or equivalently logZ(′)≤logZ()−∇(logZ())⊤(−′)+12||−′||2. Z( u )≤ Z( u)-∇( Z( u)) ( u- u )+ 12|| u- u ||^2\,. (140) Substituting this to Eq. 137, we get: ||()−′()||2≥2KL(p(⋅∣)||p′(⋅∣)),|| u( x)- u ( x)||^2≥ 2KL(p(· x)||p (· x))\,, (141) which gives the result. ∎ Appendix F Bounding Representation Similarity in Terms of Canonical Correlations We now investigate the implications for representation similarity. Recall that we consider two models specified by (,),(′,′)∈Θ( f, g),( f , g )∈ with densities p,p_ f, g and p′,′p_ f , g . Their representational dimensions are m and m′m and in particular we do not assume that they agree. We denote by L and ′ L the matrices of unembeddings, that is =((y1)…(yk))∈ℝm×kand′=(′(y1)⋯′(yk))∈ℝm′×k. L= pmatrix g(y_1)&…& g(y_k) pmatrix ^m× k L = pmatrix g (y_1)&·s& g (y_k) pmatrix ^m × k. (142) Note that the relations ()=⊤(),′()=′⊤′() u( x)= L f( x), u ( x)= L f ( x) (143) hold. We have seen in Theorem E.6 how to bound ∥()−′()∥2=∥⊤()−′⊤′()∥2 u( x)- u ( x) ^2= L f( x)- L f ( x) ^2 (144) in terms of the KL-divergence KL(p,(⋅|)||p′,′(⋅|))KL(p_ f, g(·| x)||p_ f , g (·| x)). The goal of this section is to bound the canonical correlation coefficients between the random variables with samples ((yi))1≤i≤k( g(y_i))_1≤ i≤ k and (′(yi))1≤i≤k( g (y_i))_1≤ i≤ k and the random variables f and ′ f by the expectation of the square logit difference. We denote by ρ1≥ρ2≥…≥ρmin(m,m′) _1≥ _2≥…≥ _ (m,m ) the canonical correlations of g and ′ g which are given by the singular values of (⊤)−1/2(′⊤)(′′⊤)−1/2. ( L L )^-1/2( L L )( L L )^-1/2. (145) Indeed, this holds because ⊤=kΣ,∈ℝm×m L L =k _ g, g ^m× m, ′′⊤=kΣ′,′∈ℝm′×m′ L L =k _ g , g ^m × m , and ′⊤=kΣ,′∈ℝm×m′ L L =k _ g, g ^m× m . Since the distribution p,p_ f, g is not invariant to embeddings shifts + f+ c it is natural to consider the moment-based canonical correlations of f and ′ f where instead of considering covariances one considers the second moments directly and we use the notation ,=∼p(()⊤()). M_ f, f=E_ x p_ x( f( x) f ( x)). (146) We denote the moment-based canonical correlations of f and ′ f by ρ~1≥ρ~2≥…≥ρ~min(m,m′) ρ_1≥ ρ_2≥…≥ ρ_ (m,m ) which are given by the singular values of ,−1/2,′′,′−1/2. M_ f, f^-1/2 M_ f, f M_ f , f ^-1/2. (147) The result below extends to the standard canonical correlations with minor changes which we discuss in Appendix F.4. To simplify the analysis slightly, we assume that L and ′ L have full rank m and m′m and therefore, in particular, k≥max(m,m′)+1k≥ (m,m )+1 (the +1+1 arises because the unembeddings are centred and therefore linearly dependent (∑y∈(y)=0 _y g(y)=0) so the rank can be at most k−1k-1). This implies that ⊤∈ℝm×m L L ^m× m and ′′⊤∈ℝm′×m′ L L ^m × m are invertible matrices. Similarly we assume that the moment matrices , M_ f, f and ′,′ M_ f , f defined in (146) have maximal rank m and m′m respectively. Note that this follows from the diversity condition in Asm. 2.1 under minor regularity assumptions. We expect that extensions to the rank-deficient case are possible by considering the pseudoinverses (see Marconato et al. [2025] for a full discussion of the identifiability perspective in this case). Moreover, note that this assumption is not a real restriction because we could equivalently focus on the space generated by the vectors (yi) g(y_i) and the projection of f on the orthogonal complement of this space is not identifiable anyway and similarly a degenerate , M_ f, f results in arbitrary behaviour of g on the orthogonal complement of the column space of , M_ f, g. Our upper bounds depend on diversity properties of one of the models (,)( f, g) (typically the teacher model in a distillation setup) and it turns out that the right object to consider is the logit covariance ,=∼p(()⊤())=⊤, M_ u, u=E_ x p_ x( u( x) u ( x))= L M_ f, f L (148) which measures how spread out the logit distribution is. We remark that the logits are invariant to the model symmetries (,)→(,−⊤)( f, g)→( A f, A^- g). Note that L and , M_ f, f have maximal rank m and so we denote the m-nonzero eigenvalues of , M_ u, u by μ1≥μ2≥…≥μm _1≥ _2≥…≥ _m. Note that small eigenvalues indicate that the representation does not exploit all dimensions of the embedding space and therefore this spectrum allows to quantify the diversity condition. After these preliminaries we can state the main result of this section. Theorem F.1. Consider two models (,)( f, g) and (′,′)( f , g ) with representation dimensions m and m′m and assume that , M_ f, f, L and ′,′ M_ f , f , ′ L have maximal rank m and m′m respectively. The canonical correlations ρi _i (in decreasing order) of g and ′ g satisfy the bound ∑i=1m(1−ρi2)μi≤∼p∥()−′()∥2 _i=1^m(1- _i^2) _i _ x p_ x u( x)- u ( x) ^2 (149) where μi _i are the eigenvalues of , M_ u, u in decreasing order. Similarly the moment-based canonical correlations ρ~i ρ_i (in decreasing order) of f and ′ f satisfy ∑i=1m(1−ρ~i2)μi≤∼p∥()−′()∥2 _i=1^m(1- ρ_i^2) _i _ x p_ x u( x)- u ( x) ^2 (150) where we in both cases set ρi=ρ~i=0 _i= ρ_i=0 for i>m′i>m if m>m′m>m . The goal of the next lemmas is to provide a proof of Theorem F.1, i.e., to show (149) and (150). The final summary of the proof can be found at the end of Appendix F.2. We note that the two bounds are essentially equivalent due to the symmetry of the problem in f and g, indeed note that the operation ⊤ L L just corresponds (up to a scalar) to building the covariance matrix from a random variables with finitely many values. Nevertheless, we show the two proofs separately below for the convenience of the reader. The general strategy of the proof is to regress L on ′ L and use this to decompose ()−′() u( x)- u ( x) (see Lemma F.2). Then one of the terms in the decomposition can be related to the canonical correlations ρi _i (see Lemma F.4). Finally this relation can be used to bound the canonical correlation in Lemma F.5. We then extend the results to ρ~ ρ using the same steps. F.1 Bounding similarity of unembeddings We consider the regression problem =min¯∈ℝm×m′∥−¯′∥2. B= _ B ^m× m L- B L ^2. (151) The minimizer B of this ordinary least squares problem is given by =′⊤(′′⊤)−1. B= L L ( L L )^-1. (152) We introduce the residual =−′∈ℝm×k. = L- B L ^m× k. (153) We then find the following expansion. Lemma F.2. With the notation introduced above, in particular =′⊤(′′⊤)−1 B= L L ( L L )^-1 and =−′ = L- B L we can decompose ∥()−′()∥2=∥′⊤(⊤()−′())∥2+∥⊤()∥2. u( x)- u ( x) ^2= L ( B f( x)- f ( x)) ^2+ f( x) ^2. (154) Remark F.3. This lemma can be used to bound the norm of the residual ∥ of the regression of g on ′ g . Indeed, we find that ∼p∥()−′()∥2≥∼p∥⊤()∥2=∼pTr(⊤()⊤())=Tr(⊤,)≥λmin(,)∥2. _ x p_ x u( x)- u ( x) ^2 _ x p_ x f( x) ^2=E_ x p_ xTr ( f( x) f ( x) )=Tr ( M_ f, f )≥ _ ( M_ f, f) ^2. (155) However, this bound is not invariant to model reparametrizations while the canonical correlations are. However, it is sufficient to conclude linear identifiability when the logits agree almost everywhere. Proof. Note that ′⊤=′⊤−′⊤(′′⊤)−1′′⊤=, L = L L - L L ( L L )^-1 L L =0, (156) where 0 is an m×m′m× m matrix of zeros. Then we get ∥()−′()∥2=∥⊤()−′⊤′()∥2=∥⊤()+′⊤⊤()−′⊤′()∥2=()⊤⊤()+(⊤()−′())⊤′′⊤(⊤()−′())+()⊤′⊤⏟=0(⊤()−′())+(⊤()−′())⊤′⊤⏟=0()=∥⊤()∥2+∥′⊤(⊤()−′())∥2. split u( x)- u ( x) ^2&= L f( x)- L f ( x) ^2\\ &= f( x)+ L B f( x)- L f ( x) ^2\\ &= f( x) f( x)+( B f( x)- f ( x)) L L ( B f( x)- f ( x))\\ & + f( x) L _=0( B f( x)- f ( x))+( B f( x)- f ( x)) L _=0 f( x)\\ &= f( x) ^2+ L ( B f( x)- f ( x)) ^2. split (157) This completes the proof. ∎ As a next step we have to relate the term ∥⊤(x)∥2 f(x) ^2 to the canonical correlation of (y) g(y) and ′(y) g (y). We first show the following intermediate result. Lemma F.4. Denote the eigenvalues of (⊤)−1/2⊤(⊤)−1/2( L L )^-1/2 ( L L )^-1/2 by λ1≤…≤λm _1≤…≤ _m and the squared canonical correlations of (y1),…,(yk) g(y_1),…, g(y_k) and ′(y1),…,′(yk) g (y_1),…, g (y_k) by ρ12≥…≥ρmin(m,m′)2 _1^2≥…≥ _ (m,m )^2. Then the relation λi=1−ρi2 _i=1- _i^2 (158) holds, where we set ρi=0 _i=0 for m′<i≤m <i≤ m if m′<m <m. Proof. We expand ⊤=⊤−′⊤⊤−′⊤+′′⊤⊤=⊤−′⊤(′′⊤)−1′⊤−′⊤(′′⊤)−1′⊤+′⊤(′′⊤)−1′′⊤(′′⊤)−1′⊤=⊤−′⊤(′′⊤)−1′⊤. split &= L L - L L B - B L L + B L L B \\ &= L L - L L ( L L )^-1 L L - L L ( L L )^-1 L L + L L ( L L )^-1 L L ( L L )^-1 L L \\ &= L L - L L ( L L )^-1 L L . split (159) Therefore (⊤)−1/2⊤(⊤)−1/2=(⊤)−1/2⊤(⊤)−1/2−(⊤)−1/2′⊤(′′⊤)−1′⊤(⊤)−1/2=m×m−(⊤)−1/2′⊤(′′⊤)−1/2((⊤)−1/2′⊤(′′⊤)−1/2)⊤. split( L L )^-1/2 ( L L )^-1/2&=( L L )^-1/2 L L ( L L )^-1/2-( L L )^-1/2 L L ( L L )^-1 L L ( L L )^-1/2\\ &= 1_m× m-( L L )^-1/2 L L ( L L )^-1/2 (( L L )^-1/2 L L ( L L )^-1/2 ) . split (160) Recall that by (145) the singular values ρ1,…,ρmin(m,m′) _1,…, _ (m,m ) of (⊤)−1/2′⊤(′′⊤)−1/2( L L )^-1/2 L L ( L L )^-1/2 are the canonical correlation of g and ′ g . Moreover, from (160) (and using that ⊤ A A has squared singular values as eigenvalues) we find λi=1−ρi2 _i=1- _i^2 (161) where we extend ρi=0 _i=0 for m′<i≤m <i≤ m if m≥m′m≥ m . ∎ Lemma F.5. Recall that the non-zero eigenvalues of the second moment of the logits ∼p(()⊤())∈ℝk×kE_ x p_ x( u( x) u ( x)) ^k× k are denoted by μ1≥μ2≥…≥μm _1≥ _2≥…≥ _m and the canonical correlations of g and ′ g by ρ1≥ρ2≥…≥ρmin(m,m′) _1≥ _2≥…≥ _ (m,m ). Then the following bound holds ∑i=1m(1−ρi2)μi≤∼p∥⊤()∥2. _i=1^m(1- _i^2) _i _ x p_ x f( x) ^2. (162) Proof. We rewrite using cyclicity of the trace ∥⊤()∥2=Tr(⊤(⊤)−1/2(⊤)1/2()⊤()(⊤)1/2(⊤)−1/2)=Tr(((⊤)−1/2⊤(⊤)−1/2)((⊤)1/2()⊤()(⊤)1/2)). split f( x) ^2&=Tr ( ( L L )^-1/2( L L )^1/2 f( x) f ( x)( L L )^1/2( L L )^-1/2 )\\ &=Tr ( (( L L )^-1/2 ( L L )^-1/2 ) (( L L )^1/2 f( x) f ( x)( L L )^1/2 ) ). split (163) We have seen in Lemma F.4 that the eigenvalues of (⊤)−1/2⊤(⊤)−1/2( L L )^-1/2 ( L L )^-1/2 are given (in decreasing order) by 1−ρi21- _i^2 where ρi _i are the canonical correlations of g and ′ g . We now consider the other term and claim that the eigenvalues of (⊤)1/2∼p(()⊤())(⊤)1/2 ( L L )^1/2E_ x p_ x ( f( x) f ( x) )( L L )^1/2 (164) are given by μ1≥μ2≥…≥μm _1≥ _2≥…≥ _m defined as the non-zero eigenvalues of ∼p(()⊤())=∼p(⊤()⊤())∈ℝk×k. _ x p_ x( u( x) u ( x))=E_ x p_ x ( L f( x) f ( x) L ) ^k× k. (165) Note that there are indeed at most m non-zero eigenvalues since L has rank m, potentially there are less if the image of f is restricted to a hyperplane. Since the eigenvalues of A B and B A agree for all matrices we find that the spectrum σ satisfies σ((⊤)1/2∼p(()⊤())(⊤)1/2)=σ(⊤∼p(()⊤()))=σ(∼p(⊤()⊤())) σ (( L L )^1/2E_ x p_ x ( f( x) f ( x) )( L L )^1/2 )=σ ( L L E_ x p_ x ( f( x) f ( x) ) )=σ (E_ x p_ x ( L f( x) f ( x) L ) ) (166) We can now apply von Neumann’s trace inequality (see (30)) and conclude that (note that ρi _i and μi _i where both defined as decreasing so (1−ρi2)(1- _i^2) is increasing ∑i=1m(1−ρi2)μi≤∼pTr(((⊤)−1/2⊤(⊤)−1/2)((⊤)1/2()⊤()(⊤)1/2))=∼p∥⊤()∥2. _i=1^m(1- _i^2) _i _ x p_ xTr ( (( L L )^-1/2 ( L L )^-1/2 ) (( L L )^1/2 f( x) f ( x)( L L )^1/2 ) )=E_ x p_ x f( x) ^2. (167) ∎ F.2 Bounding similarity of the embeddings We now perform essentially the same analysis to obtain bounds for the canonical correlations of f and ′ f . We present the results in slightly shorter form. We consider the regression problem ~=min¯∈ℝm×m′∼p∥()−¯′()∥2 B= _ B ^m× m E_ x p_ x f( x)- B f ( x) ^2 (168) which has the solution ~=∼p(()′()⊤)(∼p(′()′()⊤))−1. B=E_ x p_ x( f( x) f ( x) ) (E_ x p_ x( f ( x) f ( x) ) )^-1. (169) It is convenient to introduce the notation ,=∼p(()()⊤) M_ f, f=E_ x p_ x( f( x) f( x) ) (170) for the moment matrices and ,′ M_ f, f are defined similarly. Then we can write ~=,′′,′−1. B= M_ f, f M_ f , f ^-1. (171) We again consider the residual ~()=()−~′() ( x)= f( x)- B f ( x) (172) which satisfies ~,′=0 M_ , f =0 (173) where ~,′=(~()()⊤) M_ , f =E( ( x) f( x) ) is the moment matrix defined similar to (146). Similar to Lemma F.2 we get the following result. Lemma F.6. With the notation introduced above, we can decompose ∼p∥()−′()∥2=∼p(∥⊤~()∥2+∥⊤~′()−′⊤′()∥2). _ x p_ x u( x)- u ( x) ^2=E_ x p_ x ( L ( x) ^2+ L B f ( x)- L f ( x) ^2 ). (174) Remark F.7. Similar to Remark F.3 we can bound the norm of the residual ∼p∥()−′()∥2≥λmin(⊤)∼p∥~()∥2. _ x p_ x u( x)- u ( x) ^2≥ _ ( L L )E_ x p_ x ( x) ^2. (175) Proof. We calculate ∥()−′()∥2=∥⊤()−′⊤′()∥2=∥⊤~()+⊤~′()−′⊤′∥2=~()⊤⊤~()+(⊤~′()−′⊤′())⊤(⊤~′()−′⊤′())+~()⊤(⊤~′()−′⊤′())+(⊤~′()−′⊤′())⊤⊤~()=∥⊤~()∥2+∥⊤~′()−′⊤′()∥2+Tr(′()~()⊤(⊤~−′⊤))+Tr((~⊤−′)⊤⊤~()′()⊤). split u( x)- u ( x) ^2&= L f( x)- L f ( x) ^2\\ &= L ( x)+ L B f ( x)- L f ^2\\ &= ( x) L L ( x)+ ( L B f ( x)- L f ( x) ) ( L B f ( x)- L f ( x) )\\ & + ( x) L ( L B f ( x)- L f ( x) )+ ( L B f ( x)- L f ( x) ) L ( x)\\ &= L ( x) ^2+ L B f ( x)- L f ( x) ^2\\ & +Tr ( f ( x) ( x) L ( L B- L ) )+Tr ( ( B L- L ) L ( x) f ( x) ). split (176) Integrating over x we find using (173) that the cross terms vanish Tr(∼p(′()~()⊤)⏟=0(⊤~−′⊤))+Tr((~⊤−′)⊤⊤∼p(~()′()⊤)⏟=0)=0. ( E_ x p_ x ( f ( x) ( x) )_=0 L ( L B- L ) )+Tr ( ( B L- L ) L E_ x p_ x ( ( x) f ( x) )_=0 )=0. (177) And the claim follows by taking expectation in (176). ∎ We now want to relate the term ∼p(∥⊤~()∥2)E_ x p_ x ( L ( x) ^2 ) to the canonical correlation of () f( x) and ′() f ( x). For this we prove a result similar to Lemma F.4. Recall that the canonical correlation of f and ′ f , given by the singular values of ,−1/2,′′,′−1/2 M_ f, f^-1/2 M_ f, f M_ f , f ^-1/2, by ρ~1≥ρ~2≥…≥ρ~min(m,m′) ρ_1≥ ρ_2≥…≥ ρ_ (m,m ) (which we extend to 0 for i>min(m,m′)i> (m,m ) if necessary). Lemma F.8. Denote the eigenvalues of ,−1/2∼p(~()~⊤()),−1/2 M_ f, f^-1/2E_ x p_ x ( ( x) ( x) ) M_ f, f^-1/2 by λ~1≤…≤λ~m λ_1≤…≤ λ_m. Then the relation λ~i=1−ρ~i2 λ_i=1- ρ_i^2 (178) holds, where we set ρi=0 _i=0 for m′<i≤m <i≤ m if m′<m <m. Proof. We expand ∼p(~()~⊤())=∼p(()()⊤−()′()⊤′,′−1′,−,′,′−1()⊤′()+,′,′−1()⊤′()⊤′,′−1′,)=,−2,′′,′−1′,+,′′,′−1′,′′,′−1′,=,−,′′,′−1′,. splitE_ x p_ x ( ( x) ( x) )&=E_ x p_ x ( f( x) f( x) - f( x) f ( x) M_ f , f ^-1 M_ f , f- M_ f, f M_ f , f ^-1 f( x) f ( x) .\\ & .+ M_ f, f M_ f , f ^-1 f( x) f ( x) M_ f , f ^-1 M_ f , f )\\ &= M_ f, f-2 M_ f, f M_ f , f ^-1 M_ f , f+ M_ f, f M_ f , f ^-1 M_ f , f M_ f , f ^-1 M_ f , f\\ &= M_ f, f- M_ f, f M_ f , f ^-1 M_ f , f. split (179) Thus we find ,−1/2∼p(~()~⊤()),−1/2=m×m−(,−1/2,′′,′−1/2)⊤(,−1/2,′′,′−1/2). split M_ f, f^-1/2E_ x p_ x ( ( x) ( x) ) M_ f, f^-1/2&= 1_m× m- ( M_ f, f^-1/2 M_ f, f M_ f , f ^-1/2 ) ( M_ f, f^-1/2 M_ f, f M_ f , f ^-1/2 ). split (180) This then implies that λ~i=1−ρ~i2 λ_i=1- ρ_i^2 (181) where we extend ρ~i=0 ρ_i=0 for m′<i≤m <i≤ m if m≥m′m≥ m . ∎ Based on this result we can now derive a bound for canonical correlation coefficients deriving the analogue of Lemma F.5. Lemma F.9. Recall that we denoted the eigenvalues of the second moment of the logits ∼p(()⊤())∈ℝk×kE_ x p_ x( u( x) u ( x)) ^k× k by μ1≥μ2≥…≥μm _1≥ _2≥…≥ _m and the canonical correlations of f and ′ f by ρ~1≥ρ~2≥…≥ρ~min(m,m′) ρ_1≥ ρ_2≥…≥ ρ_ (m,m ) Then the following bound holds ∑i=1m(1−ρ~i2)μi≤∼p∥⊤~()∥2. _i=1^m(1- ρ_i^2) _i _ x p_ x L ( x) ^2. (182) Proof. We rewrite using cyclicity of the trace as in (163) ∥⊤~()∥2=Tr(~()⊤,−1/2,1/2⊤,1/2,−1/2~())=Tr((,−1/2~()~()⊤,−1/2)(,1/2⊤,1/2)). split L ( x) ^2&=Tr ( ( x) M_ f, f^-1/2 M_ f, f^1/2 L L M_ f, f^1/2 M_ f, f^-1/2 ( x) )\\ &=Tr ( ( M_ f, f^-1/2 ( x) ( x) M_ f, f^-1/2 ) ( M_ f, f^1/2 L L M_ f, f^1/2 ) ). split (183) Applying Lemma F.8 we find that the eigenvalues of ,−1/2~()~()⊤,−1/2 M_ f, f^-1/2 ( x) ( x) M_ f, f^-1/2 are given (in decreasing order) by 1−ρ~i21- ρ_i^2 where ρ~i ρ_i are the canonical correlations of f and ′ f . As in Lemma F.5 the other term ,1/2⊤,1/2 M_ f, f^1/2 L L M_ f, f^1/2 (184) has eigenvalues μ1≥μ2≥…≥μm _1≥ _2≥…≥ _m. Indeed, we find σ(,1/2⊤,1/2)=σ(,⊤)=σ(⊤,). σ ( M_ f, f^1/2 L L M_ f, f^1/2 )=σ ( M_ f, f L L )=σ ( L M_ f, f L ). (185) Then von Neumann’s trace inequality (30) implies ∑i=1m(1−ρ~i2)μi≤∼p∥⊤~()∥2. _i=1^m(1- ρ_i^2) _i _ x p_ x L ( x) ^2. (186) ∎ After these preparations the proof of Theorem F.1 is straightforward. Proof of Theorem F.1. Applying Lemma F.5 followed by Lemma F.2 we conclude that ∑i=1m(1−ρi2)μi≤∼p∥⊤()∥2≤∼p∥()−′()∥2 _i=1^m(1- _i^2) _i _ x p_ x f( x) ^2 _ x p_ x u( x)- u ( x) ^2 (187) which is (149). The bound (150) follows similarly from Lemma F.6 and F.9. ∎ F.3 Relating embeddings and unembeddings The results in Theorem F.1 can be used to show that f and ′ f and g and ′ g are linearly related when the model logits are similar. Concretely we bounded the canonical correlations and we can also bound the residual (see Remark F.3). However, while these are quantitative versions of the linear identifiability of f and g this alone is not sufficient to recover the identifiability result as this also requires that =~′ f= B f , =′ g= B g and ~−⊤=~ B_J^- = B. Therefore, a natural question is whether we can show quantitative bounds for ⊤~−m′×m′ B B- 1_m × m (this matrix can also be considered for m′≠m ≠ m). However, when considering the equivalent model (⊤′,−1′)( A f , A^-1 g ) for ∈GL(m′) A (m ) we find that the transition matrix becomes −1⊤~ A^-1 B B A. However, if ⊤~ B B is not a multiple of the identity we have sup∈GL(m′)∥−1⊤~∥=∞ _ A (m ) A^-1 B B A =∞ (188) and thus no meaningful bounds on ∥⊤~∥ B B in terms of ∼p∥()−′()∥2E_ x p_ x u( x)- u ( x) ^2 are possible. Instead, we can only show a weaker statement for the relation of B and ~ B. To achieve this we argue as in Lemmas F.2 and F.6. Lemma F.10. When ∼p∥′()−()∥2=0E_ x p_ x u ( x)- u( x) ^2=0, then ~⊤~=m′×m′ B B= 1_m × m . Moreover, ~⊤~ B B behaves like an approximate identity in relevant model direction in the sense that ∼p∥′⊤⊤~′()−′⊤′()∥2≤∼p∥()−′()∥2 _ x p_ x L B B f ( x)- L f ( x) ^2 _ x p_ x u( x)- u ( x) ^2 (189) and the logits ′⊤⊤~′() L B B f ( x) of the model (′,~′)( B g , B f ) also satisfy ∼p∥′⊤⊤~′()−()∥2≤4∼p∥′()−()∥2. _ x p_ x L B B f ( x)- u( x) ^2≤ 4E_ x p_ x u ( x)- u( x) ^2. (190) Remark F.11. One could derive bounds on ∥~⊤~−m′×m′∥ B B- 1_m × m when assuming lower bounds on the spectrum of the moments ′,′ M_ f , f and ′′⊤ L L , however these assumptions are again not invariant under equivalence transformations of the model. Proof. We start from (174) and decompose the second term further using ∥⊤~′()−′⊤′()∥2=∥⊤~′()+′⊤⊤~′()−′⊤′()∥2=∥⊤~′()∥2+∥′⊤⊤~′()−′⊤′()∥2. split L B f ( x)- L f ( x) ^2&= B f ( x)+ L B B f ( x)- L f ( x) ^2\\ &= B f ( x) ^2+ L B B f ( x)- L f ( x) ^2. split (191) Here the Pythagorean identity in the last step follows as in Lemma F.2 from ′⊤=0 L =0 (see (156)). From Lemma F.6 we infer that ∼p∥′⊤⊤~′()−′⊤′()∥2≤∼p∥()−′()∥2. splitE_ x p_ x L B B f ( x)- L f ( x) ^2 _ x p_ x u( x)- u ( x) ^2. split (192) This implies that ⊤~=m′×m′ B B= 1_m × m (when ′,′ M_ f , f and ⊤ L L are non-degenerate), in particular m=m′m=m . Moreover, we conclude using (a+b)2≤2a2+2b2(a+b)^2≤ 2a^2+2b^2 that ∼p∥′⊤⊤~′()−()∥2≤2∼p(∥′⊤⊤~′()−′()∥2+∥′()−()∥2)≤4∼p∥′()−()∥2. splitE_ x p_ x L B B f ( x)- u( x) ^2&≤ 2E_ x p_ x ( L B B f ( x)- u ( x) ^2+ u ( x)- u( x) ^2 )\\ &≤ 4E_ x p_ x u ( x)- u( x) ^2. split (193) In particular the logits of (′,~′)( B g , B f ) are, on average, close to the logits of the model (,)( g, f). ∎ F.4 Lower Bounds for the Mean Canonical Correlation In this section we collect applications of Theorem F.1. The proof of Theorem 3.4 can be found at the end of this section. First we show how the theorem can be used to bound the mean canonical correlation coefficient, mCCAm_CCA. Theorem F.12. Assume the hypothesis of Theorem F.1, in particular we assume that the moment matrix ,=∼p(()()⊤) M_ u, u=E_ x p_ x( u( x) u( x) ) has rank m. Denote by μmin=μm>0 _ = _m>0 the smallest non-zero eigenvalue of , M_ u, u. Then the bounds ∑i=1mρi≥∑i=1mρi2≥m−∼p∥()−′()∥2μminand _i=1^m _i≥ _i=1^m _i^2≥ m- E_ x p_ x u( x)- u ( x) ^2 _ (194) ∑i=1mρ~i≥∑i=1mρ~i2≥m−∼p∥()−′()∥2μmin _i=1^m ρ_i≥ _i=1^m ρ_i^2≥ m- E_ x p_ x u( x)- u ( x) ^2 _ (195) hold. Proof. Theorem F.1 implies ∼p∥()−′()∥2μmin≥∑i=1m(1−ρi2)μiμmin≥∑i=1m(1−ρi2) E_ x p_ x u( x)- u ( x) ^2 _ ≥ _i=1^m(1- _i^2) _i _ ≥ _i=1^m(1- _i^2) (196) where we used ρi2≤1 _i^2≤ 1 (so 1−ρi2≥01- _i^2≥ 0) and μiμmin≥1 _i _ ≥ 1 in the last step. This implies the first bound and the second estimate follows similarly. ∎ Note that there are at most min(m′,m) (m ,m) non-zero canonical correlations so the result directly implies lower bound on m′m to match the other models distribution. Or, stated equivalently, we can bound the information loss a smaller model must incur. In particular, we conclude lower bounds for the representation dimension of a student that tries to match the teacher’s logits with a given accuracy and those bounds are explicit functions of the teacher’s logits. This result can be slightly refined and we obtain the following theorem which closely resembles the reconstruction bound for PCA. Theorem F.13. Assume the hypothesis of Theorem F.1 and denote the eigenvalues of , M_ u, u in decreasing order by μi _i. Suppose the dimension m′m of the student model satisfies m′<m <m. Then we find ∼p∥()−′()∥2≥∑i=m′+1mμi. _ x p_ x u( x)- u ( x) ^2≥ _i=m +1^m _i. (197) Proof. The result follows from Theorem F.1 by using that ρi=0 _i=0 for i>m′i>m . ∎ Finally, we discuss the modifications to consider the standard covariance matrix based canonical correlations. For this, we will need the slightly stronger diversity condition that the covariance matrix ,=∼p((()−∼p(()))(()−∼p(()))⊤) _ u, u=E_ x p_ x ( ( u( x)-E_ x p_ x( u( x)) ) ( u( x)-E_ x p_ x( u( x)) ) ) (198) has rank m. Let us denote the standard (covariance-based) canonical correlations of f and ′ f by ρ¯i ρ_i. Then the following result holds. Theorem F.14. Consider two models (,)( f, g) and (′,′)( f , g ) with representation dimensions m and m′m and assume that , _ f, f, L and ′,′ _ f , f , ′ L have maximal rank m and m′m respectively. Denote by μm>0 _m>0 the smallest non-zero eigenvalue of , _ u, u. Then the bounds ∑i=1mρi≥∑i=1mρi2≥m−∼p∥()−′()∥2μmand _i=1^m _i≥ _i=1^m _i^2≥ m- E_ x p_ x u( x)- u ( x) ^2 _m (199) ∑i=1mρ¯i≥∑i=1mρ¯i2≥m−∼p∥()−′()∥2μm _i=1^m ρ_i≥ _i=1^m ρ_i^2≥ m- E_ x p_ x u( x)- u ( x) ^2 _m (200) hold. Proof. Denote the mean of the embeddings by ¯=∼p(()) f=E_ x p_ x( f( x)) and ¯′=∼p(′()) f =E_ x p_ x( f ( x)). We consider the embedding-centred models (c,)=(−¯,)( f^c, g)=( f- f, g) and ((′)c,′)=(′−¯′,′)(( f )^c, g )=( f - f , g ). The logits c u^c of the shifted model are given by c=⊤(()−¯)=()−∼p(()) u^c= L ( f( x)- f)= u( x)-E_ x p_ x( u( x)) (201) where u are the logits of the original model and a similar expression holds for ′ u . This implies c,c=, M_ u^c, u^c= _ u, u. Moreover, the moment-based and covariance-based canonical correlations agree for the shifted models (c,)( f^c, g) and ((′)c,′)(( f )^c, g ) and they further agree with the covariance-based (i.e., shift invariant) canoncial correlations of f and ′ f . We can thus now apply Theorem F.12 to the models with the shifted logits and we find ∑i=1mρ¯i≥∑i=1mρ¯i2≥m−∼p∥c()−(′)c()∥2μm≥m−∼p∥()−′()∥2μm. _i=1^m ρ_i≥ _i=1^m ρ_i^2≥ m- E_ x p_ x u^c( x)-( u )^c( x) ^2 _m≥ m- E_ x p_ x u( x)- u ( x) ^2 _m. (202) Here we used ∼p∥()−′()∥2≥∼p∥c()−(′)c()∥2 _ x p_ x u( x)- u ( x) ^2 _ x p_ x u^c( x)-( u )^c( x) ^2 (203) in the last step. The proof of (199) is similar. ∎ Proof of Theorem 3.4. The statement follows directly from Theorem F.14 when plugging in the definition (28) of mCCAm_CCA. ∎ Appendix G Example of Models with small KL Divergence and Equal Embeddings, but Dissimilar Unembeddings In this section, we illustrate why when we want to measure representational similarity that takes into account the equivalence relation of the identifiability class and why it is not enough to consider a measure on only similarity between the embeddings (or unembeddings). We do this by constructing a pair of models with equal embeddings and small KL divergence, but dissimilar unembeddings as measured with mCCAm_CCA. Note that these models will also be dissimilar in terms of drepd_rep, since drepd_rep includes the unembeddings through the unembeddings matrices. These constructed models are displayed in Figs. 4 and 4 Figure 3: Embeddings of (,)( f, g) in blue.Other coloured dots mark the unembeddings, each belonging to a label. Figure 4: Embeddings of (′,′)( f , g ) in blue. Other coloured dots mark the unembeddings. Note that brown and pink-colored unembeddings are swapped compared to Fig. 4, as are grey and yellow. Assume we have models defining a distribution over 88 labels, i.e., =y1,…,y8Y=\y_1,...,y_8\, and with representation dimensionality m=2m=2. For a subset of input data, a reference model (,)∈Θ( f, g)∈ assigns high probability to one among the first four labels, as displayed by embeddings aligning to unembeddings directions (yi) g(y_i), i=1,…,4i=1,…,4. On the remaining input points, the model assign high mass, but equal probability, to either the labels y5y_5 and y6y_6, or y7y_7 and y8y_8. This can be seen from the embeddings aligned to (y5)+(y6) g(y_5)+ g(y_6) or to (y7)+(y8) g(y_7)+ g(y_8). A second model (′,′)∈Θ( f , g )∈ has equal embeddings to (,)( f, g), that is ′()=() f ( x)= f( x) for all ∈ x , but dissimilar unembeddings. It is constructed so as to have equal unembeddings for the first 44 labels, i.e., ′(yi)=(yi) g (y_i)= g(y_i) for i=1,…,4i=1,…,4, and swapped unembeddings for y5y_5 and y6y_6 swapped and for y7y_7 and y8y_8: ′(y5)=(y6)′(y6)=(y5) and ′(y7)=(y8)′(y8)=(y7). cases g (y_5)= g(y_6)\\ g (y_6)= g(y_5) cases and cases g (y_7)= g(y_8)\\ g (y_8)= g(y_7) cases\,. (204) Despite the unembeddings of the models being dissimilar in a linear sense, we can now make dKLd_KL arbitrarily small by increasing the norm of the embeddings or the unembeddings. Notice that the embeddings that are between the unembeddings on the left are exactly in the middle in terms of angle. Therefore, since all the unembeddings have the same norm, probability values of the two most likely labels will be equal for the two models. However, increasing the norm, will not make the unembeddings of the two models closer to being linear transformations of each other. Note also that we can choose any of the first four labels y1,…,y4y_1,...,y_4, to make the ,L,L matrices with columns ~(yi) g(y_i), ~′(yi) g (y_i), and this will give us =′L=L . In particular, we have =−⊤′⊤=A=L^- L =I, so ()=′() (x)=Af (x) (205) for all x like in the identifiability result, but we do not have the corresponding part for the unembeddings. That is, ~(y)≠−⊤~′(y) g(y)≠ A^- g (y). In terms of the mean canonical correlation, mCCAm_CCA, we have for the embeddings mCCA((),′())=1m_CCA( f( x), f ( x))=1 and for the unembeddings mCCA((y),′(y))=0.67m_CCA( g(y), g (y))=0.67 (The mCCAm_CCA for the unembeddings can be made smaller by spreading out unembeddings for labels y5,y6,y7,y8y_5,y_6,y_7,y_8 further). Thus, we see that there can be a perfect linear relationship between embeddings and, even if the distributions are very close in terms of KL divergence, there is still no guarantee for the unembeddings to be similar up to the inverse linear transformation of the embeddings. Appendix H Theoretical Material for Section 3.4: The Logit Distance Bounds the Linear Identifiability Dissimilarity In this section, we will first show how the squared distance between logits can be written as a normalized distance between representations § H.1, and then show how the logit distance bounds the linear identifiability dissimilarity from Def. 3.7 in § H.2. H.1 Connection Between Distance between Logits and Representations Theorem H.1. Let p,p′p,p be as defined above, and m the dimension of the representations. Let N=(k−2m−1)N= k-2m-1. Then the squared distance between logits can be written as a normalized distance between the representations in the following sense: Let ~=⊤ L_J= U D V be the singular value decomposition of ~ L_J, and let ~=−1⊤ B_J= D^-1 U . Then we have ‖()−′()‖22=12kN∑y~∈∑⊆∖y~‖~−⊤(()−~′())‖22. \| u( x)- u ( x) \|_2^2= 12kN _ y _J \ y\ \| B_J^- ( f( x)- A_J f ( x) ) \|_2^2\;. (206) Proof. We consider the norm on the right hand side of Eq. 206: ‖~−⊤(()−~′())‖22 \| B_J^- ( f( x)- A_J f ( x) )\|_2^2 (207) =‖~−⊤~−⊤(~⊤()−~⊤′′())‖22 =\| B_J^- L_J^- ( L_J f( x)- L_J^ f ( x) )\|_2^2 (208) =‖(~~)−⊤(~⊤()−~⊤′′())‖22 =\|( B_J L_J)^- ( L_J f( x)- L_J^ f ( x) )\|_2^2 (209) Using the definition of ~=−1⊤ B_J= D^-1 U and recalling that the singular value decomposition of ~ L_J was ~=⊤ L_J= U D V , we see that ~~=−1⊤⊤=⊤ B_J L_J= D^-1 U U D V = V (210) So we have that ‖(~~)−⊤(~⊤()−~⊤′′())‖22 \|( B_J L_J)^- ( L_J f( x)- L_J^ f ( x) )\|_2^2 (211) =‖⊤(~⊤()−~⊤′′())‖22 =\| V ( L_J f( x)- L_J^ f ( x) )\|_2^2 (212) Now since ⊤ V is an orthonormal matrix, it does not change the norm of a vector. Recall, that for a vector a, we have: ‖⊤‖22=(⊤)⊤⊤=⊤⊤=⊤=‖22 \| V a\|_2^2=( V a) V a= a V V a= a a=\| a\|_2^2 (213) So we have that ‖⊤(~⊤()−~⊤′′())‖22 \| V ( L_J f( x)- L_J^ f ( x) )\|_2^2 (214) =‖~⊤()−~⊤′′()‖22 =\| L_J f( x)- L_J^ f ( x)\|_2^2 (215) So using Lemma D.4, we see that 12kN∑y~∈∑⊆∖y~∼p‖~−⊤(()−~′())‖22 12kN _ y _J \ y\E_ x p_ x\| B_J^- ( f( x)- A_J f ( x) )\|_2^2 =12kN∑y~∈∑⊆∖y~∼p‖~⊤()−~⊤′′()‖22 = 12kN _ y _J \ y\E_ x p_ x\| L_J f( x)- L_J^ f ( x)\|_2^2 (216) =‖()−′()‖22 = \| u( x)- u ( x) \|_2^2 (217) ∎ This gives us the following corollary (Corollary 3.8 from the main paper): Corollary H.2. For (,),(′,′)∈Θ( f, g),( f , g )∈ , we have drep((,),(′,′))=0⇔dlogit(p,,p′,′)=0. d_rep(( f, g),( f , g ))=0 d_logit(p_ f, g,p_ f , g )=0\,. Proof. “⟸ ”: Assume dlogit(p,p′)=0d_logit(p,p )=0. Since dlogitd_logit is a metric between probability distributions, this gives us p=p′p=p . Therefore, Corollary 3.6 gives us drep((,),(′,′))=0d_rep(( f, g),( f , g ))=0. “⟹ ”: Assume drep((,),(′,′))=0d_rep(( f, g),( f , g ))=0. From Thm. H.1 we get that dlogit(p,p′)=0d_logit(p,p )=0. ∎ H.2 Logit Distance Bounds the Linear Identifiability Dissimilarity In this section we present the full proof of Thm. 3.9, however, we first prove a lemma to help with the main result. Lemma H.3. Let (,),(,)∈Θ( f, g),( f, g)∈ . Let ~ L_J be as defined in the priliminaries Eq. 4, and let ~=⊤ L_J= U D V be the singular value decomposition of ~ L_J. Let ~=−1⊤ B_J= D^-1 U . Let σ~min _ L_J be the smallest singular value of ~ L_J, and let σmin _ be the smallest singular value of all ~ L_J. Then for every choice of labels and input, x, we have the following bounds: ‖~−⊤(()−~′())‖2σmin≥‖~−⊤(()−~′())‖2σ~min≥‖()−~′()‖2 \| B_J^- ( f( x)- A_J f ( x) )\|_2 _ ≥ \| B_J^- ( f( x)- A_J f ( x) )\|_2 _ L_J ≥\| f( x)- A_J f ( x)\|_2 (218) And similarly, if we let σ~max _ L_J be the largest singular value of ~ L_J, and σmax _ be the largest singular value of all ~ L_J, we also have: ‖~−⊤(()−~′())‖2σmax≤‖~−⊤(()−~′())‖2σ~max≤‖()−~′()‖2 \| B_J^- ( f( x)- A_J f ( x) )\|_2 _ ≤ \| B_J^- ( f( x)- A_J f ( x) )\|_2 _ L_J ≤\| f( x)- A_J f ( x)\|_2 (219) Proof. We consider the norm on the left hand side of Eq. 218:, where we recall that ~−⊤=⊤ B_J^- = D U : ‖~−⊤(()−~′())‖2 \| B_J^- ( f( x)- A_J f ( x) )\|_2 (220) =‖⊤(()−~′())‖2 =\| D U ( f( x)- A_J f ( x) )\|_2 (221) =‖(⊤()−⊤~′())‖2 = \| D ( U f( x)- U A_J f ( x) ) \|_2 (222) Now D is the diagonal matrix with the singular values of ~ L_J in the diagonal. We define min D_ to be the diagonal matrix with the minimum singular value of ~ L_J, σ~min _ L_J , in all the diagonal entries. This means that multiplying a vector with min D_ will make the norm smaller compared to multiplying with D. In other words, we have: ‖(⊤()−⊤~′())‖2≥‖min(⊤()−⊤~′())‖2 \| D ( U f( x)- U A_J f ( x) ) \|_2≥ \| D_ ( U f( x)- U A_J f ( x) ) \|_2 (223) We can calculate the effect of multiplying a vector, a, with min D_ : ‖min‖2=∑i=1D(σ~minai)2=σ~min2∑i=1Dai2=σ~min∑i=1Dai2=σ~min‖2 \| D_ a \|_2= _i=1^D( _ L_J a_i)^2= _ L_J ^2 _i=1^Da_i^2= _ L_J _i=1^Da_i^2= _ L_J \| a \|_2 (224) So we have that ‖min(⊤()−⊤~′())‖2 \| D_ ( U f( x)- U A_J f ( x) ) \|_2 (225) =σ~min‖⊤()−⊤~′()‖2 = _ L_J \| U f( x)- U A_J f ( x) \|_2 (226) =σ~min‖⊤(()−~′())‖2 = _ L_J \| U ( f( x)- A_J f ( x) ) \|_2 (227) Since ⊤ U is an orthonormal matrix, it does not change the norm of a vector, so we have that σ~min‖⊤(()−~′())‖2 _ L_J \| U ( f( x)- A_J f ( x) ) \|_2 (228) =σ~min‖()−~′()‖2 = _ L_J \| f( x)- A_J f ( x) \|_2 (229) All in all, we have that for any choice of ~ L_J and x: ‖~−⊤(()−~′())‖2≥σ~min‖()−~′()‖2 \| B_J^- ( f( x)- A_J f ( x) )\|_2≥ _ L_J \| f( x)- A_J f ( x) \|_2 (230) which means that ‖~−⊤(()−~′())‖2σ~min≥‖()−~′()‖2 \| B_J^- ( f( x)- A_J f ( x) )\|_2 _ L_J ≥ \| f( x)- A_J f ( x) \|_2 (231) And since σmin≤σ~min _ ≤ _ L_J we get the second inequality. We can make a similar argument by defining max D_ as the diagonal matrix with the maximum singular value of ~ L_J, σmax _ L , in all the diagonal entries, and go through the steps from Eq. 223 with the inequality reversed. ∎ We are now ready to prove the main result: Theorem H.4. Let p,p′p,p be as before, and m the dimension of the representations and k the number of labels. Let C=2mk−1C= 2mk-1. Let σ~min _ L_J be the smallest singular value of ~ L_J, and let σmin _ be the smallest singular value of all ~ L_J. Then dlogitd_logit, bounds the linear identifiability dissimilarity, drepd_rep, in the following way: Cdlogit(p,p′)σmin≥drep((,),(′,′)) C d_logit(p,p ) _ ≥ d_rep(( f, g),( f , g )) (232) And similarly, if we let σ~max _ L_J be the largest singular value of ~ L_J, and σmax _ be the largest singular value of all ~ L_J, we also have Cdlogit(p,p′)σmax≤drep((,),(′,′)) C d_logit(p,p ) _ ≤ d_rep(( f, g),( f , g )) (233) Proof. Let N=(k−2m−1)N= k-2m-1 and J=(k−1m)J= k-1m. We recall from Thm. H.1 that ‖()−′()‖22=12kN∑y~∈∑⊆∖y~‖~−⊤(()−~′())‖22. \| u( x)- u ( x) \|_2^2= 12kN _ y _J \ y\ \| B_J^- ( f( x)- A_J f ( x) ) \|_2^2\;. (234) Also, from Eq. 218 from Lemma H.3, we get have that ‖~−⊤(()−~′())‖2σ~min \| B_J^- ( f( x)- A_J f ( x) )\|_2 _ L_J ≥‖()−~′()‖2 ≥\| f( x)- A_J f ( x)\|_2 (235) which means that since σmin≤σ~min _ ≤ _ L_J we have that ‖()−′()‖22σmin2≥12kN∑y~∈∑⊆∖y~‖()−~′()‖22. \| u( x)- u ( x) \|_2^2 _ ^2≥ 12kN _ y _J \ y\\| f( x)- A_J f ( x)\|_2^2\;. (236) Taking the expectation over x, we get ∼px‖()−′()‖22σmin2≥12kN∑y~∈∑⊆∖y~∼px‖()−~′()‖22. _ x p_x \| u( x)- u ( x) \|_2^2 _ ^2≥ 12kN _ y _J \ y\E_ x p_x\| f( x)- A_J f ( x)\|_2^2\;. (237) Which means we have that dlogit2(p,p′)σmin2 d_logit^2(p,p ) _ ^2 ≥J2N1kJ∑y~∈∑⊆∖y~∼px‖()−~′()‖22 ≥ J2N 1kJ _ y _J \ y\E_ x p_x\| f( x)- A_J f ( x)\|_2^2 (238) =J2Ndrep2((,),(′,′)), = J2Nd_rep^2(( f, g),( f , g ))\;, (239) and therefore dlogit(p,p′)σmin d_logit(p,p ) _ ≥J2N1kJ∑y~∈∑⊆∖y~∼px‖()−~′()‖22 ≥ J2N 1kJ _ y _J \ y\E_ x p_x\| f( x)- A_J f ( x)\|_2^2 (240) =J2Ndrep((,),(′,′)). = J 2Nd_rep(( f, g),( f , g ))\;. (241) We now see that 2NJ 2NJ =2(k−2)!m!(k−m−1)!(m−1)!(k−m−1)!(k−1)! = 2(k-2)!m!(k-m-1)!(m-1)!(k-m-1)!(k-1)! (242) =2(k−2)!m(m−1)!(k−m−1)!(m−1)!(k−m−1)!(k−1)(k−2)! = 2(k-2)!m(m-1)!(k-m-1)!(m-1)!(k-m-1)!(k-1)(k-2)! (243) =2m(k−1). = 2m(k-1)\;. (244) So for C=2m(k−1)C= 2m(k-1) we get Cdlogit(p,p′)σmin C d_logit(p,p ) _ ≥drep((,),(′,′)), ≥ d_rep(( f, g),( f , g ))\;, (245) which was what we wanted to prove. If we let σ~max _ L_J be the largest singular value of ~ L_J, and σmax _ be the largest singular value of all ~ L_J, we can go through the same steps with the reversed inequality. ∎ Appendix I Proof of Theorem 4.3 In this Section we show that when dlogit(p,p′)d_logit(p,p ) between two models is small, then concepts in one model are approximately represented in another model, in particular we prove Theorem 4.3. We recall that L is the matrix of unembeddings and that a concept h defines a conditional distribution p(c∣)p_ h(c x), over C≥2C≥ 2 values. Consider a model (,)( f, g), such that h is linearly represented in f. For every c∈1,…,Cc∈\1,…,C\, we write the concept vectors c∈ℝm w_c ^m as c L α_c for some c∈ℝk α_c ^k which we call concept weights as they are the weights when writing c=∑i=1k(yi)(αc)i w_c= _i=1^k g(y_i)( _c)_i. Since we assume that L has full rank, this is possible for all c w_c. Moreover, this is a natural way to parametrize the concepts as it is invariant to linear transformations of the embedding space and concepts of interest are generally aligned with the unembeddings (y) g(y). We also define the matrix =(c)c∈1,…,C∈ℝk×C A=( α_c)_c∈\1,…,C\ ^k× C and this is related to the concept weight matrix through = W= L A. We now show that these concepts can be approximately realized on the embedding space of another model (′,′)∈Θ( f , g )∈ when the logit distance to (′,′)( f , g ) is small. Indeed, we consider the concept matrix ′=′′∈ℝm×C W = L A ^m× C (246) and therefore the individual concept vectors are given by c′=′c′. w _c= L α _c. (247) We can then define (the biases remain invariant bc=bc′b_c=b_c ) p′(c|)=exp(c′⊤′()+bc)∑j=1Cexp(j′⊤′()+j) p _ h(c| x)= ( w_c f ( x)+b_c ) _j=1^C ( w_j f ( x)+ b_j ) (248) We then get the following result. Theorem I.1. For two models (,),(′,′)∈Θ( f, g),( f , g )∈ and a concept :→Δ[C] h:X→ _[C] with C≥2C≥ 2, if h is linearly encoded in f, then min′∈ℝC×m,′∈ℝCdKL(p(⋅∣),softmax(′()+′))≤12∥op2dlogit2(p,p′). _ W ^C× m, b ^Cd_KL (p_ h(· x),softmax ( W f ( x)+ b ) )≤ 12 A _op^2d_logit^2(p,p ). (249) where A is determined from = W= L A, ∥⋅∥op · _op denotes the operator norm, where ∈ℝm×C W ^m× C is the matrix of weights c w_c from Def. 4.2 for f linearly encoding h. Proof. Since h is linearly encoded in f, we have ∈ℝm×C W ^m× C and ∈ℝC b ^C as weights and biases that gives: softmax(⊤()+)=p(c∣).softmax ( W f( x)+ b )=p_ h(c x)\,. (250) Let p′(c∣)=softmax(⊤′′()+′).p _ h(c x)=softmax ( W^ f ( x)+ b )\,. (251) We bound the minimum of dKLd_KL starting from KL divergence. We bound for fixed x using Lemma E.9 (last line) min′∈ℝm×C,′∈ℝC _ W ^m× C, b ^C (p(⋅|)||p′(⋅|))= KL(p_ h(·| x)||p _ h(·| x))= (252) =min′∈ℝm×C,′∈ℝC(softmax(⊤()+)||softmax(⊤′()+)) = _ W ^m× C, b ^C KL (softmax( W f( x)+ b)||softmax( W^ f ( x)+ b) ) (253) ≤min′∈ℝm×C,′∈ℝC12∥⊤()−⊤′′()+−′∥2 ≤ _ W ^m× C, b ^C 12 W f( x)- W^ f ( x)+ b- b ^2 (254) Next, we bound the minimum in Eq. 254 setting ′= b = b. We use that := W:= L A and ′=′′ W = L A , so we turn minimization of ′ W into a minimization of ′ A : min′∈ℝm×C12∥⊤()−⊤′′()∥2 _ W ^m× C 12 W f( x)- W^ f ( x) ^2 =min′∈ℝk×C12∥⊤⊤()−⊤′⊤′′()∥2 = _ A ^k× C 12 A L f( x)- A^ L^ f ( x) ^2 (255) ≤12∥⊤()−⊤′()∥2 ≤ 12 A u( x)- A u ( x) ^2 (256) ≤12∥⊤∥op2⋅∥()−′()∥2. ≤ 12 A _op^2· u( x)- u ( x) ^2. (257) where, in the second line, we used that min′‖⊤−⊤′′‖2≤‖⊤−⊤′‖2 _ A \| A z- A^ z \|^2≤\| A z- A z \|^2 for any ,′∈ℝk z, z ^k and, in the last line, we used that the L2L_2 norm for a matrix vector product satisfies ∥≤∥⋅∥op A v ≤ v · A _op. We end the proof by taking the expectation over ∼p x p_ x. ∎ Appendix J L1L_1 Based Loss Bounds Distribution Distance We here present the proof of the following statement: Proposition J.1. Assume that p, p′p are two models which both satisfy Assumption 3.2 for some τ>0τ>0. Then ∼p∥()−′()∥22≤2|log(τ)|∼p∥()−′()∥1. _ x p_ x u( x)- u ( x) _2^2≤ 2| (τ)|E_ x p_ x u( x)- u ( x) _1. (258) Proof. Note that 1≥p(y|)≥τ1≥ p(y| x)≥τ and thus 0≥log(p(y|))≥log(τ)0≥ (p(y| x))≥ (τ) which also implies 0≥log(p(⋅|))¯≥log(τ)0≥ (p(·| x))≥ (τ) denoting again by ⋅¯ · the average over the labels. Then we can apply (129) and get (()−′())i2=|log(p(yi|))−log(p(⋅|))¯−log(p′(yi|))+log(p′(⋅|))¯|⋅|(()−′())i|≤2|log(τ)|⋅|(()−′())i|. split( u( x)- u ( x))_i^2&= | (p(y_i| x))- (p(·| x))- (p (y_i| x))+ (p (·| x)) |·|( u( x)- u ( x))_i|\\ &≤ 2| (τ)|·|( u( x)- u ( x))_i|. split (259) We conclude that ∥()−′()∥22≤2|log(τ)|⋅∑i=1k|(()−′())i|=2|log(τ)|⋅∥()−′()∥1. u( x)- u ( x) _2^2≤ 2| (τ)|· _i=1^k|( u( x)- u ( x))_i|=2| (τ)|· u( x)- u ( x) _1. (260) Taking the expectation over p_ x we get the claim. ∎ Appendix K Additional Details on Experiments Our code is available at https://github.com/bemigini/logit-distance-bounds-rep-sim. Synth consists of 77 classes distributed over 44 spiral rays. The dataset is created to ensure that, in input space, there are not two classes that are always close to each other and that input data are not linearly separable. We generate 2d2d input data as follows: for each of the seven classes, we define four angles, each capturing the starting angle of one of the spirals. Then, for each of these rays with angle θ0 _0, varying the radius ρ∈[1,10]ρ∈[1,10], we randomly sample a point near the spirals with an angle θi=θ0+2π/T⋅ρ/10+ϵ _i= _0+2π/T·ρ/10+ε (261) where T denotes the period of the spiral and ϵ∼Unif(−σ(ρ),σ(ρ))ε (-σ(ρ),σ(ρ)) is sampled so to ensure that it never crosses other spiral rays, by regulating the interval [−σ(ρ),σ(ρ)][-σ(ρ),σ(ρ)] on the value of ρ. The dataset is visualized in Fig. 2. Training data amount to 1414k instances, while the validation and test dataset each comprise 77k data points. CIFAR. We use the CIFAR100 [Krizhevsky et al., 2009] loaded from torchvision with the original split with 500 images per class for training and 100 images per class for testing. So all in all 50,000 images for training and 10,000 images for test. We do not make a validation split for this dataset. SUB is a regenerated version of CUB200 [Wah et al., 2011], a dataset for bird classification with annotation on salient bird features. In SUB, data points are annotated with a label Y capturing the bird specie, and a concept C indicating the attribute which was modified during generation. The dataset contains a total of 38.438.4k images, splitted in 21.521.5k for training, 5.35.3k for validation, and 11.511.5k for test. It comprises 3333 annotated class labels. Each image is also annotated with a concept reflecting a salient feature of the bird that has been modified during generation, such as the color of the breast or the shape of the beak, for a total of 3333 different concept attributes. We use both these annotations to train teacher models. This enforces the teacher to both linearly encode concepts and labels in the embeddings. K.1 Architectures and hyperparameters details For Synth, we adopt an MLP constituted of 22 hidden layers, each with 512512 units and ReLU activations. Teachers and students use this architecture and map to a (m=2)(m=2) representation space. Teachers are trained for 15001500 epochs, while students are trained for 250250 epochs on the training set annotated with teacher’s probabilities. For all runs, we use a learning rate of 10−310^-3 with exponential decay on each epoch set to γ=0.995γ=0.995, and batch size of 512512. For CIFAR 100100, we use ResNet[He et al., 2016] models for the embedding part of both teacher and student, but of different sizes. For the teacher, we use a ResNet50 and for the student a ResNet18 with the same architecture as implemented by torchvision555docs.pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html. The final linear layer of the embedding model for both student and teacher goes to 5050 dimensions. For the unembeddings we use a single linear layer, giving us 100100 (one for each label) 5050-dimensional unembeddings. We initialize the teacher model with weights trained on ImageNet taken from torchvision (ResNet50_Weights.IMAGENET1K_V2) and train the teacher for 10 epochs on the train split of the CIFAR dataset. Each student is then trained for 50 epochs on the training set with either ℒlogit1L_logit^1 (L1L_1 student), dlogit2d_logit^2 (L2L_2 student), or KL divergence (KL student) between student and teacher distributions as a loss. For all runs on CIFAR, we use a learning rate of 10−310^-3 with exponential decay on each epoch set to γ=0.995γ=0.995, and batch size of 3232. For SUB, we first preprocess all images with DINOv2 [Oquab et al., 2023], mapping them to 768768 dimensional embeddings. Then, we train a shallow neural network with 10241024 units, with ReLU activations, dropout set to p=0.1p=0.1 and BatchNorm. Both teachers and students have the same architecture, and the representations for all models are (m=10)(m=10)-dimensional. Teachers and students are trained for 500500 epochs to ensure convergence and, for all runs, we use a learning rate of 5⋅10−35· 10^-3 with exponential decay on each epoch set to γ=0.995γ=0.995, and batch size of 512512. K.2 Metrics Evaluation We use sklearn.evaluation for label and concept accuracy, while we implement with torch the evaluation of dKLd_KL, dlogitd_logit, and use sklearn.cross_decomposition.CCA to evaluate mCCAm_CCA. To evaluate drepd_rep, we adopt the following procedure. Depending on the number of labels, evaluating all possible choices of pivots and subsets of labels can be unfeasible. We thus consider selecting at random the pivots and the subset of labels to reduce the computational cost. For Synth, we evaluate drepd_rep for all possible labels and pivots. For CIFAR we select all possible pivots and for each of them we consider 200200 subsets of labels selected at random among the (100−150)≈5⋅1028 100-150≈ 5· 10^28 possible combinations. For SUB, we select all possible pivots and for each of them we consider 250250 subsets of labels selected at random among the (33−110)≈6⋅107 33-110≈ 6· 10^7 possible combinations. This number of samples is very small compared to the total number of combinations possible, but the most important thing is that we get information from all possible labels and with 200200(250250) samples of 5050(1010) labels, we can be relatively sure to draw each of the 100100(3333) labels at least once. Aggregation of measures. Reported values on teachers are aggregating by evaluating the mean and standard deviations over 55 runs. For each teacher, we run 55 different seeds for both the KLKL student and the L1L_1 student, and evaluate the mean value μi _i and variance σi2 _i^2, for i∈1,…,5i∈\1,…,5\. We then aggregate all mean and variances by considering the inverse-variance weighting. The mean is given by ωi=1σi2,μ¯=∑ωiμi∑iωi, _i= 1 _i^2, μ= Σ _i _i _i _i\,, (262) whereas the standard deviation is obtained as σ¯=1∑iωi. σ= 1 _i _i\,. (263) The final results are reported as μ¯±σ¯ μ± σ for all metrics. K.3 Extended results We report here the full result of our experimental investigation on the three dataset we consider. Precision is set to 33 for all tables and metrics. Table 3: Full results on Synth and CIFAR 100. We notice that on CIFAR, L1L_1 and L2L_2 students have very similar dlogitd_logit, but the L2L_2 student has much smaller drepd_rep. This might be because L1L_1 students result in models with unembeddings which lead to lower singular values for the L matrices on this dataset. Acc(Y)(↑)( ) dKL(↓)d_KL( ) dlogit(↓)d_logit( ) drep(↓)d_rep( ) mCCAm_CCA(↑)( ) Synth !30 teach !30 0.999±0.0010.999± 0.001 !30 −- !30 −- !30 −- !30 −- 2!30 KLKL 2!30 0.999±0.0010.999± 0.001 2!30 (4.57±0.43)⋅10−2(4.57± 0.43)· 10^-2 2!30 (7.70±0.27)⋅101(7.70± 0.27)· 10^1 2!30 (6.07±0.31)⋅100(6.07± 0.31)· 10^0 2!30 0.580±0.0480.580± 0.048 2!30 L1L_1 2!30 0.999±0.0010.999± 0.001 2!30 (5.95±1.40)⋅10−3(5.95± 1.40)· 10^-3 2!30 (2.85±0.09)⋅(2.85± 0.09)· 10^0 2!30 (1.69±0.04)⋅−(1.69± 0.04)· 10^-1 2!30 0.999±0.0010.999± 0.001 !30 L2L_2 !30 0.999±0.0010.999± 0.001 !30 (1.72±0.14)⋅−(1.72± 0.14)· 10^-3 !30 (3.24±0.08)⋅100(3.24± 0.08)· 10^0 !30 (2.04±0.05)⋅10−1(2.04± 0.05)· 10^-1 !30 0.999±0.0010.999± 0.001 CIFAR !30 teach !30 0.540±0.0100.540± 0.010 !30 −- !30 −- !30 −- !30 −- 2!30 KLKL 2!30 0.480±0.0010.480± 0.001 2!30 1.29±0.0041.29± 0.004 2!30 (23.5±0.02)(23.5± 0.02) 2!30 (8.76±1.37)⋅101(8.76± 1.37)· 10^1 2!30 0.515±0.0010.515± 0.001 2!30 L1L_1 2!30 0.492±0.0010.492± 0.001 2!30 1.33±0.0031.33± 0.003 2!30 17.0±0.0217.0± 0.02 2!30 (4.56±0.60)⋅100(4.56± 0.60)· 10^0 2!30 0.767±0.0010.767± 0.001 !30 L2L_2 !30 0.493±0.0010.493± 0.001 !30 1.32±0.0051.32± 0.005 !30 17.1±0.0217.1± 0.02 !30 (6.58±1.71)⋅−(6.58± 1.71)· 10^-1 !30 0.763±0.0010.763± 0.001 Table 4: Full results on SUB. Acc(Y)(↑)( ) Acc(C)(↑)( ) dKL(↓)d_KL( ) dlogit(↓)d_logit( ) drep(↓)d_rep( ) mCCA(↑)m_CCA( ) !30 teach !30 0.915±0.0100.915± 0.010 !30 0.923±0.0110.923± 0.011 !30 −- !30 −- !30 −- !30 −- 2!30 KL 2!30 0.929±0.0010.929± 0.001 2!30 0.055±0.0010.055± 0.001 2!30 0.645±0.0070.645± 0.007 2!30 (3.73±0.05)⋅102(3.73± 0.05)· 10^2 2!30 (3.10±0.17)⋅103(3.10± 0.17)· 10^3 2!30 0.787±0.0010.787± 0.001 2!30 L1L_1 2!30 0.922±0.0010.922± 0.001 2!30 0.747±0.0090.747± 0.009 2!30 0.151±0.0020.151± 0.002 2!30 (3.40±0.01)⋅101(3.40± 0.01)· 10^1 2!30 (1.00±0.09)⋅101(1.00± 0.09)· 10^1 2!30 0.996±0.0010.996± 0.001 !30 L2L2 !30 0.919±0.0010.919± 0.001 !30 0.724±0.0060.724± 0.006 !30 0.173±0.0030.173± 0.003 !30 (3.20±0.02)⋅(3.20± 0.02)· 10^1 !30 (1.58±0.19)⋅(1.58± 0.19)· 10^0 !30 0.997±0.0010.997± 0.001